IR 05000454/2003009

IR 05000454-03-009 DRP; 05000455-03-009 DRP; on 11/03/2003 - 12/04/2003; Byron Station; Units 1 and 2. Identification and Resolution of Problems

ML040050125
Person / Time
Site:	Byron
Issue date:	01/02/2004
From:	Ann Marie Stone NRC/RGN-III/DRP/RPB3
To:	Skolds J Exelon Generation Co, Exelon Nuclear
References
IR-03-009
Download: ML040050125 (41)

Text

January 2, 2004

SUBJECT:

BYRON STATION, UNITS 1 AND 2 NRC PROBLEM IDENTIFICATION AND RESOLUTION INSPECTION REPORT 05000454/2003009(DRP); 05000455/2003009(DRP)

Dear Mr. Skolds:

On December 4, 2003, the U.S. Nuclear Regulatory Commission (NRC) completed a team inspection at the Byron Station, Units 1 and 2. The enclosed report documents the inspection findings which were discussed on December 4, 2003, with Mr. D. Hoots and other members of your staff.

This inspection was an examination of activities conducted under your license as they relate to identification and resolution of problems, and compliance with the Commissions rules and regulations and with the conditions of your operating license. Within these areas, the inspection involved selected examination of procedures and representative records, observations of activities, and interviews with personnel.

On the basis of the samples selected for review, the team concluded that, in general, problems were adequately identified, evaluated, and corrected. However, the team noted several examples where plant personnel did not identify or tolerated degraded material conditions. The team also identified two findings of very low safety significance (Green) related to various aspects of your problem identification and resolution processes. The first finding was associated with the failure to write a condition report identifying a condition adverse to quality.

As a result, a failed vibration switch associated with a reactor containment fan cooler never received the required screening in your corrective action program. At the time of the failure, your staff relied on the vibration switch to detect further degradation of the already degraded reactor containment fan cooler. The second finding was associated with a catastrophic failure of a coolant charging pump shaft. Your staff failed to implement appropriate corrective actions to preclude recurrence for this significant condition adverse to quality. Both of these findings were determined to be violations of NRC requirements. Since these conditions were determined to be of very low safety significance and have been entered into your corrective action program, the NRC is treating these findings as Non-Cited Violations in accordance with Section VI.A.1 of the NRCs Enforcement Policy. In accordance with 10 CFR 2.790 of the NRCs Rules of Practice, a copy of this letter and its enclosure and your response to this letter will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS)

component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/ RA /

Ann Marie Stone, Chief Branch 3 Division of Reactor Projects Docket Nos. 50-454; 50-455 License Nos. NPF-37; NPF-66

Enclosure:

Inspection Report 05000454/2003009(DRP); 05000455/2003009(DRP)

w/Attachment: Supplemental Information

REGION III==

Docket Nos:

50-454; 50-455 License Nos:

NPF-37; NPF-66 Report Nos:

05000454/2003009(DRP); 05000455/2003009(DRP)

Licensee:

Exelon Generation Company, LLC Facility:

Byron Station, Units 1 and 2 Location:

4450 N. German Church Road Byron, IL 61010 Dates:

November 3 through December 4, 2003 Inspectors:

J. Adams, Senior Resident Inspector - Prairie Island, Team Leader P. Snyder, Resident Inspector B. Jorgensen, Consultant C. Thompson, Illinois Emergency Management Agency Approved by:

Ann Marie Stone, Chief Branch 3 Division of Reactor Projects

Enclosure

SUMMARY OF FINDINGS

IR 05000454/2003009(DRP); 05000455/2003009(DRP); on 11/03/2003 - 12/04/2003; Byron

Station, Units 1 and 2. Identification and Resolution of Problems.

The inspection was conducted by a senior resident inspector, a resident inspector, a State of Illinois resident engineer, and a consultant. Two Green findings which were associated with Non-Cited Violations were identified. The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter (IMC) 0609, Significance Determination Process (SDP). Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.

Identification and Resolution of Problems In general, plant personnel adequately identified issues and entered them into the corrective action process at a low-level. However, the team noted numerous examples including one inspection finding, where plant personnel either did not identify or tolerated degraded equipment material conditions. The majority of issues reviewed by the team were properly categorized and evaluated by the licensee. However, the team noted that several operability evaluations were narrowly focused and lacked adequate technical justification to demonstrate continued operation of degraded equipment. Most corrective actions reviewed were appropriately implemented; however, the team identified one inspection finding regarding the prompt identification and implementation of corrective actions to preclude recurrence of a significant condition adverse to quality.

A.

Inspector-Identified and Self-Revealed Findings

Cornerstone: Mitigating Systems

Green.

The team identified a finding of very low safety significance and an associated Non-Cited Violation of 10 CFR 50, Appendix B, Criterion XVI, Corrective Action, for inadequate corrective actions to preclude repetition of a significant condition adverse to quality. The licensee failed to determine the cause and take prompt corrective actions to preclude repetition for the failure of the 2B centrifugal charging pump (CCP) shaft.

Neither the root cause report or the common cause analysis associated with this failure identified a specific root cause for the failure. Absent a root cause, the licensee presented three potential causes. The licensee implemented minimal corrective actions to address only one of the potential causes, specifically gas entrainment. Four options addressing the other two potential causes were identified and evaluated. For each of these options, the licensee determined that they were cost prohibitive and not financially justified. The team was unable to identify any corrective action planned or committed to in the licensee corrective actions program implementing actions to address the correction of the potential causes such that a high level of confidence exists that subsequent CCP shaft failures will be prevented.

The issue is more than minor because it affects the equipment performance attribute of the mitigating systems cornerstone objective to ensure the reliability of systems that respond to initiating events to prevent undesired consequences. The finding was determined to be of very low safety significance because the finding (1) did not result in a design or qualification deficiency confirmed not to result in a loss of function per Generic Letter 91-18; (2) did not represent an actual loss of safety function; (3) did not represent an actual loss of safety function of a single train for greater than the technical specification allowed outage time; (4) did not represent an actual loss of safety function of one or more non-Technical Specification trains designated as risk significant per the Maintenance Rule for greater than 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />; and (5) did not screen as potentially risk significant due to a seismic, fire, flooding, or severe weather initiating events.

(Section 4OA2.3)

Cornerstone: Barrier Integrity

Green.

The team identified a finding and associated NCV of 10 CFR 50, Appendix B,

Criterion XVI, Corrective Action, for failing to promptly identify a condition adverse to quality. Specifically, instrument maintenance personnel did not recognize that a failed vibration switch installed on the 1A Reactor Containment Fan Cooler (RCFC) was a condition adverse to quality and did not implement actions with a condition report as specified in site procedures. Absent a condition report identifying the switch failure to the corrective action process, there was no further testing or evaluation performed to determine how or why the switch failed, there was no evaluation performed to determine if the switch would have been capable to detect increasing vibration of the RCFC, there was no evaluation of extent of condition, and there was no assessment of how the failure affected the operability of the RCFC. In this case the failure of the vibration switch possessed an added significance since it was relied upon to detect further degradation of an already degraded RCFC and alert operators of potential inoperability of the 1A RCFC. The licensee had identified this switch in a compensatory measure to ensure that further degradation would not go unnoticed by operators.

The team determined that this finding was more than minor because it was associated with one of the cornerstone attributes and affected the barrier integrity cornerstone objective. Specifically, the human performance attribute of the barrier integrity cornerstone objective to provide reasonable assurance that physical design barriers (in this case the reactor containment) protect the public from radio nuclide releases caused by accidents or events. The finding was determined to be of very low safety significance because the finding (1) did not represent a degradation of the radiological barrier function provided for the control room, or auxiliary building, or spent fuel pool; (2) did not represent a degradation of the barrier function of the control room against smoke or toxic atmosphere; (3) did not represent an actual open pathway in the physical integrity of reactor containment; and (4) did not represent an actual reduction of the atmospheric pressure control function of the reactor containment. (Section 4OA2.1)

Licensee-Identified Violations

No findings of significance were identified.

REPORT DETAILS

4OA2 Problem Identification and Resolution

.1 Effectiveness of Problem Identification

a. Inspection Scope

The team conducted a review and assessment of the licensees processes for identifying and correcting problems at the Byron Station. The team reviewed previous licensee and inspector-identified issues related to the seven safety cornerstones in the Reactor Safety, Radiation Safety, and Safeguards strategic performance areas to determine if problems were appropriately identified, characterized, and entered into the corrective action program. The team reviewed selected plant procedures and program description handbooks, interviewed plant and contractor personnel, and attended various station meetings to understand the stations processes for initiating the corrective action program (CAP) and related activities. The team also reviewed Nuclear Oversight Assessments, Operating Experience Reports and trend assessments to determine if problems were being identified at the proper threshold and entered into the CAP process.

The team selected a number of condition reports (CRs) and action reports (ARs) and other corrective action documents, primarily generated since the last Problem Identification and Resolution (PI&R) inspection, for more in-depth review. Also, the team searched for items or issues which looked like potential trends and assessed whether the licensee had appropriately identified and captured these trends within the corrective action program.

The team reviewed the past performance of selected plant systems to assess equipment monitoring, evaluate maintenance rule implementation, and to identify if any issues were missed by the licensee. The systems selected were emergency diesel generators including the diesel fuel oil system, electro-hydraulic control system, chemical and volume control system (charging pump/emergency core cooling related equipment), and containment leak detection systems. As part of this assessment, the team interviewed system managers, reviewed system health reports and system monitoring programs, observed non-licensed operators during plant rounds, and performed partial system walkdowns.

The team conducted a review to determine whether the audit and self-assessment programs were effectively managed and adequately covered the subject areas. In addition, the team interviewed licensee staff regarding the audit and self-assessment programs.

The specific documents reviewed are listed in the Attachment of this report.

b. Observations and Findings

In general, plant personnel adequately identified issues and entered them into the corrective action program. In the 2 years since the previous Problem Identification and Resolution inspection, the licensee entered in excess of 9000 conditions reports into their corrective action program. However, the team concluded that a weakness existed among plant personnel associated with the identification of degraded equipment plant conditions. The team concluded that plant personnel possessed a level of tolerance for degraded plant conditions to remain uncorrected, especially if those plant conditions were perceived as or assumed to be minor or non-consequential.

b.1 Observations on Thresholds for Entering Known Problems into the Corrective Action Program The team identified a number of examples where a degraded plant condition had not been appropriately entered into the corrective action program. Nuclear Oversight assessment reports also identified a number of issues where plant personnel failed to enter degraded plant conditions. Additionally, the licensees root cause evaluation for a recent incident (the fuel handling machine collided with a reactor rod cluster control assembly change fixture basket after the limits-of-travel were bypassed to steer the machine around a reactor cavity ladder) noted that licensee staff knew of the close proximity of the ladder to the refueling machine mast, but never identified the problem in the corrective action program, even though several individuals were previously challenged with the potential for collision. (This is further discussed in NRC Special Inspection Report 05000454/2003008.)

Through interviews with plant personnel, the team noted that some plant personnel lacked confidence that minor issues would receive timely correction and perceived a level of management acceptance of degraded plant conditions based on the duration that identified problems were allowed to exist uncorrected. Plant personnel appeared to be somewhat tolerant of long standing problems which they perceived as minor or non-consequential. In several cases, it was clear that plant personnel were previously aware of the degraded conditions identified by the team. For example, during system walkdowns of the diesel generators, a team member observed that plant personnel had placed adsorbent materials under a diesel fuel oil leak but did not enter the condition into the corrective action program. Additionally, when a team member questioned a non-licensed operator on rounds about the source of oil dripping out of the auxiliary building supply fan duct, the operator said the fan bearing oilers drip oil, that oil is drawn into the fan, the oil settles out in the duct, and drips out onto the floor under the duct.

The operator then told the inspector that it had always been that way. With respect to the oil leaks described above, the conditions did not impact the operability of the associated equipment; therefore, the issues were deemed as minor.

b.2 Observations and Findings on Identifying Conditions Adverse to Quality The team identified a number of examples where individuals did not recognize problems and therefore had not initiated a condition report or work request until prompted by the team. With one exception, the conditions identified by the team were minor in nature and did not constitute violations of NRC requirements. For example, the following conditions were identified by the team members during system walkdowns, while observing operator rounds, or while performing in-office reviews of corrective action program documentation:

• The team identified that fire door ODD 317 would not close and latch without assistance due to mechanical interference. This condition was evident to any person that opened the door to access the 1B diesel generator ventilation room.

No condition report or work request was written prior to identification by the team. The team verified that the licensee entered this condition into their corrective action program with Work Request 119218.

• The team identified transient combustibles in both the 1A and 2B diesel generator rooms without transient combustible permits. A large cardboard box of cotton rags was maintained in the 1A diesel generator room to wipe up oil from numerous minor oil leaks on the engine. In the 2B diesel generator room a large roll of oil adsorbent material was found stored near the diesel generator local control panel. In neither case had plant personnel identified the absence of a transient combustible permit despite daily inspections of these rooms. The team verified that the quantity of combustible materials in the diesel rooms was bounded by the quantity of combustible materials assumed in the fire protection report. The team verified that the licensee entered this condition into their corrective action program with CR 185018.

• The team identified that a field supervisor directed the placement of a 55 gallon barrel of diesel engine lubricating oil into the 1A diesel generator room to warm up but failed to recognize the need for a transient combustible permit. The team verified that the quantity of combustible materials in the diesel room was bounded by the quantity of combustible materials assumed in the fire protection report. The fire protection report assumed four barrels of oil in a diesel room.

The team verified that the licensee entered this condition into their corrective action program with CR 185018.

• The team identified oil dripping from an auxiliary building exhaust fan ventilation duct onto the floor while observing non-licensed operator performing rounds in the auxiliary building. This was a condition that the operator was aware of but did not know if the problem was entered into the corrective action program. The team verified that the licensee had entered this condition into their corrective action program with CR 143560, dated February 7, 2003.

• The team identified a rolling tool box left unsecured and adjacent to reactor building stack process radiation monitors that went unrecognized by the operator until prompted by the inspector. This condition was corrected by the operator at the time of discovery by securing the tool box.

Although the above conditions were considered minor, one finding of very low safety significance was identified as described below.

Failure to Identify a Failed Vibration Switch as a Condition Adverse to Quality

Introduction:

A Green finding and associated NCV were identified for inadequate corrective action, in that, personnel failed to appropriately identify conditions adverse to quality associated with a failure of a vibration switch installed on the 1A RCFC.

Description:

On July 10, 2002, the 1A Reactor Containment Fan Cooler exhibited abnormal high vibration via an alarm (1VS-VP001). The high vibrations occurred during steady state fast speed fan operation. The licensee completed an operability evaluation and concluded that the fan was operable. The licensees basis for this conclusion depended on limited vibration analysis in the slow speed mode of operation (accident mode) and relied on monthly slow-speed surveillance test to detect further degradation.

On February 28, 2003, instrument mechanics performed work under Work Order 00462676-01 to support repair of the 1A RCFC. Instrument mechanics were given a work order to calibrate or replace the 1A RCFC vibration switch 1VS-VP001. In the Work Performed section of the Work Order package, the mechanics noted the following: Attempted to calibrate 1VS-VP001 but switch would not calibrate. Replaced switch with new one. CAT ID #20728." Although required by Maintenance procedure MA-AA-716-232, Proactive Maintenance, the mechanics did not initiate a condition report to document the vibration switch failure.

Because a condition report identifying the switch failure was not generated, the licensee did no further testing or evaluation to determine how or why the switch failed. No evaluation was performed to determine if the switch would have been capable to detect increasing vibration of the RCFC, the extent of condition, and there was no assessment of how the failure affected the operability of the RCFC. As stated above, the failure of the vibration switch possessed an added significance since the licensee had relied upon the switch to detect further degradation and to alert operators of a potential inoperable 1A RCFC.

Analysis:

The team determined that the failure to identify the failed vibration switch as a condition adverse to quality was a performance deficiency warranting a significance evaluation in accordance with IMC 0612, Power Reactor Inspection Reports, Appendix B, Issue Disposition Screening. The team determined that this finding was more than minor because it was associated with one of the cornerstone attributes listed in IMC 0612, Appendix B, Section C, and affected the barrier integrity cornerstone objective. Specifically, the human performance attribute of the barrier integrity cornerstone objective to provide reasonable assurance that physical design barriers (in this case the reactor containment) protect the public from radio nuclide releases caused by accidents or events was affected.

The team evaluated the finding in accordance with IMC 0609, Significance Determination Process, because the finding was associated with the barrier integrity of the containment. Using the Phase 1 screening, the team determined that the finding

(1) did not represent a degradation of the radiological barrier function provided for the control room, or auxiliary building, or spent fuel pool;

(2) did not represent a degradation of the barrier function of the control room against smoke or toxic atmosphere;

(3) did not represent an actual open pathway in the physical integrity of reactor containment; and

(4) did not represent an actual reduction of the atmospheric pressure control function of the reactor containment. The finding was determined to be of very low safety significance (Green).

Enforcement:

10 CFR 50, Appendix B, Criterion XVI, Corrective Action, states, in part, that measures shall be established to assure that conditions adverse to quality, such as failures, malfunctions, deficiencies, deviations, defective material and equipment, and nonconformance are promptly identified and corrected. Contrary to the above, on February 28, 2003, instrument mechanics did not identify that a failed vibration switch was a condition adverse to quality and did not enter this condition into the corrective action program for further evaluation as required by the CAP procedure LS-AA-125, and maintenance procedure MA-AA-716-232. Because this violation was of very low safety significance (Green), and the licensee entered the condition identified by the team into their corrective action program with CR 189008, this violation is being treated as an NCV in accordance with VI.A.1 of the NRCs Enforcement Policy (NCV 05000454/2003009-01).

b.3 Selected System Reviews The team conducted independent walkdowns of three selected systems including the emergency diesel generators and diesel fuel oil delivery system, electro-hydraulic control system, and the charging portion of the chemical and volume control system. In general, observed equipment deficiencies had been entered into the corrective action program. However, the team members identified several problems not previously entered into the corrective action program associated with the material condition of the diesel generators and associated diesel fuel oil system. For example, the team identified the following conditions during the system walkdown of the diesel generators:

• a minor diesel fuel oil leak on level switch piping associated with 1LS-DO 034; and

• loose studs on both the upper (1 of 16 studs) and lower (9 of 16 studs)diesel lubricating oil heat exchanger end bells for the 2A diesel generator.

These conditions did not render the diesel generator inoperable; therefore, their significance was minor in nature. However, not including these items in the CAP further demonstrates a tolerance for material condition discrepancies.

b.4 Nuclear Oversight The team noted through the review of corrective action program documents considerable evidence of assessment findings on the part of the Byron Nuclear Oversight (NOS) organization. The NOS group identified numerous issues across the spectrum of areas assessed.

The team noted that NOS identified many issues where plant personnel failed to recognize conditions adverse to quality and enter those conditions into the corrective action program in accordance with site procedures. These NOS identified issues supported the inspector conclusions that plant personnel lacked an appropriate sensitivity to the identification of plant problems requiring entry into the corrective action program. The following two examples from NOS were particularly notable:

• NOS conducted a review of the shift managers log for a 10-day period to verify that condition reports were written for conditions adverse to quality as defined in CAP procedures LS-AA-125 and LS-AA-125-1006. NOS identified that no condition reports were written for 12 equipment conditions logged. NOS entered these failures into the corrective action program with CR 087490.

• NOS reviewed the Byron Unit 1 refueling outage, outage control center (OCC)issues log and identified 87 issues where the condition report determination entry was left blank (52 of 87) or a condition report was required but not written (35 of 87). NOS entered these failures into the corrective action program with CR 179772.

The team concluded that the NOS organization was effectively identifying problems associated with the licensees implementation of their corrective action program. While the number of issues identified was a credit to the performance of the NOS organization, it also provided an indication of a weakness with plant personnels ability to identify problems.

.2 Review and Evaluation of Issues

a. Inspection Scope

The team reviewed previous inspection reports and other corrective action documents generated since July, 2001. The team examined selected Apparent Cause Evaluations (ACEs), Root Cause Reports (RCR), prompt investigations, operability determinations, and Common Cause Analyses (CCAs) to independently verify that identified issues were appropriately prioritized and evaluated when entered into the corrective action program.

The team reviewed data for a 5-year period for the emergency diesel generators, the diesel fuel oil system, the charging pump portion of the chemical and volume control system, containment leak detection systems, and the electro hydraulic control system.

The team focused on the technical adequacy of the cause determinations, adequacy of the extent of condition reviews including evaluations of potential common cause or generic concerns, and the appropriateness of the corrective actions. In addition, the team also assessed the adequacy of the operability and reportability determinations.

Other attributes reviewed by the team included the quality of the licensees trending of conditions and the corresponding corrective actions. The team searched for items or issues which looked like potential trends and assessed whether the licensee had appropriately identified and captured these trends within the corrective action program.

The team also assessed licensee corrective actions stemming from previous NCVs and Licensee Event Reports. The team selected several items to ensure proper implementation of the Maintenance Rule. This included verifying that the functional failures and unavailability time were properly counted and tracked.

The review included the controlling procedures, selected records of activities, and observation of various licensee meetings. In addition, the team conducted several interviews with cognizant licensee personnel. The specific documents reviewed are listed in the Attachment to this report.

b. Observations and Findings

b.1 Evaluations In general, the licensees evaluations were found to be broadly-based, technically sound, and focused on safety, with several notable exceptions discussed below.

Because the equipment in these examples remained operable, no violations of NRC requirements occurred. However, the lack of rigor in the original evaluations and the failure to adequately reassess the evaluations once new information was obtained demonstrated a weakness.

Reactor Containment Fan Cooler 1A On July 10, 2002, the 1A Reactor Containment Fan Cooler (RCFC) exhibited abnormal high vibration via an alarm (1VS-VP001). The high vibrations occurred during steady state fast speed fan operation. The operator followed the annunciator response procedure by shutting the fan off and starting a redundant fan. The operator entered the degraded fan condition into the corrective action program by writing a condition report. The initial investigation included some limited inspection and vibration testing on July 11, 2002. The fan was not declared inoperable, but it was placed on the licensees Degraded Equipment List.

An operability evaluation was completed on July 17, 2002, to support the shift managers initial determination that 1A RCFC was operable. This evaluation rested primarily on three bases. First, the degree of vibration was not excessive, as compared against generic industry criteria for rotating equipment. The licensees basis for this conclusion depended on limited vibration analysis in the slow speed mode of operation only. Second, the licensee concluded the problem was a mechanical looseness between the bearing and its housing and not a failed bearing. The licensees basis for this conclusion was developed from a limited physical inspection of the 1A RCFC and noted similarities to a previous high vibration event associated with another RCFC. The cause of the previous high vibration event was determined to be mechanical looseness of the bearing in its housing. Third, monthly slow-speed surveillance test runs would detect further degradation, should it occur. After the alarm was received with the fan in high speed on July 10, 2002, it was reset. Brief operation in slow speed (accident mode) on July 11, 2002, did not re-actuate the alarm.

From July 11, 2002, until February 24, 2003, the 1A RCFC was operated only for surveillance testing, in slow speed, and only about 15-30 minutes each month. Normal practice would have been to operate the unit 9 weeks of every 12, in high speed. Thus, this unit accumulated less than 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> of run time instead of more than 5 months.

The team did not agree that the licensees operability evaluation provided sufficient technical justification to reach the conclusion that the 1A RCFC could perform its design safety function, in the post-accident containment environment, for the assumed accident duration (mission time). The operability evaluation lacked quantitative analytical content. It reflected no assessment of available information which would have contributed to a more challenging analysis, such as historic vibration test data, vendor design performance specifications, and system design bases. The scope and degree of effort to gather information at the time of the initial vibration alarm appeared limited.

Among the parameters which might have been monitored for indications as to the cause of the problem were motor and bearing temperatures, noise, current draw, and routine vibration analysis.

Additionally, the team observed that the licensee did not establish a more conservative setpoint based on normal vibration level for slow speed fan operation. The vibration switch setpoint was left at a vibration level slightly above what was considered normal for fast speed fan operation. Typically, vibration levels are noticeably less with the RCFCs operating in slow speed. Therefore, when a fan operates in slow speed a greater margin exists between fan vibration levels and the high vibration alarm setpoint.

More degradation must occur prior to the generation of a vibration alarm. The opportunity for prompt identification of additional degradation was lost and the possibility existed that the 1A RCFC was allowed to degrade to a material condition well beyond a point that it could no longer accomplish its designed safety function under accident conditions for the required duration.

The team concluded that the licensee had not rigorously sought the best available technical information to support a timely assessment of the off-normal condition of an important barrier integrity system. As a result, the ability of the system to perform its design safety function, including operation for 30 days in the post-accident environment, was not clearly demonstrated with a high level of certainty. The team was also concerned that plant management was willing to accept an operability evaluation that lacked a rigorous, critical, and technically justified evaluation or analysis supporting the continued operability of the 1A RCFC.

The team noted that when repairs were initiated in late February 2003, workers conducting the repairs recorded that they found damage to the end of the motor rotor, and the motor shaft was worn to below the minimum required diameter. This was caused by failure of the outboard bearing in that the inner race had broken loose from the shaft and had worn against it. The repair workers also found that the vibration instrument, 1VS-VP001, would not calibrate and it had to be replaced. The as-found conditions (excepting the condition of 1VS-VP001) were reported in a condition report and an Equipment Apparent Cause Evaluation (EACE) was performed. The EACE concluded that the bearing had failed due to normal wear and tear. It also stated, based on the brief monthly test runs, that 1A RCFC, though degraded, had always been fully operable in slow speed, the accident mode. The team could not dispute this conclusion.

The team noted that the as-found conditions proved to be contrary to, or called into doubt, the bases for the presumed operability. Specifically, the problem was not mechanical looseness of the bearing in its housing. The inner race of the bearing had spun on the motor shaft. Vibration switch, VS-VP001, which had been relied on to warn of worsening vibrations, was found non-functional conditional. In addition, the amount of rotor shaft wear, as determined by measurement, required the replacement of the shaft.

The licensee had not critically and quantitatively analyzed or evaluated this new information in the EACE.

0A Essential Service Water Makeup Pump On January 29, 2002, during the performance of a surveillance test of the 0A essential service water makeup pump, the operator and system engineer noticed slightly elevated pump vibrations for about 4 minutes and the seal housing glowing red from heat. The system engineer submitted CR 092998 documenting the observed problems. On February 6, 2002, operability evaluation 02-004 was completed and concluded that the problem was due to the breakdown of the disaster bushing. The disaster bushing was not required to fulfill the safety function of the 0A essential service water makeup pump and the licensee determined that the pump remained operable.

During three subsequent operations of this pump on February 28, March 6, and March 26, 2002, additional symptoms of further degradation, not considered in the original operability evaluation, were observed by plant personnel. Two additional condition reports (097281 and 98205) were written documenting the additional observed problems. The evaluation of these two additional condition reports narrowed the cause of the pump degradation down to a problem with a pump shaft bushing. Each of these condition reports referenced operability evaluation 02-004 as basis for continued operability of the 0A essential service water makeup pump. However, no additional assessment, evaluation, or revision of the original operability evaluation were completed.

The licensees operability procedure LS-AA-105 states that structure, system, or component operability shall be re-evaluated following a change in conditions or additional information about the cause of the degradation or non-conformance becomes known. The team discussed the lack of a re-evaulation of the initial operability evaluation with the system engineer. Based on the discussions, it was clear that the system engineer could justify the continued operability of the pump in light of the additional indication of pump degradation. However, the evaluation of this information was not clearly documented in either the subsequent condition reports or the operability evaluation.

Because the equipment remained operable, the failure to document the basis for continued operation was minor in nature. However, the team concluded that the plant personnel including operators, shift managers, and plant management were willing to accept continued operability of a plant component in an environment where additional signs of degradation were evident, absent a documented rigorous, critical, and technically justified evaluation or analysis.

Containment Leak Detection Systems The Byron Station was designed with diverse systems for detection of leakage into the primary containment. One function of these systems was to meet a requirement to detect an increase in leakage of one gallon per minute (gpm) from the reactor coolant system (RCS) within one hour. The overall system included instruments to detect gaseous (PR011A) or particulate (PR011B) radioactivity in the containment atmosphere, an instrument to detect water/liquid flow into the floor drain sump (RF008), and instruments to detect floor drain sump level changes (PC002 and PC003). Recent analyses concluded that with the history of low levels of radioactive contamination in the RCS, the gaseous radiation monitors could not be relied on to meet the leak detection requirements. This was the case during the time period of interest to this inspection (July 2001 - November 2003). During this same period, there were frequent, chronic, and prolonged problems with the sump flow and level instruments.

Beginning in August 2001, leakage was detected going into the Unit 2 floor drain sump at a low rate. The source of the leak was not determined; however, the licensee concluded that it was not RCS leakage. The presence of non-RCS leaks can make prompt detection of a new RCS leak more difficult, so this is an undesirable condition.

Efforts were made to locate the leak during routine containment entries, but the source was not identified. By November 2001, the licensee developed a troubleshooting plan to specifically search for leaks. However, this plan was not fully implemented. Evidence from chemical analyses indicated that the source of the leak was service water. By that time, flow instrument 2RF008 had become unreliable (continuously alarming above one gpm) and it needed frequent flushing. This situation continued through early 2002.

On March 4, 2002, 2RF008 was (incorrectly) reading around 2 gpm (still in continuous alarm), 2PC003 was inoperable, and 2PC002 alone was being relied upon for prompt RCS leak detection. As documented in a condition report written on March 4, 2002, the licensee believed that the behavior of 2PC002 was becoming suspect in that the reading changed when the containment was vented. Technical Specifications required the particulate monitor and one flow instrument to be operable. Thereafter, significant efforts were needed to ensure at least one of the flow instruments remained operable.

On April 12, 2002, the leak detection system was placed into Maintenance Rule status (a)(1).

As the leak continued to increase, additional chemical analyses confirmed that the source of the leak was no longer believed to be service water but secondary system water. A condition report written on May 9, 2002, documented that the leak trend threatened to grow beyond 1.0 gpm before the next outage. This condition report was issued closed with no actions assigned. Five days later, a second condition report documented the same findings but produced actions to perform another search, and to develop a template for future searches. The search resulted in finding the leak on a 2A steam generator hand-hole cover. The template work was started but abandoned as impractical because the licensee believed that the steam generator leak would have been found the previous November if the search plans had been fully implemented.

Following additional weeks of effort to support periodic reliance on a single and degrading instrument, an unplanned outage in late June 2002 afforded the licensee the opportunity to make repairs to the detection systems. The bubbler tube for 2RF008 was found installed differently than the original design, such that it was behind two screens which could become partly or completely plugged and affect instrument operation or accuracy. This condition, which apparently existed from original plant construction, was corrected.

In late July 2002, only 3 weeks after the outage, 2RF008, 2PC002, and 2PC003 all were exhibiting periodic signal spiking. An operability evaluation was completed which included the information that a service water leak had been traced to the 2D RCFC and that a similar pattern of signals suggested flow surges had previously been observed (February 2002) but had been mis-diagnosed as sump weir box plugging. The detection instruments were found to be operable and to be reporting actual volume-related flow surges and level changes coming out of one RCFC. Flow into the floor drain sump gradually increased throughout the remainder of 2002, with a recurrence in December 2002 of the condition where only one instrument (2PC002) appeared to be functioning nominally.

The team did not identify examples of violations of Technical Specification action requirements applicable to the instrument systems and conditions described above.

However, the team noted a clear history of frequent, prolonged, and chronic problems with these systems, particularly in Unit 2, that challenged the licensees resources.

Symptoms of performance degradation were not always vigorously and promptly investigated to the point where they were fully understood. The licensees actions regarding these systems reflected a willingness to accept some uncertainty and to endure considerable burdens over a long period of time when multiple diverse systems were degraded and the cause(s) not clearly known.

b.2 General Corrective Action Program Implementation Observations The team determined that the licensee adequately categorized and prioritized identified issues. Daily, reports from the previous day were collected and reviewed by Departmental Corrective Action Program Coordinators. Additionally, reportability, repetitiveness, and trending were discussed where appropriate. Management evaluations were conducted daily during Management Review Committee meetings which reviewed document packages accumulated since the previous meeting. The team members attended several Management Review Committee meetings and observed processing of corrective action documents. The team did not identify instances of significant disagreement with the priority classification or disposition of the corrective action documents at the meetings attended by the team.

The team noted that the licensee performed analyses of condition reports for adverse trends. The CAP Coordinators were primary initiators of such analyses but Nuclear Oversight and the Site Corrective Action Program Administrator also identified potential adverse trends. In addition, the computerized data base software enabled CAP personnel to electronically sift and sort event reports in search of potential trends. The team did not identify any additional repetitive equipment issue or problems requiring a common cause analysis or any significant concerns associated with the common cause analyses reviewed.

The team reviewed program documents and licensee self-assessments, including assessments by the NOS organization, relating to overall conduct of the corrective action program activities at the Byron station. The team observed that self-assessments were generally challenging and addressed problem identification, evaluation, and correction.

The team members reviewed FASAs and noted that in the area of problem identification, each FASA report stated that condition reports were being written with a proper threshold. This was contradicted by the findings of the NOS audit in February 2003, which found that known deficiencies were not being reported in a condition report. The team could not determine the reason for these contradictory results. However, the teams observations were in line with those of NOS, finding a number of known deficiencies which had not been reported in a condition report.

The team did not identify programmatic failures to follow through and complete assigned corrective actions, or significant examples of corrective actions that, when implemented, did not achieve the purpose of correcting the original problem.

Effectiveness of Corrective Action

a. Inspection Scope

The team reviewed selected condition reports and associated corrective actions to evaluate the effectiveness of corrective actions, verifying that corrective actions were identified and implemented in a timely manner, commensurate with the safety significance of the issues. This included the review of corrective actions to address common cause or generic concerns. The team also verified the appropriate implementation of a sample of corrective actions. In addition, the team reviewed a sample of corrective action effectiveness reviews completed by the licensee. The selection of samples for review were based on their importance in reducing operational risks and recurring problems.

The team focused on information recorded since July 2001, but selected items were reviewed going back over a 5-year period. The team selected samples based on their importance in reducing operational risks and recurring problems. A listing of the specific documents reviewed is in the Attachment to this report.

b. Observations and Findings

The team concluded that, in general, corrective actions were adequately implemented and tracked to completion, corrective actions appeared effective in addressing the parent issue, and corrective action timeliness appeared to be commensurate with the safety significance of the issues. However, the team identified one issue of very low safety significance where appropriate corrective actions were not promptly identified and implemented.

Failure To Identify the Cause and Preclude Repetition for the Unit 2 Train B Centrifugal Charging Pump (CCP) Shaft Failure

Introduction:

The team identified a Green finding and associated NCV for the licensees failure to identify the cause of CCP shaft failures and take corrective actions to preclude repetition. The failure of the safety related Unit 2 train B CCP was considered a significant condition adverse to quality.

Background: Historically, there have been 34 previous pump shaft failures that have occurred in the industry involving pumps of the type used at Byron. Issues with stress raised areas in the shaft design were identified by Westinghouse and documented in Westinghouse Technical Bulletins as well as NRC Information Notices as early as 1980.

As documented in NRC Information Notice 80-07, Westinghouse recommended improving shaft design by increasing the radius of corners in higher stress areas of the shaft. Based on information obtained during interviews with the licensee staff, the team determined the shafts installed in the Unit 2 CCPs to be of the original shaft design made from 414 stainless steel with the improved shaft radii. The currently available shaft design consists of a new material, CA 625, that has increased toughness and a 30 percent higher yield strength.

After continued industry pump shaft failures, Westinghouse determined that maintaining vibration levels within pump manufacturer recommended levels may increase pump reliability. NRC Information Notice 94-76 included this information as well as Westinghouse recommendations to increase vibration monitoring to at least monthly and preferably every 2 weeks to increase the benefit of predictive maintenance programs.

The licensee did not change the performance frequency of their vibration monitoring surveillance and stayed at the quarterly frequency required by Section XI of the American Society of Mechanical Engineers (ASME) Boiler and Pressure Vessel Code.

The Unit 2 train B CCP was within 2 weeks of the required vibration monitoring surveillance period of Section XI of the ASME Boiler and Pressure Vessel Code prior to its failure. This suggests that had increased vibration monitoring been implemented when recommended, the catastrophic failure might have been avoided.

Description:

On November 11, 2002, the rotating element of the Unit 2 train B CCP failed during normal operation. The licensee identified that the shaft had failed during a rotational check and replaced the rotating element of the pump. The Root Cause Investigation Report associated with the Unit 2 train B CCP failure identified the root cause as high-cycle low-stress fatigue. While the licensee was investigating that failure and during the performance of a routine ASME vibration surveillance on February 25, 2003, elevated vibration levels were discovered on the 2A CCP. The licensee replaced the rotating element of the safety train A pump. The licensee initiated a common cause analysis (CCA) to investigate both shaft problems. The licensee concluded in the CCA that the root cause was indeterminate; however, potential causes for the shaft failure including the following, were identified:

• insufficient design strength in the original shaft design for abnormal stress conditions;

• sustained operation at low flow conditions; and

• potential damaging transients such as operation at zero flow, operation with a closed suction valve, and gas entrainment.

To address the potential cause of an insufficient design strength in the original shaft design for abnormal stress conditions, the licensee stated in their root cause report for the 2B CCP shaft failure that subsequent rebuilds of the rotating element will be made of the CA 625 material. However, the team was unable to identify any corrective action program commitment to replace the shafts with the improved design or a date when the actions would be completed. In fact, the licensee initiated Action Tracking Item (AT)146165-23 to evaluate several options to address the shaft failures. Option 1 was to replace the CCP shafts with the most current shaft design. In the Results Section of the licensees evaluation of option 1, the licensee stated that this option is not financially justified. The licensee has purchased one shaft of the newest material but had no current plans for its installation.

To address the potential cause of sustained operation at low flow conditions, the licensee reviewed several options. Normal steady state operation flow rate as about 200 gpm; however the ASME inservice test flow rate was 70 gpm. the licensee revised the ASME procedure to require a flow rate of 200 gpm. The team did not see much benefit in this change since the time that the CCPs operated at the 70 gpm flow rate could not be defined as sustained operation.

The licensee also evaluated a modification to the CCP recirculation line which would add a flow control valve to allow operation near the best efficiency point for the CCP, about 325 gpm. The licensee completed the evaluation of this option and determined it was cost prohibitive (Ref. AT 131171-27).

The licensee also evaluated two other options which would eliminate operations of the CCPs for normal steady state plant operations. The licensee evaluated replacing the positive displacement charging pump, long abandoned in place, with a non-safety-related centrifugal charging pump or with an another positive displacement pump.

These options would reduce the run time of the safety-related CCPs to near zero. The licensee concluded that these options were not financially justified.

Gas entrainment was identified by the pump vendor in 1999 as a problem. Prior to the CCP shaft failures, the licensee added vents to the system and added periodic ultrasonic testing to procedures to identify possible gas pockets in suction lines.

Following the November 2002 and February 2003 CCP shaft failures, the licensee determined that gas entrainment was not likely the root cause; however, the licensee took corrective actions to minimize the potential for gas entrainment into the CCP pumps. The licensee revised procedures to make venting of the pump required, not optional and to limit makeup using alternate dilute mode in an attempt to add makeup water only after it goes through the volume control tank where some degassing can occur.

Other actions proposed by the licensee in the RCR and CCA included installing additional instrumentation to implement continuous CCP monitoring. The licensee installed some vibration instrumentation on one of the CCPs that recently failed. This data, while continuously collected by a computer at the pump location, was reviewed approximately weekly by the system engineer. The licensee planned to install more vibration instrumentation on all of the pumps; however, to date no additional instrumentation has been installed. The team noted that the increased vibration instrumentation will provide more information which may be beneficial in the early identification of future failures.

The team members concluded that the licensee implemented minimal corrective actions to address only one of the potential causes, specifically gas entrainment. The potential concern of gas entrainment, as stated by the licensee in their root cause report, was determined not to be the most probable cause. As discussed above, the licensee evaluated options to address the other two potential causes, specifically,

(1) insufficient design strength in the original shaft design for abnormal stress conditions and (2)sustained operation at low flow conditions. However, the licensee determined that the options were cost prohibitive and not financially justified. The team also noted in the licensees evaluations of these options, as documented in AT 131171-27 and AT 146165-23, lacked any consideration of safety benefit in preventing future CCP shaft failures. The team was unable to identify any other corrective actions planned or committed to in the licensees corrective action program that would implement proactive actions to prevent CCP inoperability due to catastrophic shaft failures. Based on the information in the licensees corrective action program addressing the correction of this significant condition adverse to quality, the team could only conclude that the CCPs would be run to failure.

Analysis:

The team determined that the failure to initiate corrective actions to prevent recurrence of CCP inoperability due to catastrophic shaft failure was a performance deficiency warranting a significance evaluation in accordance with IMC 0612, Power Reactor Inspection Reports, Appendix B, Issue Disposition Screening. The team determined that this finding was more than minor because it was associated with one of the cornerstone attributes listed in IMC 0612, Appendix B, Section C, and affected the mitigating cornerstone objective. Specifically, the finding affected the equipment performance attribute of the mitigating systems cornerstone objective to ensure the reliability of systems that respond to initiating events to prevent undesired consequences. The team evaluated the finding in accordance with IMC 0609, Significance Determination Process. Using the Phase 1 screening, the team determined that the finding

(1) did not result in a design or qualification deficiency confirmed not to result in a loss of function per NRC Generic Letter 91-18;

(2) did not represent an actual loss of safety function;

(3) did not represent an actual loss of safety function of a single train for greater than the technical specification allowed outage time;

(4) did not represent an actual loss of safety function of one or more non-Technical Specification trains designated as risk significant per the Maintenance Rule for greater than 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />; and

(5) did not screen as potentially risk significant due to a seismic, fire, flooding, or severe weather initiating events. The team determined the finding to be of very low safety significance (Green).

Enforcement:

10 CFR 50, Appendix B, Criterion XVI, Corrective Action, states, in part, that failures are promptly corrected and in the case of significant conditions adverse to quality, measure shall assure that the cause of the condition is determined and corrective actions taken to preclude repetition. Contrary to the above, the licensee failed to identify the root cause of the 2B CCP shaft failure and therefore did not identify corrective actions to prevent recurrence. Absent the root cause, the licensee did identify three probable causes but failed to promptly implement corrective actions to address those probable causes. Because this violation was of very low safety significance, and the licensee entered the conditions into their corrective action program with CR 189373, this violation is being treated as an NCV in accordance with VI.A.1 of the NRCs Enforcement Policy (NCV 05000455/2003009-02).

.4 Assessment of Safety-Conscious Work Environment

a. Inspection Scope

The team interviewed approximately 48 members of the plant staff, representing all major work groups, and all levels of responsibility. The team conducted the interviews to assess the establishment of a safety conscious work environment. During the interviews, document reviews, and observations of activities, the team looked for evidence that plant employees might be reluctant to raise safety concerns. The interviews typically included questions similar to those listed in Appendix 1 of NRC Inspection Procedure 71152, Suggested Questions for Use in Discussions with Licensee Individuals Concerning PI&R Issues. The team also reviewed the stations procedures related to the Employee Concerns Program (ECP), and discussed the implementation of this program with the stations program coordinator.

b.

Observations No significant findings were identified. None of the plant personnel interviewed expressed any concerns regarding a safety conscious work environment. All plant personnel interviewed stated that individuals were encouraged by managers and supervisors to identify issues. The work force considered the corrective action program as an effective management tool for the identification and correction of problems, especially for the more significant problems. However, work force confidence in the program declined when it came to the resolution of issues of lower significance. Some information was received suggesting that bargaining unit and working-level personnel at the station have a limited and different view of the CAP as compared to the views of management personnel. No one believed that there were any significant issues or corrective actions being handled outside the official CAP process.

The team observed that all personnel interviewed were aware of the different avenues through which they could express concerns including the corrective action program, informing their supervisor or plant managers, contacting their employee concerns program, or coming to the NRC. Several working-level personnel said they have usually chosen to simply report issues to their immediate supervisor, rather than create a condition report themselves. The team asked several personnel to demonstrate the entry of a problem into the corrective action program. In all cases, the individuals were successful in performing this task. The licensee has provided training to all personnel concerning the electronic entry of concerns into the corrective action program but has also maintained the ability to enter problems in a hand written format for those employees not comfortable entering conditions electronically.

As was stated earlier, personnel were aware of the ECP, ECP office location, and the ECP coordinators. The team noted no reluctance on the part of plant personnel that were interviewed to use the ECP. However, none of the personnel interviewed had ever used the ECP. The team discussed the implementation of the ECP with one of the ECP coordinators. Confidentiality was a basis of the program and appeared to be rigorously maintained. The program, as described by the ECP coordinator, appeared to have been implemented in accordance with company procedures. Trends of ECP usage by employees showed a slight upward trend of the previous 3 years.

The interviewees were largely satisfied that issues reported to the CAP were eventually resolved in a satisfactory manner. Timeliness was not always what was desired. A couple of examples were stated of problems which had gone unresolved despite multiple condition reports. These issues had been assigned a low priority for action, such that a considerable time could go by before they will be acted on.

Some employees expressed concerns with staffing and work assignments at the Byron plant, which are not issues the CAP was designed to address. With many work groups reduced in size over the past couple of years, increased work for all the remaining staff was said to potentially risk more errors and to reduce the time available for tasks such as administering functions of the CAP. However, no one identified an example of staff inability or unwillingness to raise and document safety concerns due to inadequate time or resources.

Minor concerns were expressed about the tight scheduling for the processing of condition reports, and for the administrative burden associated with even a very simple issue. Some interviewees said that a problem is sometimes fixed before it is documented. Then, the condition report can be closed based on actions already taken, thereby avoiding a lot of review and evaluation and feedback steps. The team searched the condition report database, finding only about one-percent of them had been closed in this manner. Several dozen of these condition reports were reviewed in detail, and no examples were found of actions the team considered inappropriate.

4OA6 Meetings

Exit Meeting The team presented the inspection results to Mr. Dave Hoots and other members of licensee management on December 4, 2003. The licensee acknowledged the findings presented. The team confirmed with the licensee that proprietary information was examined during the inspection; however, this was not specifically discussed in this report.

ATTACHMENT:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee

S. Kuczynski, Site Vice President

D. Hoots, Plant Manager

B. Adams, Engineering Director

G. Contrady, Nuclear Oversight

D. Drawbaugh, Regulatory Assurance

D. Goldsmith, Radiation Protection Manager

W. Grundmann, Regulatory Assurance Manager

R. Irby, CAP Coordinator

K. Jury, Cantera Licensing

S. Kerr, Chemistry Manager

R. Randels, Engineering

Illinois Emergency Management Agency

J. Roman, Illinois Emergency Management Agency Resident Inspector

Nuclear Regulatory Commission

A. Stone, Chief, Reactor Projects Branch 3

R. Skokowski, Byron Senior Resident Inspector

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened and Closed

05000454/2003009-01 NCV Failure to Identify a Condition Adverse to Quality

05000455/2003009-02 NCV Failure to Identify and Implement Corrective Actions to Prevent Recurrence of a Significant Condition Adverse to Quality

Discussed

None