IR 05000298/2007010

IR 05000298-07-010; 12/03/07 - 12/06/07; Cooper Nuclear Station. Inspection Procedure Supplemental Inspection

ML080080622
Person / Time
Site:	Cooper
Issue date:	01/08/2008
From:	Hay M NRC/RGN-IV/DRP/RPB-C
To:	Minahan S Nebraska Public Power District (NPPD)
References
IR-07-010
Download: ML080080622 (16)

Text

January 8, 2008

SUBJECT:

COOPER NUCLEAR STATION - NRC SUPPLEMENTAL INSPECTION REPORT 05000298/2007010

Dear Mr. Minahan:

On December 6, 2007, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection at your Cooper Nuclear Station. The enclosed inspection report documents the inspection findings which were discussed on December 6, 2007, with you, and other members of your staff.

The inspections examined activities conducted under your license as they relate to safety and compliance with the Commission's rules and regulations and with the conditions of your license.

The inspector reviewed selected procedures and records, observed activities, and interviewed personnel.

Based on the results of this inspection, no findings of significance were identified.

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Michael C. Hay, Chief Project Branch C Division of Reactor Projects Docket: 50-298 License: DPR-46

Enclosures:

NRC Inspection Report 05000298/2007010

Nebraska Public Power District

- 2 -

REGION IV==

Docket:

50-298 License:

DPR-46 Report:

05000298/2007010 Licensee:

Nebraska Public Power District Facility:

Cooper Nuclear Station Location:

Brownville, Nebraska Dates:

December 3 through December 6, 2007 Inspector:

W. C. Walker, Senior Project Engineer, Branch C, Division of Reactor Projects (DRP)

Approved By:

M. C. Hay, Chief, Project Branch C, DRP ATTACHMENT:

Supplemental Information

Enclosure-2-SUMMARY OF FINDINGS IR05000298/2007010; 12/03/2007 - 12/06/2007; Cooper Nuclear Station. Inspection Procedure 95001 Supplemental Inspection.

The report covered a one-week period of inspection by a region-based inspector. No violations were identified. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.

Cornerstone: Mitigating Systems The U.S. Nuclear Regulatory Commission performed this supplemental inspection to assess the licensees evaluations of emergency diesel generator equipment problems associated with a performance indicator (Emergency AC Power) that crossed the Green-White threshold in the first quarter of 2007 and a related White finding involving inadequate corrective actions documented in NRC Inspection Report 05000298/2007007(ADAMS ML072290167). During this supplemental inspection, performed in accordance with Inspection Procedure 95001, the inspector determined that the licensee performed comprehensive evaluations of the three diesel generator failures that occurred since 2004. Specifically, the following three events contributed to the White performance indicator: 1) a lube oil sensing line leak in 2004 causing a failure of the diesel to run, 2) a control power failure in 2005 causing a diesel failure to start, and 3) a faulty voltage regulator circuit board in 2007 causing a diesel failure to run. On the basis of the licensees evaluations for the above events the inspector determined that the primary root causes and corrective actions for the above performance issues were appropriately evaluated and implemented.

In addition to the above issues a White finding was also identified for the January 2007 diesel failure involving inadequate corrective actions related to repeat failures of the diesel generator voltage regulator circuit board. The licensee identified that the procedure for elevating repeat occurrences of problems to a root cause evaluation was inadequate. This deficiency resulted in the licensees failure to perform a root cause evaluation following repetitive problems experienced with the emergency diesel generator voltage regulator circuit board following its installation. The root cause of the January 2007 event was determined to be: 1) a manufacturing defect of a zener diode on the voltage regulator circuit board, and 2) the process for maintaining quality of replacement safety-related components did not include the evaluation of operating experience for vendors removed from the approved suppliers list. Corrective actions to address the White finding included: 1) new procedures requiring burn in of electrical components were instituted, 2) a comprehensive review of all safety-related parts in the warehouse was conducted with holds placed on any parts that required further evaluation, and 3) the condition report procedure for escalation of repeat issues in the corrective action program was revised.

Given the licensees acceptable performance in addressing the diesel failures, the white finding associated with this issue will only be considered in assessing plant performance for a total of four quarters in accordance with the guidance in Inspection Manual Chapter (IMC) 0305, Operating Reactor Assessment Program. Since the White finding was documented in the second calendar year quarter of 2007 the finding will remain in effect until the end of the first calendar year quarter of 2008.

Enclosure-3-Report Details

INSPECTION SCOPE The U.S. Nuclear Regulatory Commission performed this supplemental inspection to assess the licensees evaluation associated with a performance indicator (Emergency AC Power) that crossed the Green-White threshold in the first quarter of 2007. The primary reason for this performance indicator being characterized as White was a second run failure of the emergency diesel generators in a 36 month period, on January 18, 2007. Overall, the Mitigating Systems Performance Index (MSPI) for the Emergency AC Power function represents three actual failures of the emergency diesel generators. This performance results in a change in core damage frequency that is characterized by industry standards as risk significant. An MSPI value of 5.7E-06/year was calculated for the three failures which exceeded the risk significance threshold of 3.4E-06/year as established by the industry and NRC. The change in core damage frequency resulting from the three failures represented by the emergency AC power MSPI value is characterized by the Reactor Oversight Process as a performance level that crosses the Green-White threshold.

In addition to the White performance indicator the NRC staff also identified a White finding and determined a corrective action violation occurred. The licensee failed to promptly identify a significant condition adverse to quality that resulted in reduced reliability of emergency diesel generator (EDG) 2. Two distinct and reasonable opportunities to identify the condition adverse to quality existed yet the condition was not promptly identified and corrected to preclude recurrence. Specifically, inadequate procedural guidance for evaluating the suitability of parts used in safety related applications presented one missed opportunity to identify that an EDG voltage regulating circuit board was defective prior to installation on November 8, 2006.

Following installation of the defective EDG 2 voltage regulator circuit board two high voltage conditions, one resulting in an EDG automatic high voltage trip, occurred on November 13, 2006. The licensees evaluation of these high voltage events missed another opportunity to identify and correct the deficient condition. The failure to identify and correct this deficient condition resulted in an additional high voltage trip of EDG 2 that occurred on January 18, 2007.

During this supplemental inspection, performed in accordance with Inspection Procedure 95001, the inspector determined that the licensee adequately determined the root causes and significant contributing causes of the Mitigating Systems Performance Index for Emergency AC Power crossing the Green-White threshold and the White finding for inadequate corrective actions, and established appropriate corrective actions to prevent recurrence.

Enclosure-4-

EVALUATION OF INSPECTION REQUIREMENTS 02.01 Problem Identification a.

Determination of who (i.e., licensee, self-revealing, or NRC) identified the issue and under what conditions This supplemental inspection focused on two diesel failures to run which took place on December 30, 2004, and January 18, 2007. The initial failure resulted from installing a diesel lube oil sensing line different than specified in design documents, making the sensing line fitting susceptible to vibration induced high cycle fatigue. The second failure was due to a manufacturing defect of a zener diode in the voltage regulator circuit. Both of the failures to run of the diesel were self-revealing. In addition the January 18, 2007, issue was NRC identified as a White finding for inadequate corrective actions due to a failure to promptly identify a significant condition adverse to quality that resulted in reduced reliability of EDG 2.

b.

Determination of how long the issue existed and prior opportunities for identification The December 2004 diesel run failure involved a leak on the lube oil sensing line which was caused by the configuration of the sensing line components being different than specified in design documents, making the fitting susceptible to vibration-induced high cycle fatigue. The design change, which was implemented in May 1989 to prevent recurring tubing failures, was improperly performed. The December 30, 2004, failure was the fourth time since 1989 that this particular fitting had required repair. The root cause team determined that a contributing cause to the December 2004 event was a failure to identify and correct the design installation error during response to previous events. Prior identification was possible on the basis of three previous maintenance activities provided opportunities to discover and correct the configuration error. The details associated with this performance deficiency are documented in NRC Inspection Report 05000298/2005005 (ADAMS ML060450109).

The January 2007 event involved a failure of the diesel to run due to a manufacturing defect of a zener diode on a voltage regulator circuit board. The board was purchased as a spare from the original equipment manufacturer in late 1973 prior to initial plant startup. The voltage regulator board remained in Nebraska Public Power Districts inventory until installed in November of 2006 after a potentiometer on the EDG 2 voltage regulator board was damaged. The voltage regulator board that failed on January 18, 2007, was the replacement for the voltage regulator board whose potentiometer was damaged on November 8, 2006. After installation of the 33 year old voltage regulator board CR-CNS-2006-08999 documents that an over-voltage event within the EDG 2 voltage circuit control system occurred. The licensee concluded that the high voltage trip occurred due to erratic potentiometer operation during the tuning process for the voltage regulator circuit board. However, laboratory testing of the voltage regulator circuit board zener diode provided definitive evidence that the overvoltage trip of EDG2 was caused by an intermittent discontinuity in a diode resulting from a manufacturing defect. On the basis of further review the NRC identified that the Basler (voltage regulator vendor) technical manual for voltage regulators and the Electrical Power Research Institute both recommended replacement of the voltage regulator card

Enclosure-5-following uncontrollable high voltage conditions. Voltage control problems on EDG 2 had been addressed using apparent cause evaluations on four separate occasions in the twelve months prior to the November 13, 2006, high voltage trip. The November 13 trip was again assigned an apparent cause evaluation versus a more thorough root cause evaluation. The licensees root cause assessment indicated that prior identification was possible if a more thorough evaluation was performed.

c.

Determination of the plant-specific risk consequences (as applicable) and compliance concerns associated with the issue The White Mitigating Systems Performance Index (MSPI) for the Emergency AC Power function represents three actual failures of the emergency diesel generators. This performance results in a change in core damage frequency that is characterized by industry standards as risk significant. An MSPI value of 5.7E-06/year was calculated for the three failures which exceeded the risk significance threshold of 3.4E-06/year as established by the industry and NRC. The change in core damage frequency resulting from the three failures represented by the emergency AC power MSPI value is characterized by the Reactor Oversight Process as a performance level that crosses the Green-White threshold.

In addition to the performance indicator turning White for the above events the NRC performed a Special Inspection to review the circumstances surrounding the January 18, 2007, diesel failure. The results were documented in NRC Inspection Report 05000298/2007007 (ADAMS ML071430289). The NRC concluded that the best characterization of risk for the January 18, 2007, diesel failure was of low to moderate safety significance (White), with a delta CDF of 1.2E-6.

02.02 Root Cause and Extent of Condition Evaluation a.

Evaluation of methods used to identify root cause(s) and contributing cause(s)

To evaluate these three emergency AC power failure events, the licensee utilized the following root cause analysis techniques:

Problem Recognition and Interim Management Investigation and Analysis Planning Cause Analysis and Determination Corrective Action Development and Assignment Corrective Action Implementation Effectiveness Measurement and Feedback The guidance for the use of cause determination techniques is contained in Administrative Procedure 0.5 Root-Cause, Root Cause Analysis Procedure, Revision 5. This procedure describes the six phases of the root cause analysis process, identifies important roles and responsibilities for the conduct of root cause analysis, and clearly delineates specific requirements to be adhered to during conduct of root cause analysis.

Enclosure-6-The inspector discussed the root cause determination process with the corrective action and assessment program manager. There are five individuals on site that are considered qualified to perform root cause analysis determinations. The qualification requirements for these individuals consists of formal classroom training, maintaining yearly qualifications, and actively participating in root cause determinations under the instruction of a qualified analyst. The inspector considered the licensees root cause determination program to be appropriate.

The inspector concluded that the quality of the root cause evaluations provided the rigor to ensure that the root cause evaluations were appropriate to determine the root causes and significant contributing causes of the diesel failures.

b.

Level of detail of the root cause evaluation The licensees root cause evaluations were thorough and identified the primary root causes of the performance issues. Specifically, for the December 30, 2004, event the configuration of the lube oil sensing line components were different than specified in design documents, making the swagelok fitting susceptible to vibration-induced high cycle fatigue. If the design change 89-022 had been properly followed the failure would not have occurred. The February 7, 2005, emergency diesel generator failure to start was a small contributor to the overall White performance indicator. It should be noted that the diode failure was a shorted diode as a result of an electrical overstress.

Regarding the January 18, 2007, diesel failure to run, a manufacturing defect in a zener diode on the voltage regulator control board was identified as the primary root cause for the failure.

The root cause for the White finding involving the failure to promptly identify a significant condition adverse to quality that resulted in reduced reliability of EDG 2 had two components. The first component was classification of a previous diesel failure in November 2006 as requiring less than a root cause evaluation even though it was a repeat occurrence. The second component involved the process for maintaining quality of replacement safety-related components did not include the evaluation of operating experience for vendors removed from the approved suppliers list.

c.

Consideration of prior occurrences of the problem and knowledge of prior operating experience The licensees evaluations included a review of internal (Cooper Nuclear Station) and external operating experience. In general, these sections included relevant operating experience along with a discussion of how this applies or does not apply to the specific situation.

d.

Consideration of potential common cause(s) and extent of condition of the problem The licensees evaluation considered the potential for common cause and extent of condition for the failure to properly implement the design change for the lube oil sensing line on the EDG, the manufacturing defects in the voltage regulator circuit boards for the EDGs, and the control of safety related parts in the warehouse. The licensees review of potential common causes for the lube oil sensing line leak determined that the most

Enclosure-7-direct causal factor for the failure was vibration and that the tubing was configured differently than specified in the approved design change. The inspector determined that the licensees extent of condition review was appropriate and thorough. The licensee performed a review of maintenance history and did not identify any concerns with other safety related systems. For the January 18, 2007, event it was determined that a random failure directly related to a manufacturing defect in the zener diode occurred.

The inspectors agreed with the licensees assessment and determined that extensive testing was required to detect the manufacturing defect in the zener diode.

The inspector concluded that the above actions combined with the corrective action program changes appropriately addressed the weaknesses identified in the root cause reports.

02.03 Corrective Actions a.

Appropriateness of corrective action(s)

The specified corrective actions were appropriate to the root and contributing causes that were identified in the root cause determination evaluations. As stated above, the root cause determinations were adequate to capture all aspects of the problems leading to the events. As a result, the specified corrective actions appeared to have addressed all possible causes and contributing causes. The licensee did identify in their root cause evaluations performed in response to the White performance indicator that prior opportunities existed to perform corrective actions which would have prevented the lube oil sensing line leak and would have potentially identified the defective voltage regulator circuit boards. The inspector agreed with this assessment and concluded that the lube oil sensing line leak was preventable had effective corrective actions been established following discovery of oil leaks at the Swagelok fitting location on four previous occasions between January 1993 and June 1999. The inspector noted that the corrective actions identified for the December 30, 2004, event addressed the design change/configuration control issues that ensured proper installation of the lube oil sensing tubing and reviewed other aspects of tubing configuration throughout the plant that would be effective in preventing recurrence.

Regarding the January 18, 2007, EDG failure to run, resulting from a defective zener diode on a voltage regulator circuit board, a corrective action was identified that may have increased the likelihood that the defective zener diode would have been identified.

Specifically, testing of the voltage regulator circuit board using a burn-in or equivalent testing may have identified the infant mortality failure prior to its occurrence.

The inspectors review of the corrective actions for the White finding determined that providing additional procedural guidance for escalation of repeat issues in the corrective action process should ensure that the appropriate level of licensee oversight is provided to ensure that a necessary root cause is performed when required. Additionally, a comprehensive review of all safety-related parts in the warehouse was conducted with holds placed on any parts that required further evaluation.

Enclosure-8-b.

Prioritization of corrective actions The inspector did not identify any specific methods utilized to prioritize the specified corrective actions based on risk significance or regulatory compliance. However, no examples of inappropriate prioritization were noted. The inspector considered the prioritization of the established corrective actions to be consistent with risk consequences.

c.

Establishment of schedule for implementing and completing the corrective actions The licensee established adequate schedules for completion of the specified corrective actions. As appropriate, some corrective actions were tied to scheduled equipment outages while others were more short term such as procedure revisions or training updates. The inspector did not identify any specific concerns with the scheduling or completion of established corrective actions.

The inspector did note that the diesel generator improvement plan regarding long term reliability improvements for the emergency diesel generators was included as a corrective action to be tracked in condition report CR-CNS-2007-01559. The inspector considered this initiative to be appropriate and should assist the licensee in the stated goal of the improvement plan to increase EDG reliability, availability, and maintenance predictability.

d.

Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence The licensee implemented a maintenance procedure change to the diesel generator electrical examination and maintenance procedure. This change included increased tasks to verify terminal tightness on diesel electrical equipment and perform thermography as a diagnostic tool to identify potential electrical problems in the EDGs prior to failure. The inspector interviewed station personnel involved in this effort and determined that this change had been implemented and was in use. The maintenance procedure specified the method, success criteria, and timing of this action in specific terms. The inspector identified no concerns in this area.

The licensee also established controls on all safety-related parts in the warehouse to ensure any parts that were not supported by documentation that ensured the parts were approved under an approved supplier program were placed on hold. The inspector reviewed this program and considered it appropriate to ensure only qualified safety-related parts would be placed in the plant.

MANAGEMENT MEETINGS Exit Meeting Summary On December 6, 2007, the inspector presented the inspection results to Mr. S. Minahan, Vice President-Nuclear and CNO, and members of his staff who acknowledged the findings. The inspector confirmed that no proprietary information was provided or examined during the inspection.

Attachment ATTACHMENTS Persons Contacted J. Bebb, Security Manager T. Carson, Maintenance Manager R. Alexander, Electrical Superintendent M. Tackett, Outage Manager K. Billesbach, Quality Assurance Manager J. Larson, Quality Assurance Supervisor G. Griffith, Reactor Engineer Manager D.L. Willis, Operations Manager V. Bhardwaj, Engineering Support Manager D. Parker, Assistant Maintenance Manager D. Werner, Operations Training Supervisor R.A. Rexroad, System Engineer M. Metzger, System Engineer P.V. Fleming, Nuclear Safety Assurance Director G. Kline, Engineering Director T. Donovan, MP&C Manager R. Estrada, Corrective Action Program Manager M. McCormack, Electrical Systems/I & C System Engineering Supervisor M. Colomb, Plant Operations General Manager B. Kirkpatrick, Licensing Specialist D. Van Der Kamp, Licensing Manager J. Flaherty, Senior Staff Licensing Engineer S. B. Minahan, Site Vice President T. Hottovy, Equipment Reliability Manager D. Montgomery, Assistant to General Manager Plant Operations Documents Reviewed Procedure 0.5 Root Cause, Revision 5 Administrative Procedure 0.5, Conduct of the Condition Report Process, Revision 59 Administrative Procedure 0.5, Condition Report Initiation, Review, and Classification, Revision 8 Administrative Procedure 0.5, Condition Report Initiation, Review, and Classification, Revision 9 System Operating Procedure 2.2.20.2, Operation of Diesel Generators From Diesel Generator Rooms, Revision 39 Human Performance High Intensity Training for Maintenance Personnel, Revision 02 Administrative Procedure 0-QA-09, Supplier Evaluation Process, Revision 22 and 23 Maintenance Procedure 7.3.8.2, Diesel Generator Electrical Examination and Maintenance, Revision 20 Root Cause Investigation: CR 2004-7947 DG1 Inoperable Due to Lube Oil Fitting Leak Root Cause Investigation: CR 2005-01360 DG1 Control Power Failure and Loss of Shutdown Cooling During Sequential Load Test Root Cause Investigation: CR 2007-00480 Unexpected Trip of Emergency Diesel Generator #2 During Surveillance Testing on January 18, 2007

Attachment-2-Root Cause Evaluation of the NRC White Performance Indicator for MSPI for EAC Exceeding the Threshold for EDG Run Demand Failures, CR-CNS-2007-01559, Revision 1 Condition Report CR-CNS-2007-05645, Gap Analysis used to evaluate the finding in the Notice of Violation against the original root causes in CR-CNS-2007-0480 and CR-CNS-2007-1559 Condition Report CR-CNS-2007-07978 Condition Report CR-CNS-2007-00058 Condition Report CR-CNS-2007-05140 Condition Report CR-CNS-2007-00114 Condition Report CR-CNS-2007-00623 Condition Report CR-CNS-2007-00898 Emergency Diesel Generator Reliability Improvement Plan, Revision 4 Diesel Generator Maintenance Improvement Team Plan Cooper Nuclear Station Equipment Trend Report, April-October 2007 Cooper Nuclear Station Monthly CAP Trend Report, October 2007