IR 05000298/2005014

NRC Special Inspection Report 05000298/2005014

ML053540128
Person / Time
Site:	Cooper
Issue date:	12/19/2005
From:	Kennedy K NRC/RGN-IV/DRP/RPB-C
To:	Edington R Nebraska Public Power District (NPPD)
References
IR-05-014
Download: ML053540128 (30)

Text

ber 19, 2005

SUBJECT:

COOPER NUCLEAR STATION - NRC SPECIAL INSPECTION REPORT 05000298/2005014

Dear Mr. Edington:

On October 21, 2005, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection at your Cooper Nuclear Station. The enclosed inspection report documents the inspection findings which were discussed on October 24, 2005, with Mr. S. Minahan, General Manager of Plant Operations, and other members of your staff. Additional in-office reviews were conducted through December 1, 2005, and the final inspection results were discussed with Mr. Minahan and your staff on December 2, 2005.

This inspection examined activities conducted under your license as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your license.

Specifically, the inspector reviewed the circumstances surrounding multiple failures of safety-related motor-operated valves during a forced outage on September 23, 2005.

Based on the results of this inspection, the NRC identified two findings which were evaluated under the risk significance determination process as having very low safety significance (Green). The NRC also determined that there were two violations associated with these findings. However, because these violations were of very low safety significance and the issues were entered into the your corrective action program, the NRC is treating these findings as noncited violations, consistent with Section VI.A.1 of the NRCs Enforcement Policy. These noncited violations are described in the subject inspection report. If you contest the violations or significance of the violations, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001, with copies to the Regional Administrator, U.S. Nuclear Regulatory Commission, Region IV, 611 Ryan Plaza Drive, Suite 400, Arlington, Texas 76011-4005; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident Inspectors at the Cooper Nuclear Station facility.

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response will be made available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records component of NRCs

Nebraska Public Power District -2-document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Should you have any questions concerning this inspection, we will be pleased to discuss them with you.

Sincerely,

/RA/

Kriss M. Kennedy, Chief Project Branch C Division of Reactor Projects Docket: 50-298 License: DPR-46

Enclosure:

NRC Inspection Report 05000298/2005014 w/attachments: Supplemental Information Special Inspection Charter Figures

REGION IV==

Docket.: 50-298 License: DPR-46 Report: 05000298/2005014 Licensee: Nebraska Public Power District Facility: Cooper Nuclear Station Location: P.O. Box 98 Brownville, Nebraska Dates: October 17 through December 1, 2005 Inspector: S. Schwind, Senior Resident Inspector Approved By: K. Kennedy, Branch C, Division of Reactor Projects Enclosure

SUMMARY OF FINDINGS

IR 05000298/2005014; 10/17/05 - 12/01/05; Cooper Nuclear Station. Other Activities.

The report documents special inspection activities conducted by a senior resident inspector.

Two Green noncited violations were identified. The significance of the issues is indicated by their color (Green, White, Yellow, or Red) and was determined by the significance determination process in Inspection Manual Chapter 0609. Findings for which the significance determination process does not apply are indicated by the severity level of the applicable violation. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.

Cornerstone: Mitigating Systems

Green.

The inspector identified a noncited violation of 10 CFR Part 50,

Appendix B, Criterion III, regarding the unauthorized modification of two safety-related motor-operated valves. On September 25, 2005, the licensee modified the mechanical interlocks inside the motor starters for these valves without following the requirements in their modification procedure. As a result, the required torque values specified in the seismic qualification report for this equipment were not used during the modification. The licensee entered this into their corrective action program as CR-CNS-2005-07542 and remounted the interlocks using the appropriate torque values.

The finding was more than minor since configuration control and the maintenance of the plants design basis is a basic principle of safe plant operation and, if left uncorrected, could become a more significant safety concern. The finding was determined to be of very low safety significance since it only involved a design or qualification deficiency that did not result in the loss of a safety function. The finding also had cross-cutting aspects associated with human performance based on the fact that appropriate administrative barriers were in place to ensure that the modifications were performed in accordance with procedures (Section 2.2).

Green.

The inspector identified a noncited violation of 10 CFR Part 50,

Appendix B, Criterion XVI, regarding inadequate corrective actions for motor-operated valve failures. Eight similar valve failures occurred over a 6-year period; however, corrective actions for those failures did not prevent two similar failures in September 2005. The licensee entered this into their corrective action program as CR-CNS-2005-06968.

The finding was more than minor since it affected the cornerstone attributes of availability and reliability of mitigating equipment as well as the operational capabilities of primary containment. The safety significance was assessed using Phase 2 of the Significance Determination Process; however, the performance deficiency did not increase the initiating event frequencies or degrade the mitigating functions described in the Phase 2 analysis. Therefore, the finding was determined to be of very low safety significance. There were also crosscutting aspects associated with problem identification and resolution based on the fact that it was within the licensee's capability to have determined and corrected the valve failure mechanism 2 months prior to the failures in September 2005, yet they failed to do so (Section 5).

REPORT DETAILS

4OA5 Other Activities

Motor-Operated Valve (MOV) Starter Failures

1. Description and Sequence of Events

On September 23, 2005, at 2:48 a.m., operators manually scrammed the reactor in response to lowering main condenser vacuum. The loss of vacuum was caused by the failure of a drain line passing through the main condenser which allowed an unacceptable amount of air in-leakage. In order to repair the leak, the licensee determined that the reactor would have to be placed in cold shutdown (<212EF) using the residual heat removal (RHR) system.

At approximately 11:49 p.m., operators attempted to close the suppression pool suction valve for RHR Pump C (RHR-MOV-MO13C) in order to align Pump C for shutdown cooling. Operators placed the control switch for the valve in the closed position but the position indication lights in the control room did not indicate any valve movement. An auxiliary operator was dispatched to the reactor building and reported an acrid odor in the vicinity of the motor control center (MCC) containing the motor starter for RHR-MOV-MO13C. At approximately the same time, the control room lost position indication for the valve due to a blown fuse in the control power supply for the motor starter.

RHR-MOV-MO13C is a containment isolation valve which is normally open in Modes 1, 2, and 3 in order to satisfy the low pressure coolant injection function of RHR by aligning the Pump C suction to the suppression pool. When RHR Pump C is aligned for shutdown cooling, this valve is required to be closed in order prevent a drain path from the reactor coolant system to the suppression pool. Based on the failure of RHR-MOV-MO13C to operate on demand, operators declared RHR Loop A inoperable, as well as the associated containment penetration, and appropriately implemented the required actions in Technical Specifications 3.5.1 and 3.6.1.3.

At approximately 2:40 a.m. on September 24, operators attempted to open the shutdown cooling suction valve for RHR Pump B (RHR-MOV-MO15B) in order to align Pump B for shutdown cooling. Operators placed the control switch in the open position but observed no indication of valve movement. The switch was taken back to the close position. A short time later, operators attempted to re-open the valve and were successful, but the decision was made not to use Pump B for shutdown cooling due to the previous failure. At approximately 6:53 a.m., operators were successful in establishing shutdown cooling using RHR Pump D.

The NRC evaluated these valve failures in accordance with Management Directive 8.3, NRC Incident Investigation Program, and determined the need to conduct a special inspection regarding the cause of the failures and the licensees corrective actions. The inspection charter is included as Attachment 2 to this report.

2. Root Cause and Immediate Corrective Actions

2.1 Root Cause for Motor Starter Failures The licensee documented the failure of RHR-MOV-MO13C in Condition Report CR-CNS-2005-06968 and the failure of RHR-MOV-MO15B in Condition Report CR-CNS-2005-07000. CR-CNS-2005-06968 was classified as a Category A condition report which was subject to a full root cause investigation. The inspector observed troubleshooting on the failed valves and reviewed the licensees root cause to assess the adequacy of the their cause determination. Condition Report CR-CNS-2005-07000 was classified as a work item only and was closed following completion of corrective maintenance on the valve.

RHR-MOV-MO13C and RHR-MOV-MO15B are safety-related MOVs, which are controlled by full voltage, reversing motor starters located in MCC Q and MCC Y, respectively. Both of these MCCs are located inside the reactor building. These starter assemblies were manufactured by Westinghouse Electric Corporation and were installed between 1986 and 1988. There are approximately 100 additional safety-related valves in the plant with identical starter assemblies. The starters contain two sets of contactors, each of which energize to operate the valve in the open or close direction. A mechanical interlock is installed between the contactors to prevent both sets of contactors from closing at the same time, which could cause damage to the valve actuator motor. The interlock also provides assurance against inadvertent valve actuation during a seismic event and serves as a time delay mechanism between the open and close strokes of a valve to provide additional electrical protection for valve actuator components. Figures 1 and 2 show a typical starter assembly and mechanical interlock.

During troubleshooting on RHR-MOV-MO13C and RHR-MOV-MO15B, the licensee determined that the mechanical interlock between the contactors in each of the starter assemblies was deformed. This deformation caused the interlock to bind such that when a valve operation was demanded from the control room, the associated contactor would energize but would not have sufficient force to move the interlock. In the case of RHR-MOV-MO13C, this binding caused the close contactor to remain energized in an attempt to close the valve until it overheated, causing the control power fuse to open. In the case of RHR-MOV-MO15B, the successful operation of the valve on the second attempt indicated that the failure mechanism was intermittent in nature. The licensee also determined the most likely cause for the deformation was a manufacturing issue with the starter assemblies. The interlock is mounted to the backplate of the starter assembly with an L bracket which is held in place by a single screw. The mounting holes for the interlocks in these two starter assemblies were off-center; therefore, external force had to be applied to the edges of the interlocks in order to mount them using these holes. Over time, the external force caused stress relaxation in the nylon plastic case of the interlock, which resulted in misalignment of the internal components and caused the interlock to bind.

The inspector concluded that this root cause was well supported by physical evidence from the failed starter assemblies as well as industry operating experience regarding similar failures on similar starter assemblies.

2.2 Immediate Corrective Actions

a. Inspection Scope

The inspector reviewed the licensees immediate corrective actions for the failures of RHR-MOV-MO13C and RHR-MOV-MO15B. This review included the work orders used to modify the motor starter assemblies, the design bases for the components which were modified, and the station procedure for configuration control and design modifications.

b. Findings

Introduction.

The inspector identified a Green noncited violation (NCV) regarding an unauthorized modification to safety-related MOVs.

Details. As an immediate corrective action for the interlock binding issue, the licensee determined that the mechanical interlocks in the starter assemblies for RHR-MOV-MO13C and RHR-MOV-MO15B should be replaced with new interlocks that were mounted in a manner that ensured no external forces were applied to their cases. In order to realign them to the desired condition, the licensee rotated the interlocks 180E and drilled new mounting holes in the backplate of the assembly which allowed installation of the interlock L brackets such that no external forces would be applied to interlock cases. This work was accomplished on September 25 using Maintenance Work Orders 4454157 (RHR-MOV-MO15B) and 4463594 (RHR-MOV-MO13C).

Engineering Procedure 3.4, Configuration Change Control, Revision 41C1, states that configuration changes to components that are safety-related shall be controlled in accordance with this procedure, 10 CFR 50.59, and 10 CFR Part 50, Appendix B. The alterations to the interlock mounting brackets constituted a configuration change to safety-related components; however, Engineering Procedure 3.4 was not used in development of the maintenance work orders nor did the licensee ensure that the requirements of 10 CFR 50.59 or 10 CFR Part 50, Appendix B, were satisfied.

Therefore, the alterations to the starter assemblies were determined by the inspector to be an unauthorized modification.

According to the Updated Final Safety Analysis Report, Section 2.3.5, the two MCCs containing these starter assemblies are required to be qualified to Seismic Class I standards. These MCCs, along with the starter assemblies were qualified to Seismic Class I standards as documented in Westinghouse Seismic Certification Report 20278-CCR-0. However, this qualification report required that fasteners of the size used to mount the interlocks be installed using 19 in-lbs of torque. Work Order 4463594 specified the mounting screw to be tightened to 17 in-lbs of torque; Work Order 4454157 provided no torque specification for the mounting screw. Therefore, the failure to maintain this design requirement invalidated the seismic qualification of these two starter assemblies.

The inspector questioned the adequacy of this unauthorized modification and, as a result, the licensee performed additional corrective maintenance on the starter assemblies which entailed loosening the mounting screws and re-tightening them using 19 in-lbs of torque. No as-found torque values were obtained during this work so the licensee was unable to assess the actual impact on the seismic qualification of the starter assemblies. In addition, the licensee reviewed this modification against the requirements of 10 CFR 50.59 and determined that prior NRC approval would not have been required for this modification. The licensee also verified the technical adequacy of the modification with the original equipment manufacturer, who concurred with the modification.

Analysis.

The unauthorized modification to RHR-MOV-MO13C and RHR-MOV-MO15B represented a performance deficiency since it was reasonably within the licensees ability to adhere to their procedural requirements for the control of plant modifications and to satisfy the requirements in 10 CFR 50.59 and 10 CFR Part 50, Appendix B. The failure to maintain seismic qualification of the mechanical interlocks could have contributed to inadvertent valve operation during a seismic event. Although RHR-MOV-MO13C is a containment isolation valve, this failure mode of both valves only affected the Mitigating Systems Cornerstone; the Barrier Integrity Cornerstone was not affected since this finding did not represent an actual breach or bypass of containment and it did not involve containment heat removal, hydrogen control, or pressure control. The finding was more than minor since configuration control and the maintenance of the plants design basis is a basic principle of safe plant operation and, if left uncorrected, could become a more significant safety concern. The finding was also determined to be of very low safety significance based on Phase 1 of the Significance Determination Process (SDP) screening, since it only involved a design or qualification deficiency that did not result in the loss of a safety function.

This finding also had crosscutting aspects associated with human performance. This assessment was based on several facts: the procedure requirements of Engineering Procedure 3.4 were unambiguous pertaining to these modifications; significant training and management emphasis have been placed on maintaining configuration control over the past 3 years; and improvements to engineering programs, including design modifications and configuration management was included in the Confirmatory Action Letter (CAL) which was closed in January 2005. In their CAL Closure Assessment Report, July 15, 2004, the licensee stated that the Strategic Improvement Plan had been largely effective in improving this area. Based on this, the inspector concluded that the only remaining barrier to errors in this area, human performance, had failed.

Enforcement.

10 CFR Part 50, Appendix B, Criterion III, Design Control, requires that measures shall be established to assure that applicable regulatory requirements and the design basis for safety-related components are correctly translated into specifications, drawings, procedures, and instructions. This requirement is implemented by Engineering Procedure 3.4, Configuration Change Control, Revision 41C1, which is mandatory for all configuration changes to safety-related components. Contrary to these requirements, the licensee failed to utilize Engineering Procedure 3.4 to modify the safety-related motor starters for RHR-MOV-MO13C and RHR-MOV-MO15B. The motor starters were modified on September 25, 2005, by Maintenance Work Orders 4454157 and 4463594 which did not comply with Engineering Procedure 3.4 and did not contain adequate information to assure that the design basis for this equipment was maintained. As a result, the seismic qualification of this equipment was invalidated.

This violation, which is of very low safety significance and was entered into the licensees corrective action program as Condition Report CR-CNS-2005-07542, is being treated as an NCV, consistent with Section VI.A of the NRC Enforcement Policy: NCV 05000298/2005014-01, Unauthorized Modification to Safety-Related Motor-Operated Valves.

3. Extent of Condition and Operability Evaluation

The inspector reviewed the licensees extent of condition evaluation and operability evaluation associated with the failed MOV motor starter assemblies. This review included field observations of approximately nine starter assembly inspections, a review of the extent of condition sample size selection criteria, and a review of the starter assembly inspection and acceptance criteria.

During the initial stages of their root cause investigation, the licensee identified a total of 108 safety-related MOVs in the plant which were controlled by similar full voltage, reversing motor starter assemblies. All of these starter assemblies were manufactured by Westinghouse and were purchased and installed between 1986 and 1988. Based on this, and the potential generic nature of the failure mechanism, the licensee selected a sample from the 108 valves for additional inspection prior to restart from the forced outage on September 25. The sample was selected based on several factors:

Historic Failure Rate: Preliminary information suggested that similar failures had occurred in the past, primarily in valves associated with the RHR system.

Location: Past failures of the starter assemblies appeared to have been clustered in MCC Q and MCC Y.

Procedures: Existing surveillance procedures were used to test the selected valves. Since these procedures typically test multiple valves, all valves in a given procedure were included in the sample.

This sampling methodology resulted in a total of 30 additional valves which were inspected and tested. The inspections included a visual inspection of the motor starter assembly for obvious signs of interlock misalignment or discoloration of components due to overheating, a valve stroke to verify that there was no apparent interference in contactor movement due to the interlock, and an audible check during the valve stroke to check for signs of contactor chatter. These inspection criteria were based on observations of the failed motor starter assemblies during troubleshooting activities. All 30 valve inspections were completed prior to restarting the reactor on September 25.

Licensee personnel did not identify any indications of starter assembly failures during these inspections. This information was used to justify continued plant operation without any further corrective maintenance to the remaining population of motor starter assemblies.

The inspector reviewed the inspection criteria and observed approximately nine inspections on September 25 and during the following 3 weeks. Based on this review, the inspector concluded that the inspections could provide valuable information to assess the operability of the motor starters; however, the inspector found no documentation regarding what constituted acceptable inspection results. During interviews with individuals responsible for the inspections and during observations of the inspections, the inspector found that the licensees acceptance criteria were not objective, were not necessarily predictive, and in some cases, would only indicate that a motor starter had already experienced some level of failure. For example, the valve stroke and audible check for contactor chatter would only indicate that an interlock was already bound to the point of causing a failure. The valve stroke by itself was not a predictive indicator of valve failure based on the intermittent nature of the failure mechanism. This was demonstrated on September 24 when RHR-MOV-MO15B failed to operate on the first attempt but was successful on the second attempt. In addition, the visual inspection for discoloration of components would only indicate that a starter assembly had already experienced some level of failure, not that a failure was imminent.

The visual inspection for signs of interlock misalignment was the only inspection criteria with the potential to predict further valve failures; however, the acceptance criteria for this inspection was subjective and was not documented. In addition, the level of rigor applied to conducting these inspections varied and, in some cases, did not appear to be adequate for detecting anomalies in the starter assemblies. During field observations of starter assembly inspections conducted on October 18, the inspector noted that maintenance personnel required engineering personnel responsible for the inspections to remain outside a 5-foot exclusion zone around the starter assembly due to arc-flash safety concerns. They stated that the engineer could enter the exclusion zone to perform the inspections, provided appropriate arc-flash protective clothing was worn.

This was not done and the engineer remained outside the exclusion zone. The inspector concluded that an adequate visual inspection could not have been performed from that distance. Furthermore, during these inspections, none of the licensee personnel had copies of the work order or inspection documentation until questioned by the inspector. Copies of the documentation were obtained prior to commencing the inspections.

In addition, according the licensees root cause investigation, a deflection of as little as 0.040 inches in the interlock case could be sufficient to cause interlock binding. Based on their own observations, the inspector found this amount of deflection difficult to observe due to wiring and other components of the starter assembly obscuring the view of the interlock. However, it was possible to observe this if the interlock was carefully inspected from close proximity and from several different angles. The inspector also found digital photographs improved the ability to assess the condition of the interlock.

The licensee did not employ any of these techniques during their visual inspections which, in some cases, consisted only of viewing the interlock from a single angle greater than 12 inches away. In the case of RHR Heat Exchanger B service water outlet Valve SW-MOV-MO89B, which was inspected on October 20, the licensees initial inspection revealed no anomalies. However, Figure 3, a digital photograph taken by the inspector, shows evidence of interlock deflection. The inspector discussed this photograph with the licensee and, as a result, Condition Report CR-CNS-2005-07746 was written to document the misalignment. This was later evaluated as acceptable. Based on these discussions, the licensee also reinspected several motor starters whose inspection results were questionable based on the lack of objective inspection criteria or the lack of rigor used to perform the initial inspections.

4. Long-Term Corrective Actions

The licensees long-term corrective actions for the motor starter assembly failures include a modification to all the mechanical interlocks used in Westinghouse full voltage, reversing motor starter assemblies at Cooper. This modification will replace all existing interlocks with new interlocks that have a slotted screw hole machined in the mounting bracket. The interlock will be mounted using the existing holes in the assembly backplate and will be adjusted so that no external force is applied to the case. The licensee believes this modification will eliminate the failure mechanism.

An additional corrective action requires that all MOVs using the Westinghouse starter assembly undergo the same inspection process as discussed in Section 3 by December 30, 2005, and every 6 months thereafter until modification of the starter assembly is complete. The licensee established a due date of February 28, 2006, for completion of the modifications. The inspector reviewed the initial schedule for inspecting the remaining population of motor starter assemblies and noted that it made no differentiation between high risk and low risk valves. After discussing this, the licensee prioritized these inspections and completed the inspections of all high risk valves on October 22.

5. Prior Opportunities to Identify Failure Mechanism

a. Inspection Scope

The inspector reviewed the history of MOV failures which exhibited similar symptoms to the failures on September 23 and 24, 2005. The review included pertinent corrective action program documents, maintenance work requests, and interviews with cognizant engineers.

b. Findings

Introduction.

The inspector identified a Green NCV regarding inadequate corrective actions for previous similar MOV failures.

Details. The inspector determined that the licensee had experienced eight MOV failures since 1999 which exhibited symptoms similar to the failures in September 2005. Based on this information, the inspector identified a number of missed opportunities for the licensee to have identified and corrected the failure mechanism prior to the most recent failures.

Valve Failure Symptoms Cause as Stated in Corrective Date Action Program RHR-MOV-MO13A 10/99 Hesitation Contactor failure HV-MOV-MO266 4/00 Failed to Stroke Mechanical interlock misalignment RHR-MOV-MO13D 10/02 Hesitation Control Switch Contacts SW-MOV-MO37 2/03 Slow Stroke None Time RHR-MOV-MO13C 9/03 Hesitation Control Switch Contacts RHR-MOV-MO13D 10/04 Hesitation Control Switch Contacts RHR-MOV-MO13D 7/05 Failed to Stroke None RHR-MOV-MO15B 7/05 Hesitation None RHR-MOV-MO13C 9/05 Failed to Stroke Mechanical Interlock Binding RHR-MOV-MO15B 9/05 Failed to Stroke Mechanical Interlock Binding Table 1: SIMILAR MOV FAILURES AT COOPER NUCLEAR STATION Table 1 summarizes the similar valve failures experienced between 1999 and 2005.

This failure data illustrates four notable examples of missed opportunities for the licensee to have identified and corrected the failure mechanism:

• On April 7, 2000, HV-MOV-MO266 (Reactor Recirculation Motor Generator Set Ventilation Outlet Isolation Valve), which is a secondary containment isolation valve, failed to stroke upon demand. Maintenance Work Request 00-1190 was initiated to troubleshoot the valve failure. This work order documented the following:

Condition: Mechanical interlock (which prevents both open and close contactors from closing at the same time) does not perform its intended function. Work Scope: Adjust Mechanical Interlock and Contactors The valve was tested successfully following this corrective maintenance. This failure was also documented in the corrective action program as Nonconformance Report 4-08192. This was dispositioned as a work item only and no formal evaluation, extent of condition review, or corrective actions resulted from it.

• On July 19, 2005, during a routine quarterly surveillance test, RHR-MOV-MO13D failed to immediately indicate valve movement when operators placed the control switch to the close position. An auxiliary operator was dispatched to the reactor building to inspect the valve and reported no abnormal conditions.

Approximately 2 to 3 minutes after the control switch was operated, the valve began to move and stroked closed. The licensee did not determine the cause of this failure; however, most of the components in the associated control circuit, including the mechanical interlock, were replaced and the valve was retested successfully. This failure was documented in Condition Report CR-CNS-2005-05183. Following the failures in September, the licensee concluded that the symptoms associated with this failure were indicative of mechanical interlock binding. Most likely, the mechanical interlock had deformed to the point where the force required to change states of the interlock was approximately equal to the force generated by the contactor coil. Therefore, when the contactor was energized by operation of the control switch, it took 2 to 3 minutes for the contactor to overcome the resistance in the interlock in order to complete the circuit for the actuator motor.

• On July 22, 2005, during a routine quarterly surveillance test, RHR-MOV-MO15B failed to immediately indicate valve movement when operators placed the control switch to the open position. The valve began to move approximately 90 seconds later. A humming noise was also heard coming from the motor starter assembly during the valve stroke. This failure was documented in Condition Report CR-CNS-2005-05261. The Suggested Action Description field of this condition report stated Problem appears to be mechanical binding in the contactors, the mechanical interlock, or the aux[iliary] contactor; recommend trending this item, and scheduling a breaker inspection for RHR-MOV-MO15B.

Since this valve is only required to be operable for shutdown cooling, no further troubleshooting or corrective maintenance was performed at this time. Following the failures in September, the licensee concluded that the symptoms associated with this failure were indicative of mechanical interlock binding, similar to the failure of RHR-MOV-MO13D.

• Several of the failures listed in Table 1 include symptoms where the valve did not begin to move immediately upon demand (hesitation) and were attributed to a black oxide layer on the contacts in the control switches. This was supported by industry operating experience regarding this particular style of switch. During discussions with the inspector, the licensee acknowledged that the symptoms of all of the failures listed in Table 1 were similar to symptoms of mechanical interlock binding but that there was no conclusive evidence. In addition, the licensee also acknowledged that the black oxide layer observed on the control switch contacts could have resulted from high electrical currents passing through the control switch while the mechanical interlock was preventing movement of the contactor. None of the failures listed in Table 1 were the subject of a full root cause investigation, with the exception of the most recent failures in September 2005. At most, these failures received less rigorous apparent cause determinations which relied upon limited use of industry operating experience and the knowledge that faulty control switches had been the apparent cause of previous failures. In the case of RHR-MOV-MO13D, there were three failures over a 3-year period, the first two of which were attributed to the control switch.

Despite having replaced the control switch twice in a 2-year period, the apparent cause for the third failure of RHR-MOV-MO13D listed the control switch as a possible cause and it was replaced for a third time.

When questioned why they were able to identify the cause of the failures following the September 2005 failures and not earlier, the licensee stated that it was because they had never performed a full root cause investigation for the past failures. Administrative Procedure 0.5CR, Condition Report Initiation, Review, and Classification, 1, provides guidance for classification of condition reports in order to determine the level of evaluation required. Category A condition reports are subject to a full root cause investigation. According to this guidance, a condition report should be classified as Category A in the following cases:

• Equipment failures that prevented fulfillment of, or have a high potential for preventing the fulfillment of, a safety function.

• Equipment failures that adversely affected proper control of a fission product barrier (e.g., primary or secondary containment).

• Trends of similar equipment failures that resulted in a significant impact to reactor safety.

Based on these statements, the inspector concluded that the licensee should have performed a root cause investigation prior to the failures in September 2005. The first seven failures listed in Table 1 involved primary or secondary containment isolation valves and the cluster of failures in the RHR system presented the high potential for challenging a safety function. Furthermore, six of the failures occurred within a 3-year period, which should have been identified as a trend of similar safety-related equipment failures.

Finally, the inspector found that sufficient industry operating experience was available regarding binding of mechanical interlocks in safety-related motor starter assemblies to have identified the failure mechanism prior to the failures in September 2005. Section 7 provides more discussion on industry operating experience.

Analysis.

The licensee failed to take corrective actions to prevent recurrence of safety-related MOV failures due to mechanical interlock binding. It was reasonably within their ability to have done so based on: their history of MOV failures; the information available in the maintenance history of the motor starters; the procedure requirements of the corrective action program; and the availability of applicable industry operating experience. Therefore, the inadequate corrective actions, which resulted in the failures of RHR-MOV-MO13C and RHR-MOV-MO15B, were considered to be a performance deficiency. This finding affected the Mitigating Systems and Barrier Integrity Cornerstones since the RHR system is required to mitigate the consequences of an accident and also RHR-MOV-MO13C is a primary containment isolation valve. The finding was more than minor since it affected the cornerstone attribute of availability and reliability of mitigating equipment and it affected the operational capability of primary containment. Since two cornerstones were affected by the finding, an SDP Phase 2 analysis was required. During the Phase 2 analysis, the inspector assumed that the performance deficiency existed for more than 30 days but that it only degraded the shutdown cooling function of the RHR system and the containment isolation function of RHR-MOV-MO13C. Since neither of these functions are described in the Risk-Informed Inspection Notebook for Cooper Nuclear Station, Revision 1, and none of the initiating event frequencies or other mitigating functions were affected, the Phase 2 analysis resulted in a finding of very low safety significance. Evaluation of the finding using MC 0609, Appendix H, was not required due to the low core damage frequency indicated by the Phase 2 analysis. These results were validated by a Senior Rector Analyst who also confirmed that the risk due to the common mode failure mechanism for all MOVs was bounded by the Phase 2 results.

This finding also had crosscutting aspects associated with problem identification and resolution. The licensee had multiple opportunities, including the failures in July 2005, to have identified and corrected the cause of the MOV failures, but failed to do so.

Enforcement.

10 CFR Part 50, Appendix B, Criterion XVI, requires that measures shall be established to assure that conditions adverse to quality, such as failures and malfunctions, are promptly identified and corrected. In the case of significant conditions adverse to quality, the measures shall assure that the cause of the condition is determined and corrective action taken to preclude repetition. Contrary to this requirement, Cooper Nuclear Station failed to identify, correct, and preclude repetition of a significant condition adverse to quality. Specifically, on September 23 and 24, 2005, Valves RHR-MO-MOV13C and RHR MO-MOV15B failed to properly operate due to binding of the mechanical interlock in each of the valves motor starter assemblies.

Cooper Nuclear Station experienced eight similar valve failures over a 6-year period, but failed to identify the root cause and failed to take corrective actions to prevent further valve failures, including the failures experienced in September 2005. This violation, which was determined to have very low safety significance and was entered into the licensees corrective action program as CR-CNS-2005-06968, is being treated as an NCV, consistent with Section VI.A of the NRC Enforcement Policy: NCV 05000298/2005014-02, Inadequate Corrective Actions for Motor-operated Valve Failures.

6. Use of Industry Operating Experience

The inspector conducted an independent search of industry operating experience to determine if similar failures had occurred elsewhere. The search identified multiple failures of Westinghouse full voltage, reversing starters, and similarly designed motor starters due to binding of the mechanical interlock. Some noteworthy examples included:

On May 31, 1988, Haddam Neck submitted Licensee Event Report 88-013-00 which reported the failure of the volume control tank isolation valve due to binding of the mechanical interlock within the reversing starter assembly. The starter assembly and interlock were provided by Westinghouse Electric Corporation.

In 1984, Rancho Seco reported the failure of a hydrogen recombiner containment isolation valve due to misalignment of the mechanical interlock in a Westinghouse reversing motor starter.

In October 2002, V.C. Summer reported that mechanical interlocks in safety-related motor starters were found to be susceptible to binding. The report was in reference to Westinghouse full voltage, reversing starter assemblies. This report also described that contactor assembly chatter was a symptom of this failure.

Cooper Nuclear Station detected contactor chatter in July 2005 on RHR-MOV-MO15B.

The inspector concluded that sufficient industry operating experience was available to indicate problems with mechanical interlock binding in these motor starter assemblies, but this experience was not used during previous investigations into MOV failures. In their CAL Closure Assessment Report, July 15, 2004, the licensee stated that the Strategic Improvement Plan had been fully effective in improving the use of operating experience to minimize the potential for adverse impacts at Cooper Nuclear Station.

CR-CNS-2005-06968 recommended further enhancements and training on the use of operating experience.

7. Potential Generic Issue

On October 12, 2005, Cooper submitted an operating experience report to alert the industry to this potential failure mechanism. In addition, the details of the failures were communicated to Westinghouse to evaluate the need to submit a report in accordance with 10 CFR Part 21.

4OA6 Meetings, Including Exit

On October 24, 2005, the preliminary results of this inspection were presented to Mr. S. Minahan and other members of his staff who acknowledged the findings.

Following additional in-office reviews, the final results of the inspection were presented to Mr. Minahan and his staff on December 2, 2005. The inspector confirmed that the supporting details in this report contained no proprietary information.

ATTACHMENT:

SUPPLEMENTAL INFORMATION

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

T. Bahensky, System Engineer

D. Barker, Corrective Action and Assessment

J. Bednar, Emergency Preparedness Manager

V. Bhardwaj, Engineering Support Manager

D. Billesbach, Electrical Maintenance Superintendent

K. Billesbach, Quality Assurance Supervisor

C. Blair, Engineer, Licensing

S. Blake, Manager, Quality Assurance

M. Boyce, Nuclear Asset Manager

K. Chambliss, Operations Manager

T. Chard, Radiological Manager

J. Christensen, General Manager of Support

D. Cook, Technical Assistant to General Manager

R. Edington, Vice President

R. Estrada, Corrective Actions Manager

K. Fili, Manager, Nuclear Projects

R. Fili, System Engineering Manager

J. Flaherty, Site Regulatory Liaison

P. Fleming, Licensing Manager

D. Kimbell, Outage Manager

G. Kline, Director, Engineering

M. McCormack, System Engineering Supervisor

S. Minahan, General Manager of Plant Operations

A. Mitchell, Design Engineering Manager

H. Northrop, Materials Manager

J. Roberts, Director, Nuclear Safety Assurance

L. Schilling, Administrative Services Manager

G. Seeman, Risk Management

R. Shaw, Shift Manager

J. Sumpter, Senior Staff Engineer, Licensing

M. Tackett, Operations Supervisor

K. Tanner, Shift Supervisor, Radiation Protection

J. Whilstler, Work Control Supervisor

D. Willis, Maintenance Manager

NRC Personnel

S. Cochrum, Senior Resident Inspector

Attachment 1

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened and Closed

05000298/2005014-01 NCV Unauthorized Modification to Safety-Related Motor-

Operated Valves (Section 2.2)05000298/2005014-02 NCV Inadequate Corrective Actions for Motor-Operated Valve

Failures (Section 5)

LIST OF ACRONYMS

CAL confirmatory action letter

CFR Code of Federal Regulations

FIN finding

MCC motor control center

MOV motor-operated valve

NCV noncited violation

NRC U.S. Nuclear Regulatory Commission

RHR residual heat removal

SDP significance determination process

Attachment 1

October 14, 2005

MEMORANDUM TO: Scott Schwind, Senior Resident Inspector

Division of Reactor Projects

FROM: Kriss

M. Kennedy, Chief, Projects Branch C /RA/

Division of Reactor Projects

SUBJECT: SPECIAL INSPECTION CHARTER TO EVALUATE MOTOR STARTER

FAILURES AT COOPER NUCLEAR STATION

In response to the failure of two 480-volt safety-related motor starters to properly operate on

September 23, 2005, a Special Inspection is being chartered. You are hereby assigned to

conduct the Special Inspection.

A. Basis

On September 23, 2005, two valves in the residual heat removal system failed to

properly operate. Motor-operated Valve RHR-MOV-MO13C, the RHR Train C

Suppression Pool Isolation Valve, failed to close when operators were aligning RHR for

shutdown cooling. The licensee inspected the 480-volt motor starter for the motor-

operated valve and found that the close contactor coil had remained energized until it

overheated and failed. Upon inspection of the mechanical interlock, the licensee

identified that an inappropriate mounting arrangement caused a slight deformation of

the nylon mechanical interlock which caused the interlock to bind and prevented the

energized motor starter from engaging properly. The motor starter mechanical interlock

device is designed to physically prevent one of the two contactors from closing when the

other is closed.

Later the same day, as operators attempted to open RHR-MOV-MO15B, the RHR Train

B Shutdown Cooling Isolation Valve to place RHR into shutdown cooling, the valve did

not immediately open when the control switch was taken to the open position. The valve

opened on the second attempt. Inspection of the motor starter associated with this

valve found that the failure mechanism was similar to that of MO13C. The mechanical

interlock was deformed due to interference with the contactors which was causing it to

bind intermittently.

The licensee believes that the failures of the motor starters may be a combination of

misalignment between the mechanical interlock and the motor starter, and degradation

of the mechanical interlock due to aging. Cooper Nuclear Station has approximately

100 valves that are the same as the two that failed on September 23.

The licensee determined that they have had a total of 7 failures of motor starters since

1988. Six of these failures have occurred in the last 6 years.

Attachment 2

Scott Schwind -2-

Management Directive 8.3, NRC Incident Investigation Program, was used to evaluate

the level of NRC response for this event. In evaluating the deterministic criteria of MD

8.3, it was determined that these motor starter failures: (1) led to the loss of a safety

function or multiple failures in systems used to mitigate an actual event; (2) involved

possible adverse generic implications; and, (3) involved repetitive failures or events

involving safety-related equipment or deficiencies in operations. Since the deterministic

criteria was met, the motor starter failures were evaluated for risk. The preliminary

Estimated Conditional Core Damage Probability was determined to be 4.1E-6.

Based on the inspection that has occurred since the failures on September 23, the

following specific concerns have been identified that warrant further inspection and

assessment:

• Following the failures on September 23, the licensee inspected 30 motor starters

for signs of failure or impending failure. Although they did not identify any

failures, the resident inspectors have concerns about the adequacy of the

inspection criteria in identifying degraded mechanical interlocks.

• The licensee has not yet developed a schedule for inspecting the remaining 70

motor starters.

• The licensee has not yet evaluated the failures for potential Part 21 notifications.

• In repairing the motor starters that failed on September 23, the NRC identified

that the licensee performed an unauthorized modification when they reinstalled

the new the mechanical interlocks.

• The adequacy of the licensees corrective actions for previous failures at CNS.

• Licensee actions taken as a result mechanical interlock failures at other plants

communicated through operating experience and generic communications.

Based on the deterministic criteria and risk insights related to the failures of the motor

starters, Region IV determined that the appropriate level of NRC response was the

conduct of a Special Inspection.

This Special Inspection is chartered to identify the circumstances surrounding this event,

determine if there are adverse generic implications, and review the licensees actions

following discovery of the condition.

B. Scope

The inspection is expected to perform data gathering and fact-finding in order to

address the following:

• Develop a complete description of the motor starter failures experienced on

September 23, 2005, and a complete sequence of events related to the event.

Attachment 2

Scott Schwind -3-

• Identify and evaluate the effectiveness of the immediate actions taken by the

licensee in response to this event. Determine if the modifications that the

licensee made to the mechanical interlocks and motor starters were conducted in

accordance with licensee procedures.

• Assess the extent of condition and the licensees operability evaluation for similar

motor starters to include seismic and environmental qualifications.

• Identify and assess additional actions planned by the licensee in response to this

event, including the timeline for their completion of these actions.

• Develop a list of failures of similar motor starters to include dates, cause of the

failures, actions taken by the licensee to correct the failures, and the

maintenance history of the failed motor starters.

• Evaluate the licensees corrective actions for previous failures including actions

taken, including the adequacy of maintenance procedures used to repair the

motor starters and the adequacy of training for personnel conducting the

maintenance.

• Evaluate pertinent industry operating experience and potential precursors to the

event, including the effectiveness of any action taken in response to the

operating experience.

• Determine if there are any generic issues related to the failure of the motor

starters. Promptly communicate any potential generic issues to regional

management.

• Assess the safety significance of any inspection findings.

C. Guidance

Inspection Procedure 93812, "Special Inspection," provides additional guidance to be

used by the Special Inspection Team. Your duties will be as described in Inspection

Procedure 93812. The inspection should emphasize fact-finding in its review of the

circumstances surrounding the event. It is not the responsibility of the team to examine

the regulatory process. Safety concerns identified that are not directly related to the

event should be reported to the Region IV office for appropriate action.

Although you have already conducted routine inspection activities related to this event,

you will formally begin the special inspection with an entrance meeting to be conducted

no later than October 17, 2005. The inspection will include a review of the results of the

licensees root cause analysis. You should brief Region IV management during the

Attachment 2

Scott Schwind -4-

course of your inspections and prior to your exit meeting. A report documenting the

results of the inspection should be issued within 30 days of the completion of the

inspection.

This Charter may be modified should you develop significant new information that

warrants review. Should you have any questions concerning this Charter, contact me at

(817) 860-8144.

cc via E-mail:

B. Mallett

T. Gwynn

J. Dixon-Herrity

A. Howell

D. Chamberlain

A. Vegel

K. Kennedy

V. Dricks

W. Maier

W. Walker

D. Terao

D. Duvigneaud

Attachment 2

Figure 1: Typical Westinghouse Full Voltage, Reversing Starter Assembly, NEMA Class 1

Figure 2: Westinghouse M34-1A Mechanical Interlock

Figure 3: Mechanical Interlock Installed in Starter Assembly for SW-MOV-MO89B