IR 05000382/2015010: Difference between revisions
StriderTol (talk | contribs) (Created page by program invented by StriderTol) |
StriderTol (talk | contribs) (StriderTol Bot change) |
||
| (11 intermediate revisions by the same user not shown) | |||
| Line 2: | Line 2: | ||
| number = ML15097A550 | | number = ML15097A550 | ||
| issue date = 04/09/2015 | | issue date = 04/09/2015 | ||
| title = | | title = IR 05000382/2015010, March 16 Through 19, 2015, Waterford Steam Electric Station, Unit 3 - Supplemental | ||
| author name = Pruett T | | author name = Pruett T | ||
| author affiliation = NRC/RGN-IV/DRP | | author affiliation = NRC/RGN-IV/DRP | ||
| addressee name = Chisum M | | addressee name = Chisum M | ||
| addressee affiliation = Entergy Operations, Inc | | addressee affiliation = Entergy Operations, Inc | ||
| docket = 05000382 | | docket = 05000382 | ||
| Line 19: | Line 19: | ||
=Text= | =Text= | ||
{{#Wiki_filter | {{#Wiki_filter:April 09, 2015 | ||
==SUBJECT:== | |||
WATERFORD STEAM ELECTRIC STATION, UNIT 3 - NRC SUPPLEMENTAL INSPECTION REPORT 05000382/2015010 | |||
==Dear Mr. Chisum:== | |||
Prior to May 26, 2013, your staff failed to establish an adequate test program to demonstrate that the train B emergency diesel generator ventilation exhaust fan would perform satisfactorily in service. This performance deficiency resulted in a failure to identify that the exhaust fan could not perform its function because it disengaged from the fan motor in April 2013. Consequently, the train B emergency diesel generator was determined to be inoperable for a period of 30 days, exceeding the Technical Specification 3.8.1 allowed outage time of 72 hours. | |||
On December 19, 2013, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection at the Waterford Steam Electric Station, Unit 3. Based on the results of this inspection, documented in NRC Inspection Report 05000382/2013008 on January 30, 2014 (ADAMS ML14030A616), and the final significance determination documented in NRC Inspection Report 05000382/2014009 on March 28, 2014 (ADAMS ML14086A768), the NRC assigned a White finding Action Matrix input to the mitigating systems cornerstone in the fourth quarter of 2013. | |||
In response to this Action Matrix input, the NRC completed a supplemental inspection using Inspection Procedure 95001, Supplemental Inspection for One or Two White Inputs in a Strategic Performance Area, on October 10, 2014, and documented the results in NRC Inspection Report 05000382/2014011 (ADAMS ML14364A412). The NRC determined that your extent of cause review was insufficient; therefore not all inspection objectives were met, and the White finding remained open and continued to receive consideration as an Action Matrix input. | |||
On March 3, 2015, you informed the NRC that the Waterford Steam Electric Station, Unit 3, was ready for a supplemental re-inspection. On March 19, 2015, the NRC completed a supplemental re-inspection and discussed the results of the inspection with you and other members of your staff. On March 24, 2015, Mr. Ryan Lantz, Chief, Reactor Projects Branch E, | |||
UNITED STATES NUCLEAR REGULATORY COMMISSION | |||
==REGION IV== | |||
1600 E LAMAR BLVD ARLINGTON, TX 76011-4511 NRC Region IV, conducted a Regulatory Performance Meeting with Mr. John Jarrell, Regulatory Assurance Manager. The topics of this meeting included the NRC inspectors observations, your staffs improved effort preparing for this inspection, and the lessons learned from the revised root cause and extent of cause evaluations. The result of the inspection is documented in the enclosed inspection report. | |||
The NRC performed this supplemental inspection to determine if (1) the root and contributing causes for the significant issues were understood, (2) the extent of condition and extent of cause for the identified issues were understood, and (3) your completed or planned corrective actions were sufficient to address and prevent repetition of the root causes and contributing causes. | |||
Your staffs evaluation identified that the two primary root causes of the White finding were that an engineering change, modification, or temporary alteration process was not effectively used to control the fastener configuration of the emergency diesel generator B exhaust fan hub (Root Cause 1); and that requirements for monitoring of supporting systems for plant components were not adequate to verify operation (Root Cause 2). Specifically, an engineering evaluation was not performed to assess the impact of adding four additional set screws to the emergency diesel generator B exhaust fan hub and spanner nut assembly in 1999, and emergency diesel generator operating procedures did not require that the emergency diesel generator exhaust fan differential pressure be monitored to verify adequate fan performance. The NRC determined that your staff identified appropriate corrective actions to revise the emergency diesel generator system operating and surveillance testing procedures to include verifying proper operation of the ventilation exhaust fan, as well as verifying that current station guidance clearly defines criteria for implementing an engineering change process. The NRC also determined that your staffs extent of cause evaluation adequately reviewed operating and testing procedures for other safety-related systems, as well as an adequate assessment of reworked threaded connections or changes in configurations of fastening components in other safety-related systems. Based on these determinations, the NRC concluded that all inspection objectives were satisfied. | |||
The NRC has determined that completed or planned corrective actions were sufficient to address the performance that led to the White finding. Therefore, the finding is closed and is not considered as an Action Matrix input as of April 1, 2015. | |||
No findings were identified during this inspection. | |||
In accordance with Title 10 of the Code of Federal Regulations 2.390, Public Inspections, Exemptions, Requests for Withholding, of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records component of the NRCs Agencywide Documents Access and Management System (ADAMS). ADAMS is accessible from the NRC website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room). | |||
Sincerely, | |||
/RA/ | |||
Troy W. Pruett, Director Division of Reactor Projects | |||
Docket No.: 50-382 License No.: NPF-38 | |||
===Enclosure:=== | ===Enclosure:=== | ||
Inspection Report 05000382/2015010 | Inspection Report 05000382/2015010 w/Attachment: Supplemental Information | ||
REGION IV== | |||
Docket No.: | |||
05000382 | |||
License No.: | |||
NPF-38 | |||
Report No.: | |||
05000382/2015010 | |||
Licensee: | |||
Entergy Operations, Inc. | |||
Facility: | |||
Waterford Steam Electric Station, Unit 3 | |||
Location: | |||
17265 River Road Killona, LA 70057 | |||
Dates: | |||
March 16 through 19, 2015 | |||
Inspector: | |||
T. Farina, Operations Engineer | |||
Approved By: | |||
Troy W. Pruett, Director Division of Reactor Projects | |||
- 2 - | |||
Enclosure | |||
=SUMMARY= | =SUMMARY= | ||
IR 05000382/2015010; 3/16/2015 | IR 05000382/2015010; 3/16/2015 - 3/19/2015; Waterford Steam Electric Station, Unit 3; | ||
Supplemental Inspection - Inspection Procedure (IP) 95001 | |||
This supplemental inspection was conducted by a reactor operations inspector from the NRCs Region IV office. No findings were identified. The significance of inspection findings is indicated by their color (Green, White, Yellow, or Red), which is determined using NRC Inspection Manual Chapter 0609, Significance Determination Process. Their cross-cutting aspects are determined using NRC Inspection Manual Chapter 0310, Aspects Within the Cross-Cutting Areas. Violations of NRC requirements are dispositioned in accordance with the NRC Enforcement Policy. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process." | |||
===Cornerstone: Mitigating Systems=== | |||
The NRC staff performed this supplemental inspection in accordance with IP 95001, Inspection for One or Two White Inputs in a Strategic Performance Area, to assess the licensees evaluation associated with the inoperability of the train B emergency diesel generator due to the failure of the train B emergency diesel generator ventilation exhaust fan in April 2013. The NRC staff previously characterized this issue as having low to moderate safety significance (White), as documented in NRC Inspection Report 05000382/2014009. The licensee identified the two primary root causes of the white finding to be that an engineering change, modification, or temporary alteration process was not effectively used to control the fastener configuration of the emergency diesel generator B exhaust fan hub (Root Cause 1); and that requirements for monitoring of supporting systems for plant components were not adequate to verify operation (Root Cause 2). Specifically, an engineering evaluation was not performed to assess the impact of adding four additional set screws to the emergency diesel generator B exhaust fan hub and spanner nut assembly in 1999, and emergency diesel generator operating procedures did not require that the emergency diesel generator exhaust fan differential pressure be monitored to verify adequate fan performance. This resulted in a mechanical failure of the fan in April, 2013, which was not detected during a surveillance test run of the emergency diesel generator. The train B emergency diesel generator was determined to have been inoperable for a period of 30 days, until the condition was corrected by replacing the emergency diesel generator exhaust fan hub assembly with new parts in their original configuration. The inspector determined that the licensee identified appropriate corrective actions to revise the emergency diesel generator system operating and surveillance testing procedures to include monitoring of the ventilation exhaust fan flow indications, as well as verifying that current station guidance clearly defines criteria for implementing an engineering change process. | |||
Given the licensees acceptable performance in addressing the inoperable emergency diesel generator and deficient testing procedures, the White finding associated with this issue is closed and is not considered in assessing plant performance as of April 1, 2015, in accordance with the guidance in IMC 0305, Operating Reactor Assessment Program. | |||
===Licensee-Identified Violations=== | |||
None. | |||
=REPORT DETAILS= | =REPORT DETAILS= | ||
==OTHER ACTIVITIES== | |||
{{a|4OA4}} | |||
==4OA4 Supplemental Inspection== | |||
{{IP sample|IP=IP 95001}} | |||
===.01 Inspection Scope=== | ===.01 Inspection Scope=== | ||
This inspection was conducted in accordance with Inspection Procedure 95001, Supplemental a White finding, which affected the mitigating systems cornerstone in the reactor safety strategic performance area. The inspection objectives were to: | This inspection was conducted in accordance with Inspection Procedure 95001, Supplemental Inspection for One or Two White Inputs in a Strategic Performance Area, to assess the licensees evaluation of a White finding, which affected the mitigating systems cornerstone in the reactor safety strategic performance area. The inspection objectives were to: | ||
provide assurance that the root causes and contributing causes of risk-significant performance issues are understood; | |||
provide assurance that the extent of condition and extent of cause of risk-significant performance issues are identified; and | |||
provide assurance that the licensees corrective actions for risk-significant performance issues are sufficient to address the root and contributing causes and to prevent recurrence. | |||
A 95001 supplemental inspection was completed on October 10, 2014, and documented in NRC Inspection Report 05000382/2014011. The NRC determined that the licensees extent of cause review was insufficient; therefore not all inspection objectives were met, and the White finding remained open and continued to receive consideration as an Action Matrix input. | |||
On March 3, 2015, the licensee informed the NRC that they were ready for the supplemental re-inspection. In preparation for the inspection, the licensee performed a root cause evaluation under Condition Report CR-WF3-2013-2530. Revision 9 of the root cause evaluation report, dated February 23, 2015, was provided to the inspector for review. The NRC inspector reviewed the results of the licensees Pre-NRC 95001 Inspection Snapshot Assessment, which was completed in February 2015. | |||
The inspector reviewed the licensees root cause evaluation in addition to other evaluations conducted in support of the root cause evaluation. The inspector reviewed corrective actions that were taken or planned to address the identified causes. The inspector also held discussions with licensee personnel to determine whether the root and contributing causes as well as the contribution of safety culture components were understood, and whether corrective actions taken or planned were appropriate to address the causes and preclude repetition. The inspector performed two system walkdowns and observed an operational surveillance from the control room. | |||
===.02 Evaluation of the Inspection Requirements=== | ===.02 Evaluation of the Inspection Requirements=== | ||
02.01 Problem Identification | 02.01 Problem Identification | ||
a. | |||
Determine whether the evaluation documented who identified the issue (i.e., licensee-identified, self-revealing, or NRC-identified) and under what conditions the issue was identified. | |||
The licensees evaluation documented that the failure of the train B emergency diesel generator (EDG) ventilation exhaust fan was identified on May 22, 2013, through the licensees troubleshooting efforts, prompted by an abnormally high room temperature while the train B EDG was operating on May 20, 2013. The high room temperature was evident to licensee personnel stationed at the EDG while it was being operated. An alarm in the control room for high temperature in the train B EDG room also alerted plant operators to the high temperature condition. The issue was therefore self-revealing. | |||
The inspector determined that the licensees evaluation adequately documented who identified the issue and under what conditions the issue was identified. | |||
b. | |||
Determine whether the evaluation documented how long the issue existed and prior opportunities for identification. | |||
The licensees evaluation included a determination of when the exhaust fan failure occurred. Based on a detailed historical review of parameters including EDG room temperature, exhaust fan motor current, and exhaust fan differential pressure, the licensee concluded that the fan failure had occurred at the start of the previous operation of the train B EDG on April 25, 2013. Therefore, the issue had existed for 27 days prior to identification. The testing of the train B EDG on April 25, 2013, during which the fan failure occurred was a prior opportunity for identification of the issue. The failure was not identified at the time it occurred due to inadequate monitoring of exhaust fan operating indications while in service. The operating and testing procedures associated with the EDG system had never required operators to monitor these indications. The inspector determined that the licensees evaluation was adequate with respect to identifying how long the issue existed and prior opportunities for identification. | |||
c. | |||
Determine whether the evaluation documented the plant-specific risk consequences, as applicable, and compliance concerns associated with the issue. | |||
The licensees evaluation included a plant-specific risk-based safety significance evaluation of the issue. The licensee determined that the train B EDG exhaust fan failure resulted in the train B EDG being in an inoperable condition during the time that the failed condition of the fan existed, and that this condition resulted in the plant being without one of its sources of emergency AC power required by the plants technical specifications for greater than the outage time allowed by the technical specification. | |||
The licensees risk evaluation concluded that the overall risk significance of the issue was of low to moderate (White) significance, which was consistent with the result of the NRCs significance determination process for the White finding as discussed in NRC Inspection Report 05000382/2013008. The inspector concluded that the licensee | |||
appropriately documented the risk consequences and compliance concerns associated with the issue. | |||
====d. Findings==== | ====d. Findings==== | ||
No findings were identified. 02.02 | No findings were identified. | ||
02.02 Root Cause, Extent of Condition, and Extent of Cause Evaluation | |||
a. | |||
Determine whether the problem was evaluated using a systematic methodology to identify the root and contributing causes. | |||
The inspectors determined that the licensee conducted a root cause evaluation in which three primary evaluation methods were used: Failure Mode Analysis, Event and Causal Factors Chart, and Why Staircase. The licensees evaluation identified two root causes and two contributing causes associated with this issue. The root causes were identified to be that an engineering change, modification, or temporary alteration process was not effectively used to control the fastener configuration of the train B EDG exhaust fan hub (Root Cause 1); and that requirements for monitoring of supporting systems for plant components were not adequate to verify operation (Root Cause 2). Specifically, an engineering evaluation was not performed to assess the impact of adding four additional set screws to the train B EDG exhaust fan hub and spanner nut assembly in 1999, and emergency diesel generator operating procedures did not require that the EDG exhaust fan differential pressure be monitored to verify adequate fan performance. This resulted in a mechanical failure of the fan in April, 2013, which was not detected during a surveillance test run of the EDG. | |||
The contributing causes identified by the licensees evaluation were associated with an inadequately-implemented engineering modification to the configuration of the fan hub connection, as well as a maintenance activity affecting the threaded connection, both of which occurred in 1999. These issues are further discussed in Section 02.02.b below. | |||
The inspector concluded that the licensee evaluated the issue using a systematic methodology to identify the root and contributing causes. | |||
b. | |||
Determine whether the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem. | |||
The licensees evaluation determined that the significance of the exhaust fan failure was increased due to the failure to identify the problem at the time it occurred during testing of the EDG system, resulting in an extended period of inoperability of the EDG. The licensee determined that the failure was not detected because available indications of ventilation exhaust flow on the plant monitoring computer and local fan differential pressure were not being monitored while the equipment was being operated. The licensee concluded that the system operating and testing procedures did not direct personnel to monitor these indications. | |||
The licensees evaluation included the development of a detailed timeline of events that contributed to the exhaust fan failure; including an evaluation of several parameters | |||
associated with the operational condition of the fan going back over 10 years. The evaluation also included a detailed review of past maintenance and engineering activities associated with the fan. This led to the determination that a combination of maintenance and engineering modification activities performed in March 1999 was the cause of the failure of the fan on April 25, 2013, in which the fan hub became disengaged from the hub sleeve of the motor shaft. Specifically, a maintenance activity was performed which involved reworking the threads of the principal threaded fastener that connects the fan with the motor shaft; this activity actually reduced the amount of stress supportable by the threaded connection, an effect which was not evaluated when the maintenance was performed. At the same time, an engineering design change was implemented that modified the configuration of the fan hub to hub sleeve connection; this design change increased the amount of stress on the threads, an effect which was also not evaluated when the design change was implemented. The combination of these activities resulted in a weaker connection that was susceptible to thread failure (low cycle fatigue loading exceeded the allowable stress level). Accordingly, the licensees evaluation identified the following two contributing causes: | |||
Work instructions and vendor manual did not provide adequate guidance for assembly and tightening of fan hub components, during maintenance on the hub assembly under Work Authorization 01177294. | |||
Reworking the threads reduced the allowable stress of the fan hub to sleeve connection. | |||
These conclusions were also supported by the results of an independent failure evaluation that was performed on the failed component by an engineering firm contracted by the licensee to determine the exact nature and cause of the failure of the fan hub to hub sleeve connection. The inspector concluded the licensees root cause evaluation was conducted to a level of detail commensurate with the significance of the problem. | |||
c. | |||
Determine whether the root cause evaluation included consideration of prior occurrences of the problem and knowledge of prior operating experience. | |||
The licensees evaluation included a review of internal and external operating experience. The licensee conducted a fleet-wide search of the Entergy corrective action program for any previously documented conditions involving ventilation fan failures similar to the failure experienced at Waterford 3 in April 2013. This review identified that a failure of the same component (train B EDG ventilation exhaust fan) occurred at Waterford 3 in 1999. This failure occurred following a maintenance that involved the replacement of the fan motor. The failure was attributed to the motor electrical leads having been inadvertently reversed, resulting in a separation of the fan hub from the hub sleeve due to a reverse starting torque that was applied when the motor was started. | |||
This event initiated the maintenance and engineering design modification activities discussed in Section 02.02.b above. | |||
The licensee also conducted an industry-wide search for operating experience involving ventilation fan failures. This review identified no prior operating experience involving a | |||
failure of the hub and sleeve threaded connection associated with this type of ventilation fan. The licensees review also consisted of direct contact with engineers at other operating plants within the industry that have similar ventilation system components. | |||
This review concluded that fan failures at other facilities were not of the same nature or connection configuration as that of the failure experienced at Waterford 3. | |||
The inspector concluded that the licensees evaluation included a consideration of prior occurrences of the problem of similar equipment failures. The inspector also concluded that the licensees evaluation included a consideration of prior operating experience. | |||
d. | |||
Determine whether the root cause evaluation addressed the extent of condition and the extent of cause of the problem. | |||
The licensee reviewed the extent of condition associated with the failure of the train B EDG ventilation exhaust fan to determine whether other similar plant components may be susceptible to the same failure mechanism. The licensees evaluation determined that the failure of the fan hub/sleeve connection was applicable to a specific configuration that is unique to this type of fan (a hydramotor-controlled variable blade pitch fan). The only two applications of this type of fan in the plant are the train A and train B EDG ventilation exhaust fans. Thus, the only other component in the plant that may be susceptible to a similar failure mechanism would be the train A EDG exhaust fan. | |||
As discussed above in Section 02.02.b, the licensees evaluation determined that the train B fan hub sleeve had a weakened threaded connection due to a 1999 maintenance activity in which the threads were reworked, as well as a design change to increase the number of set screws used to secure the hub-to-sleeve spanner nut, which increased the stress on the threaded connection. These actions were performed following a previous failure of the train B fan hub connection in 1999 that was the result of bumping the fan in the reverse rotating direction due to the motor leads having been inadvertently reversed during motor replacement, thereby damaging the hub sleeve threads and requiring rework. | |||
The licensees evaluation included a review of the maintenance history for the train A fan, which determined that the fan motor was replaced in 2003. The review further determined that: 1) the motor had not been reverse-wired, and thus the fan hub had not experienced a reverse rotation starting torque; 2) the threaded hub sleeve connection was not removed or reworked; and 3) the attachment of the spanner nut to the hub sleeve was unchanged from the original configuration of 2 set screws. An additional factor considered in the licensees evaluation was the fact that, due to differences in the ventilation system arrangements between the two trains, the train A fan motor is of a different size than the train B fan motor and produces approximately 40 percent less starting torque. A review of system engineering computer data showed that train A EDG exhaust fan differential pressure increases as expected during EDG operation. Based on the above, the licensees evaluation determined that a weakened or failed condition does not exist on the train A exhaust fan. | |||
The licensee also evaluated all fans and blowers that if failed could render an EDG inoperable. Two EDG electrical panels contain blowers for forced cooling. Of these, the | |||
train A Generator Control Panel (EG ECP6852 A) had two fans fail in December 2014, due to mechanical binding of one fan causing fuses for both fans to blow. This resulted in a prompt determination of inoperability for train A EDG. The licensee has initiated CR-WF3-2014-6086 to further evaluate the impact of these fans on EDG operability and perform a causal analysis. | |||
The licensee conducted an extent of cause evaluation for the first identified root cause, An engineering change, modification, or temporary alteration process was not effectively used to control the fastener configuration of the EDG B exhaust fan hub. | |||
Specifically, when the licensee increased the number of set screws from 2 to 6 on the fan hub assembly in 1999, it failed to perform a specific engineering evaluation of the design change, as required by the procedure in effect at the time. The licensee conducted a search of plant records for prior engineering changes involving set screw configuration to determine if any similar conditions may exist in other plant components. | |||
No other similar conditions were identified. The licensee also conducted a search of plant records for threaded fastener configuration changes. This search yielded seven relevant hits; each was determined to have been implemented using approved design change processes, including appropriate engineering evaluations where applicable. The licensee further determined that current fleet procedures would not allow a change like the one performed in 1999 to be implemented without an engineering evaluation. | |||
The licensee conducted an extent of cause evaluation for the second identified root cause, Process requirements for monitoring of supporting systems for plant components were not adequate to verify operation. Specifically, emergency diesel generator operating and testing procedures did not require the EDG exhaust fan differential pressure to be monitored, which would have identified improper operation of the exhaust fan. The licensee conducted a search of plant records over an 8 year period for events involving inadequate equipment monitoring. Twenty three condition reports were determined to be relevant to the search criteria, but none had root or apparent causes related to testing and monitoring supporting systems for plant components. The licensee then conducted a review of all Operations Technical Specification surveillances to determine if inadequate monitoring existed on other plant systems and their support systems (over 80 procedures). This review identified 14 testing and operating procedures for revision to improve monitoring of primary or supporting systems. These revisions included improved monitoring of parameters such as room cooler fan operation, pump discharge pressure, and Dry / Wet Cooling Tower fan operation, among others. | |||
The licensee conducted an extent of cause evaluation for the first contributing cause, WA 01177294 work instructions and vendor manual did not provide guidance for assembly and tightening of fan hub components. The licensee evaluated other safety related systems and determined that there were no other fans with a similar hub design as the EDG exhaust fans. The A EDG exhaust fan motor was replaced in 2003, and B EDG exhaust fan rotating assembly was replaced in 2013. Neither work order required the disassembly or reassembly of the hub assembly; both A and B EDG exhaust fans now have hub assemblies which are factory assembled in the original configuration. | |||
Both work orders were determined to contain appropriate work instructions for installation. The licensee further determined that the current fleet work planning | |||
procedure provides adequate guidance for the level of detail required in work package instructions based on the complexity and safety significance of the activity. | |||
The licensee conducted an extent of cause evaluation for the second contributing cause, Reworking the threads reduced the allowable stress of the fan hub to sleeve connection. The licensee conducted a search of plant records to identify safety-related components where fastener threads may have been reworked, which could have resulted in a reduction of thread strength or a change in thread fit clearances. This search yielded 19 relevant examples, all of which were determined by the licensee to have been resolved in an acceptable manner. Interviews with maintenance and engineering personnel by the inspector showed a sensitivity to maintenance practices involving more than minor reworking of threaded connections, with a documented preference for replacing vice reworking damaged connections. | |||
The inspector concluded that the licensees evaluation adequately addressed the extent of condition and extent of cause of the problem. | |||
e. | |||
Determine whether the root cause, extent of condition, and extent of cause evaluations appropriately considered the safety culture components as described in IMC 0310. | |||
The licensees evaluation included a review of whether a weakness in any safety culture component contributed to any causes of the issue. The licensees evaluation identified weaknesses in two safety culture components that were related to the identified second root cause of the performance deficiency (requirements for monitoring of supporting systems for plant components is not adequate to verify operation). Weaknesses in the components of resources (H.1) and documentation (H.7) within the area of human performance were identified associated with procedure quality. Procedures did not include guidance to ensure proper operation of the EDG exhaust fans during operation and testing of the EDGs. The extent of cause review further identified a total of 14 procedures requiring revision to verify operability of primary and supporting systems during testing. | |||
The inspector concluded that the licensees evaluation included an appropriate consideration of safety culture components. | |||
====f. Findings==== | ====f. Findings==== | ||
No findings were identified. 02.03 Corrective Actions | No findings were identified. | ||
02.03 Corrective Actions | |||
a. | |||
Determine whether appropriate corrective actions are specified for each root and contributing cause or that the licensee has an adequate evaluation for why no corrective actions are necessary. | |||
The licensees evaluation identified a number of corrective actions. The principal corrective actions to address the root causes were to revise the system operating Procedure OP-009-002, Emergency Diesel Generator, and surveillance testing | |||
Procedure OP-903-068, Emergency Diesel Generator and Subgroup Relay Operability Verification, to include requirements to monitor the ventilation exhaust fan differential pressure when the EDG system is operating. Additional corrective actions included the replacement of the train B EDG exhaust fan hub assembly with new parts in the original configuration (which was completed within four days of the discovery of the failed condition of the fan in order to restore the EDG to an operable condition);verification that current engineering procedures would require an engineering analysis for a similar system modification; a review of all safety-related surveillance procedures to determine if an evaluation of auxiliary system parameters should be added; and the evaluation of maintenance practices on the EDG exhaust fan components and discussion with the vendor to determine whether all appropriate maintenance activities were being performed. The inspector concluded that the identified corrective actions were appropriate and addressed the root and contributing causes. | |||
b. | |||
Determine whether corrective actions have been prioritized with consideration of risk significance and regulatory compliance. | |||
The licensees immediate corrective actions restored the EDG to an operable condition in order to restore compliance with plant technical specifications promptly upon discovery of the failure. The inspector determined that the licensee adequately prioritized the remaining corrective actions with consideration of the risk significance of the EDG system and regulatory compliance. This included appropriate actions to address a notice of violation issued by the NRC and restore compliance (see Section 02.03.e below). | |||
c. | |||
Determine whether a schedule has been established for implementing and completing the corrective actions. | |||
The inspector determined that the licensee adequately established a schedule for implementing and completing the corrective actions. The inspector noted that, as of March 19, 2015, 93 of 98 corrective actions were either complete or had a due date within 6 weeks. Further, all corrective actions to prevent recurrence (CAPRs) have been completed. | |||
d. | |||
Determine whether quantitative or qualitative measures of success have been developed for determining the effectiveness of the corrective actions to prevent recurrence. | |||
The inspector determined that the licensee had developed an effectiveness review plan to determine the method, attributes, acceptance criteria, and schedule for effectiveness reviews of the CAPRs. The inspector reviewed this plan as captured in the corrective action program as LO-WLO-2013-00149. The corrective actions that had been identified as CAPRs were the revisions to the system operating Procedure OP 009-002, Emergency Diesel Generator, and surveillance testing Procedure OP 903-068, Emergency Diesel Generator and Subgroup Relay Operability Verification, replacement of the train B EDG exhaust fan hub assembly with new parts in the original configuration; and verification that current engineering procedures clearly define the threshold for implementing Engineering Change processes. The measures | |||
for determining effectiveness included the verification that the EDG system operating procedure log sheets have been completed with appropriate flow data and acceptance criteria; review of B EDG exhaust fan hub assembly replacement documentation for proof that a factory-assembled unit was installed; and interviews with Engineering personnel to evaluate knowledge of implementation of Engineering Change processes for proposed design changes. | |||
The inspector concluded that adequate measures of success had been developed for determining the effectiveness of the corrective actions to prevent recurrence. | |||
e. | |||
Determine whether the corrective actions planned or taken adequately address the Notice of Violation that was the basis for the supplemental inspection. | |||
The NRC issued a Notice of Violation to the licensee on March 28, 2014, for the failure to perform adequate testing on the train B EDG exhaust fan to demonstrate that the exhaust fan would perform satisfactorily in service (NRC Inspection Report 05000382/2014009, ADAMS ML14086A768). During this inspection, the inspector determined that the licensee restored compliance by revising the system operating Procedure OP-009-002, Emergency Diesel Generator, and surveillance testing Procedure OP-903-068, Emergency Diesel Generator and Subgroup Relay Operability Verification, to include requirements to monitor the ventilation exhaust fan differential pressure during testing. The inspector concluded that the corrective actions taken or planned were adequate to address the Notice of Violation that was the basis for the supplemental inspection. | |||
====f. Findings==== | ====f. Findings==== | ||
No findings were identified. 02.04 Evaluation of IMC 0305 Criteria For Treatment Of Old Design Issues | No findings were identified. | ||
02.04 Evaluation of IMC 0305 Criteria For Treatment Of Old Design Issues | |||
The licensee did not request credit for self-identification of an old design issue; therefore, the risk-significant issue was not evaluated against the IMC 0305 criteria for treatment of an old design issue. | |||
{{a|4OA6}} | {{a|4OA6}} | ||
==4OA6 Meetings, Including Exit== | ==4OA6 Meetings, Including Exit== | ||
===Exit Meeting Summary=== | |||
On March 19, 2015, the inspector presented the inspection results to Mr. M. Chisum, Site Vice President, and other members of the licensee staff. The licensee acknowledged the issues presented. The inspector asked the licensee if any of the material examined during the inspection should be considered proprietary. The licensee did not identify any proprietary information. | |||
{{a|4OA7}} | |||
==4OA7 Licensee-Identified Violations== | ==4OA7 Licensee-Identified Violations== | ||
None. | None. | ||
=SUPPLEMENTAL INFORMATION= | =SUPPLEMENTAL INFORMATION= | ||
==KEY POINTS OF CONTACT== | ==KEY POINTS OF CONTACT== | ||
===Licensee Personnel=== | |||
: [[contact::M. Chisum]], Site Vice President | |||
: [[contact::M. Richey]], General Manager, Plant Operations | |||
: [[contact::R. Gilmore]], Manager, Systems and Components | |||
: [[contact::J. Jarrell]], Manager, Regulatory Assurance | |||
: [[contact::B. Lanka]], Director, Engineering | |||
: [[contact::L. Milster]], Licensing Specialist, Licensing | |||
: [[contact::R. Osborne]], Manager, Performance Improvement | |||
: [[contact::M. Chase]], Director, Regulatory & Performance Improvement | |||
: [[contact::J. Wilbur]], Senior Lead Engineer | |||
===NRC Personnel=== | |||
: [[contact::F. Ramirez]], Senior Resident Inspector | |||
: [[contact::C. Speer]], Resident Inspector | |||
==LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED== | ==LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED== | ||
===Opened=== | |||
None | |||
===Closed=== | ===Closed=== | ||
: 05000382/2013008-01 VIO Failure To Establish an Adequate Test Program to Demonstrate that the train B Emergency Diesel Generator Exhaust Fan Would Perform Satisfactorily In Service (Section 4OA4) | : 05000382/2013008-01 VIO Failure To Establish an Adequate Test Program to Demonstrate that the train B Emergency Diesel Generator Exhaust Fan Would Perform Satisfactorily In Service (Section 4OA4) | ||
===Discussed=== | ===Discussed=== | ||
None | None | ||
==LIST OF DOCUMENTS REVIEWED== | ==LIST OF DOCUMENTS REVIEWED== | ||
}} | }} | ||
Latest revision as of 13:24, 10 January 2025
| ML15097A550 | |
| Person / Time | |
|---|---|
| Site: | Waterford |
| Issue date: | 04/09/2015 |
| From: | Troy Pruett NRC/RGN-IV/DRP |
| To: | Chisum M Entergy Operations |
| Lantz R | |
| References | |
| EA-13-233 IR 2015010 | |
| Download: ML15097A550 (18) | |
Text
April 09, 2015
SUBJECT:
WATERFORD STEAM ELECTRIC STATION, UNIT 3 - NRC SUPPLEMENTAL INSPECTION REPORT 05000382/2015010
Dear Mr. Chisum:
Prior to May 26, 2013, your staff failed to establish an adequate test program to demonstrate that the train B emergency diesel generator ventilation exhaust fan would perform satisfactorily in service. This performance deficiency resulted in a failure to identify that the exhaust fan could not perform its function because it disengaged from the fan motor in April 2013. Consequently, the train B emergency diesel generator was determined to be inoperable for a period of 30 days, exceeding the Technical Specification 3.8.1 allowed outage time of 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />.
On December 19, 2013, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection at the Waterford Steam Electric Station, Unit 3. Based on the results of this inspection, documented in NRC Inspection Report 05000382/2013008 on January 30, 2014 (ADAMS ML14030A616), and the final significance determination documented in NRC Inspection Report 05000382/2014009 on March 28, 2014 (ADAMS ML14086A768), the NRC assigned a White finding Action Matrix input to the mitigating systems cornerstone in the fourth quarter of 2013.
In response to this Action Matrix input, the NRC completed a supplemental inspection using Inspection Procedure 95001, Supplemental Inspection for One or Two White Inputs in a Strategic Performance Area, on October 10, 2014, and documented the results in NRC Inspection Report 05000382/2014011 (ADAMS ML14364A412). The NRC determined that your extent of cause review was insufficient; therefore not all inspection objectives were met, and the White finding remained open and continued to receive consideration as an Action Matrix input.
On March 3, 2015, you informed the NRC that the Waterford Steam Electric Station, Unit 3, was ready for a supplemental re-inspection. On March 19, 2015, the NRC completed a supplemental re-inspection and discussed the results of the inspection with you and other members of your staff. On March 24, 2015, Mr. Ryan Lantz, Chief, Reactor Projects Branch E,
UNITED STATES NUCLEAR REGULATORY COMMISSION
REGION IV
1600 E LAMAR BLVD ARLINGTON, TX 76011-4511 NRC Region IV, conducted a Regulatory Performance Meeting with Mr. John Jarrell, Regulatory Assurance Manager. The topics of this meeting included the NRC inspectors observations, your staffs improved effort preparing for this inspection, and the lessons learned from the revised root cause and extent of cause evaluations. The result of the inspection is documented in the enclosed inspection report.
The NRC performed this supplemental inspection to determine if (1) the root and contributing causes for the significant issues were understood, (2) the extent of condition and extent of cause for the identified issues were understood, and (3) your completed or planned corrective actions were sufficient to address and prevent repetition of the root causes and contributing causes.
Your staffs evaluation identified that the two primary root causes of the White finding were that an engineering change, modification, or temporary alteration process was not effectively used to control the fastener configuration of the emergency diesel generator B exhaust fan hub (Root Cause 1); and that requirements for monitoring of supporting systems for plant components were not adequate to verify operation (Root Cause 2). Specifically, an engineering evaluation was not performed to assess the impact of adding four additional set screws to the emergency diesel generator B exhaust fan hub and spanner nut assembly in 1999, and emergency diesel generator operating procedures did not require that the emergency diesel generator exhaust fan differential pressure be monitored to verify adequate fan performance. The NRC determined that your staff identified appropriate corrective actions to revise the emergency diesel generator system operating and surveillance testing procedures to include verifying proper operation of the ventilation exhaust fan, as well as verifying that current station guidance clearly defines criteria for implementing an engineering change process. The NRC also determined that your staffs extent of cause evaluation adequately reviewed operating and testing procedures for other safety-related systems, as well as an adequate assessment of reworked threaded connections or changes in configurations of fastening components in other safety-related systems. Based on these determinations, the NRC concluded that all inspection objectives were satisfied.
The NRC has determined that completed or planned corrective actions were sufficient to address the performance that led to the White finding. Therefore, the finding is closed and is not considered as an Action Matrix input as of April 1, 2015.
No findings were identified during this inspection.
In accordance with Title 10 of the Code of Federal Regulations 2.390, Public Inspections, Exemptions, Requests for Withholding, of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records component of the NRCs Agencywide Documents Access and Management System (ADAMS). ADAMS is accessible from the NRC website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
Troy W. Pruett, Director Division of Reactor Projects
Docket No.: 50-382 License No.: NPF-38
Enclosure:
Inspection Report 05000382/2015010 w/Attachment: Supplemental Information
REGION IV==
Docket No.:
05000382
License No.:
Report No.:
Licensee:
Entergy Operations, Inc.
Facility:
Waterford Steam Electric Station, Unit 3
Location:
17265 River Road Killona, LA 70057
Dates:
March 16 through 19, 2015
Inspector:
T. Farina, Operations Engineer
Approved By:
Troy W. Pruett, Director Division of Reactor Projects
- 2 -
Enclosure
SUMMARY
IR 05000382/2015010; 3/16/2015 - 3/19/2015; Waterford Steam Electric Station, Unit 3;
Supplemental Inspection - Inspection Procedure (IP) 95001
This supplemental inspection was conducted by a reactor operations inspector from the NRCs Region IV office. No findings were identified. The significance of inspection findings is indicated by their color (Green, White, Yellow, or Red), which is determined using NRC Inspection Manual Chapter 0609, Significance Determination Process. Their cross-cutting aspects are determined using NRC Inspection Manual Chapter 0310, Aspects Within the Cross-Cutting Areas. Violations of NRC requirements are dispositioned in accordance with the NRC Enforcement Policy. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process."
Cornerstone: Mitigating Systems
The NRC staff performed this supplemental inspection in accordance with IP 95001, Inspection for One or Two White Inputs in a Strategic Performance Area, to assess the licensees evaluation associated with the inoperability of the train B emergency diesel generator due to the failure of the train B emergency diesel generator ventilation exhaust fan in April 2013. The NRC staff previously characterized this issue as having low to moderate safety significance (White), as documented in NRC Inspection Report 05000382/2014009. The licensee identified the two primary root causes of the white finding to be that an engineering change, modification, or temporary alteration process was not effectively used to control the fastener configuration of the emergency diesel generator B exhaust fan hub (Root Cause 1); and that requirements for monitoring of supporting systems for plant components were not adequate to verify operation (Root Cause 2). Specifically, an engineering evaluation was not performed to assess the impact of adding four additional set screws to the emergency diesel generator B exhaust fan hub and spanner nut assembly in 1999, and emergency diesel generator operating procedures did not require that the emergency diesel generator exhaust fan differential pressure be monitored to verify adequate fan performance. This resulted in a mechanical failure of the fan in April, 2013, which was not detected during a surveillance test run of the emergency diesel generator. The train B emergency diesel generator was determined to have been inoperable for a period of 30 days, until the condition was corrected by replacing the emergency diesel generator exhaust fan hub assembly with new parts in their original configuration. The inspector determined that the licensee identified appropriate corrective actions to revise the emergency diesel generator system operating and surveillance testing procedures to include monitoring of the ventilation exhaust fan flow indications, as well as verifying that current station guidance clearly defines criteria for implementing an engineering change process.
Given the licensees acceptable performance in addressing the inoperable emergency diesel generator and deficient testing procedures, the White finding associated with this issue is closed and is not considered in assessing plant performance as of April 1, 2015, in accordance with the guidance in IMC 0305, Operating Reactor Assessment Program.
Licensee-Identified Violations
None.
REPORT DETAILS
OTHER ACTIVITIES
4OA4 Supplemental Inspection
.01 Inspection Scope
This inspection was conducted in accordance with Inspection Procedure 95001, Supplemental Inspection for One or Two White Inputs in a Strategic Performance Area, to assess the licensees evaluation of a White finding, which affected the mitigating systems cornerstone in the reactor safety strategic performance area. The inspection objectives were to:
provide assurance that the root causes and contributing causes of risk-significant performance issues are understood;
provide assurance that the extent of condition and extent of cause of risk-significant performance issues are identified; and
provide assurance that the licensees corrective actions for risk-significant performance issues are sufficient to address the root and contributing causes and to prevent recurrence.
A 95001 supplemental inspection was completed on October 10, 2014, and documented in NRC Inspection Report 05000382/2014011. The NRC determined that the licensees extent of cause review was insufficient; therefore not all inspection objectives were met, and the White finding remained open and continued to receive consideration as an Action Matrix input.
On March 3, 2015, the licensee informed the NRC that they were ready for the supplemental re-inspection. In preparation for the inspection, the licensee performed a root cause evaluation under Condition Report CR-WF3-2013-2530. Revision 9 of the root cause evaluation report, dated February 23, 2015, was provided to the inspector for review. The NRC inspector reviewed the results of the licensees Pre-NRC 95001 Inspection Snapshot Assessment, which was completed in February 2015.
The inspector reviewed the licensees root cause evaluation in addition to other evaluations conducted in support of the root cause evaluation. The inspector reviewed corrective actions that were taken or planned to address the identified causes. The inspector also held discussions with licensee personnel to determine whether the root and contributing causes as well as the contribution of safety culture components were understood, and whether corrective actions taken or planned were appropriate to address the causes and preclude repetition. The inspector performed two system walkdowns and observed an operational surveillance from the control room.
.02 Evaluation of the Inspection Requirements
02.01 Problem Identification
a.
Determine whether the evaluation documented who identified the issue (i.e., licensee-identified, self-revealing, or NRC-identified) and under what conditions the issue was identified.
The licensees evaluation documented that the failure of the train B emergency diesel generator (EDG) ventilation exhaust fan was identified on May 22, 2013, through the licensees troubleshooting efforts, prompted by an abnormally high room temperature while the train B EDG was operating on May 20, 2013. The high room temperature was evident to licensee personnel stationed at the EDG while it was being operated. An alarm in the control room for high temperature in the train B EDG room also alerted plant operators to the high temperature condition. The issue was therefore self-revealing.
The inspector determined that the licensees evaluation adequately documented who identified the issue and under what conditions the issue was identified.
b.
Determine whether the evaluation documented how long the issue existed and prior opportunities for identification.
The licensees evaluation included a determination of when the exhaust fan failure occurred. Based on a detailed historical review of parameters including EDG room temperature, exhaust fan motor current, and exhaust fan differential pressure, the licensee concluded that the fan failure had occurred at the start of the previous operation of the train B EDG on April 25, 2013. Therefore, the issue had existed for 27 days prior to identification. The testing of the train B EDG on April 25, 2013, during which the fan failure occurred was a prior opportunity for identification of the issue. The failure was not identified at the time it occurred due to inadequate monitoring of exhaust fan operating indications while in service. The operating and testing procedures associated with the EDG system had never required operators to monitor these indications. The inspector determined that the licensees evaluation was adequate with respect to identifying how long the issue existed and prior opportunities for identification.
c.
Determine whether the evaluation documented the plant-specific risk consequences, as applicable, and compliance concerns associated with the issue.
The licensees evaluation included a plant-specific risk-based safety significance evaluation of the issue. The licensee determined that the train B EDG exhaust fan failure resulted in the train B EDG being in an inoperable condition during the time that the failed condition of the fan existed, and that this condition resulted in the plant being without one of its sources of emergency AC power required by the plants technical specifications for greater than the outage time allowed by the technical specification.
The licensees risk evaluation concluded that the overall risk significance of the issue was of low to moderate (White) significance, which was consistent with the result of the NRCs significance determination process for the White finding as discussed in NRC Inspection Report 05000382/2013008. The inspector concluded that the licensee
appropriately documented the risk consequences and compliance concerns associated with the issue.
d. Findings
No findings were identified.
02.02 Root Cause, Extent of Condition, and Extent of Cause Evaluation
a.
Determine whether the problem was evaluated using a systematic methodology to identify the root and contributing causes.
The inspectors determined that the licensee conducted a root cause evaluation in which three primary evaluation methods were used: Failure Mode Analysis, Event and Causal Factors Chart, and Why Staircase. The licensees evaluation identified two root causes and two contributing causes associated with this issue. The root causes were identified to be that an engineering change, modification, or temporary alteration process was not effectively used to control the fastener configuration of the train B EDG exhaust fan hub (Root Cause 1); and that requirements for monitoring of supporting systems for plant components were not adequate to verify operation (Root Cause 2). Specifically, an engineering evaluation was not performed to assess the impact of adding four additional set screws to the train B EDG exhaust fan hub and spanner nut assembly in 1999, and emergency diesel generator operating procedures did not require that the EDG exhaust fan differential pressure be monitored to verify adequate fan performance. This resulted in a mechanical failure of the fan in April, 2013, which was not detected during a surveillance test run of the EDG.
The contributing causes identified by the licensees evaluation were associated with an inadequately-implemented engineering modification to the configuration of the fan hub connection, as well as a maintenance activity affecting the threaded connection, both of which occurred in 1999. These issues are further discussed in Section 02.02.b below.
The inspector concluded that the licensee evaluated the issue using a systematic methodology to identify the root and contributing causes.
b.
Determine whether the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem.
The licensees evaluation determined that the significance of the exhaust fan failure was increased due to the failure to identify the problem at the time it occurred during testing of the EDG system, resulting in an extended period of inoperability of the EDG. The licensee determined that the failure was not detected because available indications of ventilation exhaust flow on the plant monitoring computer and local fan differential pressure were not being monitored while the equipment was being operated. The licensee concluded that the system operating and testing procedures did not direct personnel to monitor these indications.
The licensees evaluation included the development of a detailed timeline of events that contributed to the exhaust fan failure; including an evaluation of several parameters
associated with the operational condition of the fan going back over 10 years. The evaluation also included a detailed review of past maintenance and engineering activities associated with the fan. This led to the determination that a combination of maintenance and engineering modification activities performed in March 1999 was the cause of the failure of the fan on April 25, 2013, in which the fan hub became disengaged from the hub sleeve of the motor shaft. Specifically, a maintenance activity was performed which involved reworking the threads of the principal threaded fastener that connects the fan with the motor shaft; this activity actually reduced the amount of stress supportable by the threaded connection, an effect which was not evaluated when the maintenance was performed. At the same time, an engineering design change was implemented that modified the configuration of the fan hub to hub sleeve connection; this design change increased the amount of stress on the threads, an effect which was also not evaluated when the design change was implemented. The combination of these activities resulted in a weaker connection that was susceptible to thread failure (low cycle fatigue loading exceeded the allowable stress level). Accordingly, the licensees evaluation identified the following two contributing causes:
Work instructions and vendor manual did not provide adequate guidance for assembly and tightening of fan hub components, during maintenance on the hub assembly under Work Authorization 01177294.
Reworking the threads reduced the allowable stress of the fan hub to sleeve connection.
These conclusions were also supported by the results of an independent failure evaluation that was performed on the failed component by an engineering firm contracted by the licensee to determine the exact nature and cause of the failure of the fan hub to hub sleeve connection. The inspector concluded the licensees root cause evaluation was conducted to a level of detail commensurate with the significance of the problem.
c.
Determine whether the root cause evaluation included consideration of prior occurrences of the problem and knowledge of prior operating experience.
The licensees evaluation included a review of internal and external operating experience. The licensee conducted a fleet-wide search of the Entergy corrective action program for any previously documented conditions involving ventilation fan failures similar to the failure experienced at Waterford 3 in April 2013. This review identified that a failure of the same component (train B EDG ventilation exhaust fan) occurred at Waterford 3 in 1999. This failure occurred following a maintenance that involved the replacement of the fan motor. The failure was attributed to the motor electrical leads having been inadvertently reversed, resulting in a separation of the fan hub from the hub sleeve due to a reverse starting torque that was applied when the motor was started.
This event initiated the maintenance and engineering design modification activities discussed in Section 02.02.b above.
The licensee also conducted an industry-wide search for operating experience involving ventilation fan failures. This review identified no prior operating experience involving a
failure of the hub and sleeve threaded connection associated with this type of ventilation fan. The licensees review also consisted of direct contact with engineers at other operating plants within the industry that have similar ventilation system components.
This review concluded that fan failures at other facilities were not of the same nature or connection configuration as that of the failure experienced at Waterford 3.
The inspector concluded that the licensees evaluation included a consideration of prior occurrences of the problem of similar equipment failures. The inspector also concluded that the licensees evaluation included a consideration of prior operating experience.
d.
Determine whether the root cause evaluation addressed the extent of condition and the extent of cause of the problem.
The licensee reviewed the extent of condition associated with the failure of the train B EDG ventilation exhaust fan to determine whether other similar plant components may be susceptible to the same failure mechanism. The licensees evaluation determined that the failure of the fan hub/sleeve connection was applicable to a specific configuration that is unique to this type of fan (a hydramotor-controlled variable blade pitch fan). The only two applications of this type of fan in the plant are the train A and train B EDG ventilation exhaust fans. Thus, the only other component in the plant that may be susceptible to a similar failure mechanism would be the train A EDG exhaust fan.
As discussed above in Section 02.02.b, the licensees evaluation determined that the train B fan hub sleeve had a weakened threaded connection due to a 1999 maintenance activity in which the threads were reworked, as well as a design change to increase the number of set screws used to secure the hub-to-sleeve spanner nut, which increased the stress on the threaded connection. These actions were performed following a previous failure of the train B fan hub connection in 1999 that was the result of bumping the fan in the reverse rotating direction due to the motor leads having been inadvertently reversed during motor replacement, thereby damaging the hub sleeve threads and requiring rework.
The licensees evaluation included a review of the maintenance history for the train A fan, which determined that the fan motor was replaced in 2003. The review further determined that: 1) the motor had not been reverse-wired, and thus the fan hub had not experienced a reverse rotation starting torque; 2) the threaded hub sleeve connection was not removed or reworked; and 3) the attachment of the spanner nut to the hub sleeve was unchanged from the original configuration of 2 set screws. An additional factor considered in the licensees evaluation was the fact that, due to differences in the ventilation system arrangements between the two trains, the train A fan motor is of a different size than the train B fan motor and produces approximately 40 percent less starting torque. A review of system engineering computer data showed that train A EDG exhaust fan differential pressure increases as expected during EDG operation. Based on the above, the licensees evaluation determined that a weakened or failed condition does not exist on the train A exhaust fan.
The licensee also evaluated all fans and blowers that if failed could render an EDG inoperable. Two EDG electrical panels contain blowers for forced cooling. Of these, the
train A Generator Control Panel (EG ECP6852 A) had two fans fail in December 2014, due to mechanical binding of one fan causing fuses for both fans to blow. This resulted in a prompt determination of inoperability for train A EDG. The licensee has initiated CR-WF3-2014-6086 to further evaluate the impact of these fans on EDG operability and perform a causal analysis.
The licensee conducted an extent of cause evaluation for the first identified root cause, An engineering change, modification, or temporary alteration process was not effectively used to control the fastener configuration of the EDG B exhaust fan hub.
Specifically, when the licensee increased the number of set screws from 2 to 6 on the fan hub assembly in 1999, it failed to perform a specific engineering evaluation of the design change, as required by the procedure in effect at the time. The licensee conducted a search of plant records for prior engineering changes involving set screw configuration to determine if any similar conditions may exist in other plant components.
No other similar conditions were identified. The licensee also conducted a search of plant records for threaded fastener configuration changes. This search yielded seven relevant hits; each was determined to have been implemented using approved design change processes, including appropriate engineering evaluations where applicable. The licensee further determined that current fleet procedures would not allow a change like the one performed in 1999 to be implemented without an engineering evaluation.
The licensee conducted an extent of cause evaluation for the second identified root cause, Process requirements for monitoring of supporting systems for plant components were not adequate to verify operation. Specifically, emergency diesel generator operating and testing procedures did not require the EDG exhaust fan differential pressure to be monitored, which would have identified improper operation of the exhaust fan. The licensee conducted a search of plant records over an 8 year period for events involving inadequate equipment monitoring. Twenty three condition reports were determined to be relevant to the search criteria, but none had root or apparent causes related to testing and monitoring supporting systems for plant components. The licensee then conducted a review of all Operations Technical Specification surveillances to determine if inadequate monitoring existed on other plant systems and their support systems (over 80 procedures). This review identified 14 testing and operating procedures for revision to improve monitoring of primary or supporting systems. These revisions included improved monitoring of parameters such as room cooler fan operation, pump discharge pressure, and Dry / Wet Cooling Tower fan operation, among others.
The licensee conducted an extent of cause evaluation for the first contributing cause, WA 01177294 work instructions and vendor manual did not provide guidance for assembly and tightening of fan hub components. The licensee evaluated other safety related systems and determined that there were no other fans with a similar hub design as the EDG exhaust fans. The A EDG exhaust fan motor was replaced in 2003, and B EDG exhaust fan rotating assembly was replaced in 2013. Neither work order required the disassembly or reassembly of the hub assembly; both A and B EDG exhaust fans now have hub assemblies which are factory assembled in the original configuration.
Both work orders were determined to contain appropriate work instructions for installation. The licensee further determined that the current fleet work planning
procedure provides adequate guidance for the level of detail required in work package instructions based on the complexity and safety significance of the activity.
The licensee conducted an extent of cause evaluation for the second contributing cause, Reworking the threads reduced the allowable stress of the fan hub to sleeve connection. The licensee conducted a search of plant records to identify safety-related components where fastener threads may have been reworked, which could have resulted in a reduction of thread strength or a change in thread fit clearances. This search yielded 19 relevant examples, all of which were determined by the licensee to have been resolved in an acceptable manner. Interviews with maintenance and engineering personnel by the inspector showed a sensitivity to maintenance practices involving more than minor reworking of threaded connections, with a documented preference for replacing vice reworking damaged connections.
The inspector concluded that the licensees evaluation adequately addressed the extent of condition and extent of cause of the problem.
e.
Determine whether the root cause, extent of condition, and extent of cause evaluations appropriately considered the safety culture components as described in IMC 0310.
The licensees evaluation included a review of whether a weakness in any safety culture component contributed to any causes of the issue. The licensees evaluation identified weaknesses in two safety culture components that were related to the identified second root cause of the performance deficiency (requirements for monitoring of supporting systems for plant components is not adequate to verify operation). Weaknesses in the components of resources (H.1) and documentation (H.7) within the area of human performance were identified associated with procedure quality. Procedures did not include guidance to ensure proper operation of the EDG exhaust fans during operation and testing of the EDGs. The extent of cause review further identified a total of 14 procedures requiring revision to verify operability of primary and supporting systems during testing.
The inspector concluded that the licensees evaluation included an appropriate consideration of safety culture components.
f. Findings
No findings were identified.
02.03 Corrective Actions
a.
Determine whether appropriate corrective actions are specified for each root and contributing cause or that the licensee has an adequate evaluation for why no corrective actions are necessary.
The licensees evaluation identified a number of corrective actions. The principal corrective actions to address the root causes were to revise the system operating Procedure OP-009-002, Emergency Diesel Generator, and surveillance testing
Procedure OP-903-068, Emergency Diesel Generator and Subgroup Relay Operability Verification, to include requirements to monitor the ventilation exhaust fan differential pressure when the EDG system is operating. Additional corrective actions included the replacement of the train B EDG exhaust fan hub assembly with new parts in the original configuration (which was completed within four days of the discovery of the failed condition of the fan in order to restore the EDG to an operable condition);verification that current engineering procedures would require an engineering analysis for a similar system modification; a review of all safety-related surveillance procedures to determine if an evaluation of auxiliary system parameters should be added; and the evaluation of maintenance practices on the EDG exhaust fan components and discussion with the vendor to determine whether all appropriate maintenance activities were being performed. The inspector concluded that the identified corrective actions were appropriate and addressed the root and contributing causes.
b.
Determine whether corrective actions have been prioritized with consideration of risk significance and regulatory compliance.
The licensees immediate corrective actions restored the EDG to an operable condition in order to restore compliance with plant technical specifications promptly upon discovery of the failure. The inspector determined that the licensee adequately prioritized the remaining corrective actions with consideration of the risk significance of the EDG system and regulatory compliance. This included appropriate actions to address a notice of violation issued by the NRC and restore compliance (see Section 02.03.e below).
c.
Determine whether a schedule has been established for implementing and completing the corrective actions.
The inspector determined that the licensee adequately established a schedule for implementing and completing the corrective actions. The inspector noted that, as of March 19, 2015, 93 of 98 corrective actions were either complete or had a due date within 6 weeks. Further, all corrective actions to prevent recurrence (CAPRs) have been completed.
d.
Determine whether quantitative or qualitative measures of success have been developed for determining the effectiveness of the corrective actions to prevent recurrence.
The inspector determined that the licensee had developed an effectiveness review plan to determine the method, attributes, acceptance criteria, and schedule for effectiveness reviews of the CAPRs. The inspector reviewed this plan as captured in the corrective action program as LO-WLO-2013-00149. The corrective actions that had been identified as CAPRs were the revisions to the system operating Procedure OP 009-002, Emergency Diesel Generator, and surveillance testing Procedure OP 903-068, Emergency Diesel Generator and Subgroup Relay Operability Verification, replacement of the train B EDG exhaust fan hub assembly with new parts in the original configuration; and verification that current engineering procedures clearly define the threshold for implementing Engineering Change processes. The measures
for determining effectiveness included the verification that the EDG system operating procedure log sheets have been completed with appropriate flow data and acceptance criteria; review of B EDG exhaust fan hub assembly replacement documentation for proof that a factory-assembled unit was installed; and interviews with Engineering personnel to evaluate knowledge of implementation of Engineering Change processes for proposed design changes.
The inspector concluded that adequate measures of success had been developed for determining the effectiveness of the corrective actions to prevent recurrence.
e.
Determine whether the corrective actions planned or taken adequately address the Notice of Violation that was the basis for the supplemental inspection.
The NRC issued a Notice of Violation to the licensee on March 28, 2014, for the failure to perform adequate testing on the train B EDG exhaust fan to demonstrate that the exhaust fan would perform satisfactorily in service (NRC Inspection Report 05000382/2014009, ADAMS ML14086A768). During this inspection, the inspector determined that the licensee restored compliance by revising the system operating Procedure OP-009-002, Emergency Diesel Generator, and surveillance testing Procedure OP-903-068, Emergency Diesel Generator and Subgroup Relay Operability Verification, to include requirements to monitor the ventilation exhaust fan differential pressure during testing. The inspector concluded that the corrective actions taken or planned were adequate to address the Notice of Violation that was the basis for the supplemental inspection.
f. Findings
No findings were identified.
02.04 Evaluation of IMC 0305 Criteria For Treatment Of Old Design Issues
The licensee did not request credit for self-identification of an old design issue; therefore, the risk-significant issue was not evaluated against the IMC 0305 criteria for treatment of an old design issue.
4OA6 Meetings, Including Exit
Exit Meeting Summary
On March 19, 2015, the inspector presented the inspection results to Mr. M. Chisum, Site Vice President, and other members of the licensee staff. The licensee acknowledged the issues presented. The inspector asked the licensee if any of the material examined during the inspection should be considered proprietary. The licensee did not identify any proprietary information.
4OA7 Licensee-Identified Violations
None.
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
- M. Chisum, Site Vice President
- M. Richey, General Manager, Plant Operations
- R. Gilmore, Manager, Systems and Components
- J. Jarrell, Manager, Regulatory Assurance
- B. Lanka, Director, Engineering
- L. Milster, Licensing Specialist, Licensing
- R. Osborne, Manager, Performance Improvement
- M. Chase, Director, Regulatory & Performance Improvement
NRC Personnel
- F. Ramirez, Senior Resident Inspector
- C. Speer, Resident Inspector
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Opened
None
Closed
- 05000382/2013008-01 VIO Failure To Establish an Adequate Test Program to Demonstrate that the train B Emergency Diesel Generator Exhaust Fan Would Perform Satisfactorily In Service (Section 4OA4)
Discussed
None