05000382/LER-2015-007-01, Both Emergency Diesel Generators Declared Inoperable

Both Emergency Diesel Generators Declared Inoperable

ML16341C141
Person / Time
Site:	Waterford
Issue date:	12/06/2016
From:	Jarrell J Entergy Operations
To:	Document Control Desk, Office of Nuclear Reactor Regulation
References
LER 15-007-01
Download: ML16341C141 (10)

LER-2015-007, Both Emergency Diesel Generators Declared Inoperable

Event date:
Report date:
Reporting criterion:	10 CFR 50.73(a)(2)(v)(A), Loss of Safety Function - Shutdown the Reactor 10 CFR 50.73(a)(2)(v)(B), Loss of Safety Function - Remove Residual Heat 10 CFR 50.73(a)(2)(ii)(A), Seriously Degraded 10 CFR 50.73(a)(2)(viii)(A) 10 CFR 50.73(a)(2)(ii)(B), Unanalyzed Condition 10 CFR 50.73(a)(2)(viii)(B) 10 CFR 50.73(a)(2)(iii) 10 CFR 50.73(a)(2)(ix)(A) 10 CFR 50.73(a)(2)(iv)(A), System Actuation 10 CFR 50.73(a)(2)(x) 10 CFR 50.73(a)(2)(v), Loss of Safety Function 10 CFR 50.73(a)(2)(i)(A), Completion of TS Shutdown 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications 10 CFR 50.73(a)(2)(vii), Common Cause Inoperability 10 CFR 50.73(a)(2)(i)
3822015007R01 - NRC Website

text

Entergy Operations, Inc.

17265 River Road Killona, LA 70057-3093 jjarrel©entergy.com John P. Jarrell III Manager. Regulatory Assurance Waterford 3 10 CFR 50.73 W3F1 -2016-0072 December 6, 2016 U.S. Nuclear Regulatory Commission Attn: Document Control Desk 11555 Rockville Pike Rockvf lie, MD 20852

Subject:

Licensee Event Report (LER) 2015-007-01 Waterford Steam Electric Station, Unit 3 (Waterford 3)

Docket No. 50-382 License No. NPF-38

Dear Sir or Madam:

Entergy is hereby submitting Licensee Event Report (LER) 2015-007-01 for Waterford Steam Electric Station (Waterford 3). This is a follow-up report to provide the safety significance determination and supplemental information from the investigation into the component failure for the Emergency Diesel Generator room ventilation system.

On August 26, 2015, both Emergency Diesel Generators at Waterford 3 were declared inoperable, causing entry into Technical Specification 3.8.1.1 action f.

It was determined that this condition is reportable pursuant to 10 CFR 50.73(a)(2)(v)(A),

50.73(a)(2)(v)(B) and 10 CFR21.

This report contains no new commitments. Please contact John P. Jarrell, Regulatory Assurance Manager, at (504) 739-6685 if you have questions regarding this information.

Attachments: 1. LER-201 5-007-0 1

W3F1-2016-0072 Page 2 cc:

Mr. Kriss Kennedy, Regional Administrator U.S. NRC, Region IV RidsRgn4MailCenter@nrc.gov U.S. NRC Project Manager for Waterford 3 April.Pulvirenti@nrc.gov U.S. NRC Senior Resident Inspector for Waterford 3 Frances.Ramirez@nrc.gov Chris.Speer@nrc.gov

to W3F1-2016-0072 Licensee Event Report 2015-007-01

NRC FORM 366 (06-2016)

NRC FORM 366 U.S. NUCLEAR REGULATORY COMMISSION (06-2016)

LICENSEE EVENT REPORT (LER)

(See Page 2 for required number of digits/characters for each block)

(See NUREG-1022, R.3 for instruction and guidance for completing this form http://www.nrc/gov/reading-rm/doc-collections/nuregs/staff/sr1022/r3/)

APPROVED BY OMB: NO. 3150-0104 EXPIRES: 10/31/2018 Estimated burden per response to comply with this mandatory collection request: 80 hours9.259259e-4 days <br />0.0222 hours <br />1.322751e-4 weeks <br />3.044e-5 months <br />.

Reported lessons learned are incorporated into the licensing process and fed back to industry.

Send comments regarding burden estimate to the FOIA, Privacy and Information Collections Branch (T-5 F53), U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001, or by e-mail to Infocollects.Resource@nrc.gov, and to the Desk Officer, Office of Information and Regulatory Affairs, NEOB-10202, (3150-0104), Office of Management and Budget, Washington, DC 20503. If a means used to impose an information collection does not display a currently valid OMB control number, the NRC may not conduct or sponsor, and a person is not required to respond to, the information collection.

1. FACILITY NAME Waterford Steam Electric Station, Unit 3

2. DOCKET NUMBER 05000-382

3. PAGE 1 OF 7

4. TITLE Both Emergency Diesel Generators Declared Inoperable

5. EVENT DATE

6. LER NUMBER

7. REPORT DATE

8. OTHER FACILITIES INVOLVED MONTH DAY YEAR YEAR SEQUENTIAL NUMBER REV NO.

MONTH DAY YEAR FACILITY NAME DOCKET NUMBER 08 26 2015 2015 -

007

- 01 12 6

2016 FACILITY NAME DOCKET NUMBER

9. OPERATING MODE

11. THIS REPORT IS SUBMITTED PURSUANT TO THE REQUIREMENTS OF 10 CFR §: (Check all that apply) 1 20.2201(b) 20.2203(a)(3)(i) 50.73(a)(2)(ii)(A) 50.73(a)(2)(viii)(A) 20.2201(d) 20.2203(a)(3)(ii) 50.73(a)(2)(ii)(B) 50.73(a)(2)(viii)(B) 20.2203(a)(1) 20.2203(a)(4) 50.73(a)(2)(iii) 50.73(a)(2)(ix)(A) 20.2203(a)(2)(i) 50.36(c)(1)(i)(A) 50.73(a)(2)(iv)(A) 50.73(a)(2)(x)

10. POWER LEVEL 20.2203(a)(2)(ii) 50.36(c)(1)(ii)(A) 50.73(a)(2)(v)(A) 73.71(a)(4) 100 20.2203(a)(2)(iii) 50.36(c)(2) 50.73(a)(2)(v)(B) 73.71(a)(5) 20.2203(a)(2)(iv) 50.46(a)(3)(ii) 50.73(a)(2)(v)(C) 73.77(a)(1) 20.2203(a)(2)(v) 50.73(a)(2)(i)(A) 50.73(a)(2)(v)(D) 73.77(a)(2)(i) 20.2203(a)(2)(vi) 50.73(a)(2)(i)(B) 50.73(a)(2)(vii) 73.77(a)(5)(ii) 50.73(a)(2)(i)(C)

OTHER Specify in Abstract below or in EDG A Generator Differential EDG 'A' was being run in accordance with OP-903-115, "Train A Integrated Emergency Diesel Generator/Engineering Safety Features Test," Section 7.4, "24 hr EDG A Run with Subsequent Diesel Start" to satisfy TS SR 4.8.1.1.2.e.6. The EDG function of supplying standby electrical power on receipt of a test or emergency command signal are different in that during an Emergency Mode start of the EDG, all Test Mode trips and alarms are bypassed with the exception of overspeed and generator differential.

The direct cause for EDG A tripping on GENERATOR DIFFERENTIAL was the internal shorting of EG ECT2316 C TRANSF, NB8 Current Transformer, due to insulation failure.

The EG ECT2316 C TRANSF is a Westinghouse type KIR-60 current transformer style 7524A01G16 with serial number 28218571. There are no lot or date codes printed on the current transformer or its nameplate. The current transformer is only energized when EDG A is supplying the 3A bus.

A vendor performed a failure analysis of the failed EG ECT2316 C TRANSF, current transformer and issued a failure analysis report dated October 9, 2015. The report concluded that the failure was due to a manufacturing defect. Specifically, there were voids found in the insulation and the thickness of the insulation material around the fault area appeared reduced when compared to the other areas of the current transformer. It is believed that the thinner insulation in combination with voids increased the electrical stresses causing the insulation to break down. This eventually resulted in a fault. The insulation breakdown and resultant fault created a ground condition on the Diesel Generator bus.

Investigation to determine if any other current transformers of this make and model were installed in the plant identified that the EDG B Generator Differential Current Transformer was the same. Additionally, there are power current donut style transformers for EDG A and EDG B, nine per EDG, feeding the regulator circuit.

These are a different manufacturer, model, and construction, but are bus type current transformers that are the same age and in the same cabinet environment as the EG ECT2316 C TRANSF, NB8 Current Transformer.

EDG B HVR 501B failure On August 26, 2015, EDG 'B' was started per TS 3.8.1.1 action b.(1) which requires the demonstration of Operability of the remaining Operable EDG to preclude common mode failure of the remaining EDG. At 0740 CDT, EDG 'B' was declared inoperable and TS 3.8.1.1.f. (restore one of the inoperable EDG's to Operable status within 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> or be in at least Hot Standby within the following 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />) was entered due to the room exhaust fan not starting when the diesel engine was started. Troubleshooting determined that the EDG B room exhaust fan did not start due to HVR-501B (EG B ROOM OUTSIDE AIR INTAKE DAMPER) not opening. The solenoid on HVR-501B was replaced and tested satisfactorily. The solenoid was inspected on site both externally and internally and it was determined that there was mechanical wear on the inside of the solenoid, the air inlet valve plug was bound up inside the solenoid coil, and that the solenoid coil itself appeared to be good. Photographs of the condition were sent to an offsite vendor who concluded that rapid cycling of the solenoid valve might be the cause of the excessive wear and damage to the components.

The design of the exhaust fan inlet damper for EDG A (HVR-501A) (EG A ROOM OUTSIDE AIR INTAKE DAMPER) is different than for EDG B. HVR-501A is a large wall-mounted damper that is divided into four sections. HVR-501B is a single damper installed in its associated ductwork. There are four inlet damper sections

that make up HVR-501A. Each damper set has its own air actuator and an air supply solenoid valve. The failure of a single solenoid valve will not prevent the starting of the EDG A Exhaust Fan.

An offsite vendor performed and analysis of the solenoid associated with HVR-501B and determined that rapid cycling of the solenoid valve was the cause of the excessive wear and damage to the components. Waterfords suspicion that a faulty relay, Potter & Brumfield rotary-model MDR relays, led to rapid cycling of the valve resulting in the damage to the components is consistent with our previous findings. 600 cycles under normal conditions will not lead to the mechanical damage seen with these components. The mushrooming of the pins is due to the rapid cycling and wear of the components. The pins are not designed with this feature.

Potter & Bromfield model MDR relays have an operating history of contacts making either intermittent or high resistance connections. In one failure mode of the MDR relay, the contact rotor sticks in mid-stroke whereby a contact can be partially connecting or connecting after a delay period. A second failure mode was one in which the contacts themselves were corroded and degraded, making intermittent connection. Both of these failure modes could cause the SOV to drop out and pick up without a signal from the EDG ventilation system.

If the SOV cycles quickly from an intermittent contact, it is conceivable that the EDG room inlet damper actuator may not have time to react to the fast cycle of the solenoid valve. From the damper-closed position, with the actuator spring compressed and the air volume full of air, a quick cycle of the SOV may not allow the actuator to exhaust its air volume. Thus, the inlet damper may not move and the Control Room may have no indication of the solenoid cycling.

REPORTABLE OCCURRENCE TS 3.8.1.1 requires that two separate and independent diesel generators shall be operable, each with diesel oil feed tanks containing a minimum volume of 339 gallons of fuel, a separate diesel generator fuel oil storage tank, and a separate fuel transfer pump.

TS 3.8.1.1 Action b: With one diesel generator of 3.8.1.1.b. inoperable: (1) Demonstrate the OPERABILITY of the remaining A.C. circuits by performing SR 4.8.1.1.1.a. (separately for each offsite A.C. circuit) within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> and at least once per 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> thereafter. If the diesel generator became inoperable due to any cause other than an inoperable support system, an independently testable component, or preplanned maintenance or testing, demonstrate the OPERABILITY of the remaining OPERABLE diesel generator (unless it has been successfully tested in the last 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />) by performing SR 4.8.1.1.2.a.4. within 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> unless the absence of any potential common mode failure for the remaining diesel generator is demonstrated.

TS 3.8.1.1 Action f: With two of the above required diesel generators inoperable, demonstrate the OPERABILITY of two offsite AC circuits by performing SR 4.8.1.1.a. within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> and at least once per 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> thereafter; restore one of the inoperable diesel generators to OPERABLE status within 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> or be in at least Hot Standby within the next 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and in Cold Shutdown within the following 30 hours3.472222e-4 days <br />0.00833 hours <br />4.960317e-5 weeks <br />1.1415e-5 months <br />. Following restoration of one diesel generator follow Action statement b. with the time requirement of that action statement based on the time of initial loss of the remaining inoperable diesel generator.

This event is reportable under 10 CFR 50.73 (a)(2)(v)(A) and 10 CFR 50.73 (a)(2)(v)(D) "Any event or condition that could have prevented the fulfillment of the safety function of structures or systems that are needed to: (A)

Shut down the reactor and maintain it in a safe shutdown condition, and (D) Mitigate the consequences of an accident."

10 CFR 21 IDENTIFICATION On October 9, 2015, Waterford 3 received information from the external evaluation concerning the Generator Differential Current Transformer. The evaluation concluded that a manufacturing defect internal to the current transformer was the cause of the failure. On October 22, 2015, engineering evaluation determined the manufacturing defect could create a substantial safety hazard, as defined in 10 CFR 21, and provided the site vice president information of the defect the same day. Additional information identified in the report is as follows:

Constructor - Westinghouse Type KIR-60 current transformer, style 7524A01G16, serial number 28218571; Defect and safety hazard - There were voids found in the insulation, and the thickness of the insulation material around the fault area appeared reduced when compared to the other areas of the current transformer. There is only one transformer of this type remaining installed in the plant. Scheduled replacement is no later than November 15, 2015.

PREVIOUS OCCURRENCES

EDG A CR-WF3-2015-5610 PME performed insulation resistance (megger) testing of EDG-A circuitry associated with the differential relay to validate the received ground reading. The megger results of ~0 Meg-Ohms resistance indicate the received ground alarm is accurate. Troubleshooting determined a ground on the NB8 CT transformer had grounded the A phase generator bus. When the NB8 transformer secondary was lifted the ground indication cleared on the A phase generator bus. The NB8 transformer was replaced under WO-423632.

CR-WF3-2015-5635 PME troubleshooting of EDG-A discovered that CT NB8 A (Current Transformer) was shorted to ground. Troubleshooting determined a ground on the NB8 CT transformer had grounded the A phase generator bus. When the NB8 transformer secondary was lifted the ground indication cleared on the A phase generator bus. The NB8 transformer was replaced under WO-423632.

EDG B CR-WF3-2006-4606 Noticed inlet damper HVR501B in the open position, expected position for current plant conditions would be closed. The problem appears to be the solenoid valve not opening or one of the two permissive contacts not closing (52X or CX).

CR-WF3-2008-00753 Found the EDG 'B' outside air ventilation damper (HVR-501B) was in the 'OPEN' position rather than its normally 'CLOSED' CAUSAL FACTORS The apparent cause of the EDG A trip on generator differential is manufacturer fabrication that is less than adequate to preclude component failure. This apparent cause resulted in the condition where thinner insulation in the current transformer in combination with voids created during the manufacturing process increased the electrical stresses, thereby causing the insulation to break down. This condition eventually resulted in a fault. The insulation breakdown and resulting fault created a ground condition on the Diesel Generator bus which tripped EG EGEN0001-A, EDG A on GENERATOR DIFFERENTIAL.

The direct cause of EDG B damper HVR-501B failure to open was the failure of the solenoid valve, HVRISV501B, to isolate the air supply from the damper AOV actuator. The solenoid valve failed mechanically, internally, due to wear-out damage of moving components inside the valve housing.

The root cause of EDG B damper HVR-501B failure to open was lack of adequate preventative maintenance tasks were not put in place for Emergency Diesel Generator B inlet damper components. The EDG B air inlet damper and associated actuator, solenoid valve and relays (CX, 52X) were considered to be fail safe. These components were classified as High Critical / Low Duty Cycle. The PM templates did not require periodic replacements. An attempt was made to put a replacement task in place for HVRISV501B but the task was never performed. The AR for the new replacement task inappropriately credited a planned future replacement task that was never performed.

A contributing cause of EDG B damper HVR-501B failure to open was troubleshooting performed for the multiple failures of damper HVR-501B to close were inadequate. Between 2006 and 2008 several WOs were completed to resolve the issue with HVR-501B spuriously opening. Troubleshooting performed did not identify intermittent CX relay contacts as the cause of the issue.

CORRECTIVE ACTIONS

For EDG A Completed Actions

1. Replace Current Transformer in EDG A

2. Failure analysis on the failed EDG A Current Transformer

3. Replace Current Transformer for Generator Differential on EDG B

4. Crease a preventative maintenance task to replace the EG ECT2316 C TRANSF, NB8 Current Transformer

5. Crease a preventative maintenance task to replace power current donut style transformers for EDG A and EDG B Actions

1. Replace remaining donut style transformers on EDG A and B (nine per EDG)

For EDG B Completed Actions

1. Replace Solenoid [SOL] associated with HVR-501B

SAFETY SIGNIFICANCE

Industrial Safety: There was no industrial safety significance associated with this issue.

Radiological Safety: There was no radiological safety significance associated with this issue.

Environmental Safety: There was no environmental safety significance associated with this issue.

Nuclear Safety: The finding was determined to be of very low safety significance.

The actual consequence as stated in the problem statement was entry into a 2-hour shutdown LCO due to both emergency diesel generators being concurrently inoperable. Both emergency diesel generator issues were resolved, and the EDGs were tested successfully. Other than a short window of increased risk, there was no actual consequence to general safety of the public, nuclear safety, and radiological safety for this event. There was no consequence related to industrial safety.

If response actions are delayed, unreliable emergency diesel generators failing simultaneously during a loss of off-site power event will mean that safe shutdown cooling equipment could not be operated and the reactor will be in jeopardy of core damage. Such an event will place the general public at risk of exposure to a radiological event beyond the site boundary.

Because the EDG B inoperable condition was related to failure of the inlet damper to open, there was a high probability that the damper could be opened manually or that the room could be cooled by other means. The risk of this particular equipment failure not being recoverable was Low. EDG B would have been made operable in the case of a loss of off-site power. The current transformer failure on EDG A was an age-related failure due to a latent manufacturing inadequacy.

Because there were two simultaneous equipment failures, and given that there have been several other EDG equipment failures in recent years, the probability of more EDG failures if no actions are taken to address latent equipment weaknesses is considered high.

The actions to reduce the frequency or consequence are included in the Root Cause Evaluation Corrective Action Plan.

ADDITIONAL INFORMATION

Energy industry identification system (EIIS) codes and component function identifiers are identified in the text with brackets [ ].