ML063400561
| ML063400561 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 12/06/2006 |
| From: | Howell A NRC/RGN-IV/DRP |
| To: | James M. Levine Arizona Public Service Co |
| References | |
| EA-06-296 IR-06-012 | |
| Download: ML063400561 (36) | |
See also: IR 05000528/2006012
Text
December 6, 2006
James M. Levine, Executive
Vice President, Generation
Mail Station 7602
Arizona Public Service Company
P.O. Box 52034
Phoenix, AZ 85072-2034
SUBJECT: PALO VERDE NUCLEAR GENERATING STATION, UNITS 1, 2, and 3 - NRC
SPECIAL INSPECTION REPORT 05000528/2006012; 05000529/2006012;
Dear Mr. Levine:
On November 30, 2006, the U.S. Nuclear Regulatory Commission (NRC) completed a special
inspection at your Palo Verde Nuclear Generating Station, Units 1, 2, and 3. This inspection
examined activities associated with the Unit 3 Train A emergency diesel generator (EDG)
failures that occurred on July 25 and September 22, 2006. On both occasions the EDG failed
to produce an output voltage during testing. The NRC's initial evaluation satisfied the criteria in
NRC Management Directive 8.3, NRC Incident Investigation Program, for conducting a special
inspection. The basis for initiating this special inspection is further discussed in the Charter,
which is included as Attachment 2 to this report. The determination that the inspection would
be conducted was made by the NRC on September 29, 2006, and the inspection started on
October 2, 2006.
The enclosed special inspection report documents the inspection findings which were discussed
on November 9, 2006, with you, and other members of your staff, and on November 30, 2006,
with Mr. David Mauldin, Vice President, Engineering, and other members of your staff.
The inspection examined activities conducted under your license as they relate to safety and
compliance with the Commission's rules and regulations and with the conditions of your license.
The inspectors reviewed selected procedures and records, observed activities, and interviewed
personnel.
The attached report discusses two findings that appear to have low to moderate safety
significance (White). As described in Section 3.0 of this report, the NRC concluded that the
failure to establish appropriate instructions for performing corrective maintenance activities on a
K-1 relay resulted in the Unit 3 Train A EDG being inoperable between September 4 and 22,
2006. Additionally, the failure to identify and correct the cause of the erratic EDG K-1 relay
operation prior to installation of the relay on July 26, 2006, was identified as another
performance deficiency that contributed to the Unit 3 Train A EDG being inoperable for a period
Arizona Public Service Company
-2-
greater than the Technical Specification completion time. The safety significance of these
findings was assessed on the basis of the best available information, including influential
assumptions, using the applicable Significance Determination Process and were preliminarily
determined to be White (i.e., low to moderate safety significance) findings. Preliminarily, these
findings have a low to moderate safety significance when assuming a loss of offsite power
initiating event and the Unit 3 Train A EDG being in an unreliable condition for approximately
40 days and a nonfunctional condition for approximately 18 days. Attachment 3 of this report
provides a detailed description of the preliminary risk assessment. In accordance with NRC
Inspection Manual Chapter (IMC) 0609, Significance Determination Process, we intend to
complete our evaluation using the best available information and issue our final determination
of safety significance within 90 days of this letter.
These findings do not represent an immediate safety concern because of the corrective actions
you have taken. These actions involved inspecting, cleaning, and implementing mechanical
adjustments, as appropriate, to the operating mechanism of the EDG K-1 relays.
Also, these findings constitute apparent violations of NRC requirements and are being
considered for escalated enforcement action in accordance with the NRC Enforcement Policy.
The current Enforcement Policy is included on the NRCs web site at
http://www.nrc.gov/reading-rm/adams.html.
Before we make a final decision on this matter, we are providing you an opportunity to present
to the NRC your perspectives on the facts and assumptions, used by the NRC to arrive at the
findings and their significance, at a Regulatory Conference or in writing. As discussed during a
telephone call with Mr. Scott Bauer, Department Leader, Regulatory Affairs, we understand that
it is your intent to discuss your perspectives during a Regulatory Conference. Accordingly, a
Regulatory Conference is scheduled to be conducted in the NRC Region IV office in Arlington,
Texas, on January 16, 2007. We encourage you to submit supporting documentation at least
one week prior to the conference in an effort to make the conference more efficient and
effective. This Regulatory Conference will be open to public observation.
Since the NRC has not made a final determination in this matter, no Notice of Violation is being
issued for these inspection findings at this time. In addition, please be advised that the number
and characterization of apparent violations described in the enclosed inspection report may
change as a result of further NRC review.
The report also documents one finding with two examples involving inadequate implementation
of the operability determination process. This finding was determined to be a violation of very
low safety significance. Because of the very low safety significance and because it was entered
into your corrective action program, the NRC is treating this finding as a noncited violation
consistent with Section VI.A.1 of the NRC Enforcement Policy. If you contest the noncited
violation in this report, you should provide a response within 30 days of the date of this
inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission,
ATTN.: Document Control Desk, Washington, DC 20555-0001; and the NRC Resident
Inspector at the Palo Verde Nuclear Generating Station.
In accordance with 10 CFR 2.390 of the NRC's Rules of Practice, a copy of this letter
and its enclosure will be available electronically for public inspection in the NRC Public
Document Room or from the Publicly Available Records (PARS) component of NRC's
Arizona Public Service Company
-3-
document system (ADAMS). ADAMS is accessible from the NRC Web site at
http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
Arthur T. Howell III, Director
Division of Reactor Projects
Dockets: 50-528
50-529
50-530
Licenses: NPF-41
Enclosure:
Inspection Report 05000528/2006012; 05000529/2006012; 05000530/2006012
w/Attachment 1: Supplemental Information
Attachment 2: Special Inspection Charter
Attachment 3: Significance Determination Evaluation
cc w/enclosure:
Steve Olea
Arizona Corporation Commission
1200 W. Washington Street
Phoenix, AZ 85007
Douglas K. Porter, Senior Counsel
Southern California Edison Company
Law Department, Generation Resources
P.O. Box 800
Rosemead, CA 91770
Chairman
Maricopa County Board of Supervisors
301 W. Jefferson, 10th Floor
Phoenix, AZ 85003
Aubrey V. Godwin, Director
Arizona Radiation Regulatory Agency
4814 South 40 Street
Phoenix, AZ 85040
Craig K. Seaman, General Manager
Regulatory Affairs and
Performance Improvement
Palo Verde Nuclear Generating Station
Arizona Public Service Company
-4-
Mail Station 7636
P.O. Box 52034
Phoenix, AZ 85072-2034
Jeffrey T. Weikert
Assistant General Counsel
El Paso Electric Company
Mail Location 167
123 W. Mills
El Paso, TX 79901
John W. Schumann
Los Angeles Department of Water & Power
Southern California Public Power Authority
P.O. Box 51111, Room 1255-C
Los Angeles, CA 90051-0100
John Taylor
Public Service Company of New Mexico
2401 Aztec NE, MS Z110
Albuquerque, NM 87107-4224
Thomas D. Champ
Southern California Edison Company
5000 Pacific Coast Hwy, Bldg. D1B
San Clemente, CA 92672
Robert Henry
Salt River Project
6504 East Thomas Road
Scottsdale, AZ 85251
Brian Almon
Public Utility Commission
William B. Travis Building
P.O. Box 13326
1701 North Congress Avenue
Austin, TX 78701-3326
Karen O'Regan
Environmental Program Manager
City of Phoenix
Office of Environmental Programs
200 West Washington Street
Phoenix, AZ 85003
Matthew Benac
Assistant Vice President
Nuclear & Generation Services
Arizona Public Service Company
-5-
El Paso Electric Company
340 East Palm Lane, Suite 310
Phoenix, AZ 85004
Arizona Public Service Company
-6-
Electronic distribution by RIV:
Regional Administrator (BSM1)
DRP Director (ATH)
DRS Director (DDC)
DRS Deputy Director (RJC1)
Senior Resident Inspector (GXW2)
Branch Chief, DRP/D (TWP)
Senior Project Engineer, DRP/D (GEW)
Team Leader, DRP/TSS (RVA)
RITS Coordinator (KEG)
V. Dricks, PAO (VLD)
D. Cullison, OEDO RIV Coordinator (DGC)
ROPreports
K. S. Fuller, RC/ACES (KSF)
C. A. Carpenter, D:OE (CAC)
G. M. Vasquez (GMV)
OE:EA File (RidsOeMailCenter)
SUNSI Review Completed: __TWP_ ADAMS: / Yes
G No Initials: __TWP____
/ Publicly Available G Non-Publicly Available G Sensitive
/ Non-Sensitive
R:\\_REACTORS\\_PV\\2006\\PV2006-12RP-MCH.wpd
RIV:SPE:DRP/D RI:DRS/EB1
PE:DRP/D
C:DRP/D
SRA:DRS
ACES
MCHay
SPRutenkroger
MRBloodgood TWPruett
MFRunyan
GMVasquez
/RA/
MCHay For
MCHay For
/RA/
/RA/
/RA/
11 /27/06
11/28/06
11/28/06
11/28/06
11/28/06
11/28/06
D:DRP
ATHowell III
/RA/
12/06/06
OFFICIAL RECORD COPY
T=Telephone E=E-mail F=Fax
U.S. NUCLEAR REGULATORY COMMISSION
REGION IV
Dockets:
50-528; 50-529; 50-530
Licenses:
Report No.:
05000528/2006012; 05000529/2006012; 05000530/2006012
Licensee:
Arizona Public Service Company
Facility:
Palo Verde Nuclear Generating Station, Units 1, 2, and 3
Location:
5951 S. Wintersburg Road
Tonopah, Arizona
Dates:
October 2 through November 30, 2006
Inspectors:
M. Hay, Senior Project Engineer, Team Leader
Dr. S. Rutenkroger, Reactor Inspector, Engineering Branch 1
M. Runyan, Senior Reactor Analyst
Accompanied:
M. Bloodgood, Reactor Engineer, Nuclear Safety Professional Development
Program
Approved By:
Arthur T. Howell III, Director
Division of Reactor Projects
Enclosure
1
SUMMARY OF FINDINGS
IR 05000528/2006012; 05000529/2006012; 05000530/2006012; 10/02/2006 - 11/09/2006; Palo
Verde Nuclear Generating Station, Units 1, 2, and 3: Special Inspection in response to Unit 3
Train A EDG failures on July 25 and September 22, 2006.
The report covered a 5-day period (October 2-6, 2006) of onsite inspection, with in-office review
through November 30, 2006, by a special inspection team consisting of one senior project
engineer, one reactor inspector, one reactor engineer, and one senior reactor analyst. Three
findings were identified. The significance of most findings is indicated by its color (Green,
White, Yellow, Red) using Inspection Manual Chapter 0609, Significance Determination
Process. Findings for which the significance determination process does not apply may be
Green or be assigned a severity level after NRC management review. The NRC's program for
overseeing the safe operation of commercial nuclear power reactors is described in
NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.
Summary of Event
The NRC conducted a special inspection to better understand the circumstances surrounding
two failures of the Unit 3 Train A emergency diesel generator that occurred on July 25 and
September 22, 2006. Both failures involved the emergency diesel generator failing to obtain an
output voltage during surveillance testing because of faulty K-1 relay operation. In accordance
with NRC Management Directive 8.3, NRC Incident Investigation Program, it was determined
that this event involved repetitive failures of safety related equipment having potential adverse
generic implications and had sufficient risk significance to warrant a special inspection.
A.
NRC-Identified and Self Revealing Findings
Cornerstone: Mitigating Systems
TBD. The team identified an apparent violation of 10 CFR Part 50, Appendix B,
Criterion V, Instructions, Procedures, and Drawings, for the failure to establish
appropriate instructions for performing corrective maintenance activities on an
emergency diesel generator K-1 relay. As a result, following identification that a
replacement emergency diesel generator K-1 relay was unreliable, the licensee
performed ineffective corrective maintenance activities on this relay. This
performance deficiency contributed to the Unit 3 Train A emergency diesel
generator being inoperable between September 4 and 22, 2006, and a failure on
September 22, 2006. Immediate corrective actions included inspection,
cleaning, and/or performing mechanical adjustments on all emergency diesel
generator K-1 relays. This issue was entered into the licensees corrective
action program as Condition Report/Disposition Request 2926830.
The finding is greater than minor because it is associated with the equipment
performance cornerstone objective to ensure the availability, reliability, and
capability of systems that respond to initiating events to prevent undesirable
consequences. Using NRC Inspection Manual Chapter 0609, Significance
Determination Process, Phase 1 Worksheet, a Phase 2 evaluation was required
Enclosure
2
because the finding resulted in the loss of the safety function of the Unit 3 Train
A emergency diesel generator for greater than the Technical Specification
completion time. The Phase 2 evaluation concluded that the finding was of low
to moderate safety significance. A Phase 3 preliminary significance
determination analysis also determined the finding was of low to moderate safety
significance. The cause of this finding is related to the crosscutting element of
human performance associated with resources in that the licensee failed to
develop and implement appropriate work instructions prior to performing
corrective maintenance activities on an emergency diesel generator K-1 relay
(Section 3.0).
TBD. The team identified an apparent violation of 10 CFR Part 50, Appendix B,
Criterion XVI, Corrective Actions, for the failure to identify and correct the
cause of erratic emergency diesel generator K-1 relay operation prior to
installation of the relay on July 26, 2006. This performance deficiency
contributed to the Unit 3 Train A emergency diesel generator being inoperable
between September 4 and 22, 2006, and a failure on September 22, 2006.
Immediate corrective actions included inspection, cleaning, and performing
mechanical adjustments, as appropriate, on all emergency diesel generator K-1
relays. This issue was entered into the licensees corrective action program as
Condition Report/Disposition Request 2926830.
The finding is greater than minor because it is associated with the equipment
performance cornerstone objective to ensure the availability, reliability, and
capability of systems that respond to initiating events to prevent undesirable
consequences. Using NRC Inspection Manual Chapter 0609, Significance
Determination Process, Phase 1 Worksheet, a Phase 2 evaluation was required
because the finding resulted in the loss of the safety function of the Unit 3 Train
A emergency diesel generator for greater than the Technical Specification
allowed outage time. The Phase 2 evaluation concluded that the finding was of
low to moderate safety significance. A Phase 3 preliminary significance
determination analysis also determined the finding was of low to moderate
safety significance. The cause of this finding is related to the crosscutting
element of problem identification and resolution in that the failure to fully
evaluate and implement adequate corrective maintenance actions for the Unit 3
Train A emergency diesel generator resulted in the emergency diesel generator
being inoperable for 18 days (Section 3.0).
The team identified two examples of a noncited violation of 10 CFR Part 50,
Appendix B, Criterion V, Instructions, Procedures, and Drawings, for the failure
to perform operabilty determinations. In both examples, the licensee failed to
perform an operability determination following identification of a degraded
condition that had the potential to adversely affect the safety function of all
emergency diesel generators. Specifically, an operability determination was not
performed after identifying the failure of the Unit 3 Train A emergency diesel
generator on July 25, 2006, was potentially the result of plastic debris affecting
proper auxiliary contact operation of a K-1 relay. The licensee determined the
debris most likely originated from a modification performed on all emergency
diesel generator K-1 relays during initial plant startup. Following another failure
Enclosure
3
of the Unit 3 Train A emergency diesel generator on September 22, 2006, an
operability determination was not performed after identifying the failure was the
result of the K-1 relay actuating arm not providing adequate compression of the
auxiliary contacts. The licensee determined this degraded condition most likely
originated during implementation a modification done to all emergency diesel
generator K-1 relays during initial plant startup.
This finding is greater than minor because the failure to follow the operability
determination process, if left uncorrected, would become a more significant
safety concern in that degraded or nonconforming conditions would not be
properly evaluated. Using the Phase 1 worksheet in NRC Inspection Manual
Chapter 0609, Significance Determination Process, the finding was determined
to have very low safety significance because unreliable K-1 relay operation
resulted in no actual loss of safety function of the other five emergency diesel
generators prior to corrective actions being implemented, and the finding did not
represent a potential risk significant condition because of a seismic, flooding, or
severe weather event. This issue is documented in the licensees corrective
action program as Condition Report/Disposition Requests 2928389 and
2940558. The cause of this finding is related to the crosscutting element of
problem identification and resolution in that engineering personnel failed to
properly evaluate and perform operability determinations for identified degraded
conditions affecting the emergency diesel generators (Section 4.0).
B.
Licensee-Identified Findings
None.
Enclosure
4
REPORT DETAILS
1.0
SPECIAL INSPECTION SCOPE
The NRC conducted this special inspection to better understand the circumstances
surrounding two failures of the Unit 3 Train A emergency diesel generator (EDG) that
occurred on July 25 and September 22, 2006. Both failures involved the EDG failing to
obtain an output voltage during surveillance testing because of a faulty K-1 relay
operation. In accordance with NRC Management Directive 8.3, NRC Incident
Investigation Program, it was determined that this event met several deterministic
criteria and had sufficient risk significance to warrant a special inspection.
The team used NRC Inspection Procedure 93812, Special Inspection Procedure, to
conduct the inspection. The special inspection team reviewed procedures, corrective
action documents, and design and maintenance records for the equipment of concern.
The team interviewed key station personnel regarding the event, reviewed the root
cause analysis, and assessed the adequacy of corrective actions. A list of specific
documents reviewed is provided in Attachment 1. The charter for the special inspection
effort is provided as Attachment 2.
2.0
EVENT DESCRIPTION
Each unit at Palo Verde contains two safety-related 5500 Kw EDGs that provide standby
power for safe plant shutdown in the event the normal supply of power is lost. On
July 25, 2006, at 12:53 p.m., the Unit 3 Train A EDG failed to develop an output voltage
during a routine surveillance test. When the EDGs are secured a field shorting K-1
relay actuates to electrically short the generator field causing the generator output
voltage to collapse. A relay latching mechanism maintains the field shorted until the
EDG is started at which time the latch disengages allowing the relay to actuate and un-
short the EDG field. With the field no longer shorted the voltage regulator establishes
and controls the EDG output voltage. The licensee identified that a faulty set of auxiliary
contacts on a K-1 relay prevented the latch from disengaging that resulted in the
generator field being shorted during the start of the EDG. The licensee determined the
root cause of the auxiliary contact failure could be attributed to either plastic debris or
oxide film buildup preventing continuity across the contacts when closed. Following the
failure on July 25, 2006, the licensee replaced the failed K-1 relay with a new relay
obtained from the warehouse. During continuity checks of the new relay, the same
auxiliary contacts were identified to operate unreliably. The last available relay from the
warehouse was obtained and it also operated unreliably and it had a warped cover.
Based on no other replacement K-1 relays being available, the licensee performed
corrective maintenance activities on the first relay obtained from the warehouse in an
attempt to resolve the problem. Following these corrective maintenance activities, the
relay was successfully tested several times and the Unit 3 Train A EDG was declared
operable at 10:35 a.m. on July 26, 2006.
On September 22, 2006, the Unit 3 Train A EDG failed to develop an output voltage
following a postmaintenance surveillance test. The licensee identified that the same set
Enclosure
5
of auxiliary contacts that previously exhibited erratic behavior during continuity checks
had failed. The licensee identified that the K-1 relay actuating arm for the affected
auxiliary contact module was not providing adequate compression of the auxiliary
contacts. Corrective actions involved mechanical adjustments to the actuating arm to
provide proper auxiliary contact compression. Additional corrective actions included
inspecting, cleaning, and making mechanical adjustments, as necessary, to all other
affected EDG K-1 relays.
3.0
PERFORMANCE DEFICIENCIES RESULTING IN EDG FAILURE
a.
Inspection Scope
On July 25 and September 22, 2006, the Unit 3 Train A EDG failed to produce output
voltage during surveillance testing. The team reviewed the licensees corrective actions
following failure of the Unit 3 Train A EDG on July 25, 2006, to assess their
effectiveness with respect to preventing the subsequent failure that occurred on
September 22, 2006.
b. Observations and Findings
Introduction: The team identified two apparent violations of NRC requirements. The
team identified an apparent violation of 10 CFR Part 50, Appendix B, Criterion V,
Instructions, Procedures, and Drawings, for the failure to establish appropriate
instructions for performing corrective maintenance activities on an EDG K-1 relay. As a
result, following identification that a replacement EDG K-1 relay was unreliable, the
licensee performed ineffective corrective maintenance activities on this relay.
Additionally, the team identified an apparent violation of 10 CFR Part 50, Appendix B,
Criterion XVI, Corrective Action, involving the failure to identify the cause of a
significant condition adverse to quality and take corrective actions to preclude repetition.
Specifically, following identification that a replacement EDG K-1 relay was unreliable,
the licensee failed to identify the cause of this condition and implement adequate
corrective actions. These performance deficiencies resulted in the Unit 3 Train A EDG
being inoperable between September 4 and 22, 2006, and a failure on September 22,
2006. These issues have potential low to moderate safety significance (White).
Description: On July 25, 2006, at 12:53 p.m., the Unit 3 Train A EDG failed to develop
an output voltage during a routine surveillance test. The licensee identified that a faulty
set of auxiliary contacts on a K-1 relay resulted in the generator field being shorted
during the start of the EDG. A new K-1 relay assembly was acquired from the
warehouse and during functional testing the same auxiliary contacts exhibited erratic
operation. The team noted that Work Order 2913004 stated, in part, that the K-1 relay
failed at first. Checked and re-attempted satisfactory. Performed several times
satisfactory. System engineering showed up and after discussing the problem they
wanted to verify repeatability. Checking contact resistance was found to be erratic.
Unable to clean the contacts to get consistent readings. Determined not reliable and
ordered last K-1 relay from the warehouse.
Enclosure
6
Initial attempts, by electrical maintenance personnel, to clean the auxiliary contacts of
the first relay obtained from the warehouse consisted of using a 9-volt battery connected
across the contacts. By cycling the contacts, the licensee stated that an electrical arc
could potentially clean any oxidation affecting the contacts ability to pass current. The
licensee stated this practice was utilized because engineering would not allow intrusive
actions, such as taking the relay apart, to clean the auxiliary contacts because of
concerns with maintaining critical dimensions. The team noted the licensee had no
maintenance instructions applicable to disassembly of the relay and no detailed vendor
specifications describing the critical attributes of the device. The licensee stated that
obtaining this type of information was not possible because the equipment was
obsolete, it was commercially dedicated by a vendor, and the vendor and manufacturer
of the component were no longer in business. Based on these reasons, the licensee
stated that no maintenance activities were performed on the auxiliary contacts and that
when problems were encountered the K-1 relay was replaced as a whole unit.
After initial attempts to clean the K-1 relay auxiliary contacts using the 9-volt battery
failed, the licensee obtained the last replacement K-1 relay available onsite. Again, they
found that the same set of auxiliary contacts on this relay exhibited erratic operation
when cycled. Additionally, the cover to this relay was found in a warped condition. At
this point the licensee decided to perform intrusive cleaning of the auxiliary contacts on
the first device that they determined was unreliable following non-intrusive cleaning
efforts. The team noted that no detailed work instructions were established to perform
this activity to improve its reliable operation. As previously stated, the licensee did not
possess any detailed vendor information specifically related to performing maintenance
activities on this device. After disassembling the auxiliary contacts, cleaning, and
reassembly, the relay was tested several times and the erratic behavior was not
observed during subsequent functional tests of the relay. A postmaintenance
surveillance test of the EDG was performed satisfactorily and the equipment was
declared operable at 10:35 a.m. on July 26, 2006.
On September 22, 2006, the Unit 3 Train A EDG failed to develop an output voltage
following a postmaintenance surveillance test. The licensee identified that the same set
of auxiliary contacts that exhibited erratic behavior on the K-1 relay had failed, resulting
in the generator field being shorted during the start of the EDG. The licensee identified
that the auxiliary contacts were not held closed when the K-1 relay was energized
because of an actuating arm that was not adequately depressing the auxiliary contact
switch. The team noted that this condition most likely existed during the initial testing on
July 25, 2006, and would have contributed to the erratic operation of this contact switch
assembly. Additionally, the team determined that this condition was not identified and
corrected because instructions for performing corrective maintenance activities on the
unreliable K-1 relay were inadequate. The team noted that the last successful start of
the Unit 3 Train A EDG was on September 4, 2006.
As previously stated, the licensee initially believed the erratic behavior resulted from
oxidation of the contact surfaces which required an intrusive maintenance activity to
clean the contact surfaces. The licensee stated that contact oxidation is a common
occurrence requiring cleaning. The team reviewed all work orders associated with
replacement of the K-1 relay dating back to 1984 and noted that none of the work orders
documented erratic auxiliary contact operation because of oxidation. During the review
Enclosure
7
of work orders, the team noted that Work Order 00067739, dated December 11, 1984,
discussed troubleshooting and repair activities for a faulty auxiliary contact on the K-1
relay affecting the Unit 1 Train B EDG. This work order contained instructions to inspect
the auxiliary contact arm on the K-1 relay and make adjustments as needed for proper
contact operation per Technical Manual MO18-390. The maintenance technician
performing the work documented, Adjusted the actuation arm for the auxiliary contacts
on the left side of the K-1 contactor as required. Based on this work order, the team
determined that on this occasion the licensee had worked on the auxiliary contact
operating mechanism to ensure reliable operation. A technician involved in this
maintenance activity did not recall the specifics of the work performed on the relay nor
the use of technical information contained in the technical manual. The team reviewed
the technical manual and found that no pertinent maintenance information for the K-1
relay existed.
Although no pertinent maintenance information for the K-1 relay was identified during
review of Technical Manual MO18-390, the team noted that detailed instructions were
provided to maintenance personnel for ensuring that critical tolerances of other relays
were maintained. The team noted that the voltage controlled overcurrent relay, reverse
power relay, frequency relay, and negative-phase sequence time overcurrent relay, all
associated with the EDG voltage regulating system, contained specific installation,
operation, and maintenance instructions. The team noted that these instructions
provided detailed information for activities involving contact cleaning, contact
adjustments, operational checks, and mechanical adjustments for each particular type
of relay to ensure reliable operation.
The licensee stated that the EDG K-1 relays had a history of operating reliably. Data
collected since 1990 indicated that the EDG K-1 relays had not failed because of
auxiliary contact problems similar to the failures identified in July and September of
2006. The team determined that this reliability data further demonstrated that, when the
erratic relay operation was identified, the licensee should have recognized that
corrective measures were needed that would require appropriate instructions to ensure
future reliable operation.
The team determined that the licensees problem analysis efforts were narrowly
focused, which led them to conclude that the cause of the erratic relay operation was
oxidized contacts. The erratic operation of the K-1 relay provided an indication that
sufficient auxiliary contact continuity existed, at least intermittently, which indicated that
another failure mechanism was contributing to the unreliable K-1 relay operation. If the
licensee performed an adequate cause analysis of this significant condition adverse to
quality, then they may have identified the failure mechanism associated with the
actuating arm not providing adequate contact compression prior to installation of the
new relay on July 26, 2006.
Analysis: NRC Inspection Manual Chapter 0612, Power Reactor Inspection Reports,
defines a performance deficiency as an issue that is the result of a licensee not meeting
a requirement or standard where the cause was reasonably within the licensees ability
to foresee and correct and that should have been prevented. The licensee determined
that the K-1 relay that failed in September of 2006 was unreliable prior to placing it in
service and would require corrective maintenance. The licensee stated that
Enclosure
8
disassembly of the relay to implement intrusive corrective actions had never been
performed because of concerns with maintaining critical dimensions for reliable relay
operation. The licensee did not obtain, nor did they develop, detailed information
specific to performing corrective or preventive maintenance activities for this specific
relay. On the basis of these considerations, the team concluded that the licensees
failure to establish and implement adequate maintenance instructions to resolve the
unreliable K-1 relay condition was a performance deficiency resulting in the Unit 3 Train
A EDG being inoperable between September 4 and 22, 2006. The team determined
that the EDG was inoperable for an 18-day period on the basis that when the EDG was
shut down on September 4, 2006, the K-1 relay auxiliary contacts would have been
positioned and maintained in a state that would have resulted in a subsequent failure of
the relay to operate properly following an EDG start signal. Additionally, the team
determined that the failure to perform an adequate cause assessment of the erratic
relay operation contributed to the inoperability of the Unit 3 Train A EDG.
These findings are greater than minor because they are associated with the equipment
performance cornerstone objective to ensure the availability, reliability, and capability of
systems that respond to initiating events to prevent undesirable consequences. Using
NRC Inspection Manual Chapter 0609, Significance Determination Process, Phase 1
Worksheet, a Phase 2 analysis was required because the findings resulted in the loss of
the safety function of the Unit 3 Train A EDG for greater than the Technical Specification
completion time. The Phase 2 and 3 evaluations preliminarily concluded that the
findings were of low to moderate safety significance. (See Attachment 3 for Phase 2
and Phase 3 details.) The cause of the Criterion XVI finding is related to the
crosscutting element of problem identification and resolution in that the failure to fully
evaluate and implement adequate corrective maintenance actions for the Unit 3 Train A
EDG contributed to the EDG being inoperable for 18 days. Additionally, the cause of
the Criterion V finding is related to the crosscutting element of human performance
associated with resources in that the licensee failed to develop and implement
appropriate work instructions prior to performing corrective maintenance activities on the
subject EDG K-1 relay, which contributed to the EDG being inoperable for 18 days.
Enforcement: 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and
Drawings, states, in part, that activities affecting quality shall be prescribed by
documented instructions, procedures, or drawings of a type appropriate to the
circumstances and shall be accomplished in accordance with these instructions,
procedures, or drawings. Contrary to this, the licensee failed to develop appropriate
instructions or procedures for corrective maintenance activities on the Unit 3 Train A
EDG K-1 relay. This failure resulted in the Unit 3 Train A EDG being inoperable
between September 4 and 22, 2006. This item has been entered into the licensees
corrective action program as Condition Report/Disposition Request (CRDR) 2926830.
Pending determination of safety significance, this finding is identified as an apparent
violation (AV)05000530/2006012-01, Failure to Establish Appropriate Instructions.
10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action, states, in part, that
measures shall be established to assure that conditions adverse to quality, such as
failures, malfunctions, deficiencies, deviations, defective material and equipment, and
nonconformances are promptly identified and corrected and for significant conditions
adverse to quality, measures shall assure that the cause of the condition is determined
Enclosure
9
and corrective action taken to preclude repetition. Contrary to this, the licensee failed to
identify and correct the cause of the erratic EDG K-1 relay operation prior to installation
of the relay on July 26, 2006. This failure resulted in the Unit 3 Train A EDG being
inoperable between September 4 and 22, 2006. This item has been entered into the
licensees corrective action program as CRDR 2926830. Pending determination of
safety significance, this finding is identified as AV 05000530/2006012-02, Failure to
Identify and Correct a Condition Adverse to Quality.
4.0
Failure to Implement the Operability Determination Process
a.
Inspection Scope
The team assessed the engineering and operations departments implementation of the
operability determination (OD) process after identifying potential adverse conditions
involving reliable K-1 relay operation of the EDGs. This assessment was performed
through interviews and a review of precisely logs, ODs, and related documents. In
addition, the team conducted an independent assessment of system operability.
b.
Observations and Findings
Introduction: The team identified two examples of a Green noncited violation of 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, involving the
failure to follow the OD procedure.
Description:
Example One
Administrative Procedure 40DP-9OP26, Operability Determination and Functional
Assessment, Revision 17, Section 1.3, stated, in part, that the OD process is entered
when the ability of a Technical Specification system or component to perform its
specified safety function is called into question by the discovery of a degraded condition.
As previously discussed, on July 25, 2006, the Unit 3 Train A EDG failed to produce
output voltage during surveillance testing. The licensee identified that a faulty set of
auxiliary contacts on a K-1 relay resulted in the generator field being shorted during the
start of the EDG. The team noted that an engineering team was assigned to perform a
root cause analysis for the K-1 relay auxiliary contact failure. This root cause evaluation
was documented in CRDR 2913003. The root cause team determined the most
probable cause was due to contamination on the contact surface either from oxidation or
from pieces of plastic filings found in the contact area. The plastic pieces were believed
to have resulted from a modification of the contact block actuator implemented by
Design Change Package DCP X-PE-007. This design change added auxiliary contacts,
latching mechanisms, and spacers to all EDG K-1 relays at the facility to address
seismic concerns identified during testing of the K-1 relay during the initial plant
construction phase.
Enclosure
10
The NRC inspection team noted that the licensees root cause assessment team
concluded that only the Unit 3 Train A EDG was potentially degraded by this condition.
The evaluation stated, in part, that:
The same model/type field shorting contactor is used on all six Class 1E
EDGs. In addition, two spare K-1 relays removed from the warehouse
exhibited the same symptoms with varying resistance across the auxiliary
contacts. An inspection/test of the auxiliary contacts for all other EDGs
was performed, with no other auxiliary contact resistance problems
identified.
The NRC team was informed that the licensees inspection consisted of a functional
check of the relay and did not involve visually inspecting the auxiliary contact internals to
identify the presence of plastic filings. Based on this, the team determined that the
inspection and contact resistance testing alone failed to demonstrate why only the Unit 3
Train A EDG K-1 relay was affected since the relay had operated properly since being
placed in service on August 1, 2001. Therefore, the licensee inadequately assessed the
extent of condition of the unreliable relay operation relative to the other EDGs at the
facility.
The NRC team noted that the root cause assessment identified a degraded condition,
plastic filings in the contact module that likely affected all the facility EDGs. However,
the licensee failed to enter the OD process after identifying this potentially degraded
condition.
Example Two
As previously discussed, on September 22, 2006, at 1:18 a.m., the Unit 3 Train A EDG
failed to produce output voltage during surveillance testing. The licensee identified that
a faulty set of auxiliary contacts on a K-1 relay resulted in the generator field being
shorted during the start of the EDG. The licensee identified that the K-1 relay actuating
arm for the affected auxiliary contact module was not providing adequate compression
of the auxiliary contacts. The licensee noted that part of the actuating arm consisted of
a metal tab that was bent in a configuration that would result in less contact
compression. The licensee decided to straighten the metal tab, thereby, providing
additional contact compression. Five functional tests of the relay were performed and
the EDG was declared operable following a postmaintenance surveillance test on
September 22, 2006, at 5:48 p.m.
The team noted that CRDR 2926830 documented the licensees evaluation of this
failure. The CRDR stated, in part:
The auxiliary contacts that had failed were added to the K-1 relay by
Design Change Package DCP X-PE-007 during plant startup in response
to electrical seismic latch failures. Engineering believes that the actuator
arm mounted metal tab was initially bent because originally there were no
auxiliary contacts on that side of the K-1 relay. Following completion of
the design change, the auxiliary contacts appeared to be working properly
so the actuator arms were left bent down. Inspection of some of the K-1
Enclosure
11
relays removed from EDGs in the past found at least one that had the
actuator arm straight; however, in most cases, the actuator arm metal tab
for the auxiliary contacts were found bent down. This is a repeat failure
of Unit 3 Train A EDG to produce output voltage. Recent verification of
acceptable K-1 relay auxiliary contact continuity on the other five EDGs
provides the basis that this condition is not present on those relays.
The NRC team was informed that the licensees inspection consisted of a functional
check of the relay and did not involve visually inspecting the auxiliary contact actuating
arms to identify a bent configuration. Based on this, the team determined that the
inspection and contact resistance testing alone failed to demonstrate why only the Unit 3
Train A EDG K-1 relay was affected, since the relay had operated properly since being
placed in service on July 26, 2006. Therefore, the licensee inadequately assessed the
extent of condition of the unreliable relay operation relative to the other EDGs at the
facility.
The team noted that the engineering assessment identified a degraded condition, a bent
K-1 relay actuating arm resulting in unreliable operation, that likely affected all the facility
EDGs. The licensee failed to enter the OD process after identifying this potentially
degraded condition. Following discussions with the licensee, an OD was performed on
September 27, 2006.
In both of these examples the team determined that engineering failed to recognize that
the identified degraded conditions had the potential to adversely affect the other EDGs.
On both occasions engineering relied on continuity checks of the auxiliary contacts to
inappropriately conclude that the other EDGs were not affected. The team noted that
the testing results were pertinent to an OD assessment; however, the information did
not provide adequate justification for not implementing the OD process ensuring
reasonable assurance existed supporting operability of the other EDGs.
Analysis: The performance deficiency was associated with engineering personnel not
adequately implementing the provisions of the OD procedure following the identification
of a degraded condition. This finding was more than minor because the failure to follow
the operability determination process, if left uncorrected, would become a more
significant safety concern in that degraded or nonconforming conditions would not be
properly evaluated. Using the Phase 1 worksheet in Manual Chapter 0609,
Significance Determination Process, the finding was determined to have very low
safety significance because, although these conditions resulted in unreliable K-1 relay
operation, no actual loss of safety function occurred (with respect to the other 5 EDGs)
prior to corrective actions being implemented, and the finding did not represent a
potential risk significant condition due to a seismic, flooding, or severe weather event.
This finding involved problem identification and resolution crosscutting aspects
associated with engineering personnel failing to properly evaluate and perform
operability determinations for identified degraded conditions.
Enforcement: 10 CFR Part 50, Criterion V, Instructions, Procedures, and Drawings,
states, in part, that activities affecting quality shall be prescribed by documented
instructions, procedures, or drawings of a type appropriate to the circumstances and
shall be accomplished in accordance with these instructions, procedures, or drawings.
Enclosure
12
Administrative Procedure 40DP-9OP26, Operability Determination and Functional
Assessment, Revision 17, Section 1.3, stated, in part, that the OD process is entered
when the ability of a Technical Specification system or component to perform its
specified safety function is called into question by the discovery of a degraded condition.
Contrary to the above, on two occasions the licensee failed to enter the OD process
when the ability of a Technical Specification system or component safety function was
called into question. On July 25, 2006, following failure of the Unit 3 Train A EDG, an
OD was not performed after identifying the failure was likely because of plastic filings
affecting proper auxiliary contact operation of a K-1 relay. The filings were believed to
be from a modification that was performed on all EDG K-1 relays during initial plant
startup. On September 22, 2006, following another failure of the Unit 3 Train A EDG, an
OD was not performed after identifying the failure was the result of the K-1 relay
actuating arm not providing adequate compression of the auxiliary contacts. The
licensee determined this condition most likely resulted from a modification performed on
all EDG K-1 relays during initial plant startup. Because the finding is of very low safety
significance and has been entered into the licensees corrective action program as
CRDRs 2928389 and 2940558, this violation is being treated as a noncited violation
consistent with Section VI.A of the Enforcement Policy: Noncited Violation
05000528;05000529;05000530/2006012-03, Failure to Implement the Operability
Determination Process.
5.0
CORRECTIVE ACTIONS FOLLOWING EDG FAILURES
a.
Inspection Scope
The team assessed the licensees immediate and long-term planned corrective actions
associated with the Unit 3 Train A EDG failures that occurred on July 25 and
September 22, 2006. This assessment was performed through interviews, review of
operator logs, corrective action documents, work orders, and related documents.
b.
Observations and Findings
Following the Unit 3 Train A EDG failure on July 25, 2006, the licensee identified that
plastic filings inside the auxiliary contact module may have resulted in the failure. The
licensee concluded this material most likely was introduced during a design change
performed on all the K-1 relays and subsequently established a schedule to inspect all
the EDG auxiliary contact modules. The team noted these inspections were scheduled
to be performed November 2006 through March 2007. NRC Inspection Manual,
Part 9900, Technical Guidance, Operability Determination and Functionality
Assessments for Resolution of Degraded or Nonconforming Conditions Adverse to
Quality or Safety, Section 4.2, states, When a potential degraded or nonconforming
condition is identified, the licensee should take action without delay to confirm if a
system, structure, or component is degraded or nonconforming. The team concluded
that waiting approximately 8 months to identify whether other EDGs were affected by
this potential adverse condition was not commensurate with the safety consequences of
having a degraded EDG.
Enclosure
13
Following the September 22, 2006, Unit 3 Train A EDG failure, the licensee identified
that a bent K-1 relay actuating arm resulted in unreliable auxiliary contact operation.
Immediate corrective actions involved straightening the arm to provide additional contact
compression for the Unit 3 Train A EDG. The team noted that the licensee implemented
timely corrective actions to inspect and implement mechanical adjustments, as needed,
to all the EDG relays to ensure adequate contact compression during operation. These
actions were implemented September 27-30, 2006, and incorporated the inspections
resulting from the July failure that were originally not scheduled to be completed until
March 2007. The licensee straightened bent K-1 relay contactor arms for the Unit 2
Trains A and B EDGs, and the Unit 1 Train B EDG. The team determined that these
actions were timely and they included the inspections identified following the July failure.
Therefore, there were no regulatory findings associated with timeliness of these
corrective actions.
The team noted that long-term planned corrective actions consisted of replacing all of
the EDG automatic voltage regulators, including replacement of the K-1 relays, with a
different design. The licensee stated that they plan to have these replacement activities
accomplished during the next refueling outage for each unit.
6.0
Generic Implications
The team reviewed various NRC generic communications and operating experience
from other licensees relevant to the EDG relay failures identified at the Palo Verde
Nuclear Generating Station. No relevant similar relay failures were identified. Both the
NRC and the licensee concluded that the relay problems pertaining to ensuring
adequate contact compression is provided by the actuator arm was potentially of
generic concern. On October 21, 2006, the licensee submitted voluntary Licensee
Event Report (LER) 50-530/2006-006-00 to report this concern.
4OA3 Event Follow-up (71153)
.1
(Closed) LER 05000530/2006-006-00, Voluntary LER for Failure of Emergency
Diesel Generator to Attain Required Voltage Due to Relay Contactor
On September 22, 2006, at 1:18 a.m., the Unit 3 Train A EDG failed to produce
output voltage during surveillance testing. The licensee identified that a faulty
set of auxiliary contacts on a K-1 relay resulted in the generator field being
shorted during the start of the EDG. The licensee identified that the K-1 relay
actuating arm for the affected auxiliary contact module was not providing
adequate compression of the auxiliary contacts. The licensee noted that part of
the actuating arm consisted of a metal tab that was bent in a configuration that
would result in less contact compression. Immediate corrective actions involved
mechanical adjustments made to the actuating arm providing additional contact
compression for the Unit 3 Train A EDG. Additionally, the licensee implemented
corrective actions to inspect and make adjustments as needed to all the EDG
relays. As discussed in section 3.0 of this report, the Unit 3 Train A EDG failure
on September 22, 2006, resulted from and inadequate cause assessment and
the failure to establish appropriate corrective maintenance instructions which
Enclosure
14
resulted in a violation of Technical specification 3.8.1.B since the inoperable
EDG exceeded the completion time of 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />. The team determined that the
licensee failed to identify that 10 CFR 50.73(a)(2)(B) requires the licensee to
report any operation or condition which was prohibited by the plants Technical
Specifications. Based on the licensee having performed a voluntary LER
addressing both Unit 3 Train A EDG failures that occurred on July 25, 2006, and
September 22, 2006, the failure to make a required report in accordance with
10 CFR 50.73 constitutes a violation of minor significance that is not subject to
enforcement action in accordance with Section IV of the NRCs Enforcement.
This LER is closed.
4OA6 Meetings, Including Exit
On November 30, 2006, the inspection results were discussed with Mr. David Mauldin,
Vice President, Engineering, and other members of the plant staff. The inspectors
asked the licensee whether any of the material examined during the inspection should
be considered proprietary. No proprietary information was identified.
ATTACHMENT 1: SUPPLEMENTAL INFORMATION
ATTACHMENT 2: SPECIAL INSPECTION CHARTER
ATTACHMENT 3: SIGNIFICANCE DETERMINATION EVALUATION
Attachment 1
A1-1
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
S. Bauer, Department Leader, Regulatory Affairs
P. Borchert, Director, Operations
L. Bullington, Probabilistic Risk Analysis
C. Churchman, Director, Engineering
C. Cooper, Acting Electrical Maintenance Engineering Section Leader
E. Dutton, Nuclear Assurance Department
C. Eubanks, Vice President, Nuclear Operations
M. Green, LAN Department
J. Holmes, Section Leader
R. Henry, Site Representative
D. Leech, Department Leader, Corrective Action Program
J. Levine, Executive Vice President, Generation
C. Marschall, Consultant
D. Mauldin, Vice President, Engineering
L. Nguyen, LA Power and Water
S. Peace, Owner Services Consultant
M. Perito, Plant Manager
J. Proctor, Section Leader, Regulatory Affairs - Compliance
C. Seaman, General Manager, Regulatory Affairs and Performance Improvement
D. Steen, Senior Engineer
T. Radtke, General Manager
B. Ramey, Maintenance Engineering Department Leader
R, Schwartsbeck, Enercon Services
E. Shouse, Site Representative
G. Sowers, Section Leader, Probabilistic Risk Analysis
D. Straka, Senior Consultant, Regulatory Affairs
D. Vogt, Section Leader, Operations/Shift Technical Advisor
D. Withers, Maintenance Engineering
NRC Personnel
T. Vegel, Deputy Director, Division of Reactor Projects
G. Warnick, Senior Resident Inspector, Palo Verde Nuclear Generating Station
Attachment 1
A1-2
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Opened
Failure to Establish Appropriate Instructions05000530/2006012-02
Failure to Identify and Correct aCondition Adverse
to Quality
05000528;05000529;05000530/2006012-03
Failure to Implement the Operability Determination
Process
Closed
05000528; 05000529;05000530/2006012-03
Failure to Implement the Operability Determination
Process
05000530/2006-006-00
LER
Voluntary LER for Failure of Emergency Diesel
Generator to Attain Required Voltage Due to Relay
LIST OF DOCUMENTS REVIEWED
Drawings
13-M018-00159
C72-15000-100
D72 12200 750 Rev. E
D72-12200-710 Rev. E
D72-12200-100 Rev. B
Miscellaneous
1986 Document Specification 13-MM-0018
CES 0391-45C Seismic Test Program on Unit and Diesel Generator Control Panel
Emergency Diesel Generator Pre-operational Test
Surveillance Logs
IEEE 387 (1972 Version)
Design Change Package DCP 2SE-PE-007
Design Change Package DCP 10E-PE-007
M018-00425 Seismic Qualification Report on Triaxial Shake Table Tests of Two K-1 Relays
M018-00367 Report of Witness Tests of KSV-20-T Engine-Generator Set
Miscellaneous Documents Containing Pre-operational Test Data
Miscellaneous Control Room Log Entries for K-1 Relay Failures
Plant Change Request 86-13-PE-002
Purchasing Order 10470-13-MM-018 Documentation
Purchasing Order 33501265 Documentation
Attachment 1
A1-3
Purchasing Order 44930161 Documentation
Purchasing Order 60113782 Documentation
Portec Inc. Instruction Manual for the Static Voltage Regulator System
SFR 1PE-018
TN-E-3489
Startup Work Authorizations
SWA 15805, 1 SWA 15532, 1 SWA 15795, 1 SWA 17469, 1 SWA 16047, 2 SWA 15532, 2
SWA 15795, 2 SWA 16047, 3 SWA 15532, 3 SWA 15795, U1-SWA 15188, and U1-SWA
19219
Engineering Evaluation Requests
EER 85-PE-007
EER 85-PE-008
EER 84-PE-002
Procedures
CRAIs
2829958, 2829959, 2829960, 2829961, 2829964, 2829965, 2829967, 2829971, 2829973, and
2829974
CRDRs
43930, 51630, 51743, 53295, 53788, 55546, 58135, 59433, 90278, 115952, 160332, 2361791,
2405054, 2410347, 2432009, 2532225, 2570582, 2579229, 2582956, 2641676, 2645588,
2650009, 2660221, 2752631, 2759704, 2784750, 2847506, 2872154, 2906158, 2913003,
2926830, and 2927262
Work Orders
00003495, 67521, 67550, 119715, 360726, 360944, 412917, 1040142, 1071966, 1329487,
2361781, 2410350, 2750447, 2794362, 2902642, 2913004, 2913286, 2913287, 2913295,
2913306, 2913753, 2919666, 2919670, 2919671, 2919672, 2919673, 2919747, and 2926829
Attachment 2
A2-1
SPECIAL INSPECTION CHARTER
September 29, 2006
MEMORANDUM TO: Michael Hay, Senior Project Engineer,
Project Branch D, Division of Reactor Projects (DRP)
Dr. Scott Rutenkroger, Reactor Inspector,
Engineering Branch 1, Division of Reactor Safety
Michael Bloodgood, Reactor Engineer, Nuclear Safety Professional
Development Program, Project Branch D, DRP
FROM:
Arthur T. Howell III, Director, DRP /RA/ AVegel for
SUBJECT:
SPECIAL INSPECTION CHARTER TO EVALUATE THE PALO VERDE
NUCLEAR GENERATING STATION UNIT 3 EMERGENCY DIESEL
GENERATOR FAILURE
A Special Inspection Team is being chartered in response to the Palo Verde Nuclear
Generating Station Unit 3 Emergency Diesel Generator (EDG) failure. The diesel failed to
develop an output voltage when started for a surveillance test. The licensee determined that a
modification to the field flashing relay caused the failure. You are hereby designated as the
Special Inspection Team members. Mr. Hay is designated as the team leader. The assigned
SRA to support the team is Mike Runyan.
A.
Basis
On July 25, 2006, Unit 3, Train A, EDG failed to develop output voltage during a
surveillance test. The licensee's root cause determined plastic debris potentially
prevented auxiliary contacts from properly functioning resulting in shorting out of the
generator field during startup preventing a proper field flash. Two replacement relays
obtained from the licensee warehouse exhibited the same unreliable condition. After
performing corrective maintenance activities on one of the relays, the diesel was
subsequently tested and declared operable on July 26, 2006.
On September 22, 2006, Unit 3, Train A, EDG failed to develop output voltage during a
surveillance test. The licensee determined that the same auxiliary contact which failed
in July 2006 was faulty. The licensee identified that this failure was attributed to a bent
metal actuator arm that is used to actuate the auxiliary contacts associated with the field
shorting circuit. Additionally, the licensee determined this bent metal actuator arm
potentially exits in all six EDG's at the facility. Based on previous failures it appears this
bent arm is the underlying root cause for the field shorting auxiliary contacts failure to
operate reliably, and this condition may affect all operating EDG's at the facility.
Attachment 2
A2-2
This Special Inspection Team is chartered to review the generic impact of the relays
bent arms on the other Palo Verde Emergency Diesel Generators as well as any
potential impact on other nuclear plants. The team is also to review the design change
method and reviews that the licensee used when making the relay modifications. The
team will also review the licensees operability determination and corrective action
program for determining the root cause and correction of the diesels failure.
B.
Scope
The team is expected to address the following:
1.
Develop a complete scope of the failures of all Palo Verde Emergency Diesel
Generators to develop an output voltage.
2.
Review the extent of condition determination for this condition (current and prior
K1 relay failures) and whether the licensees actions are comprehensive. This
should include potential for other diesel failures.
3.
Review the licensees determination of the cause of any design deficiencies.
Independently verify key assumptions and facts. If available, determine if the
licensees current and prior root cause analyses and corrective actions have
addressed the extent of condition for problems with the emergency diesel
generators K1 relays.
4.
Determine if the Technical Specifications were met when the diesel failed.
5.
Review and assess the corrective actions for current and past similar failures.
6.
Review the licensees EDG operability determination to evaluate the emergency
diesel generators operability.
7.
Collect data as necessary to support a risk analysis.
8.
Determine if this issue has generic implications to other nuclear facilities.
C.
Guidance
Inspection Procedure 93812, Special Inspection, provides additional guidance to be
used by the Special Inspection Team. Your duties will be as described in Inspection
Procedure 93812. The inspection should emphasize fact-finding in its review of the
circumstances surrounding the event. It is not the responsibility of the team to examine
the regulatory process. Safety concerns identified that are not directly related to the
event should be reported to the Region IV office for appropriate action.
The Team will report to the site, conduct an entrance, and begin inspection no later than
October 4, 2006. While on site, you will provide daily status briefings to Region IV
management, who will coordinate with the Office of Nuclear Reactor Regulation, to
ensure that all other parties are kept informed. A report documenting the results of the
inspection should be issued within 30 days of the completion of the inspection.
Attachment 2
A2-3
This Charter may be modified should the team develop significant new information that
warrants review. Should you have any questions concerning this Charter, contact me at
(817) 860-8248.
Attachment 3
A3-1
SIGNIFICANCE DETERMINATION EVALUATION
Significance determination process Phase 1:
In accordance with NRC Inspection Manual Chapter 0609, Appendix A, Significance
Determination of Reactor Inspection Findings for At-Power Situations, the inspectors
conducted a significance determination Phase 1 screening and determined that the
finding resulted in loss of the safety function of the Train A emergency diesel generator
for greater than the Technical Specification allowed outage time. Therefore, a
Significance Determination Process Phase 2 evaluation was required.
Significance determination process Phase 2:
The Risk-Informed Inspection Notebook for Palo Verde Nuclear Generating Station,
Unit 3, Revision 1, September 2, 2003, was utilized for the Phase 2 evaluation of the
inoperable Train A emergency diesel generator. The following steps and the associated
findings are listed below:
Select or define the applicable initiating event scenarios:
Table 2, Initiators and System Dependency for Palo Verde Nuclear Generating
Station, Units 1, 2, and 3, was reviewed to determine that the loss of offsite
power (LOOP) initiating event scenario was the only scenario that needed to be
analyzed due to the failure of the Train A EDG.
Estimate the likelihood of scenario initiating events and conditions:
The performance deficiency was assumed to exist for 58 days. The Phase 2
analysis assumed the EDG was nonfunctional for an 18-day period, representing
the period from its last successful start on September 4, 2006, to its failed
surveillance on September 22, 2006. Using Table 1, Categories of Initiating
Events for Palo Verde Nuclear Generating Station Unit 3, the initiating event
likelihood for loss of offsite power was determined to be valued at 3. Additional
risk was accrued during the first 40 days of exposure because of a higher
likelihood of failure of the K-1 relay.
Estimate the remaining mitigation capability:
Using the SDP worksheet for a LOOP (Table 3.7, SDP Worksheet for Palo
Verde Nuclear Generating Station, Units 1, 2, and 3 - Loss of Offsite Power
(LOOP)), Sequences 1, 2, 3, 4, and 5, the following results were assigned for
each:
Sequence 1: LOOP-AFW - 6
Sequence 2: LOOP-EAC-REC3 - 8
Sequence 3: LOOP-EAC-TDAFW-REC1 - 8
Sequence 4: LOOP-EAC-SEAL-HPSI - 12
Sequence 5: LOOP-EAC-SEAL-REC1 - 10
Attachment 3
A3-2
Estimate the risk significance of the inspection finding:
NRC Inspection Manual Chapter 0609, Significance Determination Process,
Appendix A, Attachment 1, Counting Rule Worksheet, was utilized using one
sequence that resulted in a value of 6. Since step 10 was greater than zero, the
risk significance of the inspection finding was determined to be at low to
moderate safety significance (White).
Phase 3 Analysis
Description of Performance Deficiencies
On July 25, 2006, the Unit 3 Train A EDG failed to start because of a failed K1 relay.
The last two spare relays obtained from the warehouse were identified to operate
unreliabley. The licensee performed corrective maintenance on one of the relays and
following installation it failed on September 22, 2006. Two performance deficiencies
were identified that contributed to the September 22, 2006, failure. The first
performance deficiency involved the failure to establish appropriate instructions for
performing corrective maintenance activities on an emergency diesel generator K-1
relay. The second performance deficiency involved the failure to identify and correct the
cause of the erratic EDG K-1 relay operation prior to installation of the relay on July 26,
2006.
Assumptions
1.
The Unit 3 Train A EDG demand record is shown in the following table. (A failed bench
test of the new K1 relay was not credited because it may have been due to debris
buildup resulting from a long period of warehousing.)
Date
Demand Result
K-1 Relay
7/25/06
Fail to Start
Old
7/26/06
Successful Start
New
8/07/06
Successful Start
New
8/24/06
Successful Start
New
9/04/06
Successful Start
New
9/22/06
Fail to Start
New
The data was analyzed using three alternate assumptions as follows:
Attachment 3
A3-3
Deterministic Assumption
It is assumed that the as-left condition of the EDG following a run either
predisposes it to a guaranteed success or failure on the next demand. This
assumes that the success or failure on the next attempt is a function of the
as-left condition of the relay following a load run.
Based on this assumption, the Train A EDG was guaranteed to fail to start on
any demand subsequent to the last successful run on September 4, 2006, but
was likewise guaranteed to succeed on any demand prior to this date.
Therefore, the EDG is assumed to be a failed state for 18 days.
Stochastic Assumption
It is assumed that the EDG success or failure is a probabilistic event on any
given demand. This assumes that the relay is more or less in the same state
following each run, but that the as-left tolerances are so close to critical that the
chance of success or failure on the next demand is purely a probabilistic event.
Using this assumption, the Train A EDG was vulnerable to failure on any
demand following the installation of the new K1 relay on July 26, 2006. The
successful start on July 26, 2006, and prior bench/installed test successes and
failures were excluded from the data set because of possible preconditioning
effects. Therefore, three demands and one failure were left. The resulting
assumption is that the Train A EDG would have a 0.25 chance of failing to start
from a K1 relay failure in response to any demand during the 58-day period
between July 26, 2006, and September 22, 2006.
Combination Assumption
It is assumed that a stochastic mechanism existed for the first 40 days, but after
the final successful start on September 4, 2006, the EDG was guaranteed to fail
on its next demand and, therefore, was in a failed state for the final 18 days of
the exposure period.
Using this assumption, the Train A EDG had a 0.25 probability of failing to start
because of a failed K1 relay for the first 40 days and a 1.0 probability of failure
for the final 18 days of the exposure period.
2.
The analyst discarded any risk that may have accrued from September 22, 2005, to July
25, 2006, (the balance of a one-year period) because of a lack of failure history during
this period (as confirmed by a review of surveillance test results).
3.
The analyst modified the current Palo Verde SPAR model (Revision 3.21, October 28,
2005) to reflect the plant-specific LOOP frequencies listed in NUREG/CR-6890,
Reevaluation of Station Blackout Risk at Nuclear Power Plants, Volume 1, Table D-1.
This study comprised an update based on an analysis of offsite power events during
1986-2004. This change to the Palo Verde SPAR model for this specific analysis was
Attachment 3
A3-4
endorsed by a representative of the Idaho National Laboratory (INL), the NRCs
contractor for SPAR model development.
Category of LOOP
Current SPAR Value
Revised Value Based on
Plant Centered
2.07E-3/yr.
2.01E-3/yr.
Switchyard Centered
1.04E-2/yr.
9.01E-3/yr.
Grid Related
1.86E-2/yr.
4.40E-2/yr.
Weather Related
4.83E-3/yr.
3.83E-3/yr.
4.
It is assumed that the Unit 3 Train B EDG was not vulnerable to fail from the same
cause as the Train A EDG failure for the entire 58-day exposure period. This is based
on the fact that the relay actuator arm in the K1 relay for this diesel generator was
confirmed to be in a field-straightened configuration during this period. Therefore,
common cause was not invoked in the analysis and a failure probability of 1.0 was used
in lieu of TRUE. This is a key assumption with respect to the risk estimate because
common cause, if presumed, would result in a large increase in the value of the
fail-to-start common cause basic event.
5.
No specific recovery of the Train A EDG was assumed, and no changes were made to
the EDG recovery values in the SPAR model. That is, for this analysis, the analyst
assumed that the recovery probability of the EDGs was nominal.
6.
The SPAR model includes cross-connection capabilities from the other units' diesel
generators. However, the cutsets that include these basic events are very insignificant
in the analysis. Therefore, the analyst did not adjust Unit 1 and 2 EDG common cause
probabilities from the base case.
Internal Events Analysis
The Palo Verde SPAR model (Revision 3.21, October 28, 2005), modified as described
above, was used at a truncation of E-12.
Deterministic Assumption
Based on this assumption, the Train A EDG was guaranteed to fail to start on any
demand subsequent to the last successful run on September 4, 2006, but was likewise
guaranteed to succeed on any demand prior to this date. Therefore, the EDG was
assumed to be in a failed state for 18 days. The use of a "T/2" assumption is not valid
in this case because the normally open contact is assumed to be open following the
previous EDG run. This would cause the K1 unlatch coil to fail to energize for any
hypothetical demand during this period and result in failure of the EDG field flash.
Attachment 3
A3-5
In the SPAR model, the following changes were made:
EPS-DGN-FS-DGA was set to 1.0
The result in SAPHIRE is 1.047E-4/yr. A review of the cutsets revealed that several
included independent failures of Train A EDG to run as well as the test and maintenance
basic event. These cutsets were logically inconsistent:
Cutsets containing:
Value
EPS-DGN-FR-DGA
2.024E-6/yr.
EPS-DGN-TM-DGA
4.084E-7/yr.
Total
2.432E-6/yr.
Extracting these cutsets leaves a result of 1.047E-4 - 2.432E-6 = 1.023E-4/yr.
Therefore, for an 18-day exposure period, the delta-CDF of the finding is
1.023E-4yr.(18/365) = 5.0E-6/yr.
Common cause events were retained in the evaluation case in order to retain the entire
probability of failure of the Train B EDG.
Stochastic Assumption
Using this assumption, the Train A EDG was vulnerable to failure on any demand
following the installation of the new K1 relay on July 26, 2006. As discussed above, the
Train A EDG had a 0.25 probability of failing to start from a K1 relay failure in response
to any demand during the 58-day period of time between July 26, 2006, and
September 22, 2006.
In SPAR, the following changes were made:
EPS-DGN-FS-DGA was set to 0.25 + 5E-3 (base case probability) = 0.255
The result in SAPHIRE is 2.447E-5. For the 58-day exposure period, the resulting
delta-CDF is 2.447E-5 (58/365) = 3.9E-6/yr.
Combination Assumption
Using this assumption, the Train A EDG had a 0.25 probability of failing to start because
of a failed K1 relay for the first 40 days and a 1.0 probability of the same failure for the
final 18 days of the exposure period.
The result from the deterministic assumption is the same in this case. The exposure
time for the stochastic portion is set at 40 instead of 58 days. Using the results above,
the delta-CDF of the combination assumption is 5.0E-6/yr. + 40/58 (3.9.E-6/yr.) =
7.7E-6/yr.
Attachment 3
A3-6
External Events
Seismic
Palo Verde is located in a relatively stable seismic region for a plant located in the
Western USA. The Idaho National Laboratory conducted a study to predict the
frequency of a LOOP resulting from a seismic event at all US plants, as documented in
Frequency of Seismically-Induced LOOP events for SPAR models, Revision 1,
September 2005. The conclusion of this study was that the mean frequency for a
seismically-induced LOOP at Palo Verde is 5.37E-5/yr. The study concluded that the
failure of ceramic insulators would be the most likely failure mode inducing a LOOP.
For risk assessment purposes, a seismically-induced LOOP would have a recovery
profile similar to a severe weather event-induced LOOP. In SPAR (as modified above
for this analysis), the frequency of a weather-related LOOP (used for the internal events
assessment) is 3.83E-3/year. Therefore, the increase in LOOP frequency from seismic
events is not significant by itself as it relates to the risk of this finding.
The analyst also considered the possibility that an earthquake that results in a LOOP
could damage equipment (apart from the diesel generators) that could add non-
negligible risk to the finding. To address this issue, INL produced a document entitled,
Seismic Event Modeling and Seismic Risk Assessment Handbook, Revision 1,
September 2005. In particular, a LOOP resulting from an earthquake that also involved
loss of risk-significant equipment in Train B and/or loss of the gas turbine generators
could result in significant risk despite the low frequency of seismically-induced LOOPs.
Within this document, Table B-1, Generic SSC Seismic Fragilities, provides a list of
components along with the median g-force required to damage them. The following
table lists examples of the equipment of concern and the frequency of earthquakes at
Palo Verde that exceed the threshold value:
Component
High Confidence Low
Probability of Failure
Capacity (g)
Frequency of >g Earthquake
at Palo Verde
Electrical Equipment
(function during seismic
event)
0.34
1.0E-5/yr
Electrical Equipment
(function after seismic event)
0.77
1.0E-8/yr.
Battery Chargers/Inverters
0.54
1.0E-6/yr.
Batteries/Battery Racks
1.3
<9.3E-10/yr.
Diesel Generator/Support
Systems
1.06
<9.3E-10/yr.
Turbine-driven pumps
0.85
1.0E-9/yr.
Attachment 3
A3-7
Equipment success at g-forces well above the HCLPF value is possible. Based on
review of the information provided above and other information in the INL document, the
analyst concluded that earthquakes causing LOOPs and loss of other on-site equipment
would add risk small in comparison to the internal events result.
The analyst assumed (conservatively) that the gas turbine generators would be lost in a
seismic event that also causes a LOOP. To calculate the risk of the finding in light of
this assumption, the analyst ran two cases using the revised SPAR model. In both runs,
the frequency of LOOPs was set at 5.37E-5/year (frequency of seismic-induced
LOOPs), and nonrecoveries of offsite power for all relevant times (3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> and less as
well as 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />) were set to TRUE (there are no offsite recovery events within the E-12
truncation greater than 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> in the Palo Verde SPAR model except for 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />;
therefore, this change set is equivalent to assuming that offsite power following an
earthquake is not recovered). Both gas turbine generator fail-to-start events were set to
TRUE and only LOOP sequences were quantified. In the first case, the EDGs are
assumed to be nominally reliable and available. This result was 1.362E-7/yr. In the
second case, EDG A is assigned a failure probability of 1.0 for the fail-to-start event.
The result was 2.747E-6/yr. The resulting CDF of the finding attributable to a seismic
event that debilitates the gas turbine generators is therefore (2.747E-6 -
1.362E-7)(18/365) = 1.3E-7/yr.
Based on disussions, the licensees PRA assigns a value of approximately 4E-7/yr
delta-CDF for seismic events for a 18-day exposure period.
Fire
The analyst reviewed the licensee's IPEEE for Severe Accident Vulnerabilities,
June 30, 1995, to determine the risk attributable to the finding resulting from internal
fires.
A fire in Room 5B (Train B ESF switchgear room) was considered to have the largest
potential risk to the finding. A fire in this room would possibly cause a loss of offsite
power to both ESF buses. In this case, with a failure of EDG A to start, a station
blackout situation would exist. In the IPEEE, the CDF result for a fire in the Train B ESF
switchgear room was 9.73E-6/yr. The fire ignition frequency for this room was
5.5E-3/yr.
The analyst determined that the IPEEE did not contain sufficient information to quantify
the risk attributable to fires pertaining to the performance deficiency. Therefore, the
analyst requested that the licensee use its fire PRA model for this purpose. The licensee
reported that their fire PRA calculates a delta-CDF of 3.716E-6/yr for EDG A being
nonfunctional versus the base case condition. For this case, the EDG failure was
assumed to be independent in nature, the same assumption used in the SPAR analysis.
For an 18-day exposure, this would result in a delta-CDF of 1.8E-7/yr.
Attachment 3
A3-8
As a comparison, the following table shows the differences in the treatment of Room 5B
between the IPEEE and the fire PRA.
Room 5B
Fire PRA
Fire Ignition Frequency
5.5E-3/yr.
4.16E-3/yr.
9.73E-6/yr.
1.17E-7/yr.
This example shows that the CCDP of a fire in this room decreased from 1.8E-3 in the
IPEEE to 2.8E-5 in the fire PRA. This difference is not surprising because the IPEEE
was basically a screening tool that assumed worst-case bounding conditions while the
fire PRA incorporated realistic, best-estimate approximations.
Internal Flooding/High Velocity Winds/Other External Events
The analyst concluded qualitatively that no other external events would add appreciably
to the risk of the finding. The licensee reported that the risk added from internal flooding
according to their model was in the E-8 range.
Combined Risk
Using the licensee analysis results for seismic and fire events, the following table
indicates the total estimated risk of the finding for each of the assumed failure
mechanisms:
Assumption
Internal
Events
Seismic
Fire
Total Risk
Deterministic
5.0E-6
4E-7/yr
1.3E-7/yr
5.5E-6/yr
Stochastic
3.9E-6
3E-7/yr1
1.0E-7/yr1
4.3E-6/yr
Combination
7.7E-6
6E-7/yr1
2.1E-7/yr1
8.5E-6/yr
1. Seismic and fire CDFs were adjusted for the stochastic and combination
assumptions.
In accordance with IMC 0609, Appendix H, station blackout sequences, which
predominate the risk of the assessed condition, are not considered significant release
events for a large, dry containment. Therefore, large early release was considered
unimportant in this analysis.
Licensee Analysis
The analyst did not receive a detailed description of the licensees analysis, but was
informed verbally that the delta-CDF of the finding for internal events and fire, assuming
an 18-day exposure and no recovery of the Train A EDG, was approximately 1.6E-6/yr.
Attachment 3
A3-9
Adding the licensees approximate seismic risk, the overall result would be
approximately 2.0E-6/yr.
References
Palo Verde SPAR model (Revision 3.21, October 28, 2005)
NUREG/CR-6890, Reevaluation of Station Blackout Risk at Nuclear Power Plants,
Volume 1, Table D-1
Seismic Event Modeling and Seismic Risk Assessment Handbook, Revision 1,
September 2005
Palo Verde IPEEE for Severe Accident Vulnerabilities, June 30, 1995
Palo Verde Fire PRA Overview and Results, 13-NS-C072