ML063400561

IR 05000528-06-012; 05000529-06-012; 05000530-06-012; 10/02/2006 - 11/09/2006; Palo Verde Nuclear Generating Station, Units 1, 2, and 3: Special Inspection in Response to Unit 3 Train a EDG Failures on July 25 and September 22, 2006

ML063400561
Person / Time
Site:	Palo Verde
Issue date:	12/06/2006
From:	Howell A NRC/RGN-IV/DRP
To:	James M. Levine Arizona Public Service Co
References
EA-06-296 IR-06-012
Download: ML063400561 (36)

See also: IR 05000528/2006012

Text

December 6, 2006

EA-06-296

James M. Levine, Executive

Vice President, Generation

Mail Station 7602

Arizona Public Service Company

P.O. Box 52034

Phoenix, AZ 85072-2034

SUBJECT: PALO VERDE NUCLEAR GENERATING STATION, UNITS 1, 2, and 3 - NRC

SPECIAL INSPECTION REPORT 05000528/2006012; 05000529/2006012;

05000530/2006012

Dear Mr. Levine:

On November 30, 2006, the U.S. Nuclear Regulatory Commission (NRC) completed a special

inspection at your Palo Verde Nuclear Generating Station, Units 1, 2, and 3. This inspection

examined activities associated with the Unit 3 Train A emergency diesel generator (EDG)

failures that occurred on July 25 and September 22, 2006. On both occasions the EDG failed

to produce an output voltage during testing. The NRC's initial evaluation satisfied the criteria in

NRC Management Directive 8.3, NRC Incident Investigation Program, for conducting a special

inspection. The basis for initiating this special inspection is further discussed in the Charter,

which is included as Attachment 2 to this report. The determination that the inspection would

be conducted was made by the NRC on September 29, 2006, and the inspection started on

October 2, 2006.

The enclosed special inspection report documents the inspection findings which were discussed

on November 9, 2006, with you, and other members of your staff, and on November 30, 2006,

with Mr. David Mauldin, Vice President, Engineering, and other members of your staff.

The inspection examined activities conducted under your license as they relate to safety and

compliance with the Commission's rules and regulations and with the conditions of your license.

The inspectors reviewed selected procedures and records, observed activities, and interviewed

personnel.

The attached report discusses two findings that appear to have low to moderate safety

significance (White). As described in Section 3.0 of this report, the NRC concluded that the

failure to establish appropriate instructions for performing corrective maintenance activities on a

K-1 relay resulted in the Unit 3 Train A EDG being inoperable between September 4 and 22,

2006. Additionally, the failure to identify and correct the cause of the erratic EDG K-1 relay

operation prior to installation of the relay on July 26, 2006, was identified as another

performance deficiency that contributed to the Unit 3 Train A EDG being inoperable for a period

Arizona Public Service Company

-2-

greater than the Technical Specification completion time. The safety significance of these

findings was assessed on the basis of the best available information, including influential

assumptions, using the applicable Significance Determination Process and were preliminarily

determined to be White (i.e., low to moderate safety significance) findings. Preliminarily, these

findings have a low to moderate safety significance when assuming a loss of offsite power

initiating event and the Unit 3 Train A EDG being in an unreliable condition for approximately

40 days and a nonfunctional condition for approximately 18 days. Attachment 3 of this report

provides a detailed description of the preliminary risk assessment. In accordance with NRC

Inspection Manual Chapter (IMC) 0609, Significance Determination Process, we intend to

complete our evaluation using the best available information and issue our final determination

of safety significance within 90 days of this letter.

These findings do not represent an immediate safety concern because of the corrective actions

you have taken. These actions involved inspecting, cleaning, and implementing mechanical

adjustments, as appropriate, to the operating mechanism of the EDG K-1 relays.

Also, these findings constitute apparent violations of NRC requirements and are being

considered for escalated enforcement action in accordance with the NRC Enforcement Policy.

The current Enforcement Policy is included on the NRCs web site at

http://www.nrc.gov/reading-rm/adams.html.

Before we make a final decision on this matter, we are providing you an opportunity to present

to the NRC your perspectives on the facts and assumptions, used by the NRC to arrive at the

findings and their significance, at a Regulatory Conference or in writing. As discussed during a

telephone call with Mr. Scott Bauer, Department Leader, Regulatory Affairs, we understand that

it is your intent to discuss your perspectives during a Regulatory Conference. Accordingly, a

Regulatory Conference is scheduled to be conducted in the NRC Region IV office in Arlington,

Texas, on January 16, 2007. We encourage you to submit supporting documentation at least

one week prior to the conference in an effort to make the conference more efficient and

effective. This Regulatory Conference will be open to public observation.

Since the NRC has not made a final determination in this matter, no Notice of Violation is being

issued for these inspection findings at this time. In addition, please be advised that the number

and characterization of apparent violations described in the enclosed inspection report may

change as a result of further NRC review.

The report also documents one finding with two examples involving inadequate implementation

of the operability determination process. This finding was determined to be a violation of very

low safety significance. Because of the very low safety significance and because it was entered

into your corrective action program, the NRC is treating this finding as a noncited violation

consistent with Section VI.A.1 of the NRC Enforcement Policy. If you contest the noncited

violation in this report, you should provide a response within 30 days of the date of this

inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission,

ATTN.: Document Control Desk, Washington, DC 20555-0001; and the NRC Resident

Inspector at the Palo Verde Nuclear Generating Station.

In accordance with 10 CFR 2.390 of the NRC's Rules of Practice, a copy of this letter

and its enclosure will be available electronically for public inspection in the NRC Public

Document Room or from the Publicly Available Records (PARS) component of NRC's

Arizona Public Service Company

-3-

document system (ADAMS). ADAMS is accessible from the NRC Web site at

http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Arthur T. Howell III, Director

Division of Reactor Projects

Dockets: 50-528

50-529

50-530

Licenses: NPF-41

NPF-51

NPF-74

Enclosure:

Inspection Report 05000528/2006012; 05000529/2006012; 05000530/2006012

w/Attachment 1: Supplemental Information

Attachment 2: Special Inspection Charter

Attachment 3: Significance Determination Evaluation

cc w/enclosure:

Steve Olea

Arizona Corporation Commission

1200 W. Washington Street

Phoenix, AZ 85007

Douglas K. Porter, Senior Counsel

Southern California Edison Company

Law Department, Generation Resources

P.O. Box 800

Rosemead, CA 91770

Chairman

Maricopa County Board of Supervisors

301 W. Jefferson, 10th Floor

Phoenix, AZ 85003

Aubrey V. Godwin, Director

Arizona Radiation Regulatory Agency

4814 South 40 Street

Phoenix, AZ 85040

Craig K. Seaman, General Manager

Regulatory Affairs and

Performance Improvement

Palo Verde Nuclear Generating Station

Arizona Public Service Company

-4-

Mail Station 7636

P.O. Box 52034

Phoenix, AZ 85072-2034

Jeffrey T. Weikert

Assistant General Counsel

El Paso Electric Company

Mail Location 167

123 W. Mills

El Paso, TX 79901

John W. Schumann

Los Angeles Department of Water & Power

Southern California Public Power Authority

P.O. Box 51111, Room 1255-C

Los Angeles, CA 90051-0100

John Taylor

Public Service Company of New Mexico

2401 Aztec NE, MS Z110

Albuquerque, NM 87107-4224

Thomas D. Champ

Southern California Edison Company

5000 Pacific Coast Hwy, Bldg. D1B

San Clemente, CA 92672

Robert Henry

Salt River Project

6504 East Thomas Road

Scottsdale, AZ 85251

Brian Almon

Public Utility Commission

William B. Travis Building

P.O. Box 13326

1701 North Congress Avenue

Austin, TX 78701-3326

Karen O'Regan

Environmental Program Manager

City of Phoenix

Office of Environmental Programs

200 West Washington Street

Phoenix, AZ 85003

Matthew Benac

Assistant Vice President

Nuclear & Generation Services

Arizona Public Service Company

-5-

El Paso Electric Company

340 East Palm Lane, Suite 310

Phoenix, AZ 85004

Arizona Public Service Company

-6-

Electronic distribution by RIV:

Regional Administrator (BSM1)

DRP Director (ATH)

DRS Director (DDC)

DRS Deputy Director (RJC1)

Senior Resident Inspector (GXW2)

Branch Chief, DRP/D (TWP)

Senior Project Engineer, DRP/D (GEW)

Team Leader, DRP/TSS (RVA)

RITS Coordinator (KEG)

DRS STA (DAP)

V. Dricks, PAO (VLD)

D. Cullison, OEDO RIV Coordinator (DGC)

ROPreports

PV Site Secretary (PRC)

K. S. Fuller, RC/ACES (KSF)

C. A. Carpenter, D:OE (CAC)

G. M. Vasquez (GMV)

OE:EA File (RidsOeMailCenter)

SUNSI Review Completed: __TWP_ ADAMS: / Yes

G No Initials: __TWP____

/ Publicly Available G Non-Publicly Available G Sensitive

/ Non-Sensitive

R:\\_REACTORS\\_PV\\2006\\PV2006-12RP-MCH.wpd

RIV:SPE:DRP/D RI:DRS/EB1

PE:DRP/D

C:DRP/D

SRA:DRS

ACES

MCHay

SPRutenkroger

MRBloodgood TWPruett

MFRunyan

GMVasquez

/RA/

MCHay For

MCHay For

/RA/

/RA/

/RA/

11 /27/06

11/28/06

11/28/06

11/28/06

11/28/06

11/28/06

D:DRP

ATHowell III

/RA/

12/06/06

OFFICIAL RECORD COPY

T=Telephone E=E-mail F=Fax

U.S. NUCLEAR REGULATORY COMMISSION

REGION IV

Dockets:

50-528; 50-529; 50-530

Licenses:

NPF-41; NPF-51; NPF-74

Report No.:

05000528/2006012; 05000529/2006012; 05000530/2006012

Licensee:

Arizona Public Service Company

Facility:

Palo Verde Nuclear Generating Station, Units 1, 2, and 3

Location:

5951 S. Wintersburg Road

Tonopah, Arizona

Dates:

October 2 through November 30, 2006

Inspectors:

M. Hay, Senior Project Engineer, Team Leader

Dr. S. Rutenkroger, Reactor Inspector, Engineering Branch 1

M. Runyan, Senior Reactor Analyst

Accompanied:

M. Bloodgood, Reactor Engineer, Nuclear Safety Professional Development

Program

Approved By:

Arthur T. Howell III, Director

Division of Reactor Projects

Enclosure

1

SUMMARY OF FINDINGS

IR 05000528/2006012; 05000529/2006012; 05000530/2006012; 10/02/2006 - 11/09/2006; Palo

Verde Nuclear Generating Station, Units 1, 2, and 3: Special Inspection in response to Unit 3

Train A EDG failures on July 25 and September 22, 2006.

The report covered a 5-day period (October 2-6, 2006) of onsite inspection, with in-office review

through November 30, 2006, by a special inspection team consisting of one senior project

engineer, one reactor inspector, one reactor engineer, and one senior reactor analyst. Three

findings were identified. The significance of most findings is indicated by its color (Green,

White, Yellow, Red) using Inspection Manual Chapter 0609, Significance Determination

Process. Findings for which the significance determination process does not apply may be

Green or be assigned a severity level after NRC management review. The NRC's program for

overseeing the safe operation of commercial nuclear power reactors is described in

NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.

Summary of Event

The NRC conducted a special inspection to better understand the circumstances surrounding

two failures of the Unit 3 Train A emergency diesel generator that occurred on July 25 and

September 22, 2006. Both failures involved the emergency diesel generator failing to obtain an

output voltage during surveillance testing because of faulty K-1 relay operation. In accordance

with NRC Management Directive 8.3, NRC Incident Investigation Program, it was determined

that this event involved repetitive failures of safety related equipment having potential adverse

generic implications and had sufficient risk significance to warrant a special inspection.

A.

NRC-Identified and Self Revealing Findings

Cornerstone: Mitigating Systems

TBD. The team identified an apparent violation of 10 CFR Part 50, Appendix B,

Criterion V, Instructions, Procedures, and Drawings, for the failure to establish

appropriate instructions for performing corrective maintenance activities on an

emergency diesel generator K-1 relay. As a result, following identification that a

replacement emergency diesel generator K-1 relay was unreliable, the licensee

performed ineffective corrective maintenance activities on this relay. This

performance deficiency contributed to the Unit 3 Train A emergency diesel

generator being inoperable between September 4 and 22, 2006, and a failure on

September 22, 2006. Immediate corrective actions included inspection,

cleaning, and/or performing mechanical adjustments on all emergency diesel

generator K-1 relays. This issue was entered into the licensees corrective

action program as Condition Report/Disposition Request 2926830.

The finding is greater than minor because it is associated with the equipment

performance cornerstone objective to ensure the availability, reliability, and

capability of systems that respond to initiating events to prevent undesirable

consequences. Using NRC Inspection Manual Chapter 0609, Significance

Determination Process, Phase 1 Worksheet, a Phase 2 evaluation was required

Enclosure

2

because the finding resulted in the loss of the safety function of the Unit 3 Train

A emergency diesel generator for greater than the Technical Specification

completion time. The Phase 2 evaluation concluded that the finding was of low

to moderate safety significance. A Phase 3 preliminary significance

determination analysis also determined the finding was of low to moderate safety

significance. The cause of this finding is related to the crosscutting element of

human performance associated with resources in that the licensee failed to

develop and implement appropriate work instructions prior to performing

corrective maintenance activities on an emergency diesel generator K-1 relay

(Section 3.0).

TBD. The team identified an apparent violation of 10 CFR Part 50, Appendix B,

Criterion XVI, Corrective Actions, for the failure to identify and correct the

cause of erratic emergency diesel generator K-1 relay operation prior to

installation of the relay on July 26, 2006. This performance deficiency

contributed to the Unit 3 Train A emergency diesel generator being inoperable

between September 4 and 22, 2006, and a failure on September 22, 2006.

Immediate corrective actions included inspection, cleaning, and performing

mechanical adjustments, as appropriate, on all emergency diesel generator K-1

relays. This issue was entered into the licensees corrective action program as

Condition Report/Disposition Request 2926830.

The finding is greater than minor because it is associated with the equipment

performance cornerstone objective to ensure the availability, reliability, and

capability of systems that respond to initiating events to prevent undesirable

consequences. Using NRC Inspection Manual Chapter 0609, Significance

Determination Process, Phase 1 Worksheet, a Phase 2 evaluation was required

because the finding resulted in the loss of the safety function of the Unit 3 Train

A emergency diesel generator for greater than the Technical Specification

allowed outage time. The Phase 2 evaluation concluded that the finding was of

low to moderate safety significance. A Phase 3 preliminary significance

determination analysis also determined the finding was of low to moderate

safety significance. The cause of this finding is related to the crosscutting

element of problem identification and resolution in that the failure to fully

evaluate and implement adequate corrective maintenance actions for the Unit 3

Train A emergency diesel generator resulted in the emergency diesel generator

being inoperable for 18 days (Section 3.0).

The team identified two examples of a noncited violation of 10 CFR Part 50,

Appendix B, Criterion V, Instructions, Procedures, and Drawings, for the failure

to perform operabilty determinations. In both examples, the licensee failed to

perform an operability determination following identification of a degraded

condition that had the potential to adversely affect the safety function of all

emergency diesel generators. Specifically, an operability determination was not

performed after identifying the failure of the Unit 3 Train A emergency diesel

generator on July 25, 2006, was potentially the result of plastic debris affecting

proper auxiliary contact operation of a K-1 relay. The licensee determined the

debris most likely originated from a modification performed on all emergency

diesel generator K-1 relays during initial plant startup. Following another failure

Enclosure

3

of the Unit 3 Train A emergency diesel generator on September 22, 2006, an

operability determination was not performed after identifying the failure was the

result of the K-1 relay actuating arm not providing adequate compression of the

auxiliary contacts. The licensee determined this degraded condition most likely

originated during implementation a modification done to all emergency diesel

generator K-1 relays during initial plant startup.

This finding is greater than minor because the failure to follow the operability

determination process, if left uncorrected, would become a more significant

safety concern in that degraded or nonconforming conditions would not be

properly evaluated. Using the Phase 1 worksheet in NRC Inspection Manual

Chapter 0609, Significance Determination Process, the finding was determined

to have very low safety significance because unreliable K-1 relay operation

resulted in no actual loss of safety function of the other five emergency diesel

generators prior to corrective actions being implemented, and the finding did not

represent a potential risk significant condition because of a seismic, flooding, or

severe weather event. This issue is documented in the licensees corrective

action program as Condition Report/Disposition Requests 2928389 and

2940558. The cause of this finding is related to the crosscutting element of

problem identification and resolution in that engineering personnel failed to

properly evaluate and perform operability determinations for identified degraded

conditions affecting the emergency diesel generators (Section 4.0).

B.

Licensee-Identified Findings

None.

Enclosure

4

REPORT DETAILS

1.0

SPECIAL INSPECTION SCOPE

The NRC conducted this special inspection to better understand the circumstances

surrounding two failures of the Unit 3 Train A emergency diesel generator (EDG) that

occurred on July 25 and September 22, 2006. Both failures involved the EDG failing to

obtain an output voltage during surveillance testing because of a faulty K-1 relay

operation. In accordance with NRC Management Directive 8.3, NRC Incident

Investigation Program, it was determined that this event met several deterministic

criteria and had sufficient risk significance to warrant a special inspection.

The team used NRC Inspection Procedure 93812, Special Inspection Procedure, to

conduct the inspection. The special inspection team reviewed procedures, corrective

action documents, and design and maintenance records for the equipment of concern.

The team interviewed key station personnel regarding the event, reviewed the root

cause analysis, and assessed the adequacy of corrective actions. A list of specific

documents reviewed is provided in Attachment 1. The charter for the special inspection

effort is provided as Attachment 2.

2.0

EVENT DESCRIPTION

Each unit at Palo Verde contains two safety-related 5500 Kw EDGs that provide standby

power for safe plant shutdown in the event the normal supply of power is lost. On

July 25, 2006, at 12:53 p.m., the Unit 3 Train A EDG failed to develop an output voltage

during a routine surveillance test. When the EDGs are secured a field shorting K-1

relay actuates to electrically short the generator field causing the generator output

voltage to collapse. A relay latching mechanism maintains the field shorted until the

EDG is started at which time the latch disengages allowing the relay to actuate and un-

short the EDG field. With the field no longer shorted the voltage regulator establishes

and controls the EDG output voltage. The licensee identified that a faulty set of auxiliary

contacts on a K-1 relay prevented the latch from disengaging that resulted in the

generator field being shorted during the start of the EDG. The licensee determined the

root cause of the auxiliary contact failure could be attributed to either plastic debris or

oxide film buildup preventing continuity across the contacts when closed. Following the

failure on July 25, 2006, the licensee replaced the failed K-1 relay with a new relay

obtained from the warehouse. During continuity checks of the new relay, the same

auxiliary contacts were identified to operate unreliably. The last available relay from the

warehouse was obtained and it also operated unreliably and it had a warped cover.

Based on no other replacement K-1 relays being available, the licensee performed

corrective maintenance activities on the first relay obtained from the warehouse in an

attempt to resolve the problem. Following these corrective maintenance activities, the

relay was successfully tested several times and the Unit 3 Train A EDG was declared

operable at 10:35 a.m. on July 26, 2006.

On September 22, 2006, the Unit 3 Train A EDG failed to develop an output voltage

following a postmaintenance surveillance test. The licensee identified that the same set

Enclosure

5

of auxiliary contacts that previously exhibited erratic behavior during continuity checks

had failed. The licensee identified that the K-1 relay actuating arm for the affected

auxiliary contact module was not providing adequate compression of the auxiliary

contacts. Corrective actions involved mechanical adjustments to the actuating arm to

provide proper auxiliary contact compression. Additional corrective actions included

inspecting, cleaning, and making mechanical adjustments, as necessary, to all other

affected EDG K-1 relays.

3.0

PERFORMANCE DEFICIENCIES RESULTING IN EDG FAILURE

a.

Inspection Scope

On July 25 and September 22, 2006, the Unit 3 Train A EDG failed to produce output

voltage during surveillance testing. The team reviewed the licensees corrective actions

following failure of the Unit 3 Train A EDG on July 25, 2006, to assess their

effectiveness with respect to preventing the subsequent failure that occurred on

September 22, 2006.

b. Observations and Findings

Introduction: The team identified two apparent violations of NRC requirements. The

team identified an apparent violation of 10 CFR Part 50, Appendix B, Criterion V,

Instructions, Procedures, and Drawings, for the failure to establish appropriate

instructions for performing corrective maintenance activities on an EDG K-1 relay. As a

result, following identification that a replacement EDG K-1 relay was unreliable, the

licensee performed ineffective corrective maintenance activities on this relay.

Additionally, the team identified an apparent violation of 10 CFR Part 50, Appendix B,

Criterion XVI, Corrective Action, involving the failure to identify the cause of a

significant condition adverse to quality and take corrective actions to preclude repetition.

Specifically, following identification that a replacement EDG K-1 relay was unreliable,

the licensee failed to identify the cause of this condition and implement adequate

corrective actions. These performance deficiencies resulted in the Unit 3 Train A EDG

being inoperable between September 4 and 22, 2006, and a failure on September 22,

2006. These issues have potential low to moderate safety significance (White).

Description: On July 25, 2006, at 12:53 p.m., the Unit 3 Train A EDG failed to develop

an output voltage during a routine surveillance test. The licensee identified that a faulty

set of auxiliary contacts on a K-1 relay resulted in the generator field being shorted

during the start of the EDG. A new K-1 relay assembly was acquired from the

warehouse and during functional testing the same auxiliary contacts exhibited erratic

operation. The team noted that Work Order 2913004 stated, in part, that the K-1 relay

failed at first. Checked and re-attempted satisfactory. Performed several times

satisfactory. System engineering showed up and after discussing the problem they

wanted to verify repeatability. Checking contact resistance was found to be erratic.

Unable to clean the contacts to get consistent readings. Determined not reliable and

ordered last K-1 relay from the warehouse.

Enclosure

6

Initial attempts, by electrical maintenance personnel, to clean the auxiliary contacts of

the first relay obtained from the warehouse consisted of using a 9-volt battery connected

across the contacts. By cycling the contacts, the licensee stated that an electrical arc

could potentially clean any oxidation affecting the contacts ability to pass current. The

licensee stated this practice was utilized because engineering would not allow intrusive

actions, such as taking the relay apart, to clean the auxiliary contacts because of

concerns with maintaining critical dimensions. The team noted the licensee had no

maintenance instructions applicable to disassembly of the relay and no detailed vendor

specifications describing the critical attributes of the device. The licensee stated that

obtaining this type of information was not possible because the equipment was

obsolete, it was commercially dedicated by a vendor, and the vendor and manufacturer

of the component were no longer in business. Based on these reasons, the licensee

stated that no maintenance activities were performed on the auxiliary contacts and that

when problems were encountered the K-1 relay was replaced as a whole unit.

After initial attempts to clean the K-1 relay auxiliary contacts using the 9-volt battery

failed, the licensee obtained the last replacement K-1 relay available onsite. Again, they

found that the same set of auxiliary contacts on this relay exhibited erratic operation

when cycled. Additionally, the cover to this relay was found in a warped condition. At

this point the licensee decided to perform intrusive cleaning of the auxiliary contacts on

the first device that they determined was unreliable following non-intrusive cleaning

efforts. The team noted that no detailed work instructions were established to perform

this activity to improve its reliable operation. As previously stated, the licensee did not

possess any detailed vendor information specifically related to performing maintenance

activities on this device. After disassembling the auxiliary contacts, cleaning, and

reassembly, the relay was tested several times and the erratic behavior was not

observed during subsequent functional tests of the relay. A postmaintenance

surveillance test of the EDG was performed satisfactorily and the equipment was

declared operable at 10:35 a.m. on July 26, 2006.

On September 22, 2006, the Unit 3 Train A EDG failed to develop an output voltage

following a postmaintenance surveillance test. The licensee identified that the same set

of auxiliary contacts that exhibited erratic behavior on the K-1 relay had failed, resulting

in the generator field being shorted during the start of the EDG. The licensee identified

that the auxiliary contacts were not held closed when the K-1 relay was energized

because of an actuating arm that was not adequately depressing the auxiliary contact

switch. The team noted that this condition most likely existed during the initial testing on

July 25, 2006, and would have contributed to the erratic operation of this contact switch

assembly. Additionally, the team determined that this condition was not identified and

corrected because instructions for performing corrective maintenance activities on the

unreliable K-1 relay were inadequate. The team noted that the last successful start of

the Unit 3 Train A EDG was on September 4, 2006.

As previously stated, the licensee initially believed the erratic behavior resulted from

oxidation of the contact surfaces which required an intrusive maintenance activity to

clean the contact surfaces. The licensee stated that contact oxidation is a common

occurrence requiring cleaning. The team reviewed all work orders associated with

replacement of the K-1 relay dating back to 1984 and noted that none of the work orders

documented erratic auxiliary contact operation because of oxidation. During the review

Enclosure

7

of work orders, the team noted that Work Order 00067739, dated December 11, 1984,

discussed troubleshooting and repair activities for a faulty auxiliary contact on the K-1

relay affecting the Unit 1 Train B EDG. This work order contained instructions to inspect

the auxiliary contact arm on the K-1 relay and make adjustments as needed for proper

contact operation per Technical Manual MO18-390. The maintenance technician

performing the work documented, Adjusted the actuation arm for the auxiliary contacts

on the left side of the K-1 contactor as required. Based on this work order, the team

determined that on this occasion the licensee had worked on the auxiliary contact

operating mechanism to ensure reliable operation. A technician involved in this

maintenance activity did not recall the specifics of the work performed on the relay nor

the use of technical information contained in the technical manual. The team reviewed

the technical manual and found that no pertinent maintenance information for the K-1

relay existed.

Although no pertinent maintenance information for the K-1 relay was identified during

review of Technical Manual MO18-390, the team noted that detailed instructions were

provided to maintenance personnel for ensuring that critical tolerances of other relays

were maintained. The team noted that the voltage controlled overcurrent relay, reverse

power relay, frequency relay, and negative-phase sequence time overcurrent relay, all

associated with the EDG voltage regulating system, contained specific installation,

operation, and maintenance instructions. The team noted that these instructions

provided detailed information for activities involving contact cleaning, contact

adjustments, operational checks, and mechanical adjustments for each particular type

of relay to ensure reliable operation.

The licensee stated that the EDG K-1 relays had a history of operating reliably. Data

collected since 1990 indicated that the EDG K-1 relays had not failed because of

auxiliary contact problems similar to the failures identified in July and September of

2006. The team determined that this reliability data further demonstrated that, when the

erratic relay operation was identified, the licensee should have recognized that

corrective measures were needed that would require appropriate instructions to ensure

future reliable operation.

The team determined that the licensees problem analysis efforts were narrowly

focused, which led them to conclude that the cause of the erratic relay operation was

oxidized contacts. The erratic operation of the K-1 relay provided an indication that

sufficient auxiliary contact continuity existed, at least intermittently, which indicated that

another failure mechanism was contributing to the unreliable K-1 relay operation. If the

licensee performed an adequate cause analysis of this significant condition adverse to

quality, then they may have identified the failure mechanism associated with the

actuating arm not providing adequate contact compression prior to installation of the

new relay on July 26, 2006.

Analysis: NRC Inspection Manual Chapter 0612, Power Reactor Inspection Reports,

defines a performance deficiency as an issue that is the result of a licensee not meeting

a requirement or standard where the cause was reasonably within the licensees ability

to foresee and correct and that should have been prevented. The licensee determined

that the K-1 relay that failed in September of 2006 was unreliable prior to placing it in

service and would require corrective maintenance. The licensee stated that

Enclosure

8

disassembly of the relay to implement intrusive corrective actions had never been

performed because of concerns with maintaining critical dimensions for reliable relay

operation. The licensee did not obtain, nor did they develop, detailed information

specific to performing corrective or preventive maintenance activities for this specific

relay. On the basis of these considerations, the team concluded that the licensees

failure to establish and implement adequate maintenance instructions to resolve the

unreliable K-1 relay condition was a performance deficiency resulting in the Unit 3 Train

A EDG being inoperable between September 4 and 22, 2006. The team determined

that the EDG was inoperable for an 18-day period on the basis that when the EDG was

shut down on September 4, 2006, the K-1 relay auxiliary contacts would have been

positioned and maintained in a state that would have resulted in a subsequent failure of

the relay to operate properly following an EDG start signal. Additionally, the team

determined that the failure to perform an adequate cause assessment of the erratic

relay operation contributed to the inoperability of the Unit 3 Train A EDG.

These findings are greater than minor because they are associated with the equipment

performance cornerstone objective to ensure the availability, reliability, and capability of

systems that respond to initiating events to prevent undesirable consequences. Using

NRC Inspection Manual Chapter 0609, Significance Determination Process, Phase 1

Worksheet, a Phase 2 analysis was required because the findings resulted in the loss of

the safety function of the Unit 3 Train A EDG for greater than the Technical Specification

completion time. The Phase 2 and 3 evaluations preliminarily concluded that the

findings were of low to moderate safety significance. (See Attachment 3 for Phase 2

and Phase 3 details.) The cause of the Criterion XVI finding is related to the

crosscutting element of problem identification and resolution in that the failure to fully

evaluate and implement adequate corrective maintenance actions for the Unit 3 Train A

EDG contributed to the EDG being inoperable for 18 days. Additionally, the cause of

the Criterion V finding is related to the crosscutting element of human performance

associated with resources in that the licensee failed to develop and implement

appropriate work instructions prior to performing corrective maintenance activities on the

subject EDG K-1 relay, which contributed to the EDG being inoperable for 18 days.

Enforcement: 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and

Drawings, states, in part, that activities affecting quality shall be prescribed by

documented instructions, procedures, or drawings of a type appropriate to the

circumstances and shall be accomplished in accordance with these instructions,

procedures, or drawings. Contrary to this, the licensee failed to develop appropriate

instructions or procedures for corrective maintenance activities on the Unit 3 Train A

EDG K-1 relay. This failure resulted in the Unit 3 Train A EDG being inoperable

between September 4 and 22, 2006. This item has been entered into the licensees

corrective action program as Condition Report/Disposition Request (CRDR) 2926830.

Pending determination of safety significance, this finding is identified as an apparent

violation (AV)05000530/2006012-01, Failure to Establish Appropriate Instructions.

10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action, states, in part, that

measures shall be established to assure that conditions adverse to quality, such as

failures, malfunctions, deficiencies, deviations, defective material and equipment, and

nonconformances are promptly identified and corrected and for significant conditions

adverse to quality, measures shall assure that the cause of the condition is determined

Enclosure

9

and corrective action taken to preclude repetition. Contrary to this, the licensee failed to

identify and correct the cause of the erratic EDG K-1 relay operation prior to installation

of the relay on July 26, 2006. This failure resulted in the Unit 3 Train A EDG being

inoperable between September 4 and 22, 2006. This item has been entered into the

licensees corrective action program as CRDR 2926830. Pending determination of

safety significance, this finding is identified as AV 05000530/2006012-02, Failure to

Identify and Correct a Condition Adverse to Quality.

4.0

Failure to Implement the Operability Determination Process

a.

Inspection Scope

The team assessed the engineering and operations departments implementation of the

operability determination (OD) process after identifying potential adverse conditions

involving reliable K-1 relay operation of the EDGs. This assessment was performed

through interviews and a review of precisely logs, ODs, and related documents. In

addition, the team conducted an independent assessment of system operability.

b.

Observations and Findings

Introduction: The team identified two examples of a Green noncited violation of 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, involving the

failure to follow the OD procedure.

Description:

Example One

Administrative Procedure 40DP-9OP26, Operability Determination and Functional

Assessment, Revision 17, Section 1.3, stated, in part, that the OD process is entered

when the ability of a Technical Specification system or component to perform its

specified safety function is called into question by the discovery of a degraded condition.

As previously discussed, on July 25, 2006, the Unit 3 Train A EDG failed to produce

output voltage during surveillance testing. The licensee identified that a faulty set of

auxiliary contacts on a K-1 relay resulted in the generator field being shorted during the

start of the EDG. The team noted that an engineering team was assigned to perform a

root cause analysis for the K-1 relay auxiliary contact failure. This root cause evaluation

was documented in CRDR 2913003. The root cause team determined the most

probable cause was due to contamination on the contact surface either from oxidation or

from pieces of plastic filings found in the contact area. The plastic pieces were believed

to have resulted from a modification of the contact block actuator implemented by

Design Change Package DCP X-PE-007. This design change added auxiliary contacts,

latching mechanisms, and spacers to all EDG K-1 relays at the facility to address

seismic concerns identified during testing of the K-1 relay during the initial plant

construction phase.

Enclosure

10

The NRC inspection team noted that the licensees root cause assessment team

concluded that only the Unit 3 Train A EDG was potentially degraded by this condition.

The evaluation stated, in part, that:

The same model/type field shorting contactor is used on all six Class 1E

EDGs. In addition, two spare K-1 relays removed from the warehouse

exhibited the same symptoms with varying resistance across the auxiliary

contacts. An inspection/test of the auxiliary contacts for all other EDGs

was performed, with no other auxiliary contact resistance problems

identified.

The NRC team was informed that the licensees inspection consisted of a functional

check of the relay and did not involve visually inspecting the auxiliary contact internals to

identify the presence of plastic filings. Based on this, the team determined that the

inspection and contact resistance testing alone failed to demonstrate why only the Unit 3

Train A EDG K-1 relay was affected since the relay had operated properly since being

placed in service on August 1, 2001. Therefore, the licensee inadequately assessed the

extent of condition of the unreliable relay operation relative to the other EDGs at the

facility.

The NRC team noted that the root cause assessment identified a degraded condition,

plastic filings in the contact module that likely affected all the facility EDGs. However,

the licensee failed to enter the OD process after identifying this potentially degraded

condition.

Example Two

As previously discussed, on September 22, 2006, at 1:18 a.m., the Unit 3 Train A EDG

failed to produce output voltage during surveillance testing. The licensee identified that

a faulty set of auxiliary contacts on a K-1 relay resulted in the generator field being

shorted during the start of the EDG. The licensee identified that the K-1 relay actuating

arm for the affected auxiliary contact module was not providing adequate compression

of the auxiliary contacts. The licensee noted that part of the actuating arm consisted of

a metal tab that was bent in a configuration that would result in less contact

compression. The licensee decided to straighten the metal tab, thereby, providing

additional contact compression. Five functional tests of the relay were performed and

the EDG was declared operable following a postmaintenance surveillance test on

September 22, 2006, at 5:48 p.m.

The team noted that CRDR 2926830 documented the licensees evaluation of this

failure. The CRDR stated, in part:

The auxiliary contacts that had failed were added to the K-1 relay by

Design Change Package DCP X-PE-007 during plant startup in response

to electrical seismic latch failures. Engineering believes that the actuator

arm mounted metal tab was initially bent because originally there were no

auxiliary contacts on that side of the K-1 relay. Following completion of

the design change, the auxiliary contacts appeared to be working properly

so the actuator arms were left bent down. Inspection of some of the K-1

Enclosure

11

relays removed from EDGs in the past found at least one that had the

actuator arm straight; however, in most cases, the actuator arm metal tab

for the auxiliary contacts were found bent down. This is a repeat failure

of Unit 3 Train A EDG to produce output voltage. Recent verification of

acceptable K-1 relay auxiliary contact continuity on the other five EDGs

provides the basis that this condition is not present on those relays.

The NRC team was informed that the licensees inspection consisted of a functional

check of the relay and did not involve visually inspecting the auxiliary contact actuating

arms to identify a bent configuration. Based on this, the team determined that the

inspection and contact resistance testing alone failed to demonstrate why only the Unit 3

Train A EDG K-1 relay was affected, since the relay had operated properly since being

placed in service on July 26, 2006. Therefore, the licensee inadequately assessed the

extent of condition of the unreliable relay operation relative to the other EDGs at the

facility.

The team noted that the engineering assessment identified a degraded condition, a bent

K-1 relay actuating arm resulting in unreliable operation, that likely affected all the facility

EDGs. The licensee failed to enter the OD process after identifying this potentially

degraded condition. Following discussions with the licensee, an OD was performed on

September 27, 2006.

In both of these examples the team determined that engineering failed to recognize that

the identified degraded conditions had the potential to adversely affect the other EDGs.

On both occasions engineering relied on continuity checks of the auxiliary contacts to

inappropriately conclude that the other EDGs were not affected. The team noted that

the testing results were pertinent to an OD assessment; however, the information did

not provide adequate justification for not implementing the OD process ensuring

reasonable assurance existed supporting operability of the other EDGs.

Analysis: The performance deficiency was associated with engineering personnel not

adequately implementing the provisions of the OD procedure following the identification

of a degraded condition. This finding was more than minor because the failure to follow

the operability determination process, if left uncorrected, would become a more

significant safety concern in that degraded or nonconforming conditions would not be

properly evaluated. Using the Phase 1 worksheet in Manual Chapter 0609,

Significance Determination Process, the finding was determined to have very low

safety significance because, although these conditions resulted in unreliable K-1 relay

operation, no actual loss of safety function occurred (with respect to the other 5 EDGs)

prior to corrective actions being implemented, and the finding did not represent a

potential risk significant condition due to a seismic, flooding, or severe weather event.

This finding involved problem identification and resolution crosscutting aspects

associated with engineering personnel failing to properly evaluate and perform

operability determinations for identified degraded conditions.

Enforcement: 10 CFR Part 50, Criterion V, Instructions, Procedures, and Drawings,

states, in part, that activities affecting quality shall be prescribed by documented

instructions, procedures, or drawings of a type appropriate to the circumstances and

shall be accomplished in accordance with these instructions, procedures, or drawings.

Enclosure

12

Administrative Procedure 40DP-9OP26, Operability Determination and Functional

Assessment, Revision 17, Section 1.3, stated, in part, that the OD process is entered

when the ability of a Technical Specification system or component to perform its

specified safety function is called into question by the discovery of a degraded condition.

Contrary to the above, on two occasions the licensee failed to enter the OD process

when the ability of a Technical Specification system or component safety function was

called into question. On July 25, 2006, following failure of the Unit 3 Train A EDG, an

OD was not performed after identifying the failure was likely because of plastic filings

affecting proper auxiliary contact operation of a K-1 relay. The filings were believed to

be from a modification that was performed on all EDG K-1 relays during initial plant

startup. On September 22, 2006, following another failure of the Unit 3 Train A EDG, an

OD was not performed after identifying the failure was the result of the K-1 relay

actuating arm not providing adequate compression of the auxiliary contacts. The

licensee determined this condition most likely resulted from a modification performed on

all EDG K-1 relays during initial plant startup. Because the finding is of very low safety

significance and has been entered into the licensees corrective action program as

CRDRs 2928389 and 2940558, this violation is being treated as a noncited violation

consistent with Section VI.A of the Enforcement Policy: Noncited Violation

05000528;05000529;05000530/2006012-03, Failure to Implement the Operability

Determination Process.

5.0

CORRECTIVE ACTIONS FOLLOWING EDG FAILURES

a.

Inspection Scope

The team assessed the licensees immediate and long-term planned corrective actions

associated with the Unit 3 Train A EDG failures that occurred on July 25 and

September 22, 2006. This assessment was performed through interviews, review of

operator logs, corrective action documents, work orders, and related documents.

b.

Observations and Findings

Following the Unit 3 Train A EDG failure on July 25, 2006, the licensee identified that

plastic filings inside the auxiliary contact module may have resulted in the failure. The

licensee concluded this material most likely was introduced during a design change

performed on all the K-1 relays and subsequently established a schedule to inspect all

the EDG auxiliary contact modules. The team noted these inspections were scheduled

to be performed November 2006 through March 2007. NRC Inspection Manual,

Part 9900, Technical Guidance, Operability Determination and Functionality

Assessments for Resolution of Degraded or Nonconforming Conditions Adverse to

Quality or Safety, Section 4.2, states, When a potential degraded or nonconforming

condition is identified, the licensee should take action without delay to confirm if a

system, structure, or component is degraded or nonconforming. The team concluded

that waiting approximately 8 months to identify whether other EDGs were affected by

this potential adverse condition was not commensurate with the safety consequences of

having a degraded EDG.

Enclosure

13

Following the September 22, 2006, Unit 3 Train A EDG failure, the licensee identified

that a bent K-1 relay actuating arm resulted in unreliable auxiliary contact operation.

Immediate corrective actions involved straightening the arm to provide additional contact

compression for the Unit 3 Train A EDG. The team noted that the licensee implemented

timely corrective actions to inspect and implement mechanical adjustments, as needed,

to all the EDG relays to ensure adequate contact compression during operation. These

actions were implemented September 27-30, 2006, and incorporated the inspections

resulting from the July failure that were originally not scheduled to be completed until

March 2007. The licensee straightened bent K-1 relay contactor arms for the Unit 2

Trains A and B EDGs, and the Unit 1 Train B EDG. The team determined that these

actions were timely and they included the inspections identified following the July failure.

Therefore, there were no regulatory findings associated with timeliness of these

corrective actions.

The team noted that long-term planned corrective actions consisted of replacing all of

the EDG automatic voltage regulators, including replacement of the K-1 relays, with a

different design. The licensee stated that they plan to have these replacement activities

accomplished during the next refueling outage for each unit.

6.0

Generic Implications

The team reviewed various NRC generic communications and operating experience

from other licensees relevant to the EDG relay failures identified at the Palo Verde

Nuclear Generating Station. No relevant similar relay failures were identified. Both the

NRC and the licensee concluded that the relay problems pertaining to ensuring

adequate contact compression is provided by the actuator arm was potentially of

generic concern. On October 21, 2006, the licensee submitted voluntary Licensee

Event Report (LER) 50-530/2006-006-00 to report this concern.

4OA3 Event Follow-up (71153)

.1

(Closed) LER 05000530/2006-006-00, Voluntary LER for Failure of Emergency

Diesel Generator to Attain Required Voltage Due to Relay Contactor

On September 22, 2006, at 1:18 a.m., the Unit 3 Train A EDG failed to produce

output voltage during surveillance testing. The licensee identified that a faulty

set of auxiliary contacts on a K-1 relay resulted in the generator field being

shorted during the start of the EDG. The licensee identified that the K-1 relay

actuating arm for the affected auxiliary contact module was not providing

adequate compression of the auxiliary contacts. The licensee noted that part of

the actuating arm consisted of a metal tab that was bent in a configuration that

would result in less contact compression. Immediate corrective actions involved

mechanical adjustments made to the actuating arm providing additional contact

compression for the Unit 3 Train A EDG. Additionally, the licensee implemented

corrective actions to inspect and make adjustments as needed to all the EDG

relays. As discussed in section 3.0 of this report, the Unit 3 Train A EDG failure

on September 22, 2006, resulted from and inadequate cause assessment and

the failure to establish appropriate corrective maintenance instructions which

Enclosure

14

resulted in a violation of Technical specification 3.8.1.B since the inoperable

EDG exceeded the completion time of 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />. The team determined that the

licensee failed to identify that 10 CFR 50.73(a)(2)(B) requires the licensee to

report any operation or condition which was prohibited by the plants Technical

Specifications. Based on the licensee having performed a voluntary LER

addressing both Unit 3 Train A EDG failures that occurred on July 25, 2006, and

September 22, 2006, the failure to make a required report in accordance with

10 CFR 50.73 constitutes a violation of minor significance that is not subject to

enforcement action in accordance with Section IV of the NRCs Enforcement.

This LER is closed.

4OA6 Meetings, Including Exit

On November 30, 2006, the inspection results were discussed with Mr. David Mauldin,

Vice President, Engineering, and other members of the plant staff. The inspectors

asked the licensee whether any of the material examined during the inspection should

be considered proprietary. No proprietary information was identified.

ATTACHMENT 1: SUPPLEMENTAL INFORMATION

ATTACHMENT 2: SPECIAL INSPECTION CHARTER

ATTACHMENT 3: SIGNIFICANCE DETERMINATION EVALUATION

Attachment 1

A1-1

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

S. Bauer, Department Leader, Regulatory Affairs

P. Borchert, Director, Operations

L. Bullington, Probabilistic Risk Analysis

C. Churchman, Director, Engineering

C. Cooper, Acting Electrical Maintenance Engineering Section Leader

E. Dutton, Nuclear Assurance Department

C. Eubanks, Vice President, Nuclear Operations

M. Green, LAN Department

J. Holmes, Section Leader

R. Henry, Site Representative

D. Leech, Department Leader, Corrective Action Program

J. Levine, Executive Vice President, Generation

C. Marschall, Consultant

D. Mauldin, Vice President, Engineering

L. Nguyen, LA Power and Water

S. Peace, Owner Services Consultant

M. Perito, Plant Manager

J. Proctor, Section Leader, Regulatory Affairs - Compliance

C. Seaman, General Manager, Regulatory Affairs and Performance Improvement

D. Steen, Senior Engineer

T. Radtke, General Manager

B. Ramey, Maintenance Engineering Department Leader

R, Schwartsbeck, Enercon Services

E. Shouse, Site Representative

G. Sowers, Section Leader, Probabilistic Risk Analysis

D. Straka, Senior Consultant, Regulatory Affairs

D. Vogt, Section Leader, Operations/Shift Technical Advisor

D. Withers, Maintenance Engineering

NRC Personnel

T. Vegel, Deputy Director, Division of Reactor Projects

G. Warnick, Senior Resident Inspector, Palo Verde Nuclear Generating Station

Attachment 1

A1-2

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

05000530/2006012-01

AV

Failure to Establish Appropriate Instructions05000530/2006012-02

AV

Failure to Identify and Correct aCondition Adverse

to Quality

05000528;05000529;05000530/2006012-03

NCV

Failure to Implement the Operability Determination

Process

Closed

05000528; 05000529;05000530/2006012-03

NCV

Failure to Implement the Operability Determination

Process

05000530/2006-006-00

LER

Voluntary LER for Failure of Emergency Diesel

Generator to Attain Required Voltage Due to Relay

Contactor

LIST OF DOCUMENTS REVIEWED

Drawings

13-M018-00159

C72-15000-100

D72 12200 750 Rev. E

D72-12200-710 Rev. E

D72-12200-100 Rev. B

Miscellaneous

1986 Document Specification 13-MM-0018

CES 0391-45C Seismic Test Program on Unit and Diesel Generator Control Panel

Emergency Diesel Generator Pre-operational Test

Surveillance Logs

IEEE 387 (1972 Version)

Design Change Package DCP 2SE-PE-007

Design Change Package DCP 10E-PE-007

M018-00425 Seismic Qualification Report on Triaxial Shake Table Tests of Two K-1 Relays

M018-00367 Report of Witness Tests of KSV-20-T Engine-Generator Set

Miscellaneous Documents Containing Pre-operational Test Data

Miscellaneous Control Room Log Entries for K-1 Relay Failures

Plant Change Request 86-13-PE-002

Purchasing Order 10470-13-MM-018 Documentation

Purchasing Order 33501265 Documentation

Attachment 1

A1-3

Purchasing Order 44930161 Documentation

Purchasing Order 60113782 Documentation

Portec Inc. Instruction Manual for the Static Voltage Regulator System

SFR 1PE-018

TN-E-3489

Startup Work Authorizations

SWA 15805, 1 SWA 15532, 1 SWA 15795, 1 SWA 17469, 1 SWA 16047, 2 SWA 15532, 2

SWA 15795, 2 SWA 16047, 3 SWA 15532, 3 SWA 15795, U1-SWA 15188, and U1-SWA

19219

Engineering Evaluation Requests

EER 85-PE-007

EER 85-PE-008

EER 84-PE-002

Procedures

93PE-1PE01

CRAIs

2829958, 2829959, 2829960, 2829961, 2829964, 2829965, 2829967, 2829971, 2829973, and

2829974

CRDRs

43930, 51630, 51743, 53295, 53788, 55546, 58135, 59433, 90278, 115952, 160332, 2361791,

2405054, 2410347, 2432009, 2532225, 2570582, 2579229, 2582956, 2641676, 2645588,

2650009, 2660221, 2752631, 2759704, 2784750, 2847506, 2872154, 2906158, 2913003,

2926830, and 2927262

Work Orders

00003495, 67521, 67550, 119715, 360726, 360944, 412917, 1040142, 1071966, 1329487,

2361781, 2410350, 2750447, 2794362, 2902642, 2913004, 2913286, 2913287, 2913295,

2913306, 2913753, 2919666, 2919670, 2919671, 2919672, 2919673, 2919747, and 2926829

Attachment 2

A2-1

SPECIAL INSPECTION CHARTER

September 29, 2006

MEMORANDUM TO: Michael Hay, Senior Project Engineer,

Project Branch D, Division of Reactor Projects (DRP)

Dr. Scott Rutenkroger, Reactor Inspector,

Engineering Branch 1, Division of Reactor Safety

Michael Bloodgood, Reactor Engineer, Nuclear Safety Professional

Development Program, Project Branch D, DRP

FROM:

Arthur T. Howell III, Director, DRP /RA/ AVegel for

SUBJECT:

SPECIAL INSPECTION CHARTER TO EVALUATE THE PALO VERDE

NUCLEAR GENERATING STATION UNIT 3 EMERGENCY DIESEL

GENERATOR FAILURE

A Special Inspection Team is being chartered in response to the Palo Verde Nuclear

Generating Station Unit 3 Emergency Diesel Generator (EDG) failure. The diesel failed to

develop an output voltage when started for a surveillance test. The licensee determined that a

modification to the field flashing relay caused the failure. You are hereby designated as the

Special Inspection Team members. Mr. Hay is designated as the team leader. The assigned

SRA to support the team is Mike Runyan.

A.

Basis

On July 25, 2006, Unit 3, Train A, EDG failed to develop output voltage during a

surveillance test. The licensee's root cause determined plastic debris potentially

prevented auxiliary contacts from properly functioning resulting in shorting out of the

generator field during startup preventing a proper field flash. Two replacement relays

obtained from the licensee warehouse exhibited the same unreliable condition. After

performing corrective maintenance activities on one of the relays, the diesel was

subsequently tested and declared operable on July 26, 2006.

On September 22, 2006, Unit 3, Train A, EDG failed to develop output voltage during a

surveillance test. The licensee determined that the same auxiliary contact which failed

in July 2006 was faulty. The licensee identified that this failure was attributed to a bent

metal actuator arm that is used to actuate the auxiliary contacts associated with the field

shorting circuit. Additionally, the licensee determined this bent metal actuator arm

potentially exits in all six EDG's at the facility. Based on previous failures it appears this

bent arm is the underlying root cause for the field shorting auxiliary contacts failure to

operate reliably, and this condition may affect all operating EDG's at the facility.

Attachment 2

A2-2

This Special Inspection Team is chartered to review the generic impact of the relays

bent arms on the other Palo Verde Emergency Diesel Generators as well as any

potential impact on other nuclear plants. The team is also to review the design change

method and reviews that the licensee used when making the relay modifications. The

team will also review the licensees operability determination and corrective action

program for determining the root cause and correction of the diesels failure.

B.

Scope

The team is expected to address the following:

1.

Develop a complete scope of the failures of all Palo Verde Emergency Diesel

Generators to develop an output voltage.

2.

Review the extent of condition determination for this condition (current and prior

K1 relay failures) and whether the licensees actions are comprehensive. This

should include potential for other diesel failures.

3.

Review the licensees determination of the cause of any design deficiencies.

Independently verify key assumptions and facts. If available, determine if the

licensees current and prior root cause analyses and corrective actions have

addressed the extent of condition for problems with the emergency diesel

generators K1 relays.

4.

Determine if the Technical Specifications were met when the diesel failed.

5.

Review and assess the corrective actions for current and past similar failures.

6.

Review the licensees EDG operability determination to evaluate the emergency

diesel generators operability.

7.

Collect data as necessary to support a risk analysis.

8.

Determine if this issue has generic implications to other nuclear facilities.

C.

Guidance

Inspection Procedure 93812, Special Inspection, provides additional guidance to be

used by the Special Inspection Team. Your duties will be as described in Inspection

Procedure 93812. The inspection should emphasize fact-finding in its review of the

circumstances surrounding the event. It is not the responsibility of the team to examine

the regulatory process. Safety concerns identified that are not directly related to the

event should be reported to the Region IV office for appropriate action.

The Team will report to the site, conduct an entrance, and begin inspection no later than

October 4, 2006. While on site, you will provide daily status briefings to Region IV

management, who will coordinate with the Office of Nuclear Reactor Regulation, to

ensure that all other parties are kept informed. A report documenting the results of the

inspection should be issued within 30 days of the completion of the inspection.

Attachment 2

A2-3

This Charter may be modified should the team develop significant new information that

warrants review. Should you have any questions concerning this Charter, contact me at

(817) 860-8248.

Attachment 3

A3-1

SIGNIFICANCE DETERMINATION EVALUATION

Significance determination process Phase 1:

In accordance with NRC Inspection Manual Chapter 0609, Appendix A, Significance

Determination of Reactor Inspection Findings for At-Power Situations, the inspectors

conducted a significance determination Phase 1 screening and determined that the

finding resulted in loss of the safety function of the Train A emergency diesel generator

for greater than the Technical Specification allowed outage time. Therefore, a

Significance Determination Process Phase 2 evaluation was required.

Significance determination process Phase 2:

The Risk-Informed Inspection Notebook for Palo Verde Nuclear Generating Station,

Unit 3, Revision 1, September 2, 2003, was utilized for the Phase 2 evaluation of the

inoperable Train A emergency diesel generator. The following steps and the associated

findings are listed below:

Select or define the applicable initiating event scenarios:

Table 2, Initiators and System Dependency for Palo Verde Nuclear Generating

Station, Units 1, 2, and 3, was reviewed to determine that the loss of offsite

power (LOOP) initiating event scenario was the only scenario that needed to be

analyzed due to the failure of the Train A EDG.

Estimate the likelihood of scenario initiating events and conditions:

The performance deficiency was assumed to exist for 58 days. The Phase 2

analysis assumed the EDG was nonfunctional for an 18-day period, representing

the period from its last successful start on September 4, 2006, to its failed

surveillance on September 22, 2006. Using Table 1, Categories of Initiating

Events for Palo Verde Nuclear Generating Station Unit 3, the initiating event

likelihood for loss of offsite power was determined to be valued at 3. Additional

risk was accrued during the first 40 days of exposure because of a higher

likelihood of failure of the K-1 relay.

Estimate the remaining mitigation capability:

Using the SDP worksheet for a LOOP (Table 3.7, SDP Worksheet for Palo

Verde Nuclear Generating Station, Units 1, 2, and 3 - Loss of Offsite Power

(LOOP)), Sequences 1, 2, 3, 4, and 5, the following results were assigned for

each:

Sequence 1: LOOP-AFW - 6

Sequence 2: LOOP-EAC-REC3 - 8

Sequence 3: LOOP-EAC-TDAFW-REC1 - 8

Sequence 4: LOOP-EAC-SEAL-HPSI - 12

Sequence 5: LOOP-EAC-SEAL-REC1 - 10

Attachment 3

A3-2

Estimate the risk significance of the inspection finding:

NRC Inspection Manual Chapter 0609, Significance Determination Process,

Appendix A, Attachment 1, Counting Rule Worksheet, was utilized using one

sequence that resulted in a value of 6. Since step 10 was greater than zero, the

risk significance of the inspection finding was determined to be at low to

moderate safety significance (White).

Phase 3 Analysis

Description of Performance Deficiencies

On July 25, 2006, the Unit 3 Train A EDG failed to start because of a failed K1 relay.

The last two spare relays obtained from the warehouse were identified to operate

unreliabley. The licensee performed corrective maintenance on one of the relays and

following installation it failed on September 22, 2006. Two performance deficiencies

were identified that contributed to the September 22, 2006, failure. The first

performance deficiency involved the failure to establish appropriate instructions for

performing corrective maintenance activities on an emergency diesel generator K-1

relay. The second performance deficiency involved the failure to identify and correct the

cause of the erratic EDG K-1 relay operation prior to installation of the relay on July 26,

2006.

Assumptions

1.

The Unit 3 Train A EDG demand record is shown in the following table. (A failed bench

test of the new K1 relay was not credited because it may have been due to debris

buildup resulting from a long period of warehousing.)

Date

Demand Result

K-1 Relay

7/25/06

Fail to Start

Old

7/26/06

Successful Start

New

8/07/06

Successful Start

New

8/24/06

Successful Start

New

9/04/06

Successful Start

New

9/22/06

Fail to Start

New

The data was analyzed using three alternate assumptions as follows:

Attachment 3

A3-3

Deterministic Assumption

It is assumed that the as-left condition of the EDG following a run either

predisposes it to a guaranteed success or failure on the next demand. This

assumes that the success or failure on the next attempt is a function of the

as-left condition of the relay following a load run.

Based on this assumption, the Train A EDG was guaranteed to fail to start on

any demand subsequent to the last successful run on September 4, 2006, but

was likewise guaranteed to succeed on any demand prior to this date.

Therefore, the EDG is assumed to be a failed state for 18 days.

Stochastic Assumption

It is assumed that the EDG success or failure is a probabilistic event on any

given demand. This assumes that the relay is more or less in the same state

following each run, but that the as-left tolerances are so close to critical that the

chance of success or failure on the next demand is purely a probabilistic event.

Using this assumption, the Train A EDG was vulnerable to failure on any

demand following the installation of the new K1 relay on July 26, 2006. The

successful start on July 26, 2006, and prior bench/installed test successes and

failures were excluded from the data set because of possible preconditioning

effects. Therefore, three demands and one failure were left. The resulting

assumption is that the Train A EDG would have a 0.25 chance of failing to start

from a K1 relay failure in response to any demand during the 58-day period

between July 26, 2006, and September 22, 2006.

Combination Assumption

It is assumed that a stochastic mechanism existed for the first 40 days, but after

the final successful start on September 4, 2006, the EDG was guaranteed to fail

on its next demand and, therefore, was in a failed state for the final 18 days of

the exposure period.

Using this assumption, the Train A EDG had a 0.25 probability of failing to start

because of a failed K1 relay for the first 40 days and a 1.0 probability of failure

for the final 18 days of the exposure period.

2.

The analyst discarded any risk that may have accrued from September 22, 2005, to July

25, 2006, (the balance of a one-year period) because of a lack of failure history during

this period (as confirmed by a review of surveillance test results).

3.

The analyst modified the current Palo Verde SPAR model (Revision 3.21, October 28,

2005) to reflect the plant-specific LOOP frequencies listed in NUREG/CR-6890,

Reevaluation of Station Blackout Risk at Nuclear Power Plants, Volume 1, Table D-1.

This study comprised an update based on an analysis of offsite power events during

1986-2004. This change to the Palo Verde SPAR model for this specific analysis was

Attachment 3

A3-4

endorsed by a representative of the Idaho National Laboratory (INL), the NRCs

contractor for SPAR model development.

Category of LOOP

Current SPAR Value

Revised Value Based on

NUREG/CR-6890

Plant Centered

2.07E-3/yr.

2.01E-3/yr.

Switchyard Centered

1.04E-2/yr.

9.01E-3/yr.

Grid Related

1.86E-2/yr.

4.40E-2/yr.

Weather Related

4.83E-3/yr.

3.83E-3/yr.

4.

It is assumed that the Unit 3 Train B EDG was not vulnerable to fail from the same

cause as the Train A EDG failure for the entire 58-day exposure period. This is based

on the fact that the relay actuator arm in the K1 relay for this diesel generator was

confirmed to be in a field-straightened configuration during this period. Therefore,

common cause was not invoked in the analysis and a failure probability of 1.0 was used

in lieu of TRUE. This is a key assumption with respect to the risk estimate because

common cause, if presumed, would result in a large increase in the value of the

fail-to-start common cause basic event.

5.

No specific recovery of the Train A EDG was assumed, and no changes were made to

the EDG recovery values in the SPAR model. That is, for this analysis, the analyst

assumed that the recovery probability of the EDGs was nominal.

6.

The SPAR model includes cross-connection capabilities from the other units' diesel

generators. However, the cutsets that include these basic events are very insignificant

in the analysis. Therefore, the analyst did not adjust Unit 1 and 2 EDG common cause

probabilities from the base case.

Internal Events Analysis

The Palo Verde SPAR model (Revision 3.21, October 28, 2005), modified as described

above, was used at a truncation of E-12.

Deterministic Assumption

Based on this assumption, the Train A EDG was guaranteed to fail to start on any

demand subsequent to the last successful run on September 4, 2006, but was likewise

guaranteed to succeed on any demand prior to this date. Therefore, the EDG was

assumed to be in a failed state for 18 days. The use of a "T/2" assumption is not valid

in this case because the normally open contact is assumed to be open following the

previous EDG run. This would cause the K1 unlatch coil to fail to energize for any

hypothetical demand during this period and result in failure of the EDG field flash.

Attachment 3

A3-5

In the SPAR model, the following changes were made:

EPS-DGN-FS-DGA was set to 1.0

The result in SAPHIRE is 1.047E-4/yr. A review of the cutsets revealed that several

included independent failures of Train A EDG to run as well as the test and maintenance

basic event. These cutsets were logically inconsistent:

Cutsets containing:

Value

EPS-DGN-FR-DGA

2.024E-6/yr.

EPS-DGN-TM-DGA

4.084E-7/yr.

Total

2.432E-6/yr.

Extracting these cutsets leaves a result of 1.047E-4 - 2.432E-6 = 1.023E-4/yr.

Therefore, for an 18-day exposure period, the delta-CDF of the finding is

1.023E-4yr.(18/365) = 5.0E-6/yr.

Common cause events were retained in the evaluation case in order to retain the entire

probability of failure of the Train B EDG.

Stochastic Assumption

Using this assumption, the Train A EDG was vulnerable to failure on any demand

following the installation of the new K1 relay on July 26, 2006. As discussed above, the

Train A EDG had a 0.25 probability of failing to start from a K1 relay failure in response

to any demand during the 58-day period of time between July 26, 2006, and

September 22, 2006.

In SPAR, the following changes were made:

EPS-DGN-FS-DGA was set to 0.25 + 5E-3 (base case probability) = 0.255

The result in SAPHIRE is 2.447E-5. For the 58-day exposure period, the resulting

delta-CDF is 2.447E-5 (58/365) = 3.9E-6/yr.

Combination Assumption

Using this assumption, the Train A EDG had a 0.25 probability of failing to start because

of a failed K1 relay for the first 40 days and a 1.0 probability of the same failure for the

final 18 days of the exposure period.

The result from the deterministic assumption is the same in this case. The exposure

time for the stochastic portion is set at 40 instead of 58 days. Using the results above,

the delta-CDF of the combination assumption is 5.0E-6/yr. + 40/58 (3.9.E-6/yr.) =

7.7E-6/yr.

Attachment 3

A3-6

External Events

Seismic

Palo Verde is located in a relatively stable seismic region for a plant located in the

Western USA. The Idaho National Laboratory conducted a study to predict the

frequency of a LOOP resulting from a seismic event at all US plants, as documented in

Frequency of Seismically-Induced LOOP events for SPAR models, Revision 1,

September 2005. The conclusion of this study was that the mean frequency for a

seismically-induced LOOP at Palo Verde is 5.37E-5/yr. The study concluded that the

failure of ceramic insulators would be the most likely failure mode inducing a LOOP.

For risk assessment purposes, a seismically-induced LOOP would have a recovery

profile similar to a severe weather event-induced LOOP. In SPAR (as modified above

for this analysis), the frequency of a weather-related LOOP (used for the internal events

assessment) is 3.83E-3/year. Therefore, the increase in LOOP frequency from seismic

events is not significant by itself as it relates to the risk of this finding.

The analyst also considered the possibility that an earthquake that results in a LOOP

could damage equipment (apart from the diesel generators) that could add non-

negligible risk to the finding. To address this issue, INL produced a document entitled,

Seismic Event Modeling and Seismic Risk Assessment Handbook, Revision 1,

September 2005. In particular, a LOOP resulting from an earthquake that also involved

loss of risk-significant equipment in Train B and/or loss of the gas turbine generators

could result in significant risk despite the low frequency of seismically-induced LOOPs.

Within this document, Table B-1, Generic SSC Seismic Fragilities, provides a list of

components along with the median g-force required to damage them. The following

table lists examples of the equipment of concern and the frequency of earthquakes at

Palo Verde that exceed the threshold value:

Component

High Confidence Low

Probability of Failure

Capacity (g)

Frequency of >g Earthquake

at Palo Verde

Electrical Equipment

(function during seismic

event)

0.34

1.0E-5/yr

Electrical Equipment

(function after seismic event)

0.77

1.0E-8/yr.

Battery Chargers/Inverters

0.54

1.0E-6/yr.

Batteries/Battery Racks

1.3

<9.3E-10/yr.

Diesel Generator/Support

Systems

1.06

<9.3E-10/yr.

Turbine-driven pumps

0.85

1.0E-9/yr.

Attachment 3

A3-7

Equipment success at g-forces well above the HCLPF value is possible. Based on

review of the information provided above and other information in the INL document, the

analyst concluded that earthquakes causing LOOPs and loss of other on-site equipment

would add risk small in comparison to the internal events result.

The analyst assumed (conservatively) that the gas turbine generators would be lost in a

seismic event that also causes a LOOP. To calculate the risk of the finding in light of

this assumption, the analyst ran two cases using the revised SPAR model. In both runs,

the frequency of LOOPs was set at 5.37E-5/year (frequency of seismic-induced

LOOPs), and nonrecoveries of offsite power for all relevant times (3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> and less as

well as 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />) were set to TRUE (there are no offsite recovery events within the E-12

truncation greater than 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> in the Palo Verde SPAR model except for 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />;

therefore, this change set is equivalent to assuming that offsite power following an

earthquake is not recovered). Both gas turbine generator fail-to-start events were set to

TRUE and only LOOP sequences were quantified. In the first case, the EDGs are

assumed to be nominally reliable and available. This result was 1.362E-7/yr. In the

second case, EDG A is assigned a failure probability of 1.0 for the fail-to-start event.

The result was 2.747E-6/yr. The resulting CDF of the finding attributable to a seismic

event that debilitates the gas turbine generators is therefore (2.747E-6 -

1.362E-7)(18/365) = 1.3E-7/yr.

Based on disussions, the licensees PRA assigns a value of approximately 4E-7/yr

delta-CDF for seismic events for a 18-day exposure period.

Fire

The analyst reviewed the licensee's IPEEE for Severe Accident Vulnerabilities,

June 30, 1995, to determine the risk attributable to the finding resulting from internal

fires.

A fire in Room 5B (Train B ESF switchgear room) was considered to have the largest

potential risk to the finding. A fire in this room would possibly cause a loss of offsite

power to both ESF buses. In this case, with a failure of EDG A to start, a station

blackout situation would exist. In the IPEEE, the CDF result for a fire in the Train B ESF

switchgear room was 9.73E-6/yr. The fire ignition frequency for this room was

5.5E-3/yr.

The analyst determined that the IPEEE did not contain sufficient information to quantify

the risk attributable to fires pertaining to the performance deficiency. Therefore, the

analyst requested that the licensee use its fire PRA model for this purpose. The licensee

reported that their fire PRA calculates a delta-CDF of 3.716E-6/yr for EDG A being

nonfunctional versus the base case condition. For this case, the EDG failure was

assumed to be independent in nature, the same assumption used in the SPAR analysis.

For an 18-day exposure, this would result in a delta-CDF of 1.8E-7/yr.

Attachment 3

A3-8

As a comparison, the following table shows the differences in the treatment of Room 5B

between the IPEEE and the fire PRA.

Room 5B

IPEEE

Fire PRA

Fire Ignition Frequency

5.5E-3/yr.

4.16E-3/yr.

CDF

9.73E-6/yr.

1.17E-7/yr.

This example shows that the CCDP of a fire in this room decreased from 1.8E-3 in the

IPEEE to 2.8E-5 in the fire PRA. This difference is not surprising because the IPEEE

was basically a screening tool that assumed worst-case bounding conditions while the

fire PRA incorporated realistic, best-estimate approximations.

Internal Flooding/High Velocity Winds/Other External Events

The analyst concluded qualitatively that no other external events would add appreciably

to the risk of the finding. The licensee reported that the risk added from internal flooding

according to their model was in the E-8 range.

Combined Risk

Using the licensee analysis results for seismic and fire events, the following table

indicates the total estimated risk of the finding for each of the assumed failure

mechanisms:

Assumption

Internal

Events

Seismic

Fire

Total Risk

Deterministic

5.0E-6

4E-7/yr

1.3E-7/yr

5.5E-6/yr

Stochastic

3.9E-6

3E-7/yr1

1.0E-7/yr1

4.3E-6/yr

Combination

7.7E-6

6E-7/yr1

2.1E-7/yr1

8.5E-6/yr

1. Seismic and fire CDFs were adjusted for the stochastic and combination

assumptions.

Large Early Release Frequency

In accordance with IMC 0609, Appendix H, station blackout sequences, which

predominate the risk of the assessed condition, are not considered significant release

events for a large, dry containment. Therefore, large early release was considered

unimportant in this analysis.

Licensee Analysis

The analyst did not receive a detailed description of the licensees analysis, but was

informed verbally that the delta-CDF of the finding for internal events and fire, assuming

an 18-day exposure and no recovery of the Train A EDG, was approximately 1.6E-6/yr.

Attachment 3

A3-9

Adding the licensees approximate seismic risk, the overall result would be

approximately 2.0E-6/yr.

References

Palo Verde SPAR model (Revision 3.21, October 28, 2005)

NUREG/CR-6890, Reevaluation of Station Blackout Risk at Nuclear Power Plants,

Volume 1, Table D-1

Seismic Event Modeling and Seismic Risk Assessment Handbook, Revision 1,

September 2005

Palo Verde IPEEE for Severe Accident Vulnerabilities, June 30, 1995

Palo Verde Fire PRA Overview and Results, 13-NS-C072