ML072950104

IR 05000361-07-013, 05000362-07-013, on 06/27/07 - 07/02/07, San Onofre Nuclear Generating Station, Units 2, 3, and Independent Spent Fuel Storage Installation; Special Inspection in Response to an Instrument Air Header Break and Unit 2 Tri

ML072950104
Person / Time
Site:	San Onofre
Issue date:	10/19/2007
From:	Clark J NRC/RGN-IV/DRP/RPB-E
To:	Rosenblum R Southern California Edison Co
References
IR-07-013
Download: ML072950104 (76)

See also: IR 05000361/2007013

Text

October 19, 2007

Richard M. Rosenblum

Senior Vice President and

Chief Nuclear Officer

Southern California Edison Company

San Onofre Nuclear Generating Station

P.O. Box 128

San Clemente, CA 92674-0128

SUBJECT:

SAN ONOFRE NUCLEAR GENERATING STATION - NRC SPECIAL

INSPECTION REPORT 05000361/2007013; 05000362/2007013

Dear Mr. Rosenblum:

On September 13, 2007, the U.S. Nuclear Regulatory Commission (NRC) completed a special

inspection at your San Onofre Nuclear Generating Station facility. This inspection examined

activities associated with the loss of instrument air event on June 20, 2007. On this occasion,

instrument air pressure on Unit 2 dropped significantly, causing the feedwater control valves to

stop functioning and resulting in an increase in steam generator water level. Operators

manually tripped the Unit 2 reactor. The NRC's initial evaluation satisfied the criteria in NRC

Management Directive 8.3, NRC Incident Investigation Program, for conducting a special

inspection. The basis for initiating this special inspection is further discussed in the inspection

charter, which is included in this report as Attachment 2. The determination that the inspection

would be conducted was made by the NRC on June 26, 2007, and the inspection started on

June 27, 2007.

The enclosed inspection report documents the inspection findings, which were discussed on

September 13, 2007 and again on October 11, 2007, with members of your staff. The

inspection examined activities conducted under your license as they relate to safety and

compliance with the Commission's rules and regulations and with the conditions of your license.

The inspectors reviewed selected procedures and records, observed activities, and interviewed

personnel.

The report documents eight NRC identified and self-revealing findings of very low safety

significance (Green). The eight findings involved issues concerning both the failure of your

processes and programs to prevent or mitigate the loss of instrument air event, and the

subsequent failure of your staff to thoroughly evaluate operator and equipment responses

following the event. The NRC is concerned about the occurrence of this event and the less

than adequate reviews conducted by your staff, and will conduct followup baseline inspections

to verify that your corrective actions in response to this inspection are thorough and effective.

Five of the findings were determined to involve violations of NRC requirements. Because of

their very low safety significance and because they were entered into your corrective action

program, the NRC is treating these findings as noncited violations (NCVs) consistent with

Section VI.A.1 of the NRC Enforcement Policy. If you contest these NCVs, you should provide

Southern California Edison Company

- 2 -

a response within 30 days of the date of this inspection report, with the basis for your denial, to

the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington

DC 20555-0001; with copies to the Regional Administrator, U.S. Nuclear Regulatory

Commission Region IV, 611 Ryan Plaza Drive, Suite 400, Arlington, Texas, 76011-4005; the

Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington

DC 20555-0001; and the NRC Resident Inspector at the San Onofre Nuclear Generating

Station facility.

In accordance with 10 CFR 2.390 of the NRC's Rules of Practice, a copy of this letter, its

enclosure, and your response (if any) will be made available electronically for public inspection

in the NRC Public Document Room or from the Publicly Available Records (PARS) component

of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at

http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Jeffrey A. Clark, Chief

Projects Branch E

Division of Reactor Projects

Dockets: 50-361

50-362

License: NPF-10

NPF-15

Enclosure: Inspection Report 05000361/2007013; 05000362/2007013

Attachment 1: Supplemental Information

Attachment 2: Special Inspection Charter

Attachment 3: Significance Determination Evaluation

cc w/Enclosure:

Chairman, Board of Supervisors

County of San Diego

1600 Pacific Highway, Room 335

San Diego, CA 92101

Gary L. Nolff

Assistant Director-Resources

City of Riverside

3900 Main Street

Riverside, CA 92522

Mark L. Parsons

Deputy City Attorney

City of Riverside

3900 Main Street

Riverside, CA 92522

Dr. David Spath, Chief

Division of Drinking Water and

Environmental Management

California Department of Health Services

850 Marina Parkway, Bldg P, 2nd Floor

Richmond, CA 94804

Michael J. DeMarco

San Onofre Liaison

San Diego Gas & Electric Company

8315 Century Park Ct. CP21G

San Diego, CA 92123-1548

Director, Radiological Health Branch

State Department of Health Services

P.O. Box 997414 (MS 7610)

Sacramento, CA 95899-7414

Southern California Edison Company

- 3 -

Mayor

City of San Clemente

100 Avenida Presidio

San Clemente, CA 92672

James D. Boyd, Commissioner

California Energy Commission

1516 Ninth Street (MS 34)

Sacramento, CA 95814

Douglas K. Porter, Esq.

Southern California Edison Company

2244 Walnut Grove Avenue

Rosemead, CA 91770

Mr. Raymond W. Waldo, Vice President,

Nuclear Generation

Southern California Edison Company

San Onofre Nuclear Generating Station

P.O. Box 128

San Clemente, CA 92674-0128

A. Edward Scherer

Southern California Edison Company

San Onofre Nuclear Generating Station

P.O. Box 128

San Clemente, CA 92674-0128

Brian Katz

Southern California Edison Company

San Onofre Nuclear Generating Station

P.O. Box 128

San Clemente, CA 92674-0128

Mr. Steve Hsu

Department of Health Services

Radiologic Health Branch

MS 7610, P.O. Box 997414

Sacramento, CA 95899-7414

Mr. James T. Reilly

Southern California Edison Company

San Onofre Nuclear Generating Station

P.O. Box 128

San Clemente, CA 92674-0128

Southern California Edison Company

- 4 -

Electronic distribution by RIV:

Regional Administrator (EEC)

DRP Director (ATH)

DRS Director (DDC)

DRS Deputy Director (RJC1)

Senior Resident Inspector (CCO1)

Branch Chief, DRP/E (JAC)

Senior Project Engineer, DRP/E (GDR)

Team Leader, DRP/TSS (CJP)

RITS Coordinator (MSH3)

Only inspection reports to the following:

DRS STA (DAP)

V. Dricks, PAO (VLD)

D. Pelton, OEDO RIV Coordinator (DLP)

ROPreports

SO Site Secretary (vacant)

SUNSI Review Completed: _JAC__ ADAMS:  : Yes

G No Initials: __JAC_

Publicly Available G Non-Publicly Available G Sensitive

Non-Sensitive

R:\\_REACTORS\\SO\\2007\\SO2007-13RP-GBM.wpd

RIV:SRI:DRP/C

RI:DRP/E

SRI:DRP/E

SRA:DRS

C:DRP/E

GBMiller

JEJosey

CCOsterholtz

DPLoveless

JAClark

/RA/

T-GBM

E=GBM

/RA/

/RA/

10/16 /07

10/17/07

10/16/07

10/17/07

10/19/07

OFFICIAL RECORD COPY

T=Telephone E=E-mail F=Fax

Enclosure

-1-

U.S. NUCLEAR REGULATORY COMMISSION

REGION IV

Docket:

50-361, 50-362

Licenses:

NPF-10, NPF-15

Report No.:

05000361/2007013; 05000362/2007013

Licensee:

Southern California Edison Co. (SCE)

Facility:

San Onofre Nuclear Generating Station, Units 2, 3

Location:

5000 S. Pacific Coast Hwy.

San Clemente, California

Dates:

June 27 through September 13, 2007

Inspectors:

J. Josey, Resident Inspector, Project Branch E, DRP

D. Loveless, Senior Reactor Analyst

G. Miller, Senior Resident Inspector, Project Branch C, DRP

C. Osterholtz, Senior Resident Inspector, Project Branch E, DRP

M. Sitek, Resident Inspector, Project Branch E, DRP

Approved By:

Jeffrey A. Clark, Chief

Project Branch E

Division of Reactor Projects

Enclosure

-2-

SUMMARY OF FINDINGS

IR 05000361/2007013, 05000362/2007013; 06/27/07 - 07/02/07; San Onofre Nuclear

Generating Station, Units 2, 3, and Independent Spent Fuel Storage Installation; Special

Inspection in response to an instrument air header break and Unit 2 trip on June 20, 2007.

The report covered a 6-day period (June 27 - July 2, 2007) of onsite inspection, with inoffice

review through September 13, 2007, by a special inspection team consisting of one senior

resident inspector, one resident inspector, and one senior reactor analyst. Eight findings were

identified. The significance of most findings is indicated by its color (Green, White, Yellow, or

Red) using Inspection Manual Chapter 0609, Significance Determination Process. Findings

for which the significance determination process does not apply may be Green or be assigned a

severity level after NRCs management review. The NRC's program for overseeing the safe

operation of commercial nuclear power reactors is described in NUREG-1649, Reactor

Oversight Process, Revision 3, dated July 2000.

Summary of Event

The NRC conducted a special inspection to better understand the circumstances surrounding

an instrument air header break and Unit 2 trip on June 20, 2007. In accordance with NRC

Management Directive 8.3, NRC Incident Investigation Program, it was determined that this

event involved multiple failures in systems used to mitigate the effects of an actual event,

involved potential adverse generic implications, and had sufficient risk significance to warrant a

special inspection.

A.

NRC-Identified and Self-Revealing Findings

Cornerstone: Initiating Events

Green. The inspectors reviewed a self-revealing Green finding involving

ineffective corrective actions taken in response to site and industry operating

experience with instrument air header ruptures. Specifically, contrary to

Section 6.2.3 of Procedure SO-123-I-1.42, Maintenance Division Experience

Report, Revision 0, the licensee failed to implement corrective actions to prevent

recurrence for an equipment failure with the potential to cause a significant plant

transient, and failed to appropriately consider previous industry and plant

experience similar to the event. Additionally, licensee personnel failed to properly

evaluate and take corrective actions based on industry operating experience

through 2006 involving improperly made soldered joints in instrument air systems.

As a result, an additional failure of an improperly made instrument air header joint

occurred at SONGS on June 20, 2007. The licensee entered this issue in their

corrective action program as Action Request AR 070600867.

This finding was more than minor since it was associated with the equipment

reliability attribute of the initiating events cornerstone and affected the cornerstone

objective to limit the likelihood of events that upset plant stability and challenge

critical safety functions. This finding required a Phase 2 analysis per the Manual

Chapter 0609, Significance Determination Process, Phase 1 Worksheets since

the loss of instrument air is a transient initiator resulting in the loss of the

feedwater system which is part of the power conversion system which can be used

to mitigate the consequences of an accident. Based on the results of the Phase 2

Enclosure

-3-

analysis and a subsequent Phase 3 analysis, the finding was determined to be of

very low safety significance (Green) because of the availability of the diverse

auxiliary feedwater system and the ability of the operators to depressurize the

steam generators and utilize the condensate system for heat removal. These

results were evaluated by a senior reactor analyst. This finding has a crosscutting

aspect in the area of problem identification and resolution associated with

operating experience in that the licensee failed to effectively implement changes to

station processes, procedures, and equipment in response to operating

experience involving improperly made instrument air system joints P.2(b).

(Section 2.1)

Green. The inspectors identified a Green noncited violation of Technical Specification 5.5.1.1 involving the failure to meet procedural requirements

following a loss of instrument air. Specifically, operators failed to monitor nitrogen

tank levels or take precautions for the possibility of oxygen-deficient areas in the

plant following actuation of the low pressure backup nitrogen system. The

licensee entered this issue in their corrective action program as Action

Request AR 070700291.

This finding was more than minor since it was associated with the human

performance attribute of the initiating events cornerstone and affected the

cornerstone objective to limit the likelihood of events that upset plant stability and

challenge critical safety functions. This finding required a Phase 2 analysis in

accordance with the Manual Chapter 0609, Significance Determination Process,

Phase 1 Worksheets since the loss of instrument air is a transient initiator resulting

in the loss of the feedwater system which is part of the power conversion system

which can be used to mitigate the consequences of an accident. Based on the

results of the Phase 2 analysis, the finding was determined to be of very low safety

significance because of the low likelihood of a complete loss of instrument air and

the availability of the auxiliary feedwater system. The cause of this finding has a

crosscutting aspect in the area of human performance associated with resources

because licensee personnel were not adequately trained on the operation of the

low pressure nitrogen system to effectively implement the abnormal operating

instruction H.2(b). (Section 2.2)

Cornerstone: Mitigating Systems

Green. A self-revealing, Green noncited violation of 10 CFR Part 50, Appendix B,

Criterion III, Design Control, was identified when Unit 2 experienced a loss of

instrument air due to the failure of a soldered joint. Specifically, the loss of

instrument air resulted in component cooling water (CCW) Pump 024 being in a

runout condition for approximately 75 minutes due to a previous system

modification. The licensee entered this issue in their corrective action program as

Action Requests AR 070700051 and 070600872.

This finding was greater than minor because it was associated with the mitigating

systems cornerstone attribute of design control and affected the associated

cornerstone objective to ensure the availability, reliability, and capability of systems

that respond to initiating events to prevent undesirable consequences. The finding

did not affect the initiating events cornerstone functions of the component cooling

water system because the condition would only have existed given a loss of

Enclosure

-4-

instrument air initiator had already occurred. In accordance with NRC Inspection

Manual Chapter 0609, Appendix A, Phase 1 Worksheet, Significance

Determination Process (SDP) Phase 1 Screening Worksheet for the Initiating

Events, Mitigating Systems, and Barriers Cornerstones, this finding was

determined to be of very low safety significance because the finding was a design

deficiency confirmed not to result in a loss of operability per Part 9900, Technical

Guidance, Operability Determination Process for Operability and Functional

Assessment. (Section 2.3)

Green. The inspectors reviewed a self-revealing Green finding involving the failure

to take effective corrective actions for a failed control room annunciator.

Specifically, after the annunciator for actuation of the backup nitrogen supply to

the instrument air system failed to function on demand on several occasions from

1994 through 2007, the corrective actions taken by the licensee to restore the

annunciator to service were inadequate and narrowly focused. The annunciator

subsequently failed to function during the loss of instrument air event on

June 20, 2007. The licensee entered this issue in their corrective action program

as Action Request AR 070601250.

This finding was more than minor since it was associated with the human

performance attribute of the initiating events cornerstone and affected the

cornerstone objective to limit the likelihood of events that upset plant stability and

challenge critical safety functions. This finding required a Phase 2 analysis in

accordance with the Manual Chapter 0609, Significance Determination Process,

Phase 1 Worksheets since the loss of instrument air is a transient initiator resulting

in the loss of the feedwater system which is part of the power conversion system

which can be used to mitigate the consequences of an accident. Based on the

results of the Phase 2 analysis, the finding was determined to be of very low safety

significance because of the low likelihood of a complete loss of instrument air and

the availability of the auxiliary feedwater system. This finding has a crosscutting

aspect in the area of problem identification and resolution associated with the

corrective action program in that the licensee failed to thoroughly evaluate the

failed annunciator such that the resolution appropriately addressed the causes

P.2(c). (Section 2.4)

Green. The inspectors identified a Green noncited violation of Technical Specification 5.5.1.1 involving the failure to maintain an adequate abnormal

operating instruction for a loss of instrument air event. The licensee entered this

issue in their corrective action program as Action Request AR 070801151.

This finding was more than minor because it was associated with the procedure

quality attribute of the mitigating systems cornerstone and affected the

cornerstone objective to ensure the availability, reliability and capability of systems

that respond to initiating events, in that a less than adequate abnormal operating

procedure could have prevented operators from promptly tripping the reactor,

allowing conditions to continue to degrade and resulting in a demand on the

reactor protection system. Using the Significance Determination Process Phase 1

Screening Worksheet in Appendix A of Inspection Manual Chapter 0609, the

inspectors determined this finding had very low safety significance because it did

not result in an actual loss of safety function per Part 9900, Technical Guidance,

Operability Determination Process for Operability and Functional Assessment.

Enclosure

-5-

This finding has a crosscutting aspect in the area of human performance

associated with resources in that the licensee failed to provide operators with

complete, accurate, and up-to-date procedures H.2(c). (Section 2.5)

Green. A self-revealing, Green noncited violation of 10 CFR Part 55.46(c)(1) was

identified involving the licensees failure to incorporate a design change in

modeling plant response for the plant-referenced simulator. Specifically, during

operator training in the plant-referenced simulator, the controlled bleedoff valves

for the reactor coolant pumps were modeled to fail closed on a loss of instrument

air, whereas the valves in the plant remained open during an actual loss of

instrument air event on June 20, 2007. The licensee entered this issue in their

corrective action program as Action Requests AR 070600873 and 070900160.

This finding was greater than minor because it was associated with the mitigating

systems cornerstone attribute of human performance and affected the associated

cornerstone objective to ensure the availability, reliability, and capability of systems

that respond to initiating events to prevent undesirable consequences. The

inspectors evaluated this finding using the Appendix I, Licensed Operator

Requalification Significance Determination Process worksheets of Manual

Chapter 0609 because the finding is a requalification training issue related to

simulator fidelity. The finding is of very low safety significance because the

discrepancy did not have an adverse impact on operator actions such that safety

related equipment was made inoperable during normal operations or in response

to a plant transient. This finding has a crosscutting aspect in the area of human

performance associated with resources in that the licensee did not provide

operators with adequate facilities and equipment for use in operator training

H.2(d). (Section 2.6)

Green. The inspectors identified a Green noncited violation of Technical Specification 5.5.1.1 involving the failure to meet procedural requirements

governing impaired annunciators. Specifically, after the identification of a failed

annunciator, operators did not enter the annunciator in the failed annunciator log

or mark the affected annunciator window with an annunciator compensatory action

flag. The licensee entered this issue in their corrective action program as Action

Request AR 070700291.

This finding was more than minor since it was associated with the human

performance attribute of the initiating events cornerstone and affected the

cornerstone objective to limit the likelihood of events that upset plant stability and

challenge critical safety functions. This finding required a Phase 2 analysis in

accordance with the Manual Chapter 0609, Significance Determination Process,

Phase 1 Worksheets since the loss of instrument air is a transient initiator resulting

in the loss of the feedwater system which is part of the power conversion system

which can be used to mitigate the consequences of an accident. Based on the

results of the Phase 2 analysis, the finding was determined to be of very low safety

significance because of the low likelihood of a complete loss of instrument air and

the availability of the auxiliary feedwater system. This finding has a crosscutting

aspect in the area of human performance associated with resources because the

operators were not sufficiently trained to consistently implement the annunciator

operating procedure H.2(b). (Section 2.7)

Enclosure

-6-

Green. A Green self-revealing finding was identified associated with the failure of

the reactor coolant pump controlled bleed off valve to shut during a loss of

instrument air event. The licensee failed to adequately implement corrective

actions from previously evaluated industry operating experience for new valve

regulators that were installed in the unit. The licensee entered this issue in their

corrective action program as Action Request AR 070600873.

The finding was greater than minor because it was associated with the mitigating

systems cornerstone attribute of design control and affected the associated

cornerstone objective to ensure the availability, reliability, and capability of systems

that respond to initiating events to prevent undesirable consequences. Using

Manual Chapter 0609, Significance Determination Process, Phase 1 Worksheet,

the finding is determined to have very low safety significance because the

condition only affected the mitigation systems cornerstone and it was confirmed

not to result in loss of operability per Part 9900, Technical guidance, Operability

Determination Process for Operability and Functionality Assessment

(Section 2.8).

B.

Licensee-Identified Violations

None.

Enclosure

-7-

REPORT DETAILS

1.0

SPECIAL INSPECTION SCOPE

The NRC conducted a special inspection at San Onofre Generating Station (SONGS) to

better understand the circumstances surrounding the loss of instrument air event on

June 20, 2007. On this occasion, instrument air pressure on Unit 2 dropped significantly

following the separation of a 3-inch air header in the auxiliary building. This caused the

feedwater control valves to stop functioning, resulting in an uncontrolled increase in

steam generator water level. Operators manually tripped the Unit 2 reactor. In

accordance with NRC Management Directive 8.3, it was determined that this event had

sufficient risk significance to warrant a special inspection.

The team used NRC Inspection Procedure 93812, Special Inspection Procedure, to

conduct the inspection. The special inspection team reviewed procedures, corrective

action documents, operator logs, design documentation, maintenance records, and

procurement records for the instrument air system. The team interviewed various

station personnel regarding the event. The team reviewed the licencees preliminary

root cause analysis report, past failure records, extent of condition evaluation,

immediate and long term corrective actions, and industry operating experience. A list of

specific documents reviewed is provided in Attachment 1. The charter for the special

inspection is included as Attachment 2.

1.1

Event Summary

During full power operation on June 20, 2007, a 3-inch diameter instrument air line failed

at an improperly soldered joint on the Unit 2 instrument air header. The joint completely

separated, resulting in a double-ended guillotine shear of the supply header and a

complete loss of instrument air to Unit 2. The loss of instrument air pressure caused the

feedwater control valves to stop functioning, and operators manually tripped the Unit 2

reactor as a result of an uncontrolled steam generator water level increase. Although

instrument air is a shared system at SONGS, a backup nitrogen system can support

system loads on the unaffected unit following a pipe break via excess flow check valves.

As a result, the pressure drop on Unit 3 was not as significant during the event and

operators maintained control of all functions.

Operators located the failed piping in the Unit 2 turbine building and were able to isolate

the break approximately thirty minutes after the event began. Operators applied a

temporary repair to the break and restored instrument air header pressure. Subsequent

investigations identified 32 additional leaking instrument air fittings in Unit 2 and Unit 3,

possibly as a result of improper joint fabrication during initial construction. Maintenance

personnel placed structural clamps on the leaking fittings to prevent additional piping

separations until permanent repairs could be made.

The time line below describes the major events following the separation of the

instrument air header fitting on June 20, 2007.

Enclosure

-8-

June 20, 2007

2244

Instrument air dryer Temp/Level/DP HI alarm received.

Control room instrument air header pressure noted to be 80 psig and lowering.

Instrument air pressure low alarm received on Unit 2. (90 psig setpoint)

Operators entered Procedure SO23-13-5, Loss of Instrument Air.

2245

Unit 2 air operated valves begin to move on their own.

Full Flow condensate polisher demineralizer bypass valves open.

Chemical volume control system letdown flow isolates.

Unit 3 instrument air pressure low alarm received. (90 psig setpoint)

2247

Steam Generator level noted to be 82% and rising in generator E088 on Unit 2.

Operators secured charging pumps due to loss of letdown and began manually

controlling pressurizer level.

Operators bypassed instrument air dryers. Indicated instrument air header

pressure in the control room increases from 42 psig to 67 psig.

2248

Heater drain pump P059 trips.

2250

Heater drain pump P058 trips.

E088 level approaching trip set point, manually tripped the Unit 2 reactor.

Entered Procedure SO23-12-1, Standard Post Trip Actions.

2252

Steam generator E088 level exceeds 100%.

2253

Operators manually tripped both main feed pumps.

2254

Operators initiated both trains of auxiliary feedwater. (EFAS)

Steam bypass control system responding sluggishly; both reactor temperature

and pressure slightly higher than expected. Operators begin controlling pressure

and temperature using one atmospheric dump valve.

2258

Steam generator E088 level returns to less than 100%.

2303

Entered Procedure SO23-12-2, Reactor Trip Recovery.

2321

Location of instrument air header rupture identified and isolated using manual

valves. Instrument air header pressure indicated in the control room immediately

recovers from 67 psig to 108 psig (normal operating pressure). Instrument air

dryer Temp/Level/DP HI and instrument air low pressure alarms clear.

Temporary repair (soft patch) put in place on the instrument air header.

2329

Component Cooling Water (CCW) Pump A noted to be in a runout condition,

operators started CCW Pump B.

0024

NRC notified of unit trip due to uncontrolled level rise in steam generator E088

upon loss of instrument air.

0030

Procedure SO23-12-1, Loss of Instrument Air, exited.

Enclosure

-9-

1.2

Operator Response

The team assessed the response of the control room operators to the loss of instrument

air. The team reviewed operator logs, plant computer data, and strip charts to evaluate

operator performance in coping with the event and transient; verified that operator

actions were in accordance with the response required by plant procedures and training;

and verified that the licensee identified and implemented appropriate corrective actions

associated with personnel performance problems that occurred during the event. The

team also conducted interviews with each of the control room operators who were on

shift the night of the event.

The team concluded the operators acted appropriately to manually trip the Unit 2 reactor

and turbine and place the unit in a safe condition. The inspectors also concluded the

operators acted promptly and appropriately in recovering the instrument air system and

in maintaining Unit 3 at power. However, the team also identified several opportunities

for improvement in some aspects of operator response and training associated with the

event.

Through interviews with control room personnel, the inspectors noted a general

weakness in the operators understanding of the design and integrated operation of the

instrument air and low pressure nitrogen systems. As an example, several operators

erroneously stated that the respiratory/service air system was supporting the Unit 3

instrument air loads during the event, when in fact the Unit 3 loads were being supplied

by the backup nitrogen system. Several operators also believed their efforts to bypass

the instrument air filters and to place an additional dryer in service had a positive effect

on restoring instrument air pressure to Unit 2, when in reality the Unit 2 instrument air

header pressure was not recoverable due to the complete separation of the pipe header

from the supply lines.

The inspectors concluded the operators understanding of the event on June 20, and

their ability to diagnose and respond to future events involving a loss of instrument air,

were complicated by the sparse control room instrumentation provided for the

instrument air system. Specifically, operators in both control rooms are provided with

one indication each for respiratory/service air supply pressure, backup nitrogen system

supply pressure, and instrument air supply pressure. There are no indications for actual

air header pressure at the system loads for either Unit 2 or Unit 3. Additionally, the

control room indications provided real-time pressure indication only; there were no strip

charts recorders to allow prompt diagnosis of pressure trends, nor were there any

computer points available to provide pressure indication, tracking, or trending

information to the control room operators.

The inspectors reviewed Procedure SO23-13-5, Loss of Instrument Air, Revision 5,

which was the abnormal operating instruction used by the operators to respond to the

loss of instrument air pressure on June 20, 2007. The inspectors concluded that given

the limited data available to plant operators in the control rooms, the abnormal operating

instruction did not provide sufficient guidance to ensure operators would be able to take

prompt action to mitigate the effects of a loss of instrument air in all circumstances. The

inspectors determined the failure to maintain an adequate operating instruction to

respond to a loss of instrument air was a violation of Technical Specification 5.5.1.1.

This finding is described further in Section 2.5 of this report.

Enclosure

-10-

Through review of operator logs, the marked-up copy of the abnormal operating

instruction for loss of instrument air used during the event, and interviews of operators,

the inspectors identified that operators had failed to take the required actions specified

in the abnormal operating instruction for actuation of the backup nitrogen system.

Specifically, the operators did not take steps to monitor for oxygen-deficient areas of the

plant caused by nitrogen leakage from the instrument air system and did not begin

monitoring nitrogen tank levels. The inspectors also noted that control room

Annunciator 61B38, N2 SUPPLY TO INST AIR HEADER ON, failed to alarm during

the event, which, when combined with the aforementioned weak operator understanding

of the system and limited control room instrumentation, likely contributed to the

operators failure to take the actions of the abnormal operating instruction. The

inspectors noted the failure to take these actions had the potential to result in operator

injury or death from entering oxygen-deficient areas in the plant. The inspectors

determined the failure to follow the requirements of the abnormal operating instruction

was a violation of Technical Specification 5.5.1.1. This finding is discussed further in

Section 2.2 of this report.

The inspectors examined the post-trip review package assembled by the licensee

following the trip of Unit 2. The inspectors noted the post-trip review package

appropriately addressed the response and operation of safety-related plant equipment

to the event. The post-trip review also properly identified operator performance issues

associated with a missed surveillance requirement and implemented appropriate

corrective actions. The inspectors concluded the licensees post-trip review was

adequate per the guidance of Generic Letter 83-28, Required Actions Based on

Generic Implications of Salem ATWS Events. However, the inspectors identified some

weaknesses in the scope and thoroughness of the review in regard to the nonsafety-

related aspects of the event. For example, the post trip review did not identify the failure

of Annunciator 61B38 to alarm as mentioned above and described further in Section 1.4

of this report. The review package also contained a typographical error identified by the

inspectors that, had the recorded value been correct, would have indicated that the

reactor trip circuit breakers had failed to open within their design time limit. The review

also did not address the weaknesses in operator understanding of the event or the

failure of operators to follow the requirements of the abnormal operating instruction for

actuation of the backup nitrogen system as described above.

The licensee initiated a root cause evaluation to assess the above issues related to the

operator response and post-trip review for the June 20, 2007, loss of instrument air

event as part of Action Request AR 070700291.

1.3

Instrument Air System Interactions

The instrument air system at SONGS consists of three motor driven instrument air

compressors, two parallel air dryers and four parallel air filters, all of which are located in

the turbine building. Backup pressure sources for instrument air are provided by the low

pressure nitrogen system and the respiratory/service air system. The instrument air

system is designed to provide a continuous supply of filtered, dried, and essentially oil-

free air for pneumatic instruments and valves in both units.

During normal system operation, one of the compressors is in continuous operation

while the other two compressors are in standby. The standby compressors will start and

stop automatically as required to supplement the running compressor to meet system

Enclosure

-11-

demand. The instrument air header is divided between the two units by check valves

installed in the supply headers and in the unit crossover header in the radwaste building.

Non-safety related nitrogen supply lines with isolating valves and excess flow check

valves are located downstream of the unit check valves in the air supply lines to provide

a backup nitrogen supply for each units instrument air header. The excess flow check

valves isolate on high flow, which prevents a failure in one units air piping from causing

an excessive instrument air pressure drop in the other unit. A second backup supply for

the instrument air system is provided by the respiratory/service air system. The

respiratory/service air system is connected to the instrument air supply lines upstream of

the instrument air dryers.

The instrument air system supplies the motive force to all pneumatically operated valves

and instruments in both units. All pneumatically operated valves are designed to fail to

their safe position on a loss of instrument air. Pneumatic valves with a safety function

are described in Table 9.3-1 of the SONGS Final Safety Analysis Report (FSAR) and

include: saltwater cooling system isolation valves and lubrication valves, component

cooling water isolation valves, shutdown cooling heat exchanger isolation valves, safety

injection line check valve leakoff line isolation valves, safety injection tank fill and drain

lines, and auxiliary feedwater pump steam supply valves. Significant non-safety related

pneumatic valves include the chemical volume control system letdown isolation valves,

pressurizer normal spray valves, main feedwater regulating valves, steam bypass

control system valves, reactor coolant pump seal controlled bleedoff isolation valves,

and component cooling water noncritical loop isolation valves.

1.4

Plant Response

The inspectors reviewed operator logs, alarm history, and available trend information to

evaluate the plant response to the loss of instrument air header pressure to ensure that

all systems responded as designed. The inspectors concluded the instrument air

system functioned as described in the FSAR. Following the break in the Unit 2 air

header, the excess flow check valve in the backup nitrogen supply line to Unit 2 closed

to isolate the break and successfully mitigated the effect of the transient on Unit 3 as

designed. The inspectors also concluded the integrated plant response to the overall

transient also occurred as described in the FSAR, with some exceptions as noted below.

In the post trip review package, the licensee noted excessive flow existed in the

component cooling water (CCW) system for approximately 75 minutes following the

event, which placed the CCW Pump A in a runout condition. The excess system flow

resulted when the shutdown cooling heat exchanger isolation valve failed open as

designed on the loss of instrument air pressure. Although in the original plant design

the CCW noncritical loop isolation valves failed shut on a loss of instrument air pressure

to isolate the shutdown cooling heat exchanger from the system, the licensee installed a

modification in 1995 to allow the noncritical loop isolation valves to remain open in order

to maintain cooling water for the reactor coolant pumps. Consequently, the opening of

the shutdown cooling heat exchanger isolation valve placed an additional load on the

system in excess of the capacity of the operating CCW pump. The inspectors

determined this was a violation of 10 CFR Part 50, Appendix B, Criterion III, Design

Control. This finding is discussed further in Section 2.3 of this report.

In the post-trip review package and during interviews with the inspectors, the operators

noted the controlled bleed off valve for the reactor coolant pump seals remained open

Enclosure

-12-

following the loss of instrument air. During interviews with the inspectors, the operators

stated that in operator training in the site simulator, the CBO valve always failed shut on

a loss of instrument air pressure, requiring the operators to trip the reactor coolant

pumps to ensure the integrity of the reactor coolant pump seals. As a result, when the

CBO valve indicated open during the event on June 20, 2007, control room operators

requested local, independent verification of the actual position of the CBO valve. The

inspectors concluded the discrepancy between actual plant response and that modeled

in the simulator negatively impacted operator response to the loss of instrument air

event. The inspectors determined this simulator fidelity issue was a violation of 10 CFR Part 55.46. This finding is discussed further in Section 2.6 of this report.

In reviewing the cause for the failure of the CBO valve to close on the loss of instrument

air pressure, the licensee determined that the regulator for the valve had been replaced

with a new style regulator in February 2004. The new style regulator was installed

because the original model had become obsolete. Whereas air pressure would leak off

the original model regulator causing the associated valve to close on a loss of

instrument air, the new regulator contained improved seals that locked in air pressure

and allowed the associated valve to remain open. Although the licensees substitution

equivalency evaluation required a design change impact review prior to installing the

new model regulators in the plant, the engineers performing the impact review for the

CBO valve failed to review the FSAR and so did not identify that the CBO valves were

designed to fail closed on a loss of instrument air. A finding associated with the failure

to perform an adequate design change impact review is discussed further in Section 2.8

of this report.

During the loss of instrument air event, Annunciator 61B38, N2 SUPPLY TO INST AIR

HEADER ON, failed to alarm, complicating operating understanding of and response to

the event as described in Section 1.2 of this report. The licensee initiated Action

Request AR 070601250 to address the failed annunciator. While in the control room

three days later, the inspectors noticed there were no labels, warning flags, or other

devices affixed to or logged for the nonfunctional annunciator. The inspectors noted

that should a similar loss of instrument air pressure event recur, the absence of any

warning labels or other devices to alert operators to the nonfunctional annunciator could

cause the operators to fail to take the appropriate steps per the annunciator response

instruction and loss of instrument air abnormal operating instruction to monitor enclosed

spaces for oxygen concentration and monitor the nitrogen tank levels. Due to

instrument air system leakage, actuation of the backup nitrogen system without the

compensatory action of monitoring enclosed spaces for oxygen concentration could

potentially result in operator injury or death from entering oxygen-deficient areas of the

plant. This inspectors determined the failure to appropriately track the nonfunctional

control room annunciator was a violation of Technical Specification 5.5.1.1. This finding

is discussed further in Section 2.7 of this report.

1.5

Root Cause Evaluation

The inspectors reviewed the accuracy and thoroughness of the licensee cause

determination as described in the root cause evaluation, Unit 2 Instrument Air Soldered

Joint Failure, performed as part of Action Request AR 070600867. The licensees root

cause evaluation used events and causal factors analysis and failure modes and effects

Enclosure

-13-

analysis to evaluate the physical piping failure, the use of operating experience at the

site, and the implementation of the preventive maintenance program in the instrument

air system.

For the physical piping failure, the licensee performed a metallurgical analysis of the

failed joint. The analysis showed the original solder coverage within the joint was less

than 30% of the joint interface. Corrosion facilitated by residual flux in the joint

weakened the solder over time until the eventual failure of the fitting. The licensee

concluded the cause for the improper solder coverage was an improper fit up causing

an excessive gap in the joint due to poor workmanship during initial construction. The

analysis concluded the cause of the poor workmanship was a lack of supervisory

monitoring and reinforcement, and the root cause of the physical joint failure was a lack

of intrusive testing and inspections of the instrument air system during initial

construction. The analysis did not identify corrective actions specific to the identified

causes since the events examined occurred during initial construction. Corrective

actions were identified to develop an inspection plan to locate additional leaking joints

and to periodically inspect the clamps installed on improperly made joints until repairs

can be made.

The root cause analysis also examined the ineffective review and use of industry

operating experience (OE) at the site to determine why the event had not been

prevented despite the existence of sufficient OE to foresee its occurrence. The licensee

concluded the ineffective use of OE was the result of an inappropriate cultural bias in

the engineering department that led engineers to review OE from a defensive

standpoint; i.e., the goal of the engineers performing reviews was to determine why a

particular OE was not applicable to the station. The licensee determined this culture

was reinforced by insufficient site expectations for OE procedural use and

documentation. The licensee developed corrective actions to strengthen site standards

and expectations for OE procedure use and documentation and to perform

benchmarking among industry peers to incorporate best practices for OE use.

The final portion of the licensees root cause evaluation examined the lack of an

adequate preventive maintenance program for the instrument air check valves and the

backup nitrogen flow indication switch. The licensee concluded the apparent cause for

the lack of preventive maintenance tasks was ineffective supervisory monitoring. The

licensee developed corrective actions to reinforce expectations for supervisory

performance and to implement preventive maintenance tasks for the instrument air

check valves.

The inspectors reviewed the licensees root cause evaluation and determined the

metallurgical analysis and cause evaluation for the physical piping failure was thorough

and technically sound. However, the inspectors concluded that the root cause

evaluation as a whole was narrowly focused and in some cases lacked specific,

comprehensive corrective actions.

The inspectors considered the evaluation to be narrowly focused since it did not fully

address all the factors and behaviors that contributed to the nature, magnitude and

timing of the event. For example, the evaluation did not discuss in detail a precursor

event in the form of a failed thermowell fitting in the instrument air system that occurred

in 1994. The evaluation also failed to address the poor quality of the OE review

performed in 1992. The extent of condition review made only a passing reference to the

Enclosure

-14-

domestic water system and did not contain a complete discussion of systems potentially

affected by the identified root cause. Additionally, the maintenance review in the root

cause evaluation did not discuss the scoping or performance of the instrument air

system under the Maintenance Rule (10 CFR Part 50.65). Also, the extent of cause

review for the failed joint noted that site engineers had not appropriately addressed the

instrument air system under the station Equipment Reliability Improvement Program, but

the report did not appear to investigate why that had happened or whether there were

potentially other systems that may have been similarly overlooked.

The inspectors noted that the narrow focus of the report was also reflected in the

assignment of corrective actions. Specifically, the licensee identified poor supervisory

monitoring and oversight as an apparent cause for the improperly soldered joint, but

identified no corrective actions due to the age of the issue. However, later in the same

report, the licensee identified poor supervisory monitoring and oversight as an apparent

cause for the failure to establish adequate preventive maintenance tasks for instrument

air system. Corrective actions in this case to reinforce standards and improve

performance were directed only to the engineering organization.

The inspectors noted that in some cases, the causes identified in the root cause

evaluation tended to be associated with a single, broad corrective action such as

strengthen site standards. While the inspectors did not disagree with the intent of the

corrective action, the generalized language leaves the method of implementation open

to interpretation and complicates the ability of the stations assessment organization to

perform effectiveness reviews. In these cases, the inspectors concluded a set of

specific, focused, and measurable corrective actions may have been more appropriate.

1.6

Event Precursors

The team performed a search of corrective action program databases to identify

previous instrument air system piping problems that may have been precursors to the

event on June 20, 2007. The inspectors identified the potential event precursors

described below.

The inspectors noted that in June 1994, both Units 2 and 3 experienced a loss of

instrument air event due to the failure of a soldered joint retaining a threaded thermowell

attachment to the instrument air header. In the failure analysis for the fitting, the

licensee determined the cause of the separation was poor quality workmanship that

occurred during original installation. The licensee determined that when the fitting was

initially installed, it was not centered, causing an excessive gap on one side of the joint

and resulting in inadequate solder penetration. The licensee also identified that the joint

was designed to be silver brazed per the manufacturers specification and should not

have been soldered, and that this joint had most likely been leaking since original

installation since a portion of the pipe joint had no filler metal in it. Although this event

did not result in a reactor trip, the inspectors noted the cause and nature of the failure

were nearly identical to that experienced on June 20, 2007.

In April, 2007, both Unit 2 and Unit 3 experienced a loss of instrument air event due to a

trip of the running air compressors. During this event as well as during the June 1994

loss of instrument air event described above, the annunciator for actuation of the

backup nitrogen supply to the instrument air system failed to alarm in the control room.

Following the annunciator failure in 1994, licensee maintenance technicians noted that

Enclosure

-15-

the limit switch was dirty. Since a replacement limit switch was not available, the

technicians cleaned the installed limit switch and returned it to service. In April 2007,

licensee maintenance technicians noted that the travel on the switch was satisfactory

and there were no problems on the electrical part of the system. No other work was

documented. During the event on June 20, 2007, the annunciator for actuation of the

backup nitrogen system again failed to alarm in the control room. Although the

April 2007 event was caused by a compressor failure and did not result in a reactor trip,

the inspectors noted the plant response, particularly as it related to the flow switch for

actuation of the backup nitrogen system, was similar to the event on June 20, 2007. A

finding associated with the failure of the licensee to take effective corrective actions for

the nitrogen system flow switch is described in Section 2.4 of this report.

1.7

Instrument Air System Maintenance and Testing

The inspectors reviewed the licensees program for maintenance and inspection of the

instrument air system, particularly as it related to the historical health of the instrument

air compressors and piping system.

The inspectors noted the performance of the instrument air system was monitored

under performance criteria established per the guidance for Category a(2) systems

under the Maintenance Rule. During discussions with the system engineer and site

Maintenance Rule coordinator, the inspectors learned the system was under

consideration for goal setting and monitoring per Category a(1) of the Maintenance

Rule. The licensee subsequently established goals and began monitoring the

performance of the instrument air system per Category a(1). The team considered this

action appropriate. The team also noted there had been several functional failures of

the three instrument air compressors over the past two years, and at one point earlier in

the year a temporary air compressor was installed to supplement the existing instrument

air compressors. The team noted this situation was nearly identical to that described in

NRC Inspection Report 50-361:362/97-22 as indicative of poor performance requiring

the goal setting and monitoring per Category a(1) of the Maintenance Rule. Given this

operating history, the team concluded it may have been appropriate for the station to

have classified the instrument air system as Category a(1) much earlier in the year.

However, the inspectors noted that such classification would have had no impact on the

prevention or mitigation of the loss of instrument air event experienced at the station on

June 20, 2007.

The inspectors also examined maintenance work orders for individual components in the

instrument air and low pressure nitrogen systems. The inspectors noted that no

preventive maintenance actions existed for the excess flow check valves in the supply

header for backup nitrogen to the instrument air system. The inspectors considered this

inappropriate given these valves have a function credited in the FSAR to prevent a

break in one units instrument air header from causing a loss of instrument air to the

other unit. Since check valves can not be considered inherently reliable components,

the inspectors concluded the licensee had failed to perform adequate preventive

maintenance to ensure the excess flow check valves would perform their intended

function. The inspectors determined this was a violation of 10 CFR Part 50.65a(2).

However, since the excess flow check valves did perform their intended function during

the actual loss of instrument air event on June 20, 2007, the inspectors considered this

Enclosure

-16-

violation to be minor. The licensee entered this violation in their corrective action

program as Action Requests AR 070600867 and AR 070900333, and evaluated the lack

of preventive maintenance items in the instrument air system as part of the root cause

evaluation for the loss of instrument air event.

1.8

Industry Operating Experience (OE) and Potential Generic Issues

The inspectors performed searches of operating experience databases and other

sources to identify reports of similar problems, both inside and outside the nuclear

industry.

During the late 1980s and early 1990s, a significant amount of Operating

Experience (OE) identified instances where facilities had experienced transients and/or

trips due to failures of soldered joints in instrument air system piping due to poor

workmanship during initial construction. The licensee documented their review of the

industry OE in their corrective action program (CAP) as Independent Safety

Engineering Group Operating Experience Evaluation, dated January 22, 1992. In this

review, the licensee evaluated the identified causes and corrective actions from the OE

and determined that soldering at SONGS was loosely controlled and better training was

necessary for welders at the facility. However, the licensee asserted in their evaluation

that failures due to inadequate fit-up or solder penetration typically occur within a

relatively short time frame after startup. The licensee concluded that since the

instrument air system had been in service for many years at SONGS and no significant

problems had yet been identified, then no corrective actions were necessary with

respect to the installed instrument configuration. The inspectors considered this

conclusion to be without a valid technical basis. A finding associated with this evaluation

is discussed in Section 2.1 of this report.

In addition to the internal site experience described above and in Section 1.6 of this

report, the inspectors identified additional OE in the form of Licensee Event Reports

(LERs). Most notably, the inspectors reviewed LER 05000336/2006-002-00, Manual Reactor Trip Due to Trip of Both Feed Pumps Following a Loss of Instrument Air,

April 21, 2006, and LER 05000440/2006-005-00, Decreasing Instrument Air Pressure

Results in Manual Reactor Protection System Actuation, February 9, 2007. Both

reports describe reactor trips brought about by instrument air header joint separation. In

both cases, the cause of the header joint separation was inadequate workmanship

during initial construction. Though the site OE coordinator indicated the licensee

reviewed all Licensee Event Reports for applicability to SONGS, the inspectors did not

identify any documents in the licensees corrective action program that evaluated these

events. The inspectors determined the lack of documentation in the CAP indicated the

site OE organization had determined the above described events were not applicable to

SONGS. The inspectors concluded the licensee had missed multiple opportunities both

historically and recently to identify the vulnerability presented by improperly made joints

in the instrument air system.

Given the failure history described above, the inspectors concluded the construction

methods and controls in place during initial construction at SONGS were not unique.

Therefore, the potential for separation of instrument air piping due to improperly made

joints represents a potential generic concern for all facilities with instrument air systems

utilizing soldered joints in copper piping headers.

Enclosure

-17-

2.0

SPECIAL INSPECTION FINDINGS

2.1

Ineffective Corrective Actions for Instrument Air Header Ruptures

The inspectors reviewed a self-revealing Green finding involving ineffective corrective

actions taken in response to site and industry operating experience with instrument air

header ruptures. Specifically, contrary to Section 6.2.3 of Procedure SO-123-I-1.42,

Maintenance Division Experience Report, Revision 0, the licensee failed to implement

corrective actions to prevent recurrence for an equipment failure with the potential to

cause a significant plant transient, and failed to appropriately consider previous industry

and plant experience similar to the event. Additionally, licensee personnel failed to

properly evaluate and take corrective actions based on industry operating experience

through 2006 involving improperly made soldered joints in instrument air systems. As a

result, an additional failure of an improperly made instrument air header joint occurred at

SONGS on June 20, 2007.

On June 20, 2007, both Units 2 and 3 experienced a loss of instrument air event due to

the failure of a three-inch instrument air line header fitting. As a result of the break

location, a loss of manual feedwater control occurred on Unit 2 which ultimately resulted

in a manual reactor trip due to high steam generator level.

The licensee performed a Root Cause Evaluation of this event, as documented in Action

Request AR 070600867. The licensee also performed a metallurgical analysis of the

failed joint as documented in SONGS Unit 2 Instrument Air System Failed Fitting

Metallurgical Evaluation, dated June 27, 2007. During these evaluations, the licensee

determined the root cause of the event to be poor workmanship of the header joint

during initial installation. The licensees metallurgical analysis also concluded the fitting

had only thirty percent solder coverage within the joint and had likely been leaking air

since the plants first operating cycle. Subsequent investigations by the licensee

identified 32 additional joints leaking air in the instrument air headers of both units. The

licensee installed temporary structural clamps on the leaking fittings tp prevent

additional separations until permanent repairs could be made.

The inspectors reviewed the licensees root cause evaluation and metallurgical analysis

for this event. During their review of the issue, the inspectors noted that there had been

a previous similar air header failure at SONGS, and that the licensee had previously

evaluated related industry Operating Experience (OE) involving issues with soldered

joints.

During the late 1980s and early 1990s, a significant amount of Operating Experience

(OE) identified instances where facilities had experienced transients and/or trips due to

failures of soldered joints in instrument air system piping. The identified failures were

due to a lack of adequate controls during the initial makeup of soldered joints.

Specifically, inadequate fit-up of the joints or inadequate solder penetration were

identified as the causes of the failures. The licensee performed a review documented in

Independent Safety Engineering Group Operating Experience Evaluation, dated

January 22, 1992, to evaluate applicability of the OE to the facility. In this review, the

licensee evaluated the identified causes and corrective actions from the OE and

determined that soldering at SONGS was loosely controlled and better training was

necessary for welders at the facility. However, the licensee asserted in the evaluation

that any failures due to inadequate fit-up or solder penetration would typically have

Enclosure

-18-

occurred within a relatively short time frame after startup. The licensee concluded that

since the instrument air system had been in service for many years and no significant

problems had yet been identified, then no corrective actions were necessary with

respect to the installed instrument air configuration.

In June 1994, both Units 2 and 3 experienced a loss of instrument air event. The

licensee investigated the cause and determined it was due to the failure of a soldered

joint retaining a threaded attachment, a Brazolet fitting, to the air header. The Brazolet

fitting was being used for a thermowell in the instrument air header. The licensee

performed a failure analysis of the fitting documented in Failure Analysis Report

No.94-006, dated July 8, 1994, to determine the cause of the joint failure. During this

analysis, the licensee determined the cause of the soldered fitting failure was poor

quality workmanship that occurred during original installation. The licensee determined

that when the fitting was initially installed, it was not centered, but rather cocked to one

side. This was not as required by the procedure and resulted in an excessive gap on

one side of the joint. This gap deprived one side of the joint of filler material, resulting in

inadequate solder penetration. The licensee also identified that the joint was designed

to be silver brazed per the manufacturers specification and should not have been

soldered.

Based on the metallurgical analysis, the licensee determined that the joint failure in

July 1994 was due to fatigue cracking that originated at the area between the soldered

and un-soldered sections of the joint. The failure analysis also identified that this joint

had most likely been leaking since installation. This was based on the fact that a portion

of the pipe joint had no filler metal in it. Since the joint was located approximately ten

feet off the floor in a high noise area, the leakage had not been previously identified.

The failure analysis also recommended that to prevent recurrence, all brazolet fittings in

the instrument air system should be examined both for leaks and for use of solder. The

analysis further identified that properly soldered joints fittings should be able to tolerate

instrument air header pressure indefinitely; however, if leaks were found, the fittings

should be replaced using silver braze at the earliest opportunity.

The inspectors concluded the licensees evaluation of OE performed in 1992 was

inadequate in that it improperly determined that failures due to inadequate fit-up and/or

inadequate solder penetration would have occurred within a relatively short time frame.

The inspectors also determined that the licensee failed to adequately reassess this

position following the instrument air line joint failure in 1994. The inspectors noted that

as recently as 2006, the licensee had inappropriately screened additional industry OE

relating to the failure of inadequately made instrument air piping joints as not applicable

to the station. The inspectors concluded the licensee failed to take effective corrective

actions for inadequately made joints in the instrument air system since the corrective

actions for the 1994 event and in response to industry OE were narrowly focused on

soldered Brazolet fittings and failed to evaluate soldered joints as a whole.

The safety significance and enforcement aspects of this finding are described in

Sections 3.1 and 4.1, respectively.

2.2

Failure to Follow Abnormal Operating Instruction for the Loss of Instrument Air

The inspectors identified a Green noncited violation of Technical Specification 5.5.1.1

involving the failure to meet procedural requirements following a loss of instrument air.

Enclosure

-19-

Specifically, operators failed to monitor nitrogen tank levels or take precautions for the

possibility of oxygen-deficient areas in the plant following actuation of the low pressure

backup nitrogen system.

The instrument air system at SONGS utilizes a low pressure nitrogen system as a

backup pressure source. An instrument air system pressure drop below 83 psig will

automatically actuate a control valve in the nitrogen system to supplement the

instrument air from liquid nitrogen storage tanks. A flow switch downstream of the

control valve is designed to provide an annunciator in the control when the nitrogen

system is actuated. While the air systems are being supplied by nitrogen, normal

system leakage can result in oxygen-deficient areas in enclosed spaces of the plant,

and the liquid nitrogen tank levels will decrease more rapidly than usual from the

addition of the instrument air loads. The nitrogen supply lines are provided with isolating

check valves and excess flow check valves to prevent a failure in one units piping from

causing an excessive pressure drop in the other unit.

On June 20, 2007, an instrument air header ruptured in Unit 2. Although the low

pressure nitrogen system functioned as designed to provide nitrogen to the Unit 3 air

header, the control room annunciator for nitrogen system actuation did not alarm. The

instrument air header low pressure alarms did actuate on both units, and control room

operators began taking required actions per Procedure SO23-13-5, Loss of Instrument

Air, Revision 5. Although a step in the procedure directed operators to monitor nitrogen

tank levels and monitor for oxygen concentrations in enclosed spaces following nitrogen

system actuation, this step was not performed. The inspectors noted the failure to take

these actions had the potential to result in a Unit 3 trip from nitrogen tank depletion or

the injury or death of personnel from entry into oxygen-deficient spaces.

During interviews with the inspectors, several control room operators demonstrated

knowledge weaknesses related to the operation of the backup pressure sources for

instrument air. For example, several operators mistakenly stated the Unit 3 air header

pressure had been supplied by the respiratory/service air system during the event. The

inspectors concluded that although the failed annunciator likely contributed to the

operators confusion, the failure to perform the required actions of the abnormal

operating instruction resulted from the operators poor understanding of the operation of

the nitrogen backup to the instrument air system.

The safety significance and enforcement aspects of this finding are described in

Sections 3.2 and 4.2, respectively.

2.3

Inadequate Evaluation Results in Runout of Component Cooling Water Pump

A self-revealing, Green noncited violation of 10 CFR Part 50, Appendix B, Criterion III,

Design Control, was identified when Unit 2 experienced a loss of instrument air due to

the failure of a soldered joint. Specifically, the loss of instrument air resulted in

component cooling water (CCW) Pump 024 being in a runout condition for

approximately 75 minutes due to a previous system modification.

In 1995, the licensee implemented a design change to the CCW system to provide

backup nitrogen to the non-critical loop (NCL) supply and return isolation valves. The

Enclosure

-20-

design change was made to ensure that CCW flow would be maintained to the reactor

coolant pump (RCP) seals during a loss of instrument air event. This would preclude

the licensee from the need to secure the RCPs due to a loss of CCW cooling in the

event of a loss of instrument air pressure.

On June 20, 2007, the CCW system was aligned with the Train A Pump 024 in

operation with a normal full operating load on the system, including the non-critical loop.

The Train B pump was in standby. At approximately 10:45 pm, Unit 2 experienced a

loss of instrument air when a 3-inch air header fitting separated in the auxiliary building.

Following the loss of instrument air pressure, the shutdown cooling heat exchanger

isolation valves failed opened as designed. Since the NCL isolation valves remained

open, the increased system load from the shutdown cooling heat exchanger caused the

CCW pump flow rate to increase to approximately 300 gallons per minute more than its

maximum design flow limit of 16,000 gallons per minute, placing the pump in a runout

condition. The pump operated in this condition for approximately 75 minutes before

operators took action to reduce system flow rate.

The inspectors reviewed this issue and determined that the licensee had not performed

an adequate hydraulic analysis of the CCW system in 1995 when implementing the

design change to maintain the NCL supply and return isolation valves open following a

loss of instrument air. The inspectors determined that this design change directly

contributed to placing the CCW pump in a runout condition following the loss of

instrument air.

The safety significance and enforcement aspects of this finding are described in

Sections 3.3 and 4.3, respectively.

2.4

Ineffective Corrective Actions for a Failed Control Room Annunciator

The inspectors reviewed a self-revealing Green finding involving the failure to take

effective corrective actions for a failed control room annunciator. Specifically, after the

annunciator for actuation of the backup nitrogen supply to the instrument air system

failed to function on demand on several occasions from 1994 through 2007, the

corrective actions taken by the licensee to restore the annunciator to service were

inadequate and narrowly focused. The annunciator subsequently failed to function

during the loss of instrument air event on June 20, 2007.

In June 1994, SONGS Units 2 and 3 experienced a loss of instrument air event during

which the annunciator for actuation of the backup nitrogen supply to the instrument air

system failed to actuate in the control room. The licensee entered this into their

corrective action program and generated Maintenance Order 94062628000 to

investigate and correct the issue. During the investigation, licensee maintenance

technicians noted that the limit switch was dirty. Since a replacement limit switch was

not available, the licensee cleaned the installed limit switch and returned it to service.

The work order subsequently closed with no further actions taken by the licensee.

While the licensee was performing an evolution to repressurize the backup nitrogen line

to the instrument air system in May 1996, the annunciator for actuation of the backup

nitrogen supply to instrument air again failed to actuate in the control room. The

licensee entered this event into their corrective action program as Action Request

AR 960500111. In this AR the licensee determined that possible causes of the failure

Enclosure

-21-

were slow flow rate, not enough to flow to open the check valve far enough to trip the

limit switch, or the possibility of a problem with the limit switch. The licensee generated

Maintenance Order 94062628001 to investigate and correct the issue. During their

inspection, the licensee maintenance technicians found rust on the limit switch and

determined this to be the cause of the failure. The limit switch was replaced, and the

licensee verified that it worked electrically. The inspectors noted the maintenance order

called for an operational test of the limit switch, but none was performed.

In April 2007, Units 2 and 3 experienced a loss of instrument air event. During this

event, the annunciator for actuation of the backup nitrogen supply to the instrument air

system again failed to actuate in the control room. The licensee entered this into their

corrective action program as AR 070400776 and generated Maintenance Order

07041277000 to investigate and correct the issue. During their inspection, the licensee

maintenance technicians noted that the travel on the switch was satisfactory and there

were no problems on the electrical part of the system. No other work was documented.

On June 20, 2007, both Unit 2 and Unit 3 experienced a loss of instrument air pressure

due to the failure of a three-inch instrument air line header fitting. During this event, the

annunciator for actuation of the backup nitrogen supply to the instrument air system

failed to actuate in the control room again. The licensee entered this into their corrective

action program as AR 070601250.

The inspectors concluded that the licensee failed to adequately evaluate and correct the

issue associated with the limit switch. During their review, the inspectors also noted that

the licensee had not questioned or investigated the operational aspects of the limit

switch. Instead, the licensee had narrowly focused on testing only the electrical portion

of the system. The inspectors determined that the licensee had not operationally tested

the limit switch during any of their corrective actions.

The licensee subsequently performed an operational test of the limit switch. During this

testing, the licensee determined the nitrogen flow through the check valve was not

sufficient to actuate the limit switch. Consequently, the limit switch would never have

functioned to actuate its associated control room annunciator. The licensee entered this

issue into their corrective action program.

The safety significance and enforcement aspects of this finding are described in

Sections 3.4 and 4.4, respectively.

2.5

Inadequate Procedure for a Loss of Instrument Air

The inspectors identified a Green noncited violation of Technical Specification 5.5.1.1

involving the failure to maintain an adequate abnormal operating instruction for a loss of

instrument air event.

Procedure SO23-13-5, Loss of Instrument Air, Revision 5, specifies operator actions to

mitigate the effects of excessive instrument air system leakage or the loss of the

instrument air compressors. The inspectors reviewed the procedure and noted the

following:

Step 1.a of the procedure was followed by a caution stating:

Enclosure

-22-

A large break downstream of the nitrogen supply may cause the nitrogen

excess flow check valve to seat. This may be indicated by falling nitrogen

header pressure on 2/3PI-5344B, followed by rapid return to > 80 psig.

The inspectors noted the nitrogen header pressure indication on 2/3PI-5344B

was not provided with a strip chart recorder or computer point to provide trend

information and the procedure did not direct stationing a dedicated operator to

monitor the pressure instruments. The inspectors concluded that absent a

dedicated operator to observe the pressure indicator, no trend information would

be available to the control room operators to determine whether the described

pressure response had occurred.

Step 1.b of the procedure directed operators to determine whether or not the

instrument air header pressure was stable or rising. The inspectors concluded

this determination would be complicated by the lack of any available trend

information for all of the air header pressure instruments.

Step 2.a of the procedure directed operators to trip the reactors and turbines of

both units in the event of a loss of both instrument air header pressure and

nitrogen header pressure as indicated by the control room instruments. The

inspectors noted the control room instruments only provided pressure indication

for the common headers; there were no available indications for the pressure in

the individual headers of each unit. The inspectors noted that in the event the

backup nitrogen system excess flow check valves either spuriously shut or failed

to open on a complete loss of instrument air, the nitrogen header pressure

instrument would continue to indicate sufficient pressure despite the complete

depressurization of the instrument air headers in both units. In this case,

Step 2.a would direct operators to the Subsequent Actions section of the

procedure to monitor plant response instead of the more appropriate action of

Step 2.b to immediately trip both units.

The inspectors determined the above issues could result in a delay of necessary

operator response actions to mitigate the consequences of an initiating event.

The safety significance and enforcement aspects of this finding are described in

Sections 3.5 and 4.5, respectively.

2.6

Simulator Incorrectly Modeled Plant Response to Loss of Instrument Air

A self-revealing, Green noncited violation of 10 CFR Part 55.46(c)(1) was identified

involving the licensees failure to incorporate a design change in modeling plant

response for the plant-referenced simulator. Specifically, during operator training in the

plant-referenced simulator, the controlled bleedoff valves for the reactor coolant pumps

were modeled to fail closed on a loss of instrument air, whereas the valves in the plant

remained open during an actual loss of instrument air event on June 20, 2007.

The original model regulator installed on the reactor coolant pump (RCP) controlled

bleed off (CBO) Valve 2HV9218 allowed air to bleed off on a loss of instrument air,

enabling the valve actuator to move shut to its fail-safe position. Though not an

intentional design feature of the regulator, the licensee took credit for this feature to shut

Enclosure

-23-

the RCP CBO valve during a loss of instrument air. As such, the plant-referenced

simulator used for operator training modeled the valve going shut during a loss of

instrument air event.

In February 2004, the licensee replaced the existing valve regulator for the RCP CBO

valve with a new style regulator. The new regulators were used because the original

regulator was obsolete. The vendor modified the new regulators to make them leak

tight, removing the unintentional bleed off characteristic. The licensee evaluated the

change in the new regulators and determined the new regulators to be an equivalent

valve as part of Substitute Equivalency Evaluation (SEE) 020040.

On June 20, 2007, both Units 2 and 3 experienced a loss of instrument air event due to

the failure of a three-inch instrument air line header fitting. During this event, the RCP

CBO containment isolation Valve 2HV9218 failed to go closed as the operators

expected. The control room dispatched an operator at the time of the event to

investigate and determine why the valve did not go closed. The operator noted locally

that the valve was open and that pressure was present on the regulator. The control

room subsequently dispatched another operator to independently second-check the

position of the valves. The second operator also noted that the valve was open and that

pressure was present on the regulator. The operators took no other actions at the time

because closure of this valve could cause loss of CBO flow which would have required

the RCPs to be secured.

The inspectors determined that the licensee failed to update the plant-referenced

simulator following the CBO valve regulator change. As a result, operators were trained

that the RCP CBO valves would shut during a loss of instrument air. However, during

the actual loss of instrument air event on June 20 the CBO valve did not go shut as

expected which caused confusion among the operators responding to the event.

The safety significance and enforcement aspects of this finding are described in

Sections 3.6 and 4.6, respectively.

2.7

Failure to Follow Procedure for an Impaired Annunciator

The inspectors identified a Green noncited violation of Technical Specification 5.5.1.1

involving the failure to meet procedural requirements governing impaired annunciators.

Specifically, after the identification of a failed annunciator, operators did not enter the

annunciator in the failed annunciator log or mark the affected annunciator window with

an annunciator compensatory action flag.

The instrument air system at SONGS utilizes a low pressure nitrogen system as a

backup pressure source. An instrument air system pressure drop will automatically

actuate a control valve in the nitrogen system to supplement the instrument air from

liquid nitrogen storage tanks. A flow switch downstream of the control valve is designed

to provide an annunciator in the control when the nitrogen system is actuated. This

annunciator is used as a diagnostic aid and to determine operator actions in

Procedure SO23-13-5, Loss of Instrument Air, Revision 5.

On June 20, 2007, an instrument air header rupture occurred in Unit 2. The inspectors

noted that although the low pressure nitrogen system provided nitrogen to the Unit 3 air

header as designed, the control room annunciator for nitrogen system actuation did not

Enclosure

-24-

alarm. The licensee initiated AR 070601250 on June 29, 2007 to address the failed

annunciator. On July 2, the inspectors noted the failed annunciator was not included in

the impaired annunciator log. The licensee polled two shift managers and determined

that one believed the annunciator should be treated as impaired and requiring

compensatory actions per Procedure SO23-6-29, Operation of Annunciators and

Indicators, Revision 15, and the other shift manager did not. The inspectors concluded

the operators were not consistently implementing the portion of the procedure

concerning impaired annunciators. The licensee subsequently entered the annunciator

in the impaired annunciator log and took the actions specified by Procedure SO23-6-29

for an impaired annunciator.

The safety significance and enforcement aspects of this finding are described in

Sections 3.7 and 4.7, respectively.

2.8

Inadequate Implementation of Corrective Actions for Air Operated Valve Regulators

A Green self-revealing finding was identified associated with the failure of the reactor

coolant pump controlled bleed off valve to shut during a loss of instrument air event.

The licensee failed to adequately implement corrective actions from previously

evaluated industry operating experience for new valve regulators that were installed in

the unit.

In July 2002, industry Operating Experience (OE) was issued which identified potentially

undesirable consequences due a design change to air operated valve regulators that

improved leakage characteristics of the regulators. The old model regulators allowed air

pressure to bleed off on a loss of instrument air, which enabled the valve actuator to

move to its fail-safe position. This was not an intentional design feature of the regulator.

The new regulators were changed to correct this unintentional bleed off and make them

leak tight. The OE was issued to alert users to this change so that if a user had taken

credit for this unintentional bleed off they would be aware of this change and

appropriately address it.

The licensee evaluated this OE and determined that it was applicable to the station.

AR 031001558 was initiated in October 2003 to provide appropriate actions to address

any issues. The licensee identified that this change would not affect air operated valves

that have an associated positioner or controller, or valves that are configured with a

solenoid valve installed between the air regulator and actuator that receives a signal to

vent. Valves without an associated positioner, controller, solenoid valve, or other

configuration to allow for air bleed off could be affected by use of the new regulator.

During this evaluation the licensee also determined that there were not any of the new

regulators in use at the facility at the time.

The licensee had already performed Substitute Equivalency Evaluations (SEE) for

replacing some of the old style regulators with the new style. Based on the results of

the OE review the licensee determined that there was a need to revise the existing

SEEs to require design engineering and procurement engineering to perform a design

change impact review to evaluate the installation configuration. The purpose of this

review was to evaluate whether a design change would be necessary prior to installation

of the new style regulators.

Enclosure

-25-

In February 2004 the licensee replaced the old style regulator with a new style regulator

on the reactor coolant pump (RCP) controlled bleed off (CBO) Valve 2HV9218. This

was evaluated under SEE 020040.

On June 20, 2007 both Units 2 and 3 experienced a loss of instrument air event due to

the failure of a three-inch instrument air line header fitting. During this event, the RCP

CBO containment isolation valve, 2HV9218, failed to go closed as expected. The

licensee dispatched an operator at the time of the event to investigate and determine

why the valve did not go closed. The operator noted that the valve was open and that

pressure was present on the regulator. The licensee took no other actions at the time

because closure of this valve could cause loss of CBO flow which would have required

the RCPs to be secured.

The licensee performed a review of this issue as documented in AR 070600873. During

this review, the licensee determined that the valve should have shut during the loss of

instrument air event and did not because of the new style regulator that had been

installed. The licensee also identified that the SEE appropriately identified the

requirement for design engineering and procurement engineering to perform a design

change impact to evaluate the installation configuration. However, the action by

procurement engineering to require a design change review prior to installation of the

regulator failed due to a known computer software limitation, and the maintenance

engineering review inappropriately determined that there were no applications where the

old regulators did not have a solenoid or bleed off device between the regulator and

solenoid. The licensee also identified that during both of these assessments, the

engineers did not review the UFSAR or any other licensing commitments that credited

bleed down characteristics of the old regulators during a loss of instrument air.

The licensee also performed an extent of condition review to determine if there were any

other instances of these new style regulators being installed in the plant. This review

identified 37 instances of the new regulators being installed in the plant without

performance of a design change impact review.

The safety significance and enforcement aspects of this finding are described in

Sections 3.8 and 4.8, respectively.

3.0

ASSESSMENT

3.1

Ineffective Corrective Actions for Instrument Air Header Ruptures

The failure to take effective corrective actions in response to site and industry operating

experience resulting in a subsequent instrument air header failure was a performance

deficiency. This finding was more than minor since it was associated with the

equipment reliability attribute of the initiating events cornerstone and affected the

cornerstone objective to limit the likelihood of events that upset plant stability and

challenge critical safety functions. This finding required a Phase 2 analysis per the

Manual Chapter 0609, Significance Determination Process, Phase 1 Worksheets since

the loss of instrument air is a transient initiator resulting in the loss of the feedwater

system which is part of the power conversion system which can be used to mitigate the

consequences of an accident. The inspectors performed a Phase 2 analysis using

Appendix A, Technical Basis for At-Power Significance Determination Process, and the

Phase 2 worksheets for SONGS. The inspectors assumed that the exposure period

Enclosure

-26-

was greater than 30 days, that the performance deficiency increased the likelihood that

a complete loss of instrument air would occur, and that there was no affect on mitigating

systems other than those modeled in the risk-informed notebook. Details of the

Phase 2 analysis and a subsequent Phase 3 analysis are documented in Attachment 3.

Based on the results of the Phase 3 analysis, the finding was determined to be of very

low safety significance (Green) because of the availability of the diverse auxiliary

feedwater system and the ability of the operators to depressurize the steam generators

and utilize the condensate system for heat removal. These results were evaluated by a

senior reactor analyst. In addition, the senior reactor analyst determined the impact of

this performance deficiency on the likelihood of the large-early release frequency.

These evaluations indicated that the impacts were also of very low safety significance.

This finding has a crosscutting aspect in the area of problem identification and

resolution associated with operating experience in that the licensee failed to effectively

implement changes to station processes, procedures, and equipment in response to

operating experience involving improperly made instrument air system joints P.2(b).

3.2

Failure to Follow Abnormal Operating Instruction for the Loss of Instrument Air

The failure to follow station procedures to monitor nitrogen tank levels and oxygen

concentrations in enclosed rooms where operator actions may have been required was

a performance deficiency. This finding was more than minor since it was associated

with the human performance attribute of the initiating events cornerstone and affected

the cornerstone objective to limit the likelihood of events that upset plant stability and

challenge critical safety functions. This finding required a Phase 2 analysis in

accordance with the Manual Chapter 0609, Significance Determination Process,

Phase 1 Worksheets since the loss of instrument air is a transient initiator resulting in

the loss of the feedwater system which is part of the power conversion system which

can be used to mitigate the consequences of an accident. The inspectors performed a

Phase 2 analysis using Appendix A, Technical Basis for At-Power Significance

Determination Process, and the Phase 2 worksheets for SONGS. The inspectors

assumed that the exposure period was greater than 30 days, that the performance

deficiency increased the likelihood that a complete loss of instrument air would occur,

and that there was no affect on mitigating systems other than those modeled in the risk-

informed notebook. Based on the results of the Phase 2 analysis, the finding was

determined to be of very low safety significance because of the low likelihood of a

complete loss of instrument air and the availability of the auxiliary feedwater system.

These results were evaluated by a senior reactor analyst. In addition, the senior reactor

analyst determined the impact of this performance deficiency on the risk of external

events and on the likelihood of the large-early release frequency. These evaluations

indicated that the impacts were also of very low safety significance.

The cause of this finding has a crosscutting aspect in the area of human performance

associated with resources because licensee personnel were not adequately trained on

the operation of the low pressure nitrogen system to effectively implement the abnormal

operating instruction H.2(b).

Enclosure

-27-

3.3

Inadequate Evaluation Results in Runout of Component Cooling Water Pump

The failure to adequately evaluate the total system hydraulic effects prior to

implementing a design change to supply nitrogen to the NCL isolation valves was a

performance deficiency. This finding was greater than minor because it was associated

with the mitigating systems cornerstone attribute of design control and affected the

associated cornerstone objective to ensure the availability, reliability, and capability of

systems that respond to initiating events to prevent undesirable consequences. The

finding did not affect the initiating events cornerstone functions of the component

cooling water system because the condition would only have existed given a loss of

instrument air initiator had already occurred. In accordance with NRC Inspection

Manual Chapter 0609, Appendix A, Phase 1 Worksheet, Significance Determination

Process (SDP) Phase 1 Screening Worksheet for the Initiating Events, Mitigating

Systems, and Barriers Cornerstones, this finding was determined to be of very low

safety significance because the finding was a design deficiency confirmed not to result

in a loss of operability per Part 9900, Technical Guidance, Operability Determination

Process for Operability and Functional Assessment.

3.4

Ineffective Corrective Actions for a Failed Control Room Annunciator

The failure to perform adequate corrective actions for a failed control room annunciator

resulting in the failure of the annunciator to function during an actual event was a

performance deficiency. This finding was more than minor since it was associated with

the human performance attribute of the initiating events cornerstone and affected the

cornerstone objective to limit the likelihood of events that upset plant stability and

challenge critical safety functions. This finding required a Phase 2 analysis in

accordance with the Manual Chapter 0609, Significance Determination Process,

Phase 1 Worksheets since the loss of instrument air is a transient initiator resulting in

the loss of the feedwater system which is part of the power conversion system which

can be used to mitigate the consequences of an accident. The inspectors performed a

Phase 2 analysis using Appendix A, Technical Basis for At-Power Significance

Determination Process, and the Phase 2 worksheets for SONGS. The inspectors

assumed that the exposure period was greater than 30 days, that the performance

deficiency increased the likelihood that a complete loss of instrument air would occur,

and that there was no affect on mitigating systems other than those modeled in the risk-

informed notebook. Based on the results of the Phase 2 analysis, the finding was

determined to be of very low safety significance because of the low likelihood of a

complete loss of instrument air and the availability of the auxiliary feedwater system.

These results were evaluated by a senior reactor analyst. In addition, the senior reactor

analyst determined the impact of this performance deficiency on the risk of external

events and on the likelihood of the large-early release frequency. These evaluations

indicated that the impacts were also of very low safety significance.

This finding has a crosscutting aspect in the area of problem identification and

resolution associated with the corrective action program in that the licensee failed to

thoroughly evaluate the failed annunciator such that the resolution appropriately

addressed the causes P.2(c).

Enclosure

-28-

3.5

Inadequate Procedure for a Loss of Instrument Air

The failure to provide adequate procedural guidance to immediately diagnose and

properly respond to an initiating event was a performance deficiency. This finding was

more than minor because it was associated with the procedure quality attribute of the

mitigating systems cornerstone and affected the cornerstone objective to ensure the

availability, reliability and capability of systems that respond to initiating events, in that a

less than adequate abnormal operating procedure could have prevented operators from

promptly tripping the reactor, allowing conditions to continue to degrade and resulting in

a demand on the reactor protection system. Using the Significance Determination

Process Phase 1 Screening Worksheet in Appendix A of Inspection Manual Chapter 0609, the inspectors determined this finding had very low safety significance because it

did not result in an actual loss of safety function per Part 9900, Technical Guidance,

Operability Determination Process for Operability and Functional Assessment.

This finding has a crosscutting aspect in the area of human performance associated

with resources in that the licensee failed to provide operators with complete, accurate,

and up-to-date procedures H.2(c).

3.6

Simulator Incorrectly Modeled Plant Response to Loss of Instrument Air

The failure to ensure that the plant-referenced simulator correctly replicated expected

plant response to transient conditions was a performance deficiency. This finding was

greater than minor because it was associated with the mitigating systems cornerstone

attribute of human performance and affected the associated cornerstone objective to

ensure the availability, reliability, and capability of systems that respond to initiating

events to prevent undesirable consequences. The inspectors evaluated this finding

using the Appendix I, Licensed Operator Requalification Significance Determination

Process worksheets of Manual Chapter 0609 because the finding is a requalification

training issue related to simulator fidelity. Block 12 of the Appendix I flow chart requires

the inspector to determine if deviations between the plant and simulator could result in

negative training or could have a negative impact on operator actions. Negative

Training is defined, in a later version of the standard (ANSI 3.5-1998), as training on a

simulator whose configuration or performance leads the operator to incorrect response

or understanding of the reference unit. The licensee has committed to this version of

the ANS/ANSI standard for its simulator testing program for the plant-referenced

simulator. During the event of June 20, 2007, operators were influenced by negative

training on the simulator to question control room indications and locally independently

verify valve positions because valves in the plant failed to respond to a loss of

instrument air as modeled in the simulator. Therefore, differences between the simulator

and plant did have a negative impact on operator actions. The finding is of very low

safety significance because the discrepancy did not have an adverse impact on operator

actions such that safety related equipment was made inoperable during normal

operations or in response to a plant transient.

This finding has a crosscutting aspect in the area of human performance associated

with resources in that the licensee did not provide operators with adequate facilities and

equipment for use in operator training H.2(d).

Enclosure

-29-

3.7

Failure to Follow Procedure for an Impaired Annunciator

The failure to follow station procedures resulting in an untracked nonfunctional

annunciator was a performance deficiency. This finding was more than minor since it

was associated with the human performance attribute of the initiating events

cornerstone and affected the cornerstone objective to limit the likelihood of events that

upset plant stability and challenge critical safety functions. This finding required a

Phase 2 analysis in accordance with the Manual Chapter 0609, Significance

Determination Process, Phase 1 Worksheets since the loss of instrument air is a

transient initiator resulting in the loss of the feedwater system which is part of the power

conversion system which can be used to mitigate the consequences of an accident.

The inspectors performed a Phase 2 analysis using Appendix A, Technical Basis for

At-Power Significance Determination Process, and the Phase 2 worksheets for

SONGS. The inspectors assumed that the exposure period was greater than 30 days,

that the performance deficiency increased the likelihood that a complete loss of

instrument air would occur, and that there was no affect on mitigating systems other

than those modeled in the risk-informed notebook. Based on the results of the Phase 2

analysis, the finding was determined to be of very low safety significance because of the

low likelihood of a complete loss of instrument air and the availability of the auxiliary

feedwater system. These results were evaluated by a senior reactor analyst. In

addition, the senior reactor analyst determined the impact of this performance deficiency

on the risk of external events and on the likelihood of the large-early release frequency.

These evaluations indicated that the impacts were also of very low safety significance.

This finding has a crosscutting aspect in the area of human performance associated

with resources because the operators were not sufficiently trained to consistently

implement the annunciator operating procedure H.2(b).

3.8

Inadequate Implementation of Corrective Actions for Air Operated Valve Regulators

The failure to adequately implement corrective actions from industry OE to perform a

design change impact review was a performance deficiency. The finding was greater

than minor because it was associated with the mitigating systems cornerstone attribute

of design control and affected the associated cornerstone objective to ensure the

availability, reliability, and capability of systems that respond to initiating events to

prevent undesirable consequences. Using Manual Chapter 0609, Significance

Determination Process, Phase 1 Worksheet, the finding is determined to have very low

safety significance because the condition only affected the mitigation systems

cornerstone and it was confirmed not to result in loss of operability per Part 9900,

Technical guidance, Operability Determination Process for Operability and Functionality

Assessment.

4.0

ENFORCEMENT

4.1

Ineffective Corrective Actions for Instrument Air Header Ruptures

No violation of regulatory requirements occurred since the affected equipment was not

safety-related. This finding was entered into the licensees corrective action program as

Action Request AR 070600867 and is identified as FIN 05000361;362/2007013-01,

Ineffective Corrective Actions for Instrument Air Header Ruptures.

Enclosure

-30-

4.2

Failure to Follow Abnormal Operating Instruction for the Loss of Instrument Air

Technical Specification 5.5.1.1 requires written procedures to be implemented as

recommended by Regulatory Guide 1.33, Revision 2, Appendix A, February 1978.

Section 6.b of Appendix A recommends procedures governing actions to be taken on a

loss of instrument air. Step 3.h of Procedure SO23-13-5, Loss of Instrument Air,

Revision 5, required notification of all building operators of the possibility of oxygen

deficient areas, monitoring of enclosed spaces for oxygen levels prior to entry, and

monitoring of liquid nitrogen inventory following actuation of the backup nitrogen system

following an instrument air leak. Contrary to this requirement, operators failed to

implement these actions following actuation of the nitrogen system due to an instrument

air line break on June 20, 2007. Because this violation was of very low safety

significance and was entered in the corrective action program as Action Request

AR 070700291, this violation is being treated as an NCV consistent with Section VI.A.1

of the NRC Enforcement Policy: NCV 05000361;362/2007013-02, Failure to Follow

Abnormal Operating Instruction for the Loss of Instrument Air.

4.3

Inadequate Evaluation Results in Runout of Component Cooling Water Pump

10 CFR Part 50, Appendix B, Criterion III, Design Control, requires, in part, that

measures be established to assure that applicable regulatory requirements and the

design basis, as specified in the license application, are correctly translated into

specifications, drawings, procedures, and instructions. It further states that design

control measures shall provide for verifying or checking the adequacy of design, such as

by the performance of design reviews, by the use of alternate or simplified calculational

methods, or by the performance of a suitable testing program. Contrary to the above,

the licensee failed to verify the adequacy of the design associated with the modification

of the CCW NCL isolation valves installed in 1995. Because this finding is of very low

safety significance and has been entered into the corrective action program as Action

Requests AR 070700051 and 070600872, this violation is being treated as an NCV

consistent with Section VI.A of the NRC Enforcement Policy:

NCV 05000361;362/2007013-03, Inadequate Evaluation Results in Runout of

Component Cooling Water Pump.

4.4

Ineffective Corrective Actions for a Failed Control Room Annunciator

No violations of NRC requirements were identified during the review of this issue

because instrument air is not a safety related system. The licensee entered this issue

into the corrective action program as Action Request AR 070601250:

FIN 05000361;362/2007013-04, Ineffective Corrective Actions for a Failed Control

Room Annunciator.

4.5

Inadequate Procedure for a Loss of Instrument Air

Technical Specification 5.5.1.1 requires written procedures to be implemented as

recommended by Regulatory Guide 1.33, Quality Assurance Program Requirements,

Revision 2, Appendix A, February 1978. Section 6.b of Appendix A of Regulatory Guide 1.33 recommends procedures governing actions to be taken on a loss of instrument air.

American National Standard ANS-3.2, Administrative Controls and Quality Assurance

for the Operational Phase of Nuclear Power Plants, February 1976, describes the

requirements for the quality of the procedures specified in Regulatory Guide 1.33.

Enclosure

-31-

Section 5.3.9.1(4) of Standard ANS-3.2 requires emergency procedures to specify

immediate actions for operation of controls or confirmation of automatic actions that are

required to stop the degradation of conditions and mitigate consequences. Contrary to

this requirement, since its release on May 4, 2006, Revision 5 of Procedure SO23-13-5,

Loss of Instrument Air, did not provide adequate guidance for operators to immediately

diagnose and properly respond to a complete loss of the instrument air system.

Because this violation was of very low safety significance and was entered in the

corrective action program as Action Request AR 070801151, this violation is being

treated as an NCV consistent with Section VI.A.1 of the NRC Enforcement Policy:

NCV 05000361;362/2007013-05, Inadequate Procedure for Loss of Instrument Air.

4.6

Simulator Incorrectly Modeled Plant Response to Loss of Instrument Air

10 CFR Part 55.46(c)(1) requires, in part, that the plant-referenced simulator must

demonstrate expected plant response to transient conditions. Contrary to this

requirement, the response modeled by the licensees simulator for the reactor coolant

pump controlled bleedoff valves did not demonstrate expected plant response to the

June 20, 2007 loss of instrument air event. Because this violation was of very low safety

significance and was entered in the corrective action program as Action Requests

AR 070600873 and 070900160, this violation is being treated as an NCV consistent with

Section VI.A.1 of the NRC Enforcement Policy: NCV 05000361;362/2007013-06,

Simulator Incorrectly Modeled Plant Response to Loss of Instrument Air.

4.7

Failure to Follow Procedure for an Impaired Annunciator

Technical Specification 5.5.1.1 requires written procedures to be implemented as

recommended by Regulatory Guide 1.33, Revision 2, Appendix A, February 1978.

Section 1.c of Appendix A recommends administrative procedures governing equipment

control. Section 6.2.2 of Procedure SO23-6-29, Operation of Annunciators and

Indicators, Revision 15, required tracking of impaired annunciators requiring

compensatory actions. Contrary to this requirement, operators failed to track an

impaired annunciator from June 29 to July 2, 2007. Because this violation was of very

low safety significance and was entered in the corrective action program as Action

Request AR 070700291, this violation is being treated as an NCV consistent with

Section VI.A.1 of the NRC Enforcement Policy: NCV 05000361;362/2007013-07,

Failure to Follow Procedure for an Impaired Annunciator.

4.8

Inadequate Implementation of Corrective Actions for Air Operated Valve Regulators

No violation of regulatory requirements occurred. This finding was entered in the

licensees corrective action program as Action Request AR 070600873:

FIN 05000361;362/2007013-07, Inadequate Implementation of Corrective Actions for

Air Operated Valve Regulators.

Enclosure

-32-

4OA6 Meetings, Including Exit

On July 2, 2007, and September 13, 2007, the results of this inspection were presented

to Dr. R. Waldo, Vice President Nuclear Generation, and other members of his staff who

acknowledged the findings. Additionally on October 11, 2007, the final results of this

inspection were presented to A. Scherer, Manager, Nuclear Regulatory Affairs, and

other members of his staff who acknowledged the findings. The inspector confirmed

that no proprietary material was examined during the inspection.

ATTACHMENT 1: SUPPLEMENTAL INFORMATION

ATTACHMENT 2: SPECIAL INSPECTION CHARTER

ATTACHMENT 3: SIGNIFICANCE DETERMINATION EVALUATION

Attachment 1

A1-1

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

K. Flynn, Site Operating Experience Coordinator

S. Gardner, Engineer, Nuclear Regulatory Affairs

B. Katz, Vice President, Nuclear Oversight and Regulatory Affairs

L. Kelly, Engineer, Nuclear Regulatory Affairs

M. Love, Manager, Maintenance

M. Mostafa, Consulting Engineer

K. Rauch, Operations Training Manager

A. Scherer, Manager, Nuclear Regulatory Affairs

P. Schofield, System Maintenance Engineer Supervisor

J. Summy, System Engineering Manager

D. Tuttle, Systems Engineer

T. Vogt, Manager, Operations

R. Waldo, Vice President, Nuclear Generation

D. Wilcockson, Manager, Plant Operations

C. Williams, Manager, Compliance

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened and Closed

05000361;

362/2007013-01

FIN

Ineffective Corrective Actions for Instrument Air Header

Ruptures

05000361;

362/2007013-02

NCV

Failure to Follow Abnormal Operating Instruction for the

Loss of Instrument Air

05000361;

362/2007013-03

NCV

Inadequate Evaluation Results in Runout of Component

Cooling Water Pump

05000361;

362/2007013-04

FIN

Ineffective Corrective Actions for a Failed Control Room

Annunciator

05000361;

362/2007013-05

NCV

Inadequate Procedure for a Loss of Instrument Air

05000361;

362/2007013-06

NCV

Simulator Incorrectly Modeled Plant Response to a Loss of

Instrument Air

05000361;

362/2007013-07

NCV

Failure to Follow Procedure for an Impaired Annunciator

05000361;

362/2007013-08

FIN

Inadequate Implementation of Corrective Actions for Air

Operated Valve Regulators

Attachment 1

A1-2

LIST OF DOCUMENTS REVIEWED

Procedures

Number

Title

Revision

SO23-6-29

Operation of Annunciators and Indicators

15

SO23-13-5

Loss of Instrument Air

5

SO123-0-A8

Trip/Transient and Event Review

1

SO23-12-1

Standard Post Trip Actions

21

SO23-12-2

Reactor Trip Recovery

18

SO23-1-1

Instrument Air System Operation

17

SO123-XV-5.3

Maintenance Rule Program

9

SO23-6-29

Operation of Annunciators and Indicators

15

SO123-XV-50.39.1

Division Investigative Reports

0

SO123-I-1.42

Maintenance Division Experience Report

0

SO123-XV-50

Corrective Action Process

6

Action Requests

070400754

060101956

070400766

070600867

051000080

050600477

070600877

050901305

041101801

060101956

050901037

070600914

070600867

070600872

960500111

031001558

041000977

010601495

070600873

041002146

061001297

070600914

040901643

060101956

070500196

070600870

070601250

070501276

070400776

060200898

Work Orders/Maintenance Work Orders

07061161000

06012099000

94062628000

94062628001

07041277000

07061216000

Attachment 1

A1-3

07041921000

06020520000

05111108000

05111009000

05061587000

05061588000

06020580000

05111105000

05111106000

05111107000

07051780000

05061583000

05061589000

05061590000

06021071000

07050347000

05061584000

05061585000

05061586000

07061216000

07061324000

07061325000

Drawings

Number

Title

Revision

F-10946M

Component Cooling Water System No. 1203

21

F-10543M

Component Cooling Water System No. 1203

15

40191A

Compressed Air System

15

40191B

Compressed Air System

21

40191C

Compressed Air System

19

40191D

Compressed Air System

34

40191DSO3

Compressed Air System

19

40191F

Compressed Air System

13

40191X

Compressed Air System

2

40190C

Respiratory Service Air

22

40192A

Auxiliary Gas System

20

40192B

Auxiliary Gas System

16

40191GSO3

Instrument Air Distribution

7

40191E-10

Instrument Air System

10

40191G

Instrument Air Distribution

8

Calculations

M-DSC-429, Evaluation of Joint Restraint Clamp on Instrument Air Piping, Revision 0

M-0091, Backup Nitrogen for the Instrument Air System Equipment Sizing, Revision 0

IPE-HC-006, Operator Action Summary Data Sheet Post-Initiator Human Error Probability

Calculation Worksheet

Attachment 1

A1-4

Miscellaneous Information

PRA-07-007, PRA Preliminary Evaluation of Loss of Instrument Air Event Resulting in Unit 2

Trip, Dated June 22, 2007

Operator Action Summary Data Sheet Post-Initiator Human Error Probability Calculation

Worksheet

DBD-SO23-540, Instrument Air/Dedicated Backup Nitrogen System, Revision 6

Engineering Change Package 070600914-6, Revision 0

SONGS 2 Instrument Air System Failed Fitting Metallurgical Evaluation

SONGS System Health Reports for the Instrument Air System and Vendor Owned Nitrogen

Package

Failure Analysis Report No.94-006, Failure Analysis of the Instrument Air Fitting for

Temperature Gauge 2/3TI5380

Failure Analysis Report No.94-009, Failure Analysis of the Instrument Air Fitting for

Temperature Gauge 2/3TI5380, Supplement 1, Dated October 3, 1994

DBD-SO23-540, Instrument Air/Dedicated Backup Nitrogen Systems, Revision 6

Maintenance Rule Guide Book, Dated February 2004

SD-SO23-400, Component Cooling Water System, Revision 6

Meeting Agenda Maintenance Rule Expert Panel, Dated June 21, 2007

Regulatory Guide 1.160, Monitoring the Effectiveness of Maintenance at Nuclear Power Plants,

Revision 2

Licensee Event Report No. 2007-001-01, Revision 1, Dated August 24, 2004

Substitute Equivalency Evaluation 020040, Substitute 67CFR-237 Series Regulator for 67AFR-

237, Revision 1

A2-1

Attachment 2

June 26, 2007

MEMORANDUM TO: Geoffrey Miller, Senior Resident Inspector, Grand Gulf

Jeffrey Josey, Resident Inspector, Arkansas Nuclear One

FROM:

Arthur T. Howell III, Director, Division of Reactor Projects /RA AVegel for/

SUBJECT:

SPECIAL INSPECTION CHARTER TO EVALUATE THE SAN ONOFRE

NUCLEAR GENERATING STATION INSTRUMENT AIR FAILURE

A Special Inspection Team is being chartered in response to the Unit 2 San Onofre Nuclear

Generating Station loss of instrument air event on June 20, 2007. You are hereby designated

as the Special Inspection Team members. Mr. Miller is designated as the team leader. The

assigned SRA to support the team is David Loveless.

A.

Basis

On June 20, 2007, a 3-inch diameter instrument air line failed. At SONGS, instrument

air is a shared system, but the system is equipped with certain protective features

(excess flow check valves) to ensure that a failure in the piping system on one unit does

not significantly affect instrument air pressure on the other unit. On Unit 2, instrument

air pressure dropped significantly, from approximately 110 psig to about 43 psig. The

loss of instrument air pressure caused the feedwater control valves to stop functioning

and water level in the steam generators increased in an uncontrolled manner.

Operators manually tripped the reactor. The operators also lost control of the steam

dumps to the condenser (the normal heat removal method) and controlled steam

generator pressure and decay heat removal using the steam generator atmospheric

dump valves. The chemical and volume control system letdown function auto-isolated

and operators manually controlled pressurizer level with a charging pump. On Unit 3,

the pressure drop was not as significant but appeared to be more than expected.

However, Unit 3 Operators maintained control of all functions during the event.

Operators were able to isolate the failed instrument air line approximately 30 minutes

later and regained control of the Unit 2 condenser steam dumps.

During post-trip discussions with the operators, one operator stated that they had

experienced other instrument air piping failures but the affected piping was much

smaller and did not significantly challenge plant operations. One such failure occurred

in 1994.

This Special Inspection Team is chartered to review the circumstances related to

historical and present instrument air piping problems and assess the effectiveness of the

A2-2

Attachment 2

licensees actions for resolving these problems. The team will also assess the

effectiveness of the immediate actions taken by the licensee in response to the loss of

instrument air event on June 20, 2007.

B.

Scope

The team is expected to address the following:

1.

Develop a chronology (time-line) that includes significant event elements.

2.

Evaluate the operator response to the event. Ensure that operators responded

in accordance with plant procedures and took appropriate mitigating actions.

3.

Develop an understanding of the interface between instrument air and other risk

important systems, including the possible reliance, either short term or long term,

of safety related components on instrument air.

4.

Evaluate the plant response to the event. Ensure that all systems responded as

designed. In particular, verify that design provisions, intended to prevent failure

in one units piping from causing an excessive pressure drop in the other unit,

worked properly (see UFSAR Sections 9.3.1.1.E and 9.3.1.2.3).

5.

Assess the licensees root cause determination for the instrument air piping

failure, the extent of condition review, the common cause evaluation and

corrective measures. Evaluate whether the timeliness of the corrective

measures are consistent with the safety significance of the problems.

6.

Identify previous instrument air piping problems that may have been precursors

to the June 20 event, including one event in 1994. Evaluate the licensees

corrective measures and extent of condition reviews for those problems.

7.

Evaluate the licensees instrument air system maintenance and testing

programs. Verify that the programs are adequate and that the licensee is

following the program provisions. Pay particular attention to the historical health

of the instrument air compressors and piping system.

8.

Evaluate pertinent industry operating experience that represent potential

precursors to the June 20 event, including the effectiveness of licensee actions

taken in response to the operating experience. As a minimum include Generic Letter 88-14, Instrument Air Supply System Problems Affecting Safety-Related

Equipment, including the licensees response to the generic letter; and NRC

Information Notice 2002-29,Recent Design Problems in Safety Functions of

Pneumatic Systems. You may also use NUREG 1837, Regulatory

Effectiveness Assessment of Generic Issue 43 and Generic Letter 88-14, to aid

in your assessment. The NUREG can be found at:

http://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr1837/sr1837.pdf

A2-3

Attachment 2

9.

Determine if there are any potential generic issues related to the failure of the

SONGS instrument air piping. Promptly communicate any potential generic

issues to Region IV management.

10.

Collect data as necessary to support a risk analysis. Work closely with the

Senior Reactor Analyst during this inspection.

C.

Guidance

Inspection Procedure 93812, Special Inspection, provides additional guidance to be

used by the Special Inspection Team. Your duties will be as described in Inspection

Procedure 93812. The inspection should emphasize fact-finding in its review of the

circumstances surrounding the event. It is not the responsibility of the team to examine

the regulatory process. Safety concerns identified that are not directly related to the

event should be reported to the Region IV office for appropriate action.

The Team will report to the site, conduct an entrance, and begin inspection no later than

June 27, 2007. While on site, you will provide daily status briefings to Region IV

management, who will coordinate with the Office of Nuclear Reactor Regulation, to

ensure that all other parties are kept informed. A report documenting the results of the

inspection should be issued within 30 days of the completion of the inspection.

This Charter may be modified should the team develop significant new information that

warrants review. Should you have any questions concerning this Charter, contact me at

(817) 860-8147.

A3-1

Attachment 3

ATTACHMENT 3

SIGNIFICANCE DETERMINATION EVALUATION

San Onofre Nuclear Generating Station

Failure of Instrument Air System Header

Phase 3 Analysis

A.

Brief Description of Issue

On June 20, 2007, instrument air pressure at San Onofre Unit 2 dropped significantly

following the separation of a 3-inch fitting in the system air header located in the

auxiliary building. This caused the feedwater control valves to stop functioning, resulting

in an uncontrolled increase in steam generator water level. Operators manually tripped

the Unit 2 reactor. The loss of instrument air caused containment isolations and a loss

of most power conversion system functions.

The licensee performed a metallurgical analysis of the failed joint and determined that

the cause of the failure was poor workmanship during initial installation. The analysis

concluded that the joint was most likely leaking since initial plant startup because, during

original installation the brazing activity resulted in inadequate solder coverage and the

connection had continued to deteriorate throughout the life of the plant. During a

walkdown of the system in both units, licensee personnel discovered that 32 other large

fittings were leaking at the joint.

A special inspection team reviewed the licensees root cause evaluation and

metallurgical evaluation for this event. During their review, the team noted that there

had been a previous similar air header failure at San Onofre in June 1994. At that time,

both Units 2 and 3 experienced a loss of instrument air following the failure of an

improperly soldered joint. A metallurgical analysis conducted in 1994 concluded that

this joint had also likely been leaking since initial startup from inadequate solder

coverage.

A large amount of industry operating experience has been available that deals with

soldered joint issues. During the original evaluation of related operational experience

reports, done by the licensee in 1992, they failed to properly assess the impact to San

Onofre. Engineers had determined that if failures were going to have occurred because

of inadequate fit-up and/or solder penetration, the failures would have occurred within a

relatively short time frame. Therefore, they assumed that related industry experience

was not applicable to San Onofre. The team also determined that the licensee failed to

adequately reassess their position when they experienced an air line joint failure in

1994, and as a result, failed to take effective corrective actions following that failure.

B.

Statement of the Performance Deficiency

The licensee failed to take effective corrective actions in response to the failure of an

improperly made soldered joint in the instrument air header affecting both units at San

A3-2

Attachment 3

Onofre in June 1994. Specifically, contrary to Section 6.2.3 of

Procedure SO-123-I-1.42, Maintenance Division Experience Report, Revision 0, the

licensee failed to implement corrective actions to prevent recurrence for an equipment

failure with the potential to cause a significant plant transient, and failed to appropriately

consider previous industry and plant experience similar to the event. Additionally,

licensee personnel failed to properly evaluate and take corrective actions based on

industry operating experience through 2006 involving improperly made soldered joints in

instrument air systems. As a result, an additional failure of an improperly made

instrument air header joint occurred at San Onofre on June 20, 2007, resulting in a

complete loss of instrument air to Unit 2.

C.

Significance Determination Basis

1.

Phase 1 screening logic, results and assumptions

In accordance with NRC Inspection Manual Chapter 0612, Appendix B, "Issue

Screening," the team determined that this finding represented a licensee

performance deficiency. The team then determined that the issue was more

than minor because the finding was associated with the equipment performance

attribute and affected the initiating events cornerstone objective to limit the

likelihood of those events that upset plant stability and challenge critical safety

functions during shutdown as well as power operations.

The team evaluated this finding using the, "SDP Phase 1 Screening Worksheet

for the Initiating Events, Mitigating Systems, and Barriers Cornerstones,"

provided in Manual Chapter 0609, Appendix A, "Significance Determination of

Reactor Inspection Findings for At-Power Situations." A Phase 2 estimation was

required because the associated performance deficiency represented an

increase in both the likelihood of a reactor trip and the probability that the power

conversion system would be unavailable.

2.

Phase 2 Risk Estimation

In accordance with Manual Chapter 0609, Appendix A, Attachment 1, User

Guidance for Significance Determination of Reactor Inspection Findings for At-

Power Situations, the team evaluated the subject findings using the Risk-

Informed Inspection Notebook for San Onofre Nuclear Generating Station

(SONGS) Units 2 and 3, Revision 2.1. The dominant affected accident

sequences are provided in Table 1. The team assumed that the exposure period

was greater than 30 days, that the performance deficiency increased the

likelihood that a complete loss of instrument air would occur, and that there was

no affect on mitigating systems other than that modeled in the risk-informed

notebook.

A3-3

Attachment 3

TABLE 1

Increased Likelihood of a Complete Loss of Instrument Air

Phase 2 Sequences

Initiating Event

Sequence

Mitigating Functions

Results

Loss of Instrument

Air

1

LOIA-AFW

7

2

LOIA-RCPTRIP-HPR

9

3

LOIA-RCPTRIP-CNT

9

4

LOIA-RCPTRIP-EIHP

9

Using the counting rule worksheet, the result from this estimation indicated that

this finding was of very low safety significance (GREEN).

A senior reactor analyst reviewed the Phase 2 estimation and determined that

the risk-informed notebook and the licensees PRA had a common error that

significantly underestimated the risk of this deficiency. The loss of instrument air

initiating event frequency had been established as 6.4 x 10-5/year by assuming

that a loss of all active instrument air system components as well as a loss of the

backup nitrogen system was required to realize a complete loss of instrument

air. However, there are many system breaches and other passive component

failures that would prevent the backup nitrogen system from performing its

function.

Therefore, the analyst determined that the finding should be evaluated using the

Phase 3 process.

3.

Phase 3 Analysis

The analyst quantified the change in risk of the subject performance deficiency

as indicated in the paragraphs below. The change in internal event risk was

estimated as 4.7 x 10-7 over an entire assessment period. The risk related to

seismic events changed by 4.1 x 10-7 and that related to internal fires by

9.5 x 10-9. This resulted in a total change in CDF of 8.9 x 10-7. Therefore, the

analyst determined that the subject finding was of very low safety significance

(Green).

Internal Initiating Events:

The following techniques were used in this evaluation.

a.

The analyst quantified the internal risk using the Standardized Plant

Analysis Risk (SPAR) Model for San Onofre 2 & 3, Revision 3.21, created

in October 2005. The analyst modeled a loss of instrument air by

assuming that the affect was equivalent to a transient with a complete

loss of the power conversion system with the possibility of a recovery of

A3-4

Attachment 3

condensate by bypassing the feedwater isolation and depressurizing the

steam generators. The likelihood of a loss of instrument air was then

increased to 6.8 x 10-2/year as a result of the performance deficiency.

The resulting quantification indicated an increase over the baseline core

damage frequency of 4.7 x 10-7 over a 365-day exposure.

b.

The licensee developed a model for analyzing the internal risk associated

with the event using the current version of their PRA model. The licensee

revised the model by setting the loss of instrument air initiating event

frequency to 0.5 indicating that operators could have recovered the

condition prior to a reactor trip. Additionally, the licensee estimated that

50% of the transients initiated by an instrument air system breach would

be recovered by operators prior to core damage. No other initiators were

considered to be applicable to this condition.

Licensee analysts changed their model to show that the controlled

bleedoff valves remained open. As documented in Licensee Event

Report 50-361/2007001, the licensees conditional core damage

probability (CCDP) for this event was 3.3 x 10-6.

At the analysts request, the licensee provided a CCDP for a generic loss

of instrument air. The value provided by their model was 4.85 x 10-6.

The analyst converted the licensees CCDP to a change in the core

damage frequency by multiplying it with the calculated initiating event

frequency. The result was 3.29 x 10-7 over the 1-year assessment

period. This tends to corroborate the analysts value and suggests that

the initiating event frequency is the primary difference between the two

values.

In performing the Phase 3 evaluation, the following influential assumptions were

made by the analyst:

a.

The failure of a large fitting in the instrument air system at the San

Onofre site would cause a reactor trip on the subject unit at least 1/2 the

time. This was based on the 2 historical failures in the units.

b.

This condition existed for many years for both units and should be

evaluated over the most recent 1-year assessment period.

c.

The baseline failure rate of the instrument air system should have been

the boolean combination of the system components including the backup

instrument air system. Given intact system piping, this was calculated by

the licensee to be 6.4 x 10-5/year.

d.

The instrument air system had been functional for 22.09 reactor-years.

This value is the number of years that either or both reactors were critical.

Such an assumption was used because the system is common to both

units.

A3-5

Attachment 3

e.

Because the condition causing failures in the instrument air headers was,

in part, an aging issue and because there were 32 additional leaking

fittings identified in the system, the analyst assumed that an additional

failure was eminent prior to repair (eg: 3 failures were assumed to have

occurred).

f.

Upon loss of instrument air, the condensate system is potentially still

available given operators depressurize the reactor coolant system and

manually realign the condensate system to bypass the feedwater

regulating valves.

The following calculations were performed during this analysis:

a.

The analyst calculated the revised likelihood of a loss of instrument air.

The result of 1.36 x 10-1 was calculated based on the 2 historical

breaches of the system, an additional postulated breach to account for

the aging affect, and a service time of 22.09 reactor-years.

b.

As stated above, historically San Onofre has had 2 events that involved

breaches of an instrument air header. On one of these occasions,

operators were able to identify and limit the leak to prevent a reactor

transient. Therefore, the initiating event likelihood was reduced by 50%

to 6.8 x 10-2.

c.

The analyst estimated the nonrecovery probability for the operators

depressurizing the reactor coolant system and feeding the steam

generators using the condensate system. Three components went into

this analysis: 1) the human error probability calculated using the SPAR-H

method; 2) the probability that the atmospheric dump system failed; and

3) the probability that the condensate system failed mechanically. The

last two probabilities were calculated by solving appropriate portions of

the SPAR fault trees. The overall nonrecovery probability was calculated

to be 9.3 x 10-2.

d.

The analyst used the SPAR model to quantify the internal change in risk

for a loss of instrument air by modeling a loss of condensate, bypass

capability, and main feedwater. The analyst set all initiators to the house

event, FALSE, with the exception of transients. The transient initiator

was used as a surrogate initiator for the loss of instrument air, and the

initiating event frequency was set to the calculated frequency above.

The analyst also provided for recovery of the condensate system by

adding a basic event to the COND fault tree. This basic event COND-

RECOVERY was added under the AND gate, COND-SYS-4. This gate

then indicated that the failure of both feedwater pathways as well as

nonrecovery of the condensate system via the alternate pathway were

required to fail the condensate function.

A3-6

Attachment 3

The changes to basic events used for this model are shown on Table 2.

TABLE 2

Changes to SPAR Model Basic Events

Basic Event

Initial Value

Adjusted Value

IE-TRANS

7.0 x 10-1

6.8 x 10-2

All Other Initiators

Nominal

FALSE

MFW-AOV-CF-SGS

2.7 x 10-5

TRUE

MFW-AOV-OC-4048

7.2 x 10-6

TRUE

MFW-AOV-OC-4052

7.2 x 10-6

TRUE

MFW-SYS-AVAILABLE

0.8

1.0

MFW-SYS-UNAVAIL

0.2

FALSE

MSS-TBV-CF-TBVS

2.6 x 10-6

TRUE

COND-RECOVERY

N/A

9.3 x 10-2

The model was then quantified. The case core damage frequency (CDF)

was computed to be 4.7 x 10-7/year and the baseline CDF, using the

baseline initiating event frequency of 6.4 x 10-5/year, was computed to be

4.1 x 10-10/year. The change in CDF (CDF) was then calculated by

subtracting the baseline CDF from the case CDF. This resulted in a

CDF for the increased likelihood for a loss of instrument air of 4.7 x 10-7

over a 365-day exposure. The dominant sequences from the SPAR and

licensee models are documented in Table 3.

Table 3

Phase 3 Dominant Accident Sequences

Model

Initiating Event

Sequence

Contribution

SPAR 3.21

Loss of Instrument Air

AFW,

Condensate

4.5 x 10-7

Licensee's

Revised

Loss of Instrument Air

AFW, Failure to

Depressurize

3.1 x 10-7

A3-7

Attachment 3

External Initiating Events:

The analyst used the following methods for determining the change in risk from

external events. The change in risk from an increase in the frequency of a loss

of instrument air was estimated to be 4.2 x 10-7 for a 365-day period. The

methods used are documented below:

a.

Fire

The analyst used the San Onofre IPEEE to estimate the change in risk

resulting from internal fire. The only fire areas where risk could be

increased by the subject improperly soldered fittings would be those

containing instrument air header piping. As the limiting area, the analyst

reviewed the licensees evaluation of Fire Area 2-TB-148. The fire

ignition frequency for this area was 4.5 x 10-2/yr. The analyst assumed

that only 0.1 of the fires would grow to a size that could impact the

instrument air system (severity factor) and that about 50 percent of the

fires would cause a weakening of the improperly soldered fitting joints

without causing baseline failure of the system. Using a conditional core

damage probability of 4.2 x 10-6, the change in core damage frequency

from the subject performance deficiency related to a turbine building fire

was estimated as 9.5 x 10-9 over the 365-day exposure period.

b.

Seismic

The analyst determined that, for the subject performance deficiency to

affect the core damage frequency, a seismic event must result in a failure

of an instrument air system header fitting without otherwise affecting

instrument air system components.

To estimate the baseline seismic failure of the system, the analyst used

the seismic fragility of the air-operated valves which were the least

durable components in the system as designed. The analyst evaluated

the subject performance deficiency by determining each of the following

parameters for any seismic event producing a given range of median

acceleration "a" [SE(a)]:

1.

The frequency of the seismic event SE(a) (SE(a)) ;

2.

The probability that a system header fitting fails (PHeader-SE(a));

3.

The probability that an air-operated valve fails (P System-SE(a));

4.

The conditional change in CDP (CCDPSE(a))

The CDF for the acceleration range in question (CDFSE(a)) can then be

quantified as follows:

CDFSE(a) = SE(a) * PHeader-SE(a) * (1 - PSystem-SE(a)) * CCDPSE(a)

A3-8

Attachment 3

Given that each range a was selected by the analyst specifically to be

independent of all other ranges, the total increase in risk, CDF, can be

quantified by summing the CDFSE(a) for each range evaluated as follows:

6

CDF = 3 CDFSE(a)

a=.03

over the range of SE(a).

1.

Frequency of the Seismic Event

NRC research data indicated that seismic events of 0.05g or less have

little to no impact on internal plant equipment. The analyst assumed

that seismic events less than 0.03g do not directly affect the plant.

The analyst assumed that seismic events greater than 6.0g lead to

core damage. The analyst therefore examined seismic events in the

range of 0.03g to 6.0g.

The analyst divided that range of seismic events into segments (called

"bins" hereafter); specifically, seismic events from 0.03g to 0.1g were

binned by hundredths, seismic events from 0.1g to 1.0g were binned

by tenths, and seismic events from 1.0g to 6.0g were binned by ones.

In order to determine the frequency of a seismic event for a specific

range of ground motion (g values), the analyst used a plot provided by

the licensee and obtained values for the frequency of the seismic

event that generates a level of ground motion (in peak ground

acceleration) that exceeds the lower value in each of the bins. The

analyst then calculated the difference in these frequency of

exceedance values to obtain the frequency of seismic events for the

binned seismic event ranges.

For example, according to the San Onofre curves, the frequency of

exceedance for a 0.6g seismic event is estimated at 3.9 x 10-2/yr and a

0.7g seismic event at 3.5 x 10-2/yr. The frequency of seismic events

with median acceleration in the range of 0.6g to 0.7g [SE(0.6-0.7)]

equals the difference, or 4.0 x 10-3/yr.

2.

Probability of a Header Fitting Failure

Given that the historical header failures were the result of insufficient

solder coverage and were caused by slow degradation via air leaks,

the analyst assumed that a moderately large earthquake could result

in the failure of a leaking header fitting. Therefore, the analyst used a

median seismic fragility of a long, brittle component as an estimate of

the fragility of the 32 degraded instrument air header fittings. The

seismic fragility selected was 0.3g.

A3-9

Attachment 3

The analyst obtained data on switchyard components from the Risk

Assessment of Operating Events Handbook; Volume 2, External

Events, Revision 4, which referenced generic fragility values listed in:

<

NUREG/CR-6544, Methodology for Analyzing Precursors to

Earthquake-Initiated and Fire-Initiated Accident Sequences,

April 1998; and

<

NUREG/CR-4550, Vols 3 and 4 part 3, Analysis of Core

Damage Frequency: Surry / Peach Bottom, 1986

The references describe the mean failure probability for various

equipment using the following equation:

Pfail(a) = [ ln(a/am) / (r

2 + u

2)1/2]

Where is the standard normal cumulative distribution

function and

a =

median acceleration level of the seismic event;

am=

median of the component fragility;

r=

logarithmic standard deviation representing random

uncertainty;

u=

logarithmic standard deviation representing systematic

or modeling uncertainty.

In order to calculate the probability that a degraded fitting would fail

given a seismic event, the analyst used the following generic seismic

fragilities:

am = 0.3g

r = 0.30

u = 0.45

Using the above normal cumulative distribution function equation the

analyst determined the conditional probability of failure given a seismic

event. For each of the bins the calculation was performed substituting

for the variable "a" (median acceleration level) the acceleration levels

obtained from the bins described above. The following table shows

the results of the calculation for various acceleration levels.

Median Acceleration Level/Probability of Failure

0.03g

3.6 x 10-5

0.3g

6.1 x 10-1

1.0g

1.0

0.07g

5.2 x 10-3

0.7g

9.5 x 10-1

A3-10

Attachment 3

3.

Probability that Air-Operated Valves Fail

In order to calculate the probability that the instrument air system

would fail during a given seismic event for reasons other than

improperly soldered fittings, the analyst used the following generic

seismic fragilities for air-operated valves:

am = 3.8g

r = 0.35

u = 0.50

Using the above standard normal cumulative distribution function

equation, the analyst determined the conditional probability that the

instrument air system would fail from failure of system valves given a

seismic event for each of the bins. The calculation was performed

substituting for the variable "a" (median acceleration level) the

acceleration levels obtained from the bins described above. The

following table shows the results of the calculation for various

acceleration levels.

Median Acceleration Level/

Probability of Air-Operated Valve Failure

0.03g

7.9 x 10-15

0.3g

4.7 x 10-5

1.0g

6.4 x 10-2

0.07g

6.3 x 10-11

0.7g

3.9 x 10-3

4.

Conditional Change in Core Damage Probability

The analyst evaluated the spectrum of seismic initiators to determine

the resultant impact on the reliability and availability of mitigating

systems affecting the subject performance deficiency.

The analyst used the SPAR model, to perform the Phase 3 evaluation.

The analyst started with the model discussed above used to quantify

the change in risk from internal events. However, the analyst set the

initiating event frequency for a transient to 1.0 and all other initiating

event probabilities in the SPAR model to zero. Because of the very

narrow time windows discussed for condensate recovery, and the

added burdens on operators both emotionally and physically following

a seismic event, the analyst set the nonrecovery probability for the

condensate system to 1.0. The SPAR model showed the resultant

core damage probability as 1.02 x 10-4, which represented the value

used in the above equation.

A3-11

Attachment 3

The SPAR Model was then requantified indicating no loss of

instrument air. The CCDP for this baseline condition was 4.35 x 10-7.

Therefore, the change in core damage probability is:

CCDPSE(a) = 1.02 x 10-4 - 4.35 x 10-7 = 1.02 x 10-4

Phase 3 Seismic Results

Given the assumptions previously discussed, the total increase in core

damage frequency was estimated to be about 4.1 x 10-7 for seismic events

ranging from 0.03g to 6.0g.

c.

Winds, Floods, and Other External Events

The analyst reviewed the IPEEE and determined that no other credible

scenarios initiated by high winds, floods, fire, and other external events could

initiate a failure of the degraded instrument air header fittings. Therefore, the

analyst concluded that external events other than internal fires and seismic

events were not significant contributors to risk for this finding.

Risk Contribution from Large Early Release Frequency (LERF):

Using IMC 0609 Appendix H, the SRA determined that this was a Type A finding for a

large dry containment. For PWR plants with large dry containments, only findings

related to accident categories ISLOCA and SGTR have the potential to impact LERF.

In addition, an important insight from the IPE program and other PRAs is that the

conditional probability of early containment failure is less than 0.1 for core damage

scenarios that leave the RCS at high pressure (>250 psi) at the time of reactor vessel

breach. Since this finding is not related to ISLOCA or SGTR, and the core damage

scenarios for this finding leave the RCS at high pressure, the analyst concluded that

LERF is not a significant contributor to the risk associated with this finding.