ML072950104
| ML072950104 | |
| Person / Time | |
|---|---|
| Site: | San Onofre  |
| Issue date: | 10/19/2007 |
| From: | Clark J NRC/RGN-IV/DRP/RPB-E |
| To: | Rosenblum R Southern California Edison Co |
| References | |
| IR-07-013 | |
| Download: ML072950104 (76) | |
See also: IR 05000361/2007013
Text
October 19, 2007
Richard M. Rosenblum
Senior Vice President and
Chief Nuclear Officer
Southern California Edison Company
San Onofre Nuclear Generating Station
P.O. Box 128
San Clemente, CA 92674-0128
SUBJECT:
SAN ONOFRE NUCLEAR GENERATING STATION - NRC SPECIAL
INSPECTION REPORT 05000361/2007013; 05000362/2007013
Dear Mr. Rosenblum:
On September 13, 2007, the U.S. Nuclear Regulatory Commission (NRC) completed a special
inspection at your San Onofre Nuclear Generating Station facility. This inspection examined
activities associated with the loss of instrument air event on June 20, 2007. On this occasion,
instrument air pressure on Unit 2 dropped significantly, causing the feedwater control valves to
stop functioning and resulting in an increase in steam generator water level. Operators
manually tripped the Unit 2 reactor. The NRC's initial evaluation satisfied the criteria in NRC
Management Directive 8.3, NRC Incident Investigation Program, for conducting a special
inspection. The basis for initiating this special inspection is further discussed in the inspection
charter, which is included in this report as Attachment 2. The determination that the inspection
would be conducted was made by the NRC on June 26, 2007, and the inspection started on
June 27, 2007.
The enclosed inspection report documents the inspection findings, which were discussed on
September 13, 2007 and again on October 11, 2007, with members of your staff. The
inspection examined activities conducted under your license as they relate to safety and
compliance with the Commission's rules and regulations and with the conditions of your license.
The inspectors reviewed selected procedures and records, observed activities, and interviewed
personnel.
The report documents eight NRC identified and self-revealing findings of very low safety
significance (Green). The eight findings involved issues concerning both the failure of your
processes and programs to prevent or mitigate the loss of instrument air event, and the
subsequent failure of your staff to thoroughly evaluate operator and equipment responses
following the event. The NRC is concerned about the occurrence of this event and the less
than adequate reviews conducted by your staff, and will conduct followup baseline inspections
to verify that your corrective actions in response to this inspection are thorough and effective.
Five of the findings were determined to involve violations of NRC requirements. Because of
their very low safety significance and because they were entered into your corrective action
program, the NRC is treating these findings as noncited violations (NCVs) consistent with
Section VI.A.1 of the NRC Enforcement Policy. If you contest these NCVs, you should provide
Southern California Edison Company
- 2 -
a response within 30 days of the date of this inspection report, with the basis for your denial, to
the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington
DC 20555-0001; with copies to the Regional Administrator, U.S. Nuclear Regulatory
Commission Region IV, 611 Ryan Plaza Drive, Suite 400, Arlington, Texas, 76011-4005; the
Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington
DC 20555-0001; and the NRC Resident Inspector at the San Onofre Nuclear Generating
Station facility.
In accordance with 10 CFR 2.390 of the NRC's Rules of Practice, a copy of this letter, its
enclosure, and your response (if any) will be made available electronically for public inspection
in the NRC Public Document Room or from the Publicly Available Records (PARS) component
of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at
http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
Jeffrey A. Clark, Chief
Projects Branch E
Division of Reactor Projects
Dockets: 50-361
50-362
License: NPF-10
Enclosure: Inspection Report 05000361/2007013; 05000362/2007013
Attachment 1: Supplemental Information
Attachment 2: Special Inspection Charter
Attachment 3: Significance Determination Evaluation
cc w/Enclosure:
Chairman, Board of Supervisors
County of San Diego
1600 Pacific Highway, Room 335
San Diego, CA 92101
Gary L. Nolff
Assistant Director-Resources
City of Riverside
3900 Main Street
Riverside, CA 92522
Mark L. Parsons
Deputy City Attorney
City of Riverside
3900 Main Street
Riverside, CA 92522
Dr. David Spath, Chief
Division of Drinking Water and
Environmental Management
California Department of Health Services
850 Marina Parkway, Bldg P, 2nd Floor
Richmond, CA 94804
Michael J. DeMarco
San Onofre Liaison
San Diego Gas & Electric Company
8315 Century Park Ct. CP21G
San Diego, CA 92123-1548
Director, Radiological Health Branch
State Department of Health Services
P.O. Box 997414 (MS 7610)
Sacramento, CA 95899-7414
Southern California Edison Company
- 3 -
Mayor
City of San Clemente
100 Avenida Presidio
San Clemente, CA 92672
James D. Boyd, Commissioner
California Energy Commission
1516 Ninth Street (MS 34)
Sacramento, CA 95814
Douglas K. Porter, Esq.
Southern California Edison Company
2244 Walnut Grove Avenue
Rosemead, CA 91770
Mr. Raymond W. Waldo, Vice President,
Nuclear Generation
Southern California Edison Company
San Onofre Nuclear Generating Station
P.O. Box 128
San Clemente, CA 92674-0128
A. Edward Scherer
Southern California Edison Company
San Onofre Nuclear Generating Station
P.O. Box 128
San Clemente, CA 92674-0128
Brian Katz
Southern California Edison Company
San Onofre Nuclear Generating Station
P.O. Box 128
San Clemente, CA 92674-0128
Mr. Steve Hsu
Department of Health Services
Radiologic Health Branch
MS 7610, P.O. Box 997414
Sacramento, CA 95899-7414
Mr. James T. Reilly
Southern California Edison Company
San Onofre Nuclear Generating Station
P.O. Box 128
San Clemente, CA 92674-0128
Southern California Edison Company
- 4 -
Electronic distribution by RIV:
Regional Administrator (EEC)
DRP Director (ATH)
DRS Director (DDC)
DRS Deputy Director (RJC1)
Senior Resident Inspector (CCO1)
Branch Chief, DRP/E (JAC)
Senior Project Engineer, DRP/E (GDR)
Team Leader, DRP/TSS (CJP)
RITS Coordinator (MSH3)
Only inspection reports to the following:
V. Dricks, PAO (VLD)
D. Pelton, OEDO RIV Coordinator (DLP)
ROPreports
SO Site Secretary (vacant)
SUNSI Review Completed: _JAC__ ADAMS: : Yes
G No Initials: __JAC_
- Publicly Available G Non-Publicly Available G Sensitive
- Non-Sensitive
R:\\_REACTORS\\SO\\2007\\SO2007-13RP-GBM.wpd
RIV:SRI:DRP/C
RI:DRP/E
SRI:DRP/E
SRA:DRS
C:DRP/E
GBMiller
JEJosey
CCOsterholtz
DPLoveless
JAClark
/RA/
T-GBM
E=GBM
/RA/
/RA/
10/16 /07
10/17/07
10/16/07
10/17/07
10/19/07
OFFICIAL RECORD COPY
T=Telephone E=E-mail F=Fax
Enclosure
-1-
U.S. NUCLEAR REGULATORY COMMISSION
REGION IV
Docket:
50-361, 50-362
Licenses:
Report No.:
05000361/2007013; 05000362/2007013
Licensee:
Southern California Edison Co. (SCE)
Facility:
San Onofre Nuclear Generating Station, Units 2, 3
Location:
5000 S. Pacific Coast Hwy.
San Clemente, California
Dates:
June 27 through September 13, 2007
Inspectors:
J. Josey, Resident Inspector, Project Branch E, DRP
D. Loveless, Senior Reactor Analyst
G. Miller, Senior Resident Inspector, Project Branch C, DRP
C. Osterholtz, Senior Resident Inspector, Project Branch E, DRP
M. Sitek, Resident Inspector, Project Branch E, DRP
Approved By:
Jeffrey A. Clark, Chief
Project Branch E
Division of Reactor Projects
Enclosure
-2-
SUMMARY OF FINDINGS
IR 05000361/2007013, 05000362/2007013; 06/27/07 - 07/02/07; San Onofre Nuclear
Generating Station, Units 2, 3, and Independent Spent Fuel Storage Installation; Special
Inspection in response to an instrument air header break and Unit 2 trip on June 20, 2007.
The report covered a 6-day period (June 27 - July 2, 2007) of onsite inspection, with inoffice
review through September 13, 2007, by a special inspection team consisting of one senior
resident inspector, one resident inspector, and one senior reactor analyst. Eight findings were
identified. The significance of most findings is indicated by its color (Green, White, Yellow, or
Red) using Inspection Manual Chapter 0609, Significance Determination Process. Findings
for which the significance determination process does not apply may be Green or be assigned a
severity level after NRCs management review. The NRC's program for overseeing the safe
operation of commercial nuclear power reactors is described in NUREG-1649, Reactor
Oversight Process, Revision 3, dated July 2000.
Summary of Event
The NRC conducted a special inspection to better understand the circumstances surrounding
an instrument air header break and Unit 2 trip on June 20, 2007. In accordance with NRC
Management Directive 8.3, NRC Incident Investigation Program, it was determined that this
event involved multiple failures in systems used to mitigate the effects of an actual event,
involved potential adverse generic implications, and had sufficient risk significance to warrant a
special inspection.
A.
NRC-Identified and Self-Revealing Findings
Cornerstone: Initiating Events
Green. The inspectors reviewed a self-revealing Green finding involving
ineffective corrective actions taken in response to site and industry operating
experience with instrument air header ruptures. Specifically, contrary to
Section 6.2.3 of Procedure SO-123-I-1.42, Maintenance Division Experience
Report, Revision 0, the licensee failed to implement corrective actions to prevent
recurrence for an equipment failure with the potential to cause a significant plant
transient, and failed to appropriately consider previous industry and plant
experience similar to the event. Additionally, licensee personnel failed to properly
evaluate and take corrective actions based on industry operating experience
through 2006 involving improperly made soldered joints in instrument air systems.
As a result, an additional failure of an improperly made instrument air header joint
occurred at SONGS on June 20, 2007. The licensee entered this issue in their
corrective action program as Action Request AR 070600867.
This finding was more than minor since it was associated with the equipment
reliability attribute of the initiating events cornerstone and affected the cornerstone
objective to limit the likelihood of events that upset plant stability and challenge
critical safety functions. This finding required a Phase 2 analysis per the Manual
Chapter 0609, Significance Determination Process, Phase 1 Worksheets since
the loss of instrument air is a transient initiator resulting in the loss of the
feedwater system which is part of the power conversion system which can be used
to mitigate the consequences of an accident. Based on the results of the Phase 2
Enclosure
-3-
analysis and a subsequent Phase 3 analysis, the finding was determined to be of
very low safety significance (Green) because of the availability of the diverse
auxiliary feedwater system and the ability of the operators to depressurize the
steam generators and utilize the condensate system for heat removal. These
results were evaluated by a senior reactor analyst. This finding has a crosscutting
aspect in the area of problem identification and resolution associated with
operating experience in that the licensee failed to effectively implement changes to
station processes, procedures, and equipment in response to operating
experience involving improperly made instrument air system joints P.2(b).
(Section 2.1)
Green. The inspectors identified a Green noncited violation of Technical Specification 5.5.1.1 involving the failure to meet procedural requirements
following a loss of instrument air. Specifically, operators failed to monitor nitrogen
tank levels or take precautions for the possibility of oxygen-deficient areas in the
plant following actuation of the low pressure backup nitrogen system. The
licensee entered this issue in their corrective action program as Action
Request AR 070700291.
This finding was more than minor since it was associated with the human
performance attribute of the initiating events cornerstone and affected the
cornerstone objective to limit the likelihood of events that upset plant stability and
challenge critical safety functions. This finding required a Phase 2 analysis in
accordance with the Manual Chapter 0609, Significance Determination Process,
Phase 1 Worksheets since the loss of instrument air is a transient initiator resulting
in the loss of the feedwater system which is part of the power conversion system
which can be used to mitigate the consequences of an accident. Based on the
results of the Phase 2 analysis, the finding was determined to be of very low safety
significance because of the low likelihood of a complete loss of instrument air and
the availability of the auxiliary feedwater system. The cause of this finding has a
crosscutting aspect in the area of human performance associated with resources
because licensee personnel were not adequately trained on the operation of the
low pressure nitrogen system to effectively implement the abnormal operating
instruction H.2(b). (Section 2.2)
Cornerstone: Mitigating Systems
Green. A self-revealing, Green noncited violation of 10 CFR Part 50, Appendix B,
Criterion III, Design Control, was identified when Unit 2 experienced a loss of
instrument air due to the failure of a soldered joint. Specifically, the loss of
instrument air resulted in component cooling water (CCW) Pump 024 being in a
runout condition for approximately 75 minutes due to a previous system
modification. The licensee entered this issue in their corrective action program as
Action Requests AR 070700051 and 070600872.
This finding was greater than minor because it was associated with the mitigating
systems cornerstone attribute of design control and affected the associated
cornerstone objective to ensure the availability, reliability, and capability of systems
that respond to initiating events to prevent undesirable consequences. The finding
did not affect the initiating events cornerstone functions of the component cooling
water system because the condition would only have existed given a loss of
Enclosure
-4-
instrument air initiator had already occurred. In accordance with NRC Inspection
Manual Chapter 0609, Appendix A, Phase 1 Worksheet, Significance
Determination Process (SDP) Phase 1 Screening Worksheet for the Initiating
Events, Mitigating Systems, and Barriers Cornerstones, this finding was
determined to be of very low safety significance because the finding was a design
deficiency confirmed not to result in a loss of operability per Part 9900, Technical
Guidance, Operability Determination Process for Operability and Functional
Assessment. (Section 2.3)
Green. The inspectors reviewed a self-revealing Green finding involving the failure
to take effective corrective actions for a failed control room annunciator.
Specifically, after the annunciator for actuation of the backup nitrogen supply to
the instrument air system failed to function on demand on several occasions from
1994 through 2007, the corrective actions taken by the licensee to restore the
annunciator to service were inadequate and narrowly focused. The annunciator
subsequently failed to function during the loss of instrument air event on
June 20, 2007. The licensee entered this issue in their corrective action program
as Action Request AR 070601250.
This finding was more than minor since it was associated with the human
performance attribute of the initiating events cornerstone and affected the
cornerstone objective to limit the likelihood of events that upset plant stability and
challenge critical safety functions. This finding required a Phase 2 analysis in
accordance with the Manual Chapter 0609, Significance Determination Process,
Phase 1 Worksheets since the loss of instrument air is a transient initiator resulting
in the loss of the feedwater system which is part of the power conversion system
which can be used to mitigate the consequences of an accident. Based on the
results of the Phase 2 analysis, the finding was determined to be of very low safety
significance because of the low likelihood of a complete loss of instrument air and
the availability of the auxiliary feedwater system. This finding has a crosscutting
aspect in the area of problem identification and resolution associated with the
corrective action program in that the licensee failed to thoroughly evaluate the
failed annunciator such that the resolution appropriately addressed the causes
P.2(c). (Section 2.4)
Green. The inspectors identified a Green noncited violation of Technical Specification 5.5.1.1 involving the failure to maintain an adequate abnormal
operating instruction for a loss of instrument air event. The licensee entered this
issue in their corrective action program as Action Request AR 070801151.
This finding was more than minor because it was associated with the procedure
quality attribute of the mitigating systems cornerstone and affected the
cornerstone objective to ensure the availability, reliability and capability of systems
that respond to initiating events, in that a less than adequate abnormal operating
procedure could have prevented operators from promptly tripping the reactor,
allowing conditions to continue to degrade and resulting in a demand on the
reactor protection system. Using the Significance Determination Process Phase 1
Screening Worksheet in Appendix A of Inspection Manual Chapter 0609, the
inspectors determined this finding had very low safety significance because it did
not result in an actual loss of safety function per Part 9900, Technical Guidance,
Operability Determination Process for Operability and Functional Assessment.
Enclosure
-5-
This finding has a crosscutting aspect in the area of human performance
associated with resources in that the licensee failed to provide operators with
complete, accurate, and up-to-date procedures H.2(c). (Section 2.5)
Green. A self-revealing, Green noncited violation of 10 CFR Part 55.46(c)(1) was
identified involving the licensees failure to incorporate a design change in
modeling plant response for the plant-referenced simulator. Specifically, during
operator training in the plant-referenced simulator, the controlled bleedoff valves
for the reactor coolant pumps were modeled to fail closed on a loss of instrument
air, whereas the valves in the plant remained open during an actual loss of
instrument air event on June 20, 2007. The licensee entered this issue in their
corrective action program as Action Requests AR 070600873 and 070900160.
This finding was greater than minor because it was associated with the mitigating
systems cornerstone attribute of human performance and affected the associated
cornerstone objective to ensure the availability, reliability, and capability of systems
that respond to initiating events to prevent undesirable consequences. The
inspectors evaluated this finding using the Appendix I, Licensed Operator
Requalification Significance Determination Process worksheets of Manual
Chapter 0609 because the finding is a requalification training issue related to
simulator fidelity. The finding is of very low safety significance because the
discrepancy did not have an adverse impact on operator actions such that safety
related equipment was made inoperable during normal operations or in response
to a plant transient. This finding has a crosscutting aspect in the area of human
performance associated with resources in that the licensee did not provide
operators with adequate facilities and equipment for use in operator training
H.2(d). (Section 2.6)
Green. The inspectors identified a Green noncited violation of Technical Specification 5.5.1.1 involving the failure to meet procedural requirements
governing impaired annunciators. Specifically, after the identification of a failed
annunciator, operators did not enter the annunciator in the failed annunciator log
or mark the affected annunciator window with an annunciator compensatory action
flag. The licensee entered this issue in their corrective action program as Action
Request AR 070700291.
This finding was more than minor since it was associated with the human
performance attribute of the initiating events cornerstone and affected the
cornerstone objective to limit the likelihood of events that upset plant stability and
challenge critical safety functions. This finding required a Phase 2 analysis in
accordance with the Manual Chapter 0609, Significance Determination Process,
Phase 1 Worksheets since the loss of instrument air is a transient initiator resulting
in the loss of the feedwater system which is part of the power conversion system
which can be used to mitigate the consequences of an accident. Based on the
results of the Phase 2 analysis, the finding was determined to be of very low safety
significance because of the low likelihood of a complete loss of instrument air and
the availability of the auxiliary feedwater system. This finding has a crosscutting
aspect in the area of human performance associated with resources because the
operators were not sufficiently trained to consistently implement the annunciator
operating procedure H.2(b). (Section 2.7)
Enclosure
-6-
Green. A Green self-revealing finding was identified associated with the failure of
the reactor coolant pump controlled bleed off valve to shut during a loss of
instrument air event. The licensee failed to adequately implement corrective
actions from previously evaluated industry operating experience for new valve
regulators that were installed in the unit. The licensee entered this issue in their
corrective action program as Action Request AR 070600873.
The finding was greater than minor because it was associated with the mitigating
systems cornerstone attribute of design control and affected the associated
cornerstone objective to ensure the availability, reliability, and capability of systems
that respond to initiating events to prevent undesirable consequences. Using
Manual Chapter 0609, Significance Determination Process, Phase 1 Worksheet,
the finding is determined to have very low safety significance because the
condition only affected the mitigation systems cornerstone and it was confirmed
not to result in loss of operability per Part 9900, Technical guidance, Operability
Determination Process for Operability and Functionality Assessment
(Section 2.8).
B.
Licensee-Identified Violations
None.
Enclosure
-7-
REPORT DETAILS
1.0
SPECIAL INSPECTION SCOPE
The NRC conducted a special inspection at San Onofre Generating Station (SONGS) to
better understand the circumstances surrounding the loss of instrument air event on
June 20, 2007. On this occasion, instrument air pressure on Unit 2 dropped significantly
following the separation of a 3-inch air header in the auxiliary building. This caused the
feedwater control valves to stop functioning, resulting in an uncontrolled increase in
steam generator water level. Operators manually tripped the Unit 2 reactor. In
accordance with NRC Management Directive 8.3, it was determined that this event had
sufficient risk significance to warrant a special inspection.
The team used NRC Inspection Procedure 93812, Special Inspection Procedure, to
conduct the inspection. The special inspection team reviewed procedures, corrective
action documents, operator logs, design documentation, maintenance records, and
procurement records for the instrument air system. The team interviewed various
station personnel regarding the event. The team reviewed the licencees preliminary
root cause analysis report, past failure records, extent of condition evaluation,
immediate and long term corrective actions, and industry operating experience. A list of
specific documents reviewed is provided in Attachment 1. The charter for the special
inspection is included as Attachment 2.
1.1
Event Summary
During full power operation on June 20, 2007, a 3-inch diameter instrument air line failed
at an improperly soldered joint on the Unit 2 instrument air header. The joint completely
separated, resulting in a double-ended guillotine shear of the supply header and a
complete loss of instrument air to Unit 2. The loss of instrument air pressure caused the
feedwater control valves to stop functioning, and operators manually tripped the Unit 2
reactor as a result of an uncontrolled steam generator water level increase. Although
instrument air is a shared system at SONGS, a backup nitrogen system can support
system loads on the unaffected unit following a pipe break via excess flow check valves.
As a result, the pressure drop on Unit 3 was not as significant during the event and
operators maintained control of all functions.
Operators located the failed piping in the Unit 2 turbine building and were able to isolate
the break approximately thirty minutes after the event began. Operators applied a
temporary repair to the break and restored instrument air header pressure. Subsequent
investigations identified 32 additional leaking instrument air fittings in Unit 2 and Unit 3,
possibly as a result of improper joint fabrication during initial construction. Maintenance
personnel placed structural clamps on the leaking fittings to prevent additional piping
separations until permanent repairs could be made.
The time line below describes the major events following the separation of the
instrument air header fitting on June 20, 2007.
Enclosure
-8-
June 20, 2007
2244
Instrument air dryer Temp/Level/DP HI alarm received.
Control room instrument air header pressure noted to be 80 psig and lowering.
Instrument air pressure low alarm received on Unit 2. (90 psig setpoint)
Operators entered Procedure SO23-13-5, Loss of Instrument Air.
2245
Unit 2 air operated valves begin to move on their own.
Full Flow condensate polisher demineralizer bypass valves open.
Chemical volume control system letdown flow isolates.
Unit 3 instrument air pressure low alarm received. (90 psig setpoint)
2247
Steam Generator level noted to be 82% and rising in generator E088 on Unit 2.
Operators secured charging pumps due to loss of letdown and began manually
controlling pressurizer level.
Operators bypassed instrument air dryers. Indicated instrument air header
pressure in the control room increases from 42 psig to 67 psig.
2248
Heater drain pump P059 trips.
2250
Heater drain pump P058 trips.
E088 level approaching trip set point, manually tripped the Unit 2 reactor.
Entered Procedure SO23-12-1, Standard Post Trip Actions.
2252
Steam generator E088 level exceeds 100%.
2253
Operators manually tripped both main feed pumps.
2254
Operators initiated both trains of auxiliary feedwater. (EFAS)
Steam bypass control system responding sluggishly; both reactor temperature
and pressure slightly higher than expected. Operators begin controlling pressure
and temperature using one atmospheric dump valve.
2258
Steam generator E088 level returns to less than 100%.
2303
Entered Procedure SO23-12-2, Reactor Trip Recovery.
2321
Location of instrument air header rupture identified and isolated using manual
valves. Instrument air header pressure indicated in the control room immediately
recovers from 67 psig to 108 psig (normal operating pressure). Instrument air
dryer Temp/Level/DP HI and instrument air low pressure alarms clear.
Temporary repair (soft patch) put in place on the instrument air header.
2329
Component Cooling Water (CCW) Pump A noted to be in a runout condition,
operators started CCW Pump B.
0024
NRC notified of unit trip due to uncontrolled level rise in steam generator E088
upon loss of instrument air.
0030
Procedure SO23-12-1, Loss of Instrument Air, exited.
Enclosure
-9-
1.2
Operator Response
The team assessed the response of the control room operators to the loss of instrument
air. The team reviewed operator logs, plant computer data, and strip charts to evaluate
operator performance in coping with the event and transient; verified that operator
actions were in accordance with the response required by plant procedures and training;
and verified that the licensee identified and implemented appropriate corrective actions
associated with personnel performance problems that occurred during the event. The
team also conducted interviews with each of the control room operators who were on
shift the night of the event.
The team concluded the operators acted appropriately to manually trip the Unit 2 reactor
and turbine and place the unit in a safe condition. The inspectors also concluded the
operators acted promptly and appropriately in recovering the instrument air system and
in maintaining Unit 3 at power. However, the team also identified several opportunities
for improvement in some aspects of operator response and training associated with the
event.
Through interviews with control room personnel, the inspectors noted a general
weakness in the operators understanding of the design and integrated operation of the
instrument air and low pressure nitrogen systems. As an example, several operators
erroneously stated that the respiratory/service air system was supporting the Unit 3
instrument air loads during the event, when in fact the Unit 3 loads were being supplied
by the backup nitrogen system. Several operators also believed their efforts to bypass
the instrument air filters and to place an additional dryer in service had a positive effect
on restoring instrument air pressure to Unit 2, when in reality the Unit 2 instrument air
header pressure was not recoverable due to the complete separation of the pipe header
from the supply lines.
The inspectors concluded the operators understanding of the event on June 20, and
their ability to diagnose and respond to future events involving a loss of instrument air,
were complicated by the sparse control room instrumentation provided for the
instrument air system. Specifically, operators in both control rooms are provided with
one indication each for respiratory/service air supply pressure, backup nitrogen system
supply pressure, and instrument air supply pressure. There are no indications for actual
air header pressure at the system loads for either Unit 2 or Unit 3. Additionally, the
control room indications provided real-time pressure indication only; there were no strip
charts recorders to allow prompt diagnosis of pressure trends, nor were there any
computer points available to provide pressure indication, tracking, or trending
information to the control room operators.
The inspectors reviewed Procedure SO23-13-5, Loss of Instrument Air, Revision 5,
which was the abnormal operating instruction used by the operators to respond to the
loss of instrument air pressure on June 20, 2007. The inspectors concluded that given
the limited data available to plant operators in the control rooms, the abnormal operating
instruction did not provide sufficient guidance to ensure operators would be able to take
prompt action to mitigate the effects of a loss of instrument air in all circumstances. The
inspectors determined the failure to maintain an adequate operating instruction to
respond to a loss of instrument air was a violation of Technical Specification 5.5.1.1.
This finding is described further in Section 2.5 of this report.
Enclosure
-10-
Through review of operator logs, the marked-up copy of the abnormal operating
instruction for loss of instrument air used during the event, and interviews of operators,
the inspectors identified that operators had failed to take the required actions specified
in the abnormal operating instruction for actuation of the backup nitrogen system.
Specifically, the operators did not take steps to monitor for oxygen-deficient areas of the
plant caused by nitrogen leakage from the instrument air system and did not begin
monitoring nitrogen tank levels. The inspectors also noted that control room
Annunciator 61B38, N2 SUPPLY TO INST AIR HEADER ON, failed to alarm during
the event, which, when combined with the aforementioned weak operator understanding
of the system and limited control room instrumentation, likely contributed to the
operators failure to take the actions of the abnormal operating instruction. The
inspectors noted the failure to take these actions had the potential to result in operator
injury or death from entering oxygen-deficient areas in the plant. The inspectors
determined the failure to follow the requirements of the abnormal operating instruction
was a violation of Technical Specification 5.5.1.1. This finding is discussed further in
Section 2.2 of this report.
The inspectors examined the post-trip review package assembled by the licensee
following the trip of Unit 2. The inspectors noted the post-trip review package
appropriately addressed the response and operation of safety-related plant equipment
to the event. The post-trip review also properly identified operator performance issues
associated with a missed surveillance requirement and implemented appropriate
corrective actions. The inspectors concluded the licensees post-trip review was
adequate per the guidance of Generic Letter 83-28, Required Actions Based on
Generic Implications of Salem ATWS Events. However, the inspectors identified some
weaknesses in the scope and thoroughness of the review in regard to the nonsafety-
related aspects of the event. For example, the post trip review did not identify the failure
of Annunciator 61B38 to alarm as mentioned above and described further in Section 1.4
of this report. The review package also contained a typographical error identified by the
inspectors that, had the recorded value been correct, would have indicated that the
reactor trip circuit breakers had failed to open within their design time limit. The review
also did not address the weaknesses in operator understanding of the event or the
failure of operators to follow the requirements of the abnormal operating instruction for
actuation of the backup nitrogen system as described above.
The licensee initiated a root cause evaluation to assess the above issues related to the
operator response and post-trip review for the June 20, 2007, loss of instrument air
event as part of Action Request AR 070700291.
1.3
Instrument Air System Interactions
The instrument air system at SONGS consists of three motor driven instrument air
compressors, two parallel air dryers and four parallel air filters, all of which are located in
the turbine building. Backup pressure sources for instrument air are provided by the low
pressure nitrogen system and the respiratory/service air system. The instrument air
system is designed to provide a continuous supply of filtered, dried, and essentially oil-
free air for pneumatic instruments and valves in both units.
During normal system operation, one of the compressors is in continuous operation
while the other two compressors are in standby. The standby compressors will start and
stop automatically as required to supplement the running compressor to meet system
Enclosure
-11-
demand. The instrument air header is divided between the two units by check valves
installed in the supply headers and in the unit crossover header in the radwaste building.
Non-safety related nitrogen supply lines with isolating valves and excess flow check
valves are located downstream of the unit check valves in the air supply lines to provide
a backup nitrogen supply for each units instrument air header. The excess flow check
valves isolate on high flow, which prevents a failure in one units air piping from causing
an excessive instrument air pressure drop in the other unit. A second backup supply for
the instrument air system is provided by the respiratory/service air system. The
respiratory/service air system is connected to the instrument air supply lines upstream of
the instrument air dryers.
The instrument air system supplies the motive force to all pneumatically operated valves
and instruments in both units. All pneumatically operated valves are designed to fail to
their safe position on a loss of instrument air. Pneumatic valves with a safety function
are described in Table 9.3-1 of the SONGS Final Safety Analysis Report (FSAR) and
include: saltwater cooling system isolation valves and lubrication valves, component
cooling water isolation valves, shutdown cooling heat exchanger isolation valves, safety
injection line check valve leakoff line isolation valves, safety injection tank fill and drain
lines, and auxiliary feedwater pump steam supply valves. Significant non-safety related
pneumatic valves include the chemical volume control system letdown isolation valves,
pressurizer normal spray valves, main feedwater regulating valves, steam bypass
control system valves, reactor coolant pump seal controlled bleedoff isolation valves,
and component cooling water noncritical loop isolation valves.
1.4
Plant Response
The inspectors reviewed operator logs, alarm history, and available trend information to
evaluate the plant response to the loss of instrument air header pressure to ensure that
all systems responded as designed. The inspectors concluded the instrument air
system functioned as described in the FSAR. Following the break in the Unit 2 air
header, the excess flow check valve in the backup nitrogen supply line to Unit 2 closed
to isolate the break and successfully mitigated the effect of the transient on Unit 3 as
designed. The inspectors also concluded the integrated plant response to the overall
transient also occurred as described in the FSAR, with some exceptions as noted below.
In the post trip review package, the licensee noted excessive flow existed in the
component cooling water (CCW) system for approximately 75 minutes following the
event, which placed the CCW Pump A in a runout condition. The excess system flow
resulted when the shutdown cooling heat exchanger isolation valve failed open as
designed on the loss of instrument air pressure. Although in the original plant design
the CCW noncritical loop isolation valves failed shut on a loss of instrument air pressure
to isolate the shutdown cooling heat exchanger from the system, the licensee installed a
modification in 1995 to allow the noncritical loop isolation valves to remain open in order
to maintain cooling water for the reactor coolant pumps. Consequently, the opening of
the shutdown cooling heat exchanger isolation valve placed an additional load on the
system in excess of the capacity of the operating CCW pump. The inspectors
determined this was a violation of 10 CFR Part 50, Appendix B, Criterion III, Design
Control. This finding is discussed further in Section 2.3 of this report.
In the post-trip review package and during interviews with the inspectors, the operators
noted the controlled bleed off valve for the reactor coolant pump seals remained open
Enclosure
-12-
following the loss of instrument air. During interviews with the inspectors, the operators
stated that in operator training in the site simulator, the CBO valve always failed shut on
a loss of instrument air pressure, requiring the operators to trip the reactor coolant
pumps to ensure the integrity of the reactor coolant pump seals. As a result, when the
CBO valve indicated open during the event on June 20, 2007, control room operators
requested local, independent verification of the actual position of the CBO valve. The
inspectors concluded the discrepancy between actual plant response and that modeled
in the simulator negatively impacted operator response to the loss of instrument air
event. The inspectors determined this simulator fidelity issue was a violation of 10 CFR Part 55.46. This finding is discussed further in Section 2.6 of this report.
In reviewing the cause for the failure of the CBO valve to close on the loss of instrument
air pressure, the licensee determined that the regulator for the valve had been replaced
with a new style regulator in February 2004. The new style regulator was installed
because the original model had become obsolete. Whereas air pressure would leak off
the original model regulator causing the associated valve to close on a loss of
instrument air, the new regulator contained improved seals that locked in air pressure
and allowed the associated valve to remain open. Although the licensees substitution
equivalency evaluation required a design change impact review prior to installing the
new model regulators in the plant, the engineers performing the impact review for the
CBO valve failed to review the FSAR and so did not identify that the CBO valves were
designed to fail closed on a loss of instrument air. A finding associated with the failure
to perform an adequate design change impact review is discussed further in Section 2.8
of this report.
During the loss of instrument air event, Annunciator 61B38, N2 SUPPLY TO INST AIR
HEADER ON, failed to alarm, complicating operating understanding of and response to
the event as described in Section 1.2 of this report. The licensee initiated Action
Request AR 070601250 to address the failed annunciator. While in the control room
three days later, the inspectors noticed there were no labels, warning flags, or other
devices affixed to or logged for the nonfunctional annunciator. The inspectors noted
that should a similar loss of instrument air pressure event recur, the absence of any
warning labels or other devices to alert operators to the nonfunctional annunciator could
cause the operators to fail to take the appropriate steps per the annunciator response
instruction and loss of instrument air abnormal operating instruction to monitor enclosed
spaces for oxygen concentration and monitor the nitrogen tank levels. Due to
instrument air system leakage, actuation of the backup nitrogen system without the
compensatory action of monitoring enclosed spaces for oxygen concentration could
potentially result in operator injury or death from entering oxygen-deficient areas of the
plant. This inspectors determined the failure to appropriately track the nonfunctional
control room annunciator was a violation of Technical Specification 5.5.1.1. This finding
is discussed further in Section 2.7 of this report.
1.5
Root Cause Evaluation
The inspectors reviewed the accuracy and thoroughness of the licensee cause
determination as described in the root cause evaluation, Unit 2 Instrument Air Soldered
Joint Failure, performed as part of Action Request AR 070600867. The licensees root
cause evaluation used events and causal factors analysis and failure modes and effects
Enclosure
-13-
analysis to evaluate the physical piping failure, the use of operating experience at the
site, and the implementation of the preventive maintenance program in the instrument
air system.
For the physical piping failure, the licensee performed a metallurgical analysis of the
failed joint. The analysis showed the original solder coverage within the joint was less
than 30% of the joint interface. Corrosion facilitated by residual flux in the joint
weakened the solder over time until the eventual failure of the fitting. The licensee
concluded the cause for the improper solder coverage was an improper fit up causing
an excessive gap in the joint due to poor workmanship during initial construction. The
analysis concluded the cause of the poor workmanship was a lack of supervisory
monitoring and reinforcement, and the root cause of the physical joint failure was a lack
of intrusive testing and inspections of the instrument air system during initial
construction. The analysis did not identify corrective actions specific to the identified
causes since the events examined occurred during initial construction. Corrective
actions were identified to develop an inspection plan to locate additional leaking joints
and to periodically inspect the clamps installed on improperly made joints until repairs
can be made.
The root cause analysis also examined the ineffective review and use of industry
operating experience (OE) at the site to determine why the event had not been
prevented despite the existence of sufficient OE to foresee its occurrence. The licensee
concluded the ineffective use of OE was the result of an inappropriate cultural bias in
the engineering department that led engineers to review OE from a defensive
standpoint; i.e., the goal of the engineers performing reviews was to determine why a
particular OE was not applicable to the station. The licensee determined this culture
was reinforced by insufficient site expectations for OE procedural use and
documentation. The licensee developed corrective actions to strengthen site standards
and expectations for OE procedure use and documentation and to perform
benchmarking among industry peers to incorporate best practices for OE use.
The final portion of the licensees root cause evaluation examined the lack of an
adequate preventive maintenance program for the instrument air check valves and the
backup nitrogen flow indication switch. The licensee concluded the apparent cause for
the lack of preventive maintenance tasks was ineffective supervisory monitoring. The
licensee developed corrective actions to reinforce expectations for supervisory
performance and to implement preventive maintenance tasks for the instrument air
The inspectors reviewed the licensees root cause evaluation and determined the
metallurgical analysis and cause evaluation for the physical piping failure was thorough
and technically sound. However, the inspectors concluded that the root cause
evaluation as a whole was narrowly focused and in some cases lacked specific,
comprehensive corrective actions.
The inspectors considered the evaluation to be narrowly focused since it did not fully
address all the factors and behaviors that contributed to the nature, magnitude and
timing of the event. For example, the evaluation did not discuss in detail a precursor
event in the form of a failed thermowell fitting in the instrument air system that occurred
in 1994. The evaluation also failed to address the poor quality of the OE review
performed in 1992. The extent of condition review made only a passing reference to the
Enclosure
-14-
domestic water system and did not contain a complete discussion of systems potentially
affected by the identified root cause. Additionally, the maintenance review in the root
cause evaluation did not discuss the scoping or performance of the instrument air
system under the Maintenance Rule (10 CFR Part 50.65). Also, the extent of cause
review for the failed joint noted that site engineers had not appropriately addressed the
instrument air system under the station Equipment Reliability Improvement Program, but
the report did not appear to investigate why that had happened or whether there were
potentially other systems that may have been similarly overlooked.
The inspectors noted that the narrow focus of the report was also reflected in the
assignment of corrective actions. Specifically, the licensee identified poor supervisory
monitoring and oversight as an apparent cause for the improperly soldered joint, but
identified no corrective actions due to the age of the issue. However, later in the same
report, the licensee identified poor supervisory monitoring and oversight as an apparent
cause for the failure to establish adequate preventive maintenance tasks for instrument
air system. Corrective actions in this case to reinforce standards and improve
performance were directed only to the engineering organization.
The inspectors noted that in some cases, the causes identified in the root cause
evaluation tended to be associated with a single, broad corrective action such as
strengthen site standards. While the inspectors did not disagree with the intent of the
corrective action, the generalized language leaves the method of implementation open
to interpretation and complicates the ability of the stations assessment organization to
perform effectiveness reviews. In these cases, the inspectors concluded a set of
specific, focused, and measurable corrective actions may have been more appropriate.
1.6
Event Precursors
The team performed a search of corrective action program databases to identify
previous instrument air system piping problems that may have been precursors to the
event on June 20, 2007. The inspectors identified the potential event precursors
described below.
The inspectors noted that in June 1994, both Units 2 and 3 experienced a loss of
instrument air event due to the failure of a soldered joint retaining a threaded thermowell
attachment to the instrument air header. In the failure analysis for the fitting, the
licensee determined the cause of the separation was poor quality workmanship that
occurred during original installation. The licensee determined that when the fitting was
initially installed, it was not centered, causing an excessive gap on one side of the joint
and resulting in inadequate solder penetration. The licensee also identified that the joint
was designed to be silver brazed per the manufacturers specification and should not
have been soldered, and that this joint had most likely been leaking since original
installation since a portion of the pipe joint had no filler metal in it. Although this event
did not result in a reactor trip, the inspectors noted the cause and nature of the failure
were nearly identical to that experienced on June 20, 2007.
In April, 2007, both Unit 2 and Unit 3 experienced a loss of instrument air event due to a
trip of the running air compressors. During this event as well as during the June 1994
loss of instrument air event described above, the annunciator for actuation of the
backup nitrogen supply to the instrument air system failed to alarm in the control room.
Following the annunciator failure in 1994, licensee maintenance technicians noted that
Enclosure
-15-
the limit switch was dirty. Since a replacement limit switch was not available, the
technicians cleaned the installed limit switch and returned it to service. In April 2007,
licensee maintenance technicians noted that the travel on the switch was satisfactory
and there were no problems on the electrical part of the system. No other work was
documented. During the event on June 20, 2007, the annunciator for actuation of the
backup nitrogen system again failed to alarm in the control room. Although the
April 2007 event was caused by a compressor failure and did not result in a reactor trip,
the inspectors noted the plant response, particularly as it related to the flow switch for
actuation of the backup nitrogen system, was similar to the event on June 20, 2007. A
finding associated with the failure of the licensee to take effective corrective actions for
the nitrogen system flow switch is described in Section 2.4 of this report.
1.7
Instrument Air System Maintenance and Testing
The inspectors reviewed the licensees program for maintenance and inspection of the
instrument air system, particularly as it related to the historical health of the instrument
air compressors and piping system.
The inspectors noted the performance of the instrument air system was monitored
under performance criteria established per the guidance for Category a(2) systems
under the Maintenance Rule. During discussions with the system engineer and site
Maintenance Rule coordinator, the inspectors learned the system was under
consideration for goal setting and monitoring per Category a(1) of the Maintenance
Rule. The licensee subsequently established goals and began monitoring the
performance of the instrument air system per Category a(1). The team considered this
action appropriate. The team also noted there had been several functional failures of
the three instrument air compressors over the past two years, and at one point earlier in
the year a temporary air compressor was installed to supplement the existing instrument
air compressors. The team noted this situation was nearly identical to that described in
NRC Inspection Report 50-361:362/97-22 as indicative of poor performance requiring
the goal setting and monitoring per Category a(1) of the Maintenance Rule. Given this
operating history, the team concluded it may have been appropriate for the station to
have classified the instrument air system as Category a(1) much earlier in the year.
However, the inspectors noted that such classification would have had no impact on the
prevention or mitigation of the loss of instrument air event experienced at the station on
June 20, 2007.
The inspectors also examined maintenance work orders for individual components in the
instrument air and low pressure nitrogen systems. The inspectors noted that no
preventive maintenance actions existed for the excess flow check valves in the supply
header for backup nitrogen to the instrument air system. The inspectors considered this
inappropriate given these valves have a function credited in the FSAR to prevent a
break in one units instrument air header from causing a loss of instrument air to the
other unit. Since check valves can not be considered inherently reliable components,
the inspectors concluded the licensee had failed to perform adequate preventive
maintenance to ensure the excess flow check valves would perform their intended
function. The inspectors determined this was a violation of 10 CFR Part 50.65a(2).
However, since the excess flow check valves did perform their intended function during
the actual loss of instrument air event on June 20, 2007, the inspectors considered this
Enclosure
-16-
violation to be minor. The licensee entered this violation in their corrective action
program as Action Requests AR 070600867 and AR 070900333, and evaluated the lack
of preventive maintenance items in the instrument air system as part of the root cause
evaluation for the loss of instrument air event.
1.8
Industry Operating Experience (OE) and Potential Generic Issues
The inspectors performed searches of operating experience databases and other
sources to identify reports of similar problems, both inside and outside the nuclear
industry.
During the late 1980s and early 1990s, a significant amount of Operating
Experience (OE) identified instances where facilities had experienced transients and/or
trips due to failures of soldered joints in instrument air system piping due to poor
workmanship during initial construction. The licensee documented their review of the
industry OE in their corrective action program (CAP) as Independent Safety
Engineering Group Operating Experience Evaluation, dated January 22, 1992. In this
review, the licensee evaluated the identified causes and corrective actions from the OE
and determined that soldering at SONGS was loosely controlled and better training was
necessary for welders at the facility. However, the licensee asserted in their evaluation
that failures due to inadequate fit-up or solder penetration typically occur within a
relatively short time frame after startup. The licensee concluded that since the
instrument air system had been in service for many years at SONGS and no significant
problems had yet been identified, then no corrective actions were necessary with
respect to the installed instrument configuration. The inspectors considered this
conclusion to be without a valid technical basis. A finding associated with this evaluation
is discussed in Section 2.1 of this report.
In addition to the internal site experience described above and in Section 1.6 of this
report, the inspectors identified additional OE in the form of Licensee Event Reports
(LERs). Most notably, the inspectors reviewed LER 05000336/2006-002-00, Manual Reactor Trip Due to Trip of Both Feed Pumps Following a Loss of Instrument Air,
April 21, 2006, and LER 05000440/2006-005-00, Decreasing Instrument Air Pressure
Results in Manual Reactor Protection System Actuation, February 9, 2007. Both
reports describe reactor trips brought about by instrument air header joint separation. In
both cases, the cause of the header joint separation was inadequate workmanship
during initial construction. Though the site OE coordinator indicated the licensee
reviewed all Licensee Event Reports for applicability to SONGS, the inspectors did not
identify any documents in the licensees corrective action program that evaluated these
events. The inspectors determined the lack of documentation in the CAP indicated the
site OE organization had determined the above described events were not applicable to
SONGS. The inspectors concluded the licensee had missed multiple opportunities both
historically and recently to identify the vulnerability presented by improperly made joints
in the instrument air system.
Given the failure history described above, the inspectors concluded the construction
methods and controls in place during initial construction at SONGS were not unique.
Therefore, the potential for separation of instrument air piping due to improperly made
joints represents a potential generic concern for all facilities with instrument air systems
utilizing soldered joints in copper piping headers.
Enclosure
-17-
2.0
SPECIAL INSPECTION FINDINGS
2.1
Ineffective Corrective Actions for Instrument Air Header Ruptures
The inspectors reviewed a self-revealing Green finding involving ineffective corrective
actions taken in response to site and industry operating experience with instrument air
header ruptures. Specifically, contrary to Section 6.2.3 of Procedure SO-123-I-1.42,
Maintenance Division Experience Report, Revision 0, the licensee failed to implement
corrective actions to prevent recurrence for an equipment failure with the potential to
cause a significant plant transient, and failed to appropriately consider previous industry
and plant experience similar to the event. Additionally, licensee personnel failed to
properly evaluate and take corrective actions based on industry operating experience
through 2006 involving improperly made soldered joints in instrument air systems. As a
result, an additional failure of an improperly made instrument air header joint occurred at
SONGS on June 20, 2007.
On June 20, 2007, both Units 2 and 3 experienced a loss of instrument air event due to
the failure of a three-inch instrument air line header fitting. As a result of the break
location, a loss of manual feedwater control occurred on Unit 2 which ultimately resulted
in a manual reactor trip due to high steam generator level.
The licensee performed a Root Cause Evaluation of this event, as documented in Action
Request AR 070600867. The licensee also performed a metallurgical analysis of the
failed joint as documented in SONGS Unit 2 Instrument Air System Failed Fitting
Metallurgical Evaluation, dated June 27, 2007. During these evaluations, the licensee
determined the root cause of the event to be poor workmanship of the header joint
during initial installation. The licensees metallurgical analysis also concluded the fitting
had only thirty percent solder coverage within the joint and had likely been leaking air
since the plants first operating cycle. Subsequent investigations by the licensee
identified 32 additional joints leaking air in the instrument air headers of both units. The
licensee installed temporary structural clamps on the leaking fittings tp prevent
additional separations until permanent repairs could be made.
The inspectors reviewed the licensees root cause evaluation and metallurgical analysis
for this event. During their review of the issue, the inspectors noted that there had been
a previous similar air header failure at SONGS, and that the licensee had previously
evaluated related industry Operating Experience (OE) involving issues with soldered
joints.
During the late 1980s and early 1990s, a significant amount of Operating Experience
(OE) identified instances where facilities had experienced transients and/or trips due to
failures of soldered joints in instrument air system piping. The identified failures were
due to a lack of adequate controls during the initial makeup of soldered joints.
Specifically, inadequate fit-up of the joints or inadequate solder penetration were
identified as the causes of the failures. The licensee performed a review documented in
Independent Safety Engineering Group Operating Experience Evaluation, dated
January 22, 1992, to evaluate applicability of the OE to the facility. In this review, the
licensee evaluated the identified causes and corrective actions from the OE and
determined that soldering at SONGS was loosely controlled and better training was
necessary for welders at the facility. However, the licensee asserted in the evaluation
that any failures due to inadequate fit-up or solder penetration would typically have
Enclosure
-18-
occurred within a relatively short time frame after startup. The licensee concluded that
since the instrument air system had been in service for many years and no significant
problems had yet been identified, then no corrective actions were necessary with
respect to the installed instrument air configuration.
In June 1994, both Units 2 and 3 experienced a loss of instrument air event. The
licensee investigated the cause and determined it was due to the failure of a soldered
joint retaining a threaded attachment, a Brazolet fitting, to the air header. The Brazolet
fitting was being used for a thermowell in the instrument air header. The licensee
performed a failure analysis of the fitting documented in Failure Analysis Report
No.94-006, dated July 8, 1994, to determine the cause of the joint failure. During this
analysis, the licensee determined the cause of the soldered fitting failure was poor
quality workmanship that occurred during original installation. The licensee determined
that when the fitting was initially installed, it was not centered, but rather cocked to one
side. This was not as required by the procedure and resulted in an excessive gap on
one side of the joint. This gap deprived one side of the joint of filler material, resulting in
inadequate solder penetration. The licensee also identified that the joint was designed
to be silver brazed per the manufacturers specification and should not have been
soldered.
Based on the metallurgical analysis, the licensee determined that the joint failure in
July 1994 was due to fatigue cracking that originated at the area between the soldered
and un-soldered sections of the joint. The failure analysis also identified that this joint
had most likely been leaking since installation. This was based on the fact that a portion
of the pipe joint had no filler metal in it. Since the joint was located approximately ten
feet off the floor in a high noise area, the leakage had not been previously identified.
The failure analysis also recommended that to prevent recurrence, all brazolet fittings in
the instrument air system should be examined both for leaks and for use of solder. The
analysis further identified that properly soldered joints fittings should be able to tolerate
instrument air header pressure indefinitely; however, if leaks were found, the fittings
should be replaced using silver braze at the earliest opportunity.
The inspectors concluded the licensees evaluation of OE performed in 1992 was
inadequate in that it improperly determined that failures due to inadequate fit-up and/or
inadequate solder penetration would have occurred within a relatively short time frame.
The inspectors also determined that the licensee failed to adequately reassess this
position following the instrument air line joint failure in 1994. The inspectors noted that
as recently as 2006, the licensee had inappropriately screened additional industry OE
relating to the failure of inadequately made instrument air piping joints as not applicable
to the station. The inspectors concluded the licensee failed to take effective corrective
actions for inadequately made joints in the instrument air system since the corrective
actions for the 1994 event and in response to industry OE were narrowly focused on
soldered Brazolet fittings and failed to evaluate soldered joints as a whole.
The safety significance and enforcement aspects of this finding are described in
Sections 3.1 and 4.1, respectively.
2.2
Failure to Follow Abnormal Operating Instruction for the Loss of Instrument Air
The inspectors identified a Green noncited violation of Technical Specification 5.5.1.1
involving the failure to meet procedural requirements following a loss of instrument air.
Enclosure
-19-
Specifically, operators failed to monitor nitrogen tank levels or take precautions for the
possibility of oxygen-deficient areas in the plant following actuation of the low pressure
backup nitrogen system.
The instrument air system at SONGS utilizes a low pressure nitrogen system as a
backup pressure source. An instrument air system pressure drop below 83 psig will
automatically actuate a control valve in the nitrogen system to supplement the
instrument air from liquid nitrogen storage tanks. A flow switch downstream of the
control valve is designed to provide an annunciator in the control when the nitrogen
system is actuated. While the air systems are being supplied by nitrogen, normal
system leakage can result in oxygen-deficient areas in enclosed spaces of the plant,
and the liquid nitrogen tank levels will decrease more rapidly than usual from the
addition of the instrument air loads. The nitrogen supply lines are provided with isolating
check valves and excess flow check valves to prevent a failure in one units piping from
causing an excessive pressure drop in the other unit.
On June 20, 2007, an instrument air header ruptured in Unit 2. Although the low
pressure nitrogen system functioned as designed to provide nitrogen to the Unit 3 air
header, the control room annunciator for nitrogen system actuation did not alarm. The
instrument air header low pressure alarms did actuate on both units, and control room
operators began taking required actions per Procedure SO23-13-5, Loss of Instrument
Air, Revision 5. Although a step in the procedure directed operators to monitor nitrogen
tank levels and monitor for oxygen concentrations in enclosed spaces following nitrogen
system actuation, this step was not performed. The inspectors noted the failure to take
these actions had the potential to result in a Unit 3 trip from nitrogen tank depletion or
the injury or death of personnel from entry into oxygen-deficient spaces.
During interviews with the inspectors, several control room operators demonstrated
knowledge weaknesses related to the operation of the backup pressure sources for
instrument air. For example, several operators mistakenly stated the Unit 3 air header
pressure had been supplied by the respiratory/service air system during the event. The
inspectors concluded that although the failed annunciator likely contributed to the
operators confusion, the failure to perform the required actions of the abnormal
operating instruction resulted from the operators poor understanding of the operation of
the nitrogen backup to the instrument air system.
The safety significance and enforcement aspects of this finding are described in
Sections 3.2 and 4.2, respectively.
2.3
Inadequate Evaluation Results in Runout of Component Cooling Water Pump
A self-revealing, Green noncited violation of 10 CFR Part 50, Appendix B, Criterion III,
Design Control, was identified when Unit 2 experienced a loss of instrument air due to
the failure of a soldered joint. Specifically, the loss of instrument air resulted in
component cooling water (CCW) Pump 024 being in a runout condition for
approximately 75 minutes due to a previous system modification.
In 1995, the licensee implemented a design change to the CCW system to provide
backup nitrogen to the non-critical loop (NCL) supply and return isolation valves. The
Enclosure
-20-
design change was made to ensure that CCW flow would be maintained to the reactor
coolant pump (RCP) seals during a loss of instrument air event. This would preclude
the licensee from the need to secure the RCPs due to a loss of CCW cooling in the
event of a loss of instrument air pressure.
On June 20, 2007, the CCW system was aligned with the Train A Pump 024 in
operation with a normal full operating load on the system, including the non-critical loop.
The Train B pump was in standby. At approximately 10:45 pm, Unit 2 experienced a
loss of instrument air when a 3-inch air header fitting separated in the auxiliary building.
Following the loss of instrument air pressure, the shutdown cooling heat exchanger
isolation valves failed opened as designed. Since the NCL isolation valves remained
open, the increased system load from the shutdown cooling heat exchanger caused the
CCW pump flow rate to increase to approximately 300 gallons per minute more than its
maximum design flow limit of 16,000 gallons per minute, placing the pump in a runout
condition. The pump operated in this condition for approximately 75 minutes before
operators took action to reduce system flow rate.
The inspectors reviewed this issue and determined that the licensee had not performed
an adequate hydraulic analysis of the CCW system in 1995 when implementing the
design change to maintain the NCL supply and return isolation valves open following a
loss of instrument air. The inspectors determined that this design change directly
contributed to placing the CCW pump in a runout condition following the loss of
instrument air.
The safety significance and enforcement aspects of this finding are described in
Sections 3.3 and 4.3, respectively.
2.4
Ineffective Corrective Actions for a Failed Control Room Annunciator
The inspectors reviewed a self-revealing Green finding involving the failure to take
effective corrective actions for a failed control room annunciator. Specifically, after the
annunciator for actuation of the backup nitrogen supply to the instrument air system
failed to function on demand on several occasions from 1994 through 2007, the
corrective actions taken by the licensee to restore the annunciator to service were
inadequate and narrowly focused. The annunciator subsequently failed to function
during the loss of instrument air event on June 20, 2007.
In June 1994, SONGS Units 2 and 3 experienced a loss of instrument air event during
which the annunciator for actuation of the backup nitrogen supply to the instrument air
system failed to actuate in the control room. The licensee entered this into their
corrective action program and generated Maintenance Order 94062628000 to
investigate and correct the issue. During the investigation, licensee maintenance
technicians noted that the limit switch was dirty. Since a replacement limit switch was
not available, the licensee cleaned the installed limit switch and returned it to service.
The work order subsequently closed with no further actions taken by the licensee.
While the licensee was performing an evolution to repressurize the backup nitrogen line
to the instrument air system in May 1996, the annunciator for actuation of the backup
nitrogen supply to instrument air again failed to actuate in the control room. The
licensee entered this event into their corrective action program as Action Request
AR 960500111. In this AR the licensee determined that possible causes of the failure
Enclosure
-21-
were slow flow rate, not enough to flow to open the check valve far enough to trip the
limit switch, or the possibility of a problem with the limit switch. The licensee generated
Maintenance Order 94062628001 to investigate and correct the issue. During their
inspection, the licensee maintenance technicians found rust on the limit switch and
determined this to be the cause of the failure. The limit switch was replaced, and the
licensee verified that it worked electrically. The inspectors noted the maintenance order
called for an operational test of the limit switch, but none was performed.
In April 2007, Units 2 and 3 experienced a loss of instrument air event. During this
event, the annunciator for actuation of the backup nitrogen supply to the instrument air
system again failed to actuate in the control room. The licensee entered this into their
corrective action program as AR 070400776 and generated Maintenance Order
07041277000 to investigate and correct the issue. During their inspection, the licensee
maintenance technicians noted that the travel on the switch was satisfactory and there
were no problems on the electrical part of the system. No other work was documented.
On June 20, 2007, both Unit 2 and Unit 3 experienced a loss of instrument air pressure
due to the failure of a three-inch instrument air line header fitting. During this event, the
annunciator for actuation of the backup nitrogen supply to the instrument air system
failed to actuate in the control room again. The licensee entered this into their corrective
action program as AR 070601250.
The inspectors concluded that the licensee failed to adequately evaluate and correct the
issue associated with the limit switch. During their review, the inspectors also noted that
the licensee had not questioned or investigated the operational aspects of the limit
switch. Instead, the licensee had narrowly focused on testing only the electrical portion
of the system. The inspectors determined that the licensee had not operationally tested
the limit switch during any of their corrective actions.
The licensee subsequently performed an operational test of the limit switch. During this
testing, the licensee determined the nitrogen flow through the check valve was not
sufficient to actuate the limit switch. Consequently, the limit switch would never have
functioned to actuate its associated control room annunciator. The licensee entered this
issue into their corrective action program.
The safety significance and enforcement aspects of this finding are described in
Sections 3.4 and 4.4, respectively.
2.5
Inadequate Procedure for a Loss of Instrument Air
The inspectors identified a Green noncited violation of Technical Specification 5.5.1.1
involving the failure to maintain an adequate abnormal operating instruction for a loss of
instrument air event.
Procedure SO23-13-5, Loss of Instrument Air, Revision 5, specifies operator actions to
mitigate the effects of excessive instrument air system leakage or the loss of the
instrument air compressors. The inspectors reviewed the procedure and noted the
following:
Step 1.a of the procedure was followed by a caution stating:
Enclosure
-22-
A large break downstream of the nitrogen supply may cause the nitrogen
excess flow check valve to seat. This may be indicated by falling nitrogen
header pressure on 2/3PI-5344B, followed by rapid return to > 80 psig.
The inspectors noted the nitrogen header pressure indication on 2/3PI-5344B
was not provided with a strip chart recorder or computer point to provide trend
information and the procedure did not direct stationing a dedicated operator to
monitor the pressure instruments. The inspectors concluded that absent a
dedicated operator to observe the pressure indicator, no trend information would
be available to the control room operators to determine whether the described
pressure response had occurred.
Step 1.b of the procedure directed operators to determine whether or not the
instrument air header pressure was stable or rising. The inspectors concluded
this determination would be complicated by the lack of any available trend
information for all of the air header pressure instruments.
Step 2.a of the procedure directed operators to trip the reactors and turbines of
both units in the event of a loss of both instrument air header pressure and
nitrogen header pressure as indicated by the control room instruments. The
inspectors noted the control room instruments only provided pressure indication
for the common headers; there were no available indications for the pressure in
the individual headers of each unit. The inspectors noted that in the event the
backup nitrogen system excess flow check valves either spuriously shut or failed
to open on a complete loss of instrument air, the nitrogen header pressure
instrument would continue to indicate sufficient pressure despite the complete
depressurization of the instrument air headers in both units. In this case,
Step 2.a would direct operators to the Subsequent Actions section of the
procedure to monitor plant response instead of the more appropriate action of
Step 2.b to immediately trip both units.
The inspectors determined the above issues could result in a delay of necessary
operator response actions to mitigate the consequences of an initiating event.
The safety significance and enforcement aspects of this finding are described in
Sections 3.5 and 4.5, respectively.
2.6
Simulator Incorrectly Modeled Plant Response to Loss of Instrument Air
A self-revealing, Green noncited violation of 10 CFR Part 55.46(c)(1) was identified
involving the licensees failure to incorporate a design change in modeling plant
response for the plant-referenced simulator. Specifically, during operator training in the
plant-referenced simulator, the controlled bleedoff valves for the reactor coolant pumps
were modeled to fail closed on a loss of instrument air, whereas the valves in the plant
remained open during an actual loss of instrument air event on June 20, 2007.
The original model regulator installed on the reactor coolant pump (RCP) controlled
bleed off (CBO) Valve 2HV9218 allowed air to bleed off on a loss of instrument air,
enabling the valve actuator to move shut to its fail-safe position. Though not an
intentional design feature of the regulator, the licensee took credit for this feature to shut
Enclosure
-23-
the RCP CBO valve during a loss of instrument air. As such, the plant-referenced
simulator used for operator training modeled the valve going shut during a loss of
instrument air event.
In February 2004, the licensee replaced the existing valve regulator for the RCP CBO
valve with a new style regulator. The new regulators were used because the original
regulator was obsolete. The vendor modified the new regulators to make them leak
tight, removing the unintentional bleed off characteristic. The licensee evaluated the
change in the new regulators and determined the new regulators to be an equivalent
valve as part of Substitute Equivalency Evaluation (SEE) 020040.
On June 20, 2007, both Units 2 and 3 experienced a loss of instrument air event due to
the failure of a three-inch instrument air line header fitting. During this event, the RCP
CBO containment isolation Valve 2HV9218 failed to go closed as the operators
expected. The control room dispatched an operator at the time of the event to
investigate and determine why the valve did not go closed. The operator noted locally
that the valve was open and that pressure was present on the regulator. The control
room subsequently dispatched another operator to independently second-check the
position of the valves. The second operator also noted that the valve was open and that
pressure was present on the regulator. The operators took no other actions at the time
because closure of this valve could cause loss of CBO flow which would have required
the RCPs to be secured.
The inspectors determined that the licensee failed to update the plant-referenced
simulator following the CBO valve regulator change. As a result, operators were trained
that the RCP CBO valves would shut during a loss of instrument air. However, during
the actual loss of instrument air event on June 20 the CBO valve did not go shut as
expected which caused confusion among the operators responding to the event.
The safety significance and enforcement aspects of this finding are described in
Sections 3.6 and 4.6, respectively.
2.7
Failure to Follow Procedure for an Impaired Annunciator
The inspectors identified a Green noncited violation of Technical Specification 5.5.1.1
involving the failure to meet procedural requirements governing impaired annunciators.
Specifically, after the identification of a failed annunciator, operators did not enter the
annunciator in the failed annunciator log or mark the affected annunciator window with
an annunciator compensatory action flag.
The instrument air system at SONGS utilizes a low pressure nitrogen system as a
backup pressure source. An instrument air system pressure drop will automatically
actuate a control valve in the nitrogen system to supplement the instrument air from
liquid nitrogen storage tanks. A flow switch downstream of the control valve is designed
to provide an annunciator in the control when the nitrogen system is actuated. This
annunciator is used as a diagnostic aid and to determine operator actions in
Procedure SO23-13-5, Loss of Instrument Air, Revision 5.
On June 20, 2007, an instrument air header rupture occurred in Unit 2. The inspectors
noted that although the low pressure nitrogen system provided nitrogen to the Unit 3 air
header as designed, the control room annunciator for nitrogen system actuation did not
Enclosure
-24-
alarm. The licensee initiated AR 070601250 on June 29, 2007 to address the failed
annunciator. On July 2, the inspectors noted the failed annunciator was not included in
the impaired annunciator log. The licensee polled two shift managers and determined
that one believed the annunciator should be treated as impaired and requiring
compensatory actions per Procedure SO23-6-29, Operation of Annunciators and
Indicators, Revision 15, and the other shift manager did not. The inspectors concluded
the operators were not consistently implementing the portion of the procedure
concerning impaired annunciators. The licensee subsequently entered the annunciator
in the impaired annunciator log and took the actions specified by Procedure SO23-6-29
for an impaired annunciator.
The safety significance and enforcement aspects of this finding are described in
Sections 3.7 and 4.7, respectively.
2.8
Inadequate Implementation of Corrective Actions for Air Operated Valve Regulators
A Green self-revealing finding was identified associated with the failure of the reactor
coolant pump controlled bleed off valve to shut during a loss of instrument air event.
The licensee failed to adequately implement corrective actions from previously
evaluated industry operating experience for new valve regulators that were installed in
the unit.
In July 2002, industry Operating Experience (OE) was issued which identified potentially
undesirable consequences due a design change to air operated valve regulators that
improved leakage characteristics of the regulators. The old model regulators allowed air
pressure to bleed off on a loss of instrument air, which enabled the valve actuator to
move to its fail-safe position. This was not an intentional design feature of the regulator.
The new regulators were changed to correct this unintentional bleed off and make them
leak tight. The OE was issued to alert users to this change so that if a user had taken
credit for this unintentional bleed off they would be aware of this change and
appropriately address it.
The licensee evaluated this OE and determined that it was applicable to the station.
AR 031001558 was initiated in October 2003 to provide appropriate actions to address
any issues. The licensee identified that this change would not affect air operated valves
that have an associated positioner or controller, or valves that are configured with a
solenoid valve installed between the air regulator and actuator that receives a signal to
vent. Valves without an associated positioner, controller, solenoid valve, or other
configuration to allow for air bleed off could be affected by use of the new regulator.
During this evaluation the licensee also determined that there were not any of the new
regulators in use at the facility at the time.
The licensee had already performed Substitute Equivalency Evaluations (SEE) for
replacing some of the old style regulators with the new style. Based on the results of
the OE review the licensee determined that there was a need to revise the existing
SEEs to require design engineering and procurement engineering to perform a design
change impact review to evaluate the installation configuration. The purpose of this
review was to evaluate whether a design change would be necessary prior to installation
of the new style regulators.
Enclosure
-25-
In February 2004 the licensee replaced the old style regulator with a new style regulator
on the reactor coolant pump (RCP) controlled bleed off (CBO) Valve 2HV9218. This
was evaluated under SEE 020040.
On June 20, 2007 both Units 2 and 3 experienced a loss of instrument air event due to
the failure of a three-inch instrument air line header fitting. During this event, the RCP
CBO containment isolation valve, 2HV9218, failed to go closed as expected. The
licensee dispatched an operator at the time of the event to investigate and determine
why the valve did not go closed. The operator noted that the valve was open and that
pressure was present on the regulator. The licensee took no other actions at the time
because closure of this valve could cause loss of CBO flow which would have required
the RCPs to be secured.
The licensee performed a review of this issue as documented in AR 070600873. During
this review, the licensee determined that the valve should have shut during the loss of
instrument air event and did not because of the new style regulator that had been
installed. The licensee also identified that the SEE appropriately identified the
requirement for design engineering and procurement engineering to perform a design
change impact to evaluate the installation configuration. However, the action by
procurement engineering to require a design change review prior to installation of the
regulator failed due to a known computer software limitation, and the maintenance
engineering review inappropriately determined that there were no applications where the
old regulators did not have a solenoid or bleed off device between the regulator and
solenoid. The licensee also identified that during both of these assessments, the
engineers did not review the UFSAR or any other licensing commitments that credited
bleed down characteristics of the old regulators during a loss of instrument air.
The licensee also performed an extent of condition review to determine if there were any
other instances of these new style regulators being installed in the plant. This review
identified 37 instances of the new regulators being installed in the plant without
performance of a design change impact review.
The safety significance and enforcement aspects of this finding are described in
Sections 3.8 and 4.8, respectively.
3.0
ASSESSMENT
3.1
Ineffective Corrective Actions for Instrument Air Header Ruptures
The failure to take effective corrective actions in response to site and industry operating
experience resulting in a subsequent instrument air header failure was a performance
deficiency. This finding was more than minor since it was associated with the
equipment reliability attribute of the initiating events cornerstone and affected the
cornerstone objective to limit the likelihood of events that upset plant stability and
challenge critical safety functions. This finding required a Phase 2 analysis per the
Manual Chapter 0609, Significance Determination Process, Phase 1 Worksheets since
the loss of instrument air is a transient initiator resulting in the loss of the feedwater
system which is part of the power conversion system which can be used to mitigate the
consequences of an accident. The inspectors performed a Phase 2 analysis using
Appendix A, Technical Basis for At-Power Significance Determination Process, and the
Phase 2 worksheets for SONGS. The inspectors assumed that the exposure period
Enclosure
-26-
was greater than 30 days, that the performance deficiency increased the likelihood that
a complete loss of instrument air would occur, and that there was no affect on mitigating
systems other than those modeled in the risk-informed notebook. Details of the
Phase 2 analysis and a subsequent Phase 3 analysis are documented in Attachment 3.
Based on the results of the Phase 3 analysis, the finding was determined to be of very
low safety significance (Green) because of the availability of the diverse auxiliary
feedwater system and the ability of the operators to depressurize the steam generators
and utilize the condensate system for heat removal. These results were evaluated by a
senior reactor analyst. In addition, the senior reactor analyst determined the impact of
this performance deficiency on the likelihood of the large-early release frequency.
These evaluations indicated that the impacts were also of very low safety significance.
This finding has a crosscutting aspect in the area of problem identification and
resolution associated with operating experience in that the licensee failed to effectively
implement changes to station processes, procedures, and equipment in response to
operating experience involving improperly made instrument air system joints P.2(b).
3.2
Failure to Follow Abnormal Operating Instruction for the Loss of Instrument Air
The failure to follow station procedures to monitor nitrogen tank levels and oxygen
concentrations in enclosed rooms where operator actions may have been required was
a performance deficiency. This finding was more than minor since it was associated
with the human performance attribute of the initiating events cornerstone and affected
the cornerstone objective to limit the likelihood of events that upset plant stability and
challenge critical safety functions. This finding required a Phase 2 analysis in
accordance with the Manual Chapter 0609, Significance Determination Process,
Phase 1 Worksheets since the loss of instrument air is a transient initiator resulting in
the loss of the feedwater system which is part of the power conversion system which
can be used to mitigate the consequences of an accident. The inspectors performed a
Phase 2 analysis using Appendix A, Technical Basis for At-Power Significance
Determination Process, and the Phase 2 worksheets for SONGS. The inspectors
assumed that the exposure period was greater than 30 days, that the performance
deficiency increased the likelihood that a complete loss of instrument air would occur,
and that there was no affect on mitigating systems other than those modeled in the risk-
informed notebook. Based on the results of the Phase 2 analysis, the finding was
determined to be of very low safety significance because of the low likelihood of a
complete loss of instrument air and the availability of the auxiliary feedwater system.
These results were evaluated by a senior reactor analyst. In addition, the senior reactor
analyst determined the impact of this performance deficiency on the risk of external
events and on the likelihood of the large-early release frequency. These evaluations
indicated that the impacts were also of very low safety significance.
The cause of this finding has a crosscutting aspect in the area of human performance
associated with resources because licensee personnel were not adequately trained on
the operation of the low pressure nitrogen system to effectively implement the abnormal
operating instruction H.2(b).
Enclosure
-27-
3.3
Inadequate Evaluation Results in Runout of Component Cooling Water Pump
The failure to adequately evaluate the total system hydraulic effects prior to
implementing a design change to supply nitrogen to the NCL isolation valves was a
performance deficiency. This finding was greater than minor because it was associated
with the mitigating systems cornerstone attribute of design control and affected the
associated cornerstone objective to ensure the availability, reliability, and capability of
systems that respond to initiating events to prevent undesirable consequences. The
finding did not affect the initiating events cornerstone functions of the component
cooling water system because the condition would only have existed given a loss of
instrument air initiator had already occurred. In accordance with NRC Inspection
Manual Chapter 0609, Appendix A, Phase 1 Worksheet, Significance Determination
Process (SDP) Phase 1 Screening Worksheet for the Initiating Events, Mitigating
Systems, and Barriers Cornerstones, this finding was determined to be of very low
safety significance because the finding was a design deficiency confirmed not to result
in a loss of operability per Part 9900, Technical Guidance, Operability Determination
Process for Operability and Functional Assessment.
3.4
Ineffective Corrective Actions for a Failed Control Room Annunciator
The failure to perform adequate corrective actions for a failed control room annunciator
resulting in the failure of the annunciator to function during an actual event was a
performance deficiency. This finding was more than minor since it was associated with
the human performance attribute of the initiating events cornerstone and affected the
cornerstone objective to limit the likelihood of events that upset plant stability and
challenge critical safety functions. This finding required a Phase 2 analysis in
accordance with the Manual Chapter 0609, Significance Determination Process,
Phase 1 Worksheets since the loss of instrument air is a transient initiator resulting in
the loss of the feedwater system which is part of the power conversion system which
can be used to mitigate the consequences of an accident. The inspectors performed a
Phase 2 analysis using Appendix A, Technical Basis for At-Power Significance
Determination Process, and the Phase 2 worksheets for SONGS. The inspectors
assumed that the exposure period was greater than 30 days, that the performance
deficiency increased the likelihood that a complete loss of instrument air would occur,
and that there was no affect on mitigating systems other than those modeled in the risk-
informed notebook. Based on the results of the Phase 2 analysis, the finding was
determined to be of very low safety significance because of the low likelihood of a
complete loss of instrument air and the availability of the auxiliary feedwater system.
These results were evaluated by a senior reactor analyst. In addition, the senior reactor
analyst determined the impact of this performance deficiency on the risk of external
events and on the likelihood of the large-early release frequency. These evaluations
indicated that the impacts were also of very low safety significance.
This finding has a crosscutting aspect in the area of problem identification and
resolution associated with the corrective action program in that the licensee failed to
thoroughly evaluate the failed annunciator such that the resolution appropriately
addressed the causes P.2(c).
Enclosure
-28-
3.5
Inadequate Procedure for a Loss of Instrument Air
The failure to provide adequate procedural guidance to immediately diagnose and
properly respond to an initiating event was a performance deficiency. This finding was
more than minor because it was associated with the procedure quality attribute of the
mitigating systems cornerstone and affected the cornerstone objective to ensure the
availability, reliability and capability of systems that respond to initiating events, in that a
less than adequate abnormal operating procedure could have prevented operators from
promptly tripping the reactor, allowing conditions to continue to degrade and resulting in
a demand on the reactor protection system. Using the Significance Determination
Process Phase 1 Screening Worksheet in Appendix A of Inspection Manual Chapter 0609, the inspectors determined this finding had very low safety significance because it
did not result in an actual loss of safety function per Part 9900, Technical Guidance,
Operability Determination Process for Operability and Functional Assessment.
This finding has a crosscutting aspect in the area of human performance associated
with resources in that the licensee failed to provide operators with complete, accurate,
and up-to-date procedures H.2(c).
3.6
Simulator Incorrectly Modeled Plant Response to Loss of Instrument Air
The failure to ensure that the plant-referenced simulator correctly replicated expected
plant response to transient conditions was a performance deficiency. This finding was
greater than minor because it was associated with the mitigating systems cornerstone
attribute of human performance and affected the associated cornerstone objective to
ensure the availability, reliability, and capability of systems that respond to initiating
events to prevent undesirable consequences. The inspectors evaluated this finding
using the Appendix I, Licensed Operator Requalification Significance Determination
Process worksheets of Manual Chapter 0609 because the finding is a requalification
training issue related to simulator fidelity. Block 12 of the Appendix I flow chart requires
the inspector to determine if deviations between the plant and simulator could result in
negative training or could have a negative impact on operator actions. Negative
Training is defined, in a later version of the standard (ANSI 3.5-1998), as training on a
simulator whose configuration or performance leads the operator to incorrect response
or understanding of the reference unit. The licensee has committed to this version of
the ANS/ANSI standard for its simulator testing program for the plant-referenced
simulator. During the event of June 20, 2007, operators were influenced by negative
training on the simulator to question control room indications and locally independently
verify valve positions because valves in the plant failed to respond to a loss of
instrument air as modeled in the simulator. Therefore, differences between the simulator
and plant did have a negative impact on operator actions. The finding is of very low
safety significance because the discrepancy did not have an adverse impact on operator
actions such that safety related equipment was made inoperable during normal
operations or in response to a plant transient.
This finding has a crosscutting aspect in the area of human performance associated
with resources in that the licensee did not provide operators with adequate facilities and
equipment for use in operator training H.2(d).
Enclosure
-29-
3.7
Failure to Follow Procedure for an Impaired Annunciator
The failure to follow station procedures resulting in an untracked nonfunctional
annunciator was a performance deficiency. This finding was more than minor since it
was associated with the human performance attribute of the initiating events
cornerstone and affected the cornerstone objective to limit the likelihood of events that
upset plant stability and challenge critical safety functions. This finding required a
Phase 2 analysis in accordance with the Manual Chapter 0609, Significance
Determination Process, Phase 1 Worksheets since the loss of instrument air is a
transient initiator resulting in the loss of the feedwater system which is part of the power
conversion system which can be used to mitigate the consequences of an accident.
The inspectors performed a Phase 2 analysis using Appendix A, Technical Basis for
At-Power Significance Determination Process, and the Phase 2 worksheets for
SONGS. The inspectors assumed that the exposure period was greater than 30 days,
that the performance deficiency increased the likelihood that a complete loss of
instrument air would occur, and that there was no affect on mitigating systems other
than those modeled in the risk-informed notebook. Based on the results of the Phase 2
analysis, the finding was determined to be of very low safety significance because of the
low likelihood of a complete loss of instrument air and the availability of the auxiliary
feedwater system. These results were evaluated by a senior reactor analyst. In
addition, the senior reactor analyst determined the impact of this performance deficiency
on the risk of external events and on the likelihood of the large-early release frequency.
These evaluations indicated that the impacts were also of very low safety significance.
This finding has a crosscutting aspect in the area of human performance associated
with resources because the operators were not sufficiently trained to consistently
implement the annunciator operating procedure H.2(b).
3.8
Inadequate Implementation of Corrective Actions for Air Operated Valve Regulators
The failure to adequately implement corrective actions from industry OE to perform a
design change impact review was a performance deficiency. The finding was greater
than minor because it was associated with the mitigating systems cornerstone attribute
of design control and affected the associated cornerstone objective to ensure the
availability, reliability, and capability of systems that respond to initiating events to
prevent undesirable consequences. Using Manual Chapter 0609, Significance
Determination Process, Phase 1 Worksheet, the finding is determined to have very low
safety significance because the condition only affected the mitigation systems
cornerstone and it was confirmed not to result in loss of operability per Part 9900,
Technical guidance, Operability Determination Process for Operability and Functionality
Assessment.
4.0
ENFORCEMENT
4.1
Ineffective Corrective Actions for Instrument Air Header Ruptures
No violation of regulatory requirements occurred since the affected equipment was not
safety-related. This finding was entered into the licensees corrective action program as
Action Request AR 070600867 and is identified as FIN 05000361;362/2007013-01,
Ineffective Corrective Actions for Instrument Air Header Ruptures.
Enclosure
-30-
4.2
Failure to Follow Abnormal Operating Instruction for the Loss of Instrument Air
Technical Specification 5.5.1.1 requires written procedures to be implemented as
recommended by Regulatory Guide 1.33, Revision 2, Appendix A, February 1978.
Section 6.b of Appendix A recommends procedures governing actions to be taken on a
loss of instrument air. Step 3.h of Procedure SO23-13-5, Loss of Instrument Air,
Revision 5, required notification of all building operators of the possibility of oxygen
deficient areas, monitoring of enclosed spaces for oxygen levels prior to entry, and
monitoring of liquid nitrogen inventory following actuation of the backup nitrogen system
following an instrument air leak. Contrary to this requirement, operators failed to
implement these actions following actuation of the nitrogen system due to an instrument
air line break on June 20, 2007. Because this violation was of very low safety
significance and was entered in the corrective action program as Action Request
AR 070700291, this violation is being treated as an NCV consistent with Section VI.A.1
of the NRC Enforcement Policy: NCV 05000361;362/2007013-02, Failure to Follow
Abnormal Operating Instruction for the Loss of Instrument Air.
4.3
Inadequate Evaluation Results in Runout of Component Cooling Water Pump
10 CFR Part 50, Appendix B, Criterion III, Design Control, requires, in part, that
measures be established to assure that applicable regulatory requirements and the
design basis, as specified in the license application, are correctly translated into
specifications, drawings, procedures, and instructions. It further states that design
control measures shall provide for verifying or checking the adequacy of design, such as
by the performance of design reviews, by the use of alternate or simplified calculational
methods, or by the performance of a suitable testing program. Contrary to the above,
the licensee failed to verify the adequacy of the design associated with the modification
of the CCW NCL isolation valves installed in 1995. Because this finding is of very low
safety significance and has been entered into the corrective action program as Action
Requests AR 070700051 and 070600872, this violation is being treated as an NCV
consistent with Section VI.A of the NRC Enforcement Policy:
NCV 05000361;362/2007013-03, Inadequate Evaluation Results in Runout of
Component Cooling Water Pump.
4.4
Ineffective Corrective Actions for a Failed Control Room Annunciator
No violations of NRC requirements were identified during the review of this issue
because instrument air is not a safety related system. The licensee entered this issue
into the corrective action program as Action Request AR 070601250:
FIN 05000361;362/2007013-04, Ineffective Corrective Actions for a Failed Control
Room Annunciator.
4.5
Inadequate Procedure for a Loss of Instrument Air
Technical Specification 5.5.1.1 requires written procedures to be implemented as
recommended by Regulatory Guide 1.33, Quality Assurance Program Requirements,
Revision 2, Appendix A, February 1978. Section 6.b of Appendix A of Regulatory Guide 1.33 recommends procedures governing actions to be taken on a loss of instrument air.
American National Standard ANS-3.2, Administrative Controls and Quality Assurance
for the Operational Phase of Nuclear Power Plants, February 1976, describes the
requirements for the quality of the procedures specified in Regulatory Guide 1.33.
Enclosure
-31-
Section 5.3.9.1(4) of Standard ANS-3.2 requires emergency procedures to specify
immediate actions for operation of controls or confirmation of automatic actions that are
required to stop the degradation of conditions and mitigate consequences. Contrary to
this requirement, since its release on May 4, 2006, Revision 5 of Procedure SO23-13-5,
Loss of Instrument Air, did not provide adequate guidance for operators to immediately
diagnose and properly respond to a complete loss of the instrument air system.
Because this violation was of very low safety significance and was entered in the
corrective action program as Action Request AR 070801151, this violation is being
treated as an NCV consistent with Section VI.A.1 of the NRC Enforcement Policy:
NCV 05000361;362/2007013-05, Inadequate Procedure for Loss of Instrument Air.
4.6
Simulator Incorrectly Modeled Plant Response to Loss of Instrument Air
10 CFR Part 55.46(c)(1) requires, in part, that the plant-referenced simulator must
demonstrate expected plant response to transient conditions. Contrary to this
requirement, the response modeled by the licensees simulator for the reactor coolant
pump controlled bleedoff valves did not demonstrate expected plant response to the
June 20, 2007 loss of instrument air event. Because this violation was of very low safety
significance and was entered in the corrective action program as Action Requests
AR 070600873 and 070900160, this violation is being treated as an NCV consistent with
Section VI.A.1 of the NRC Enforcement Policy: NCV 05000361;362/2007013-06,
Simulator Incorrectly Modeled Plant Response to Loss of Instrument Air.
4.7
Failure to Follow Procedure for an Impaired Annunciator
Technical Specification 5.5.1.1 requires written procedures to be implemented as
recommended by Regulatory Guide 1.33, Revision 2, Appendix A, February 1978.
Section 1.c of Appendix A recommends administrative procedures governing equipment
control. Section 6.2.2 of Procedure SO23-6-29, Operation of Annunciators and
Indicators, Revision 15, required tracking of impaired annunciators requiring
compensatory actions. Contrary to this requirement, operators failed to track an
impaired annunciator from June 29 to July 2, 2007. Because this violation was of very
low safety significance and was entered in the corrective action program as Action
Request AR 070700291, this violation is being treated as an NCV consistent with
Section VI.A.1 of the NRC Enforcement Policy: NCV 05000361;362/2007013-07,
Failure to Follow Procedure for an Impaired Annunciator.
4.8
Inadequate Implementation of Corrective Actions for Air Operated Valve Regulators
No violation of regulatory requirements occurred. This finding was entered in the
licensees corrective action program as Action Request AR 070600873:
FIN 05000361;362/2007013-07, Inadequate Implementation of Corrective Actions for
Air Operated Valve Regulators.
Enclosure
-32-
4OA6 Meetings, Including Exit
On July 2, 2007, and September 13, 2007, the results of this inspection were presented
to Dr. R. Waldo, Vice President Nuclear Generation, and other members of his staff who
acknowledged the findings. Additionally on October 11, 2007, the final results of this
inspection were presented to A. Scherer, Manager, Nuclear Regulatory Affairs, and
other members of his staff who acknowledged the findings. The inspector confirmed
that no proprietary material was examined during the inspection.
ATTACHMENT 1: SUPPLEMENTAL INFORMATION
ATTACHMENT 2: SPECIAL INSPECTION CHARTER
ATTACHMENT 3: SIGNIFICANCE DETERMINATION EVALUATION
Attachment 1
A1-1
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
K. Flynn, Site Operating Experience Coordinator
S. Gardner, Engineer, Nuclear Regulatory Affairs
B. Katz, Vice President, Nuclear Oversight and Regulatory Affairs
L. Kelly, Engineer, Nuclear Regulatory Affairs
M. Love, Manager, Maintenance
M. Mostafa, Consulting Engineer
K. Rauch, Operations Training Manager
A. Scherer, Manager, Nuclear Regulatory Affairs
P. Schofield, System Maintenance Engineer Supervisor
J. Summy, System Engineering Manager
D. Tuttle, Systems Engineer
T. Vogt, Manager, Operations
R. Waldo, Vice President, Nuclear Generation
D. Wilcockson, Manager, Plant Operations
C. Williams, Manager, Compliance
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Opened and Closed
05000361;
362/2007013-01
Ineffective Corrective Actions for Instrument Air Header
Ruptures
05000361;
362/2007013-02
Failure to Follow Abnormal Operating Instruction for the
Loss of Instrument Air
05000361;
362/2007013-03
Inadequate Evaluation Results in Runout of Component
Cooling Water Pump
05000361;
362/2007013-04
Ineffective Corrective Actions for a Failed Control Room
05000361;
362/2007013-05
Inadequate Procedure for a Loss of Instrument Air
05000361;
362/2007013-06
Simulator Incorrectly Modeled Plant Response to a Loss of
Instrument Air
05000361;
362/2007013-07
Failure to Follow Procedure for an Impaired Annunciator
05000361;
362/2007013-08
Inadequate Implementation of Corrective Actions for Air
Operated Valve Regulators
Attachment 1
A1-2
LIST OF DOCUMENTS REVIEWED
Procedures
Number
Title
Revision
SO23-6-29
Operation of Annunciators and Indicators
15
SO23-13-5
Loss of Instrument Air
5
SO123-0-A8
Trip/Transient and Event Review
1
SO23-12-1
Standard Post Trip Actions
21
SO23-12-2
Reactor Trip Recovery
18
SO23-1-1
Instrument Air System Operation
17
SO123-XV-5.3
9
SO23-6-29
Operation of Annunciators and Indicators
15
SO123-XV-50.39.1
Division Investigative Reports
0
SO123-I-1.42
Maintenance Division Experience Report
0
SO123-XV-50
Corrective Action Process
6
Action Requests
070400754
060101956
070400766
070600867
051000080
050600477
070600877
050901305
041101801
060101956
050901037
070600914
070600867
070600872
960500111
031001558
041000977
010601495
070600873
041002146
061001297
070600914
040901643
060101956
070500196
070600870
070601250
070501276
070400776
060200898
Work Orders/Maintenance Work Orders
07061161000
06012099000
94062628000
94062628001
07041277000
07061216000
Attachment 1
A1-3
07041921000
06020520000
05111108000
05111009000
05061587000
05061588000
06020580000
05111105000
05111106000
05111107000
07051780000
05061583000
05061589000
05061590000
06021071000
07050347000
05061584000
05061585000
05061586000
07061216000
07061324000
07061325000
Drawings
Number
Title
Revision
F-10946M
Component Cooling Water System No. 1203
21
F-10543M
Component Cooling Water System No. 1203
15
40191A
Compressed Air System
15
40191B
Compressed Air System
21
40191C
Compressed Air System
19
40191D
Compressed Air System
34
40191DSO3
Compressed Air System
19
40191F
Compressed Air System
13
40191X
Compressed Air System
2
40190C
Respiratory Service Air
22
40192A
Auxiliary Gas System
20
40192B
Auxiliary Gas System
16
40191GSO3
Instrument Air Distribution
7
40191E-10
Instrument Air System
10
40191G
Instrument Air Distribution
8
Calculations
M-DSC-429, Evaluation of Joint Restraint Clamp on Instrument Air Piping, Revision 0
M-0091, Backup Nitrogen for the Instrument Air System Equipment Sizing, Revision 0
IPE-HC-006, Operator Action Summary Data Sheet Post-Initiator Human Error Probability
Calculation Worksheet
Attachment 1
A1-4
Miscellaneous Information
PRA-07-007, PRA Preliminary Evaluation of Loss of Instrument Air Event Resulting in Unit 2
Trip, Dated June 22, 2007
Operator Action Summary Data Sheet Post-Initiator Human Error Probability Calculation
Worksheet
DBD-SO23-540, Instrument Air/Dedicated Backup Nitrogen System, Revision 6
Engineering Change Package 070600914-6, Revision 0
SONGS 2 Instrument Air System Failed Fitting Metallurgical Evaluation
SONGS System Health Reports for the Instrument Air System and Vendor Owned Nitrogen
Package
Failure Analysis Report No.94-006, Failure Analysis of the Instrument Air Fitting for
Temperature Gauge 2/3TI5380
Failure Analysis Report No.94-009, Failure Analysis of the Instrument Air Fitting for
Temperature Gauge 2/3TI5380, Supplement 1, Dated October 3, 1994
DBD-SO23-540, Instrument Air/Dedicated Backup Nitrogen Systems, Revision 6
Maintenance Rule Guide Book, Dated February 2004
SD-SO23-400, Component Cooling Water System, Revision 6
Meeting Agenda Maintenance Rule Expert Panel, Dated June 21, 2007
Regulatory Guide 1.160, Monitoring the Effectiveness of Maintenance at Nuclear Power Plants,
Revision 2
Licensee Event Report No. 2007-001-01, Revision 1, Dated August 24, 2004
Substitute Equivalency Evaluation 020040, Substitute 67CFR-237 Series Regulator for 67AFR-
237, Revision 1
A2-1
Attachment 2
June 26, 2007
MEMORANDUM TO: Geoffrey Miller, Senior Resident Inspector, Grand Gulf
Jeffrey Josey, Resident Inspector, Arkansas Nuclear One
FROM:
Arthur T. Howell III, Director, Division of Reactor Projects /RA AVegel for/
SUBJECT:
SPECIAL INSPECTION CHARTER TO EVALUATE THE SAN ONOFRE
NUCLEAR GENERATING STATION INSTRUMENT AIR FAILURE
A Special Inspection Team is being chartered in response to the Unit 2 San Onofre Nuclear
Generating Station loss of instrument air event on June 20, 2007. You are hereby designated
as the Special Inspection Team members. Mr. Miller is designated as the team leader. The
assigned SRA to support the team is David Loveless.
A.
Basis
On June 20, 2007, a 3-inch diameter instrument air line failed. At SONGS, instrument
air is a shared system, but the system is equipped with certain protective features
(excess flow check valves) to ensure that a failure in the piping system on one unit does
not significantly affect instrument air pressure on the other unit. On Unit 2, instrument
air pressure dropped significantly, from approximately 110 psig to about 43 psig. The
loss of instrument air pressure caused the feedwater control valves to stop functioning
and water level in the steam generators increased in an uncontrolled manner.
Operators manually tripped the reactor. The operators also lost control of the steam
dumps to the condenser (the normal heat removal method) and controlled steam
generator pressure and decay heat removal using the steam generator atmospheric
dump valves. The chemical and volume control system letdown function auto-isolated
and operators manually controlled pressurizer level with a charging pump. On Unit 3,
the pressure drop was not as significant but appeared to be more than expected.
However, Unit 3 Operators maintained control of all functions during the event.
Operators were able to isolate the failed instrument air line approximately 30 minutes
later and regained control of the Unit 2 condenser steam dumps.
During post-trip discussions with the operators, one operator stated that they had
experienced other instrument air piping failures but the affected piping was much
smaller and did not significantly challenge plant operations. One such failure occurred
in 1994.
This Special Inspection Team is chartered to review the circumstances related to
historical and present instrument air piping problems and assess the effectiveness of the
A2-2
Attachment 2
licensees actions for resolving these problems. The team will also assess the
effectiveness of the immediate actions taken by the licensee in response to the loss of
instrument air event on June 20, 2007.
B.
Scope
The team is expected to address the following:
1.
Develop a chronology (time-line) that includes significant event elements.
2.
Evaluate the operator response to the event. Ensure that operators responded
in accordance with plant procedures and took appropriate mitigating actions.
3.
Develop an understanding of the interface between instrument air and other risk
important systems, including the possible reliance, either short term or long term,
of safety related components on instrument air.
4.
Evaluate the plant response to the event. Ensure that all systems responded as
designed. In particular, verify that design provisions, intended to prevent failure
in one units piping from causing an excessive pressure drop in the other unit,
worked properly (see UFSAR Sections 9.3.1.1.E and 9.3.1.2.3).
5.
Assess the licensees root cause determination for the instrument air piping
failure, the extent of condition review, the common cause evaluation and
corrective measures. Evaluate whether the timeliness of the corrective
measures are consistent with the safety significance of the problems.
6.
Identify previous instrument air piping problems that may have been precursors
to the June 20 event, including one event in 1994. Evaluate the licensees
corrective measures and extent of condition reviews for those problems.
7.
Evaluate the licensees instrument air system maintenance and testing
programs. Verify that the programs are adequate and that the licensee is
following the program provisions. Pay particular attention to the historical health
of the instrument air compressors and piping system.
8.
Evaluate pertinent industry operating experience that represent potential
precursors to the June 20 event, including the effectiveness of licensee actions
taken in response to the operating experience. As a minimum include Generic Letter 88-14, Instrument Air Supply System Problems Affecting Safety-Related
Equipment, including the licensees response to the generic letter; and NRC
Information Notice 2002-29,Recent Design Problems in Safety Functions of
Pneumatic Systems. You may also use NUREG 1837, Regulatory
Effectiveness Assessment of Generic Issue 43 and Generic Letter 88-14, to aid
in your assessment. The NUREG can be found at:
http://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr1837/sr1837.pdf
A2-3
Attachment 2
9.
Determine if there are any potential generic issues related to the failure of the
SONGS instrument air piping. Promptly communicate any potential generic
issues to Region IV management.
10.
Collect data as necessary to support a risk analysis. Work closely with the
Senior Reactor Analyst during this inspection.
C.
Guidance
Inspection Procedure 93812, Special Inspection, provides additional guidance to be
used by the Special Inspection Team. Your duties will be as described in Inspection
Procedure 93812. The inspection should emphasize fact-finding in its review of the
circumstances surrounding the event. It is not the responsibility of the team to examine
the regulatory process. Safety concerns identified that are not directly related to the
event should be reported to the Region IV office for appropriate action.
The Team will report to the site, conduct an entrance, and begin inspection no later than
June 27, 2007. While on site, you will provide daily status briefings to Region IV
management, who will coordinate with the Office of Nuclear Reactor Regulation, to
ensure that all other parties are kept informed. A report documenting the results of the
inspection should be issued within 30 days of the completion of the inspection.
This Charter may be modified should the team develop significant new information that
warrants review. Should you have any questions concerning this Charter, contact me at
(817) 860-8147.
A3-1
Attachment 3
ATTACHMENT 3
SIGNIFICANCE DETERMINATION EVALUATION
San Onofre Nuclear Generating Station
Failure of Instrument Air System Header
Phase 3 Analysis
A.
Brief Description of Issue
On June 20, 2007, instrument air pressure at San Onofre Unit 2 dropped significantly
following the separation of a 3-inch fitting in the system air header located in the
auxiliary building. This caused the feedwater control valves to stop functioning, resulting
in an uncontrolled increase in steam generator water level. Operators manually tripped
the Unit 2 reactor. The loss of instrument air caused containment isolations and a loss
of most power conversion system functions.
The licensee performed a metallurgical analysis of the failed joint and determined that
the cause of the failure was poor workmanship during initial installation. The analysis
concluded that the joint was most likely leaking since initial plant startup because, during
original installation the brazing activity resulted in inadequate solder coverage and the
connection had continued to deteriorate throughout the life of the plant. During a
walkdown of the system in both units, licensee personnel discovered that 32 other large
fittings were leaking at the joint.
A special inspection team reviewed the licensees root cause evaluation and
metallurgical evaluation for this event. During their review, the team noted that there
had been a previous similar air header failure at San Onofre in June 1994. At that time,
both Units 2 and 3 experienced a loss of instrument air following the failure of an
improperly soldered joint. A metallurgical analysis conducted in 1994 concluded that
this joint had also likely been leaking since initial startup from inadequate solder
coverage.
A large amount of industry operating experience has been available that deals with
soldered joint issues. During the original evaluation of related operational experience
reports, done by the licensee in 1992, they failed to properly assess the impact to San
Onofre. Engineers had determined that if failures were going to have occurred because
of inadequate fit-up and/or solder penetration, the failures would have occurred within a
relatively short time frame. Therefore, they assumed that related industry experience
was not applicable to San Onofre. The team also determined that the licensee failed to
adequately reassess their position when they experienced an air line joint failure in
1994, and as a result, failed to take effective corrective actions following that failure.
B.
Statement of the Performance Deficiency
The licensee failed to take effective corrective actions in response to the failure of an
improperly made soldered joint in the instrument air header affecting both units at San
A3-2
Attachment 3
Onofre in June 1994. Specifically, contrary to Section 6.2.3 of
Procedure SO-123-I-1.42, Maintenance Division Experience Report, Revision 0, the
licensee failed to implement corrective actions to prevent recurrence for an equipment
failure with the potential to cause a significant plant transient, and failed to appropriately
consider previous industry and plant experience similar to the event. Additionally,
licensee personnel failed to properly evaluate and take corrective actions based on
industry operating experience through 2006 involving improperly made soldered joints in
instrument air systems. As a result, an additional failure of an improperly made
instrument air header joint occurred at San Onofre on June 20, 2007, resulting in a
complete loss of instrument air to Unit 2.
C.
Significance Determination Basis
1.
Phase 1 screening logic, results and assumptions
In accordance with NRC Inspection Manual Chapter 0612, Appendix B, "Issue
Screening," the team determined that this finding represented a licensee
performance deficiency. The team then determined that the issue was more
than minor because the finding was associated with the equipment performance
attribute and affected the initiating events cornerstone objective to limit the
likelihood of those events that upset plant stability and challenge critical safety
functions during shutdown as well as power operations.
The team evaluated this finding using the, "SDP Phase 1 Screening Worksheet
for the Initiating Events, Mitigating Systems, and Barriers Cornerstones,"
provided in Manual Chapter 0609, Appendix A, "Significance Determination of
Reactor Inspection Findings for At-Power Situations." A Phase 2 estimation was
required because the associated performance deficiency represented an
increase in both the likelihood of a reactor trip and the probability that the power
conversion system would be unavailable.
2.
Phase 2 Risk Estimation
In accordance with Manual Chapter 0609, Appendix A, Attachment 1, User
Guidance for Significance Determination of Reactor Inspection Findings for At-
Power Situations, the team evaluated the subject findings using the Risk-
Informed Inspection Notebook for San Onofre Nuclear Generating Station
(SONGS) Units 2 and 3, Revision 2.1. The dominant affected accident
sequences are provided in Table 1. The team assumed that the exposure period
was greater than 30 days, that the performance deficiency increased the
likelihood that a complete loss of instrument air would occur, and that there was
no affect on mitigating systems other than that modeled in the risk-informed
notebook.
A3-3
Attachment 3
TABLE 1
Increased Likelihood of a Complete Loss of Instrument Air
Phase 2 Sequences
Initiating Event
Sequence
Mitigating Functions
Results
Loss of Instrument
Air
1
LOIA-AFW
7
2
LOIA-RCPTRIP-HPR
9
3
LOIA-RCPTRIP-CNT
9
4
LOIA-RCPTRIP-EIHP
9
Using the counting rule worksheet, the result from this estimation indicated that
this finding was of very low safety significance (GREEN).
A senior reactor analyst reviewed the Phase 2 estimation and determined that
the risk-informed notebook and the licensees PRA had a common error that
significantly underestimated the risk of this deficiency. The loss of instrument air
initiating event frequency had been established as 6.4 x 10-5/year by assuming
that a loss of all active instrument air system components as well as a loss of the
backup nitrogen system was required to realize a complete loss of instrument
air. However, there are many system breaches and other passive component
failures that would prevent the backup nitrogen system from performing its
function.
Therefore, the analyst determined that the finding should be evaluated using the
Phase 3 process.
3.
Phase 3 Analysis
The analyst quantified the change in risk of the subject performance deficiency
as indicated in the paragraphs below. The change in internal event risk was
estimated as 4.7 x 10-7 over an entire assessment period. The risk related to
seismic events changed by 4.1 x 10-7 and that related to internal fires by
9.5 x 10-9. This resulted in a total change in CDF of 8.9 x 10-7. Therefore, the
analyst determined that the subject finding was of very low safety significance
(Green).
Internal Initiating Events:
The following techniques were used in this evaluation.
a.
The analyst quantified the internal risk using the Standardized Plant
Analysis Risk (SPAR) Model for San Onofre 2 & 3, Revision 3.21, created
in October 2005. The analyst modeled a loss of instrument air by
assuming that the affect was equivalent to a transient with a complete
loss of the power conversion system with the possibility of a recovery of
A3-4
Attachment 3
condensate by bypassing the feedwater isolation and depressurizing the
steam generators. The likelihood of a loss of instrument air was then
increased to 6.8 x 10-2/year as a result of the performance deficiency.
The resulting quantification indicated an increase over the baseline core
damage frequency of 4.7 x 10-7 over a 365-day exposure.
b.
The licensee developed a model for analyzing the internal risk associated
with the event using the current version of their PRA model. The licensee
revised the model by setting the loss of instrument air initiating event
frequency to 0.5 indicating that operators could have recovered the
condition prior to a reactor trip. Additionally, the licensee estimated that
50% of the transients initiated by an instrument air system breach would
be recovered by operators prior to core damage. No other initiators were
considered to be applicable to this condition.
Licensee analysts changed their model to show that the controlled
bleedoff valves remained open. As documented in Licensee Event
Report 50-361/2007001, the licensees conditional core damage
probability (CCDP) for this event was 3.3 x 10-6.
At the analysts request, the licensee provided a CCDP for a generic loss
of instrument air. The value provided by their model was 4.85 x 10-6.
The analyst converted the licensees CCDP to a change in the core
damage frequency by multiplying it with the calculated initiating event
frequency. The result was 3.29 x 10-7 over the 1-year assessment
period. This tends to corroborate the analysts value and suggests that
the initiating event frequency is the primary difference between the two
values.
In performing the Phase 3 evaluation, the following influential assumptions were
made by the analyst:
a.
The failure of a large fitting in the instrument air system at the San
Onofre site would cause a reactor trip on the subject unit at least 1/2 the
time. This was based on the 2 historical failures in the units.
b.
This condition existed for many years for both units and should be
evaluated over the most recent 1-year assessment period.
c.
The baseline failure rate of the instrument air system should have been
the boolean combination of the system components including the backup
instrument air system. Given intact system piping, this was calculated by
the licensee to be 6.4 x 10-5/year.
d.
The instrument air system had been functional for 22.09 reactor-years.
This value is the number of years that either or both reactors were critical.
Such an assumption was used because the system is common to both
units.
A3-5
Attachment 3
e.
Because the condition causing failures in the instrument air headers was,
in part, an aging issue and because there were 32 additional leaking
fittings identified in the system, the analyst assumed that an additional
failure was eminent prior to repair (eg: 3 failures were assumed to have
occurred).
f.
Upon loss of instrument air, the condensate system is potentially still
available given operators depressurize the reactor coolant system and
manually realign the condensate system to bypass the feedwater
regulating valves.
The following calculations were performed during this analysis:
a.
The analyst calculated the revised likelihood of a loss of instrument air.
The result of 1.36 x 10-1 was calculated based on the 2 historical
breaches of the system, an additional postulated breach to account for
the aging affect, and a service time of 22.09 reactor-years.
b.
As stated above, historically San Onofre has had 2 events that involved
breaches of an instrument air header. On one of these occasions,
operators were able to identify and limit the leak to prevent a reactor
transient. Therefore, the initiating event likelihood was reduced by 50%
to 6.8 x 10-2.
c.
The analyst estimated the nonrecovery probability for the operators
depressurizing the reactor coolant system and feeding the steam
generators using the condensate system. Three components went into
this analysis: 1) the human error probability calculated using the SPAR-H
method; 2) the probability that the atmospheric dump system failed; and
3) the probability that the condensate system failed mechanically. The
last two probabilities were calculated by solving appropriate portions of
the SPAR fault trees. The overall nonrecovery probability was calculated
to be 9.3 x 10-2.
d.
The analyst used the SPAR model to quantify the internal change in risk
for a loss of instrument air by modeling a loss of condensate, bypass
capability, and main feedwater. The analyst set all initiators to the house
event, FALSE, with the exception of transients. The transient initiator
was used as a surrogate initiator for the loss of instrument air, and the
initiating event frequency was set to the calculated frequency above.
The analyst also provided for recovery of the condensate system by
adding a basic event to the COND fault tree. This basic event COND-
RECOVERY was added under the AND gate, COND-SYS-4. This gate
then indicated that the failure of both feedwater pathways as well as
nonrecovery of the condensate system via the alternate pathway were
required to fail the condensate function.
A3-6
Attachment 3
The changes to basic events used for this model are shown on Table 2.
TABLE 2
Changes to SPAR Model Basic Events
Basic Event
Initial Value
Adjusted Value
IE-TRANS
7.0 x 10-1
6.8 x 10-2
All Other Initiators
Nominal
FALSE
MFW-AOV-CF-SGS
2.7 x 10-5
TRUE
MFW-AOV-OC-4048
7.2 x 10-6
TRUE
MFW-AOV-OC-4052
7.2 x 10-6
TRUE
MFW-SYS-AVAILABLE
0.8
1.0
MFW-SYS-UNAVAIL
0.2
FALSE
MSS-TBV-CF-TBVS
2.6 x 10-6
TRUE
COND-RECOVERY
N/A
9.3 x 10-2
The model was then quantified. The case core damage frequency (CDF)
was computed to be 4.7 x 10-7/year and the baseline CDF, using the
baseline initiating event frequency of 6.4 x 10-5/year, was computed to be
4.1 x 10-10/year. The change in CDF (CDF) was then calculated by
subtracting the baseline CDF from the case CDF. This resulted in a
CDF for the increased likelihood for a loss of instrument air of 4.7 x 10-7
over a 365-day exposure. The dominant sequences from the SPAR and
licensee models are documented in Table 3.
Table 3
Phase 3 Dominant Accident Sequences
Model
Initiating Event
Sequence
Contribution
SPAR 3.21
Loss of Instrument Air
AFW,
Condensate
4.5 x 10-7
Licensee's
Revised
Loss of Instrument Air
AFW, Failure to
Depressurize
3.1 x 10-7
A3-7
Attachment 3
External Initiating Events:
The analyst used the following methods for determining the change in risk from
external events. The change in risk from an increase in the frequency of a loss
of instrument air was estimated to be 4.2 x 10-7 for a 365-day period. The
methods used are documented below:
a.
Fire
The analyst used the San Onofre IPEEE to estimate the change in risk
resulting from internal fire. The only fire areas where risk could be
increased by the subject improperly soldered fittings would be those
containing instrument air header piping. As the limiting area, the analyst
reviewed the licensees evaluation of Fire Area 2-TB-148. The fire
ignition frequency for this area was 4.5 x 10-2/yr. The analyst assumed
that only 0.1 of the fires would grow to a size that could impact the
instrument air system (severity factor) and that about 50 percent of the
fires would cause a weakening of the improperly soldered fitting joints
without causing baseline failure of the system. Using a conditional core
damage probability of 4.2 x 10-6, the change in core damage frequency
from the subject performance deficiency related to a turbine building fire
was estimated as 9.5 x 10-9 over the 365-day exposure period.
b.
Seismic
The analyst determined that, for the subject performance deficiency to
affect the core damage frequency, a seismic event must result in a failure
of an instrument air system header fitting without otherwise affecting
instrument air system components.
To estimate the baseline seismic failure of the system, the analyst used
the seismic fragility of the air-operated valves which were the least
durable components in the system as designed. The analyst evaluated
the subject performance deficiency by determining each of the following
parameters for any seismic event producing a given range of median
acceleration "a" [SE(a)]:
1.
The frequency of the seismic event SE(a) (SE(a)) ;
2.
The probability that a system header fitting fails (PHeader-SE(a));
3.
The probability that an air-operated valve fails (P System-SE(a));
4.
The conditional change in CDP (CCDPSE(a))
The CDF for the acceleration range in question (CDFSE(a)) can then be
quantified as follows:
CDFSE(a) = SE(a) * PHeader-SE(a) * (1 - PSystem-SE(a)) * CCDPSE(a)
A3-8
Attachment 3
Given that each range a was selected by the analyst specifically to be
independent of all other ranges, the total increase in risk, CDF, can be
quantified by summing the CDFSE(a) for each range evaluated as follows:
6
CDF = 3 CDFSE(a)
a=.03
over the range of SE(a).
1.
Frequency of the Seismic Event
NRC research data indicated that seismic events of 0.05g or less have
little to no impact on internal plant equipment. The analyst assumed
that seismic events less than 0.03g do not directly affect the plant.
The analyst assumed that seismic events greater than 6.0g lead to
core damage. The analyst therefore examined seismic events in the
range of 0.03g to 6.0g.
The analyst divided that range of seismic events into segments (called
"bins" hereafter); specifically, seismic events from 0.03g to 0.1g were
binned by hundredths, seismic events from 0.1g to 1.0g were binned
by tenths, and seismic events from 1.0g to 6.0g were binned by ones.
In order to determine the frequency of a seismic event for a specific
range of ground motion (g values), the analyst used a plot provided by
the licensee and obtained values for the frequency of the seismic
event that generates a level of ground motion (in peak ground
acceleration) that exceeds the lower value in each of the bins. The
analyst then calculated the difference in these frequency of
exceedance values to obtain the frequency of seismic events for the
binned seismic event ranges.
For example, according to the San Onofre curves, the frequency of
exceedance for a 0.6g seismic event is estimated at 3.9 x 10-2/yr and a
0.7g seismic event at 3.5 x 10-2/yr. The frequency of seismic events
with median acceleration in the range of 0.6g to 0.7g [SE(0.6-0.7)]
equals the difference, or 4.0 x 10-3/yr.
2.
Probability of a Header Fitting Failure
Given that the historical header failures were the result of insufficient
solder coverage and were caused by slow degradation via air leaks,
the analyst assumed that a moderately large earthquake could result
in the failure of a leaking header fitting. Therefore, the analyst used a
median seismic fragility of a long, brittle component as an estimate of
the fragility of the 32 degraded instrument air header fittings. The
seismic fragility selected was 0.3g.
A3-9
Attachment 3
The analyst obtained data on switchyard components from the Risk
Assessment of Operating Events Handbook; Volume 2, External
Events, Revision 4, which referenced generic fragility values listed in:
<
NUREG/CR-6544, Methodology for Analyzing Precursors to
Earthquake-Initiated and Fire-Initiated Accident Sequences,
April 1998; and
<
NUREG/CR-4550, Vols 3 and 4 part 3, Analysis of Core
Damage Frequency: Surry / Peach Bottom, 1986
The references describe the mean failure probability for various
equipment using the following equation:
Pfail(a) = [ ln(a/am) / (r
2 + u
2)1/2]
Where is the standard normal cumulative distribution
function and
a =
median acceleration level of the seismic event;
am=
median of the component fragility;
r=
logarithmic standard deviation representing random
uncertainty;
u=
logarithmic standard deviation representing systematic
or modeling uncertainty.
In order to calculate the probability that a degraded fitting would fail
given a seismic event, the analyst used the following generic seismic
fragilities:
am = 0.3g
r = 0.30
u = 0.45
Using the above normal cumulative distribution function equation the
analyst determined the conditional probability of failure given a seismic
event. For each of the bins the calculation was performed substituting
for the variable "a" (median acceleration level) the acceleration levels
obtained from the bins described above. The following table shows
the results of the calculation for various acceleration levels.
Median Acceleration Level/Probability of Failure
0.03g
3.6 x 10-5
0.3g
6.1 x 10-1
1.0g
1.0
0.07g
5.2 x 10-3
0.7g
9.5 x 10-1
A3-10
Attachment 3
3.
Probability that Air-Operated Valves Fail
In order to calculate the probability that the instrument air system
would fail during a given seismic event for reasons other than
improperly soldered fittings, the analyst used the following generic
seismic fragilities for air-operated valves:
am = 3.8g
r = 0.35
u = 0.50
Using the above standard normal cumulative distribution function
equation, the analyst determined the conditional probability that the
instrument air system would fail from failure of system valves given a
seismic event for each of the bins. The calculation was performed
substituting for the variable "a" (median acceleration level) the
acceleration levels obtained from the bins described above. The
following table shows the results of the calculation for various
acceleration levels.
Median Acceleration Level/
Probability of Air-Operated Valve Failure
0.03g
7.9 x 10-15
0.3g
4.7 x 10-5
1.0g
6.4 x 10-2
0.07g
6.3 x 10-11
0.7g
3.9 x 10-3
4.
Conditional Change in Core Damage Probability
The analyst evaluated the spectrum of seismic initiators to determine
the resultant impact on the reliability and availability of mitigating
systems affecting the subject performance deficiency.
The analyst used the SPAR model, to perform the Phase 3 evaluation.
The analyst started with the model discussed above used to quantify
the change in risk from internal events. However, the analyst set the
initiating event frequency for a transient to 1.0 and all other initiating
event probabilities in the SPAR model to zero. Because of the very
narrow time windows discussed for condensate recovery, and the
added burdens on operators both emotionally and physically following
a seismic event, the analyst set the nonrecovery probability for the
condensate system to 1.0. The SPAR model showed the resultant
core damage probability as 1.02 x 10-4, which represented the value
used in the above equation.
A3-11
Attachment 3
The SPAR Model was then requantified indicating no loss of
instrument air. The CCDP for this baseline condition was 4.35 x 10-7.
Therefore, the change in core damage probability is:
CCDPSE(a) = 1.02 x 10-4 - 4.35 x 10-7 = 1.02 x 10-4
Phase 3 Seismic Results
Given the assumptions previously discussed, the total increase in core
damage frequency was estimated to be about 4.1 x 10-7 for seismic events
ranging from 0.03g to 6.0g.
c.
Winds, Floods, and Other External Events
The analyst reviewed the IPEEE and determined that no other credible
scenarios initiated by high winds, floods, fire, and other external events could
initiate a failure of the degraded instrument air header fittings. Therefore, the
analyst concluded that external events other than internal fires and seismic
events were not significant contributors to risk for this finding.
Risk Contribution from Large Early Release Frequency (LERF):
Using IMC 0609 Appendix H, the SRA determined that this was a Type A finding for a
large dry containment. For PWR plants with large dry containments, only findings
related to accident categories ISLOCA and SGTR have the potential to impact LERF.
In addition, an important insight from the IPE program and other PRAs is that the
conditional probability of early containment failure is less than 0.1 for core damage
scenarios that leave the RCS at high pressure (>250 psi) at the time of reactor vessel
breach. Since this finding is not related to ISLOCA or SGTR, and the core damage
scenarios for this finding leave the RCS at high pressure, the analyst concluded that
LERF is not a significant contributor to the risk associated with this finding.