IR 05000361/2003011

IR 05000361-03-011 & 05000362-03-011, on 10/06/03 to 10/10/03, San Onofre Nuclear Generating Station

ML033110291
Person / Time
Site:	San Onofre
Issue date:	11/07/2003
From:	Kennedy K Division of Nuclear Materials Safety IV
To:	Ray H Southern California Edison Co
References
IR-03-011
Download: ML033110291 (19)

Text

ber 7, 2003

SUBJECT:

SAN ONOFRE NUCLEAR GENERATING STATION - NRC SUPPLEMENTAL INSPECTION REPORT 05000361/2003011 AND 05000362/2003011

Dear Mr. Ray:

On October 10, 2003, the NRC completed a supplemental inspection at your San Onofre Nuclear Generating Station, Units 2 and 3, facility. The enclosed report documents the inspection findings which were discussed on October 10, 2003, with members of your staff.

As required by the NRC Reactor Oversight Process Action Matrix, this supplemental inspection was performed in accordance with Inspection Procedure 95001. The purpose of the inspection was to examine the causes for and actions taken related to the performance indicator for unplanned scrams per 7000 critical hours crossing the threshold from Green (very low risk significance) to White (low to moderate risk significance) for Unit 2. This supplemental inspection was conducted to provide assurance that the root causes and contributing causes of the events resulting in the White performance indicator are understood, to independently assess the extent of condition, and to provide assurance that the corrective actions for risk significant performance issues are sufficient to address the root causes and contributing causes and to prevent recurrence. The inspection consisted of selected examination of representative records and interviews with personnel.

The NRC concluded that your staff performed thorough evaluations for each of the four Unit 2 reactor trips and performed a thorough and broad-based self-assessment to identify any performance and process issues that should be addressed as a result of the performance indicator crossing the threshold from Green to White.

In accordance with 10 CFR 2.790 of the NRCs "Rules of Practice," a copy of this letter, its enclosure, and your response will be made available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Southern California Edison Co. -2-Should you have any questions concerning this inspection, we will be pleased to discuss them with you.

Sincerely,

/RA/

Kriss M. Kennedy, Chief Project Branch C Division of Reactor Projects Dockets: 50-361 50-362 Licenses: NPF-10 NPF-15 Enclosure:

NRC Inspection Report 05000361/2003011; 05000362/2003011 w/attachment: Supplemental Information cc w/enclosure:

Chairman, Board of Supervisors County of San Diego 1600 Pacific Highway, Room 335 San Diego, CA 92101 Gary L. Nolff Power Projects/Contracts Manager Riverside Public Utilities 2911 Adams Street Riverside, CA 92504 Eileen M. Teichert, Esq.

Supervising Deputy City Attorney City of Riverside 3900 Main Street Riverside, CA 92522 Joseph J. Wambold, Vice President Southern California Edison Company San Onofre Nuclear Generating Station P.O. Box 128 San Clemente, CA 92674-0128

Southern California Edison Co. -3-David Spath, Chief Division of Drinking Water and Environmental Management California Department of Health Services P.O. Box 942732 Sacramento, CA 94234-7320 Michael R. Olson San Onofre Liaison San Diego Gas & Electric Company P.O. Box 1831 San Diego, CA 92112-4150 Ed Bailey, Radiation Control Program Director Radiologic Health Branch California Department of Health Services P.O. Box 942732 (MS 178)

Sacramento, CA 94234-7320 Mayor City of San Clemente 100 Avenida Presidio San Clemente, CA 92672 James D. Boyd, Commissioner California Energy Commission 1516 Ninth Street (MS 34)

Sacramento, CA 95814 Douglas K. Porter, Esq.

Southern California Edison Company 2244 Walnut Grove Avenue Rosemead, CA 91770 Dwight E. Nunn, Vice President Southern California Edison Company San Onofre Nuclear Generating Station P.O. Box 128 San Clemente, CA 92674-0128 D

SUMMARY OF FINDINGS

IR05000361/2003011; IR05000362/2003011; 10/06/2003-10/10/2003; San Onofre Nuclear

Generating Station; Units 2&3. Supplemental inspection.

Cornerstone: Initiating Events

The U.S. Nuclear Regulatory Commission performed this supplemental inspection to assess the licensees evaluations associated with three unplanned reactor trips of Unit 2 during calender year 2002 and one unplanned reactor trip of Unit 2 during calendar year 2003. The cumulative effect of these trips was that the Performance Indicator for unplanned scrams per 7000 critical hours crossed the threshold from Green (very low risk significance) to White (low to moderate risk significance) for the first quarter of calendar year 2003. The licensee performed individual root cause evaluations for three of the four trips. The root cause evaluation for the trip on November 2, 2002, will be performed following disassembly and inspection of the failed component during the next scheduled outage. Licensee Event Report 05000361/2002007-00 associated with this trip remains open and will be reviewed by the NRC following completion of the licensees root cause evaluation. In addition to the individual trip evaluations, the licensee performed a self assessment evaluation to identify any performance and process issues that led to the White performance indicator. During this supplemental inspection, performed in accordance with Inspection Procedure 95001, the inspector determined that the licensee performed a comprehensive and thorough evaluation in which specific problems were identified, an adequate root cause evaluation was performed, and corrective actions were taken or planned to prevent recurrence.

Report Details 01

INSPECTION SCOPE

The U.S. Nuclear Regulatory Commission (NRC) performed this supplemental inspection to assess the licensees evaluation associated with a Performance Indicator (PI) that crossed the threshold from Green to White. The PI was for unplanned scrams per 7000 critical hours for Unit 2 and was related to the initiating event cornerstone in the reactor safety strategic performance area. The PI was White for the first quarter of 2003.

San Onofre Nuclear Generating Station (SONGS) Unit 2 experienced three unplanned reactor trips in 2002 and one unplanned reactor trip in 2003. The cumulative effect of these trips was to cause the PI to cross the threshold from Green to White. The inspector reviewed the licensees actions associated with these four events and conducted interviews of licensee personnel.

The licensee performed a self-assessment evaluation to identify performance and process issues led to the White PI. The scope of the licensees examination was significantly broader than the scope of this supplemental inspection. The inspector reviewed this self-assessment.

EVALUATION OF INSPECTION REQUIREMENTS 02.01 Problem Identification a. Method of identification The PI crossed the threshold from Green to White during the first quarter of 2003 as a result of an unplanned trip on February 3, 2003. Prior plant trips had occurred on June 30, November 2, and November 4, 2002. A brief description of each trip from the associated Licensee Event Report (LER) and Action Request (AR) is given below. For each trip the event was self-revealing and there were no indications of the impending failure prior to the event.

On February 4, 2003, the licensee initiated AR 030200408 to perform a self-assessment evaluation in response to the negative trend in plant performance indicated in part by the unplanned reactor trips and the resulting White PI. The root causes and corrective actions developed in this AR are discussed in Sections 02.02 and 02.03, respectively.

.1 June 30, 2002: Inadequate Procedure Results in Incorrect Adjustment of the Steam

Bypass Control System Causing a Reactor Trip (LER 05000361/2002003-00, AR 020602197)

Description.

On June 30, 2002, plant operators were performing startup testing with the reactor at about 18 percent thermal power. In response to a small change in steam flow, the steam bypass control system (SBCS) inappropriately quick-opened all four steam bypass valves. This caused steam pressure to decrease to the low steam generator pressure trip setpoint, resulting in a reactor trip.

Cause.

The cause of the improper operation of the SBCS was an incorrect calibration of the SBCS control circuitry resulting from an inadequate maintenance procedure. The licensee identified a lack of guidance for the revision and review of maintenance calibration procedures as a root cause for the inadequate procedure.

Corrective Action. The licensee developed guidance for the future review of maintenance calibration procedures, provided training to maintenance and engineering personnel, and confirmed the adequacy of the existing calibration procedures. The licensee also revised the calibration procedure and properly calibrated the SBCS in both Units.

.2 November 2, 2002: Main Feedwater (FW) Controller Fault Causes Loss of Main

Feedwater and Reactor Trip on Low Steam Generator Level (LER 05000361/2002006- 00, AR 021100079)

Description.

On November 2, 2002, a FW controller in Unit 2 failed, causing the associated regulating valve (2FV1111) to close, resulting in a reactor trip from 100 percent power on low steam generator level.

Cause.

The licensee identified the root cause of the trip as an age-related failure of an operational amplifier in the FW control system master controller proportional plus integral control card.

Corrective Action. The licensee replaced the control cards for both FW regulating valves in Unit 2. The cards on Unit 3 were not replaced since they were significantly newer than those on Unit 2. The licensee also developed and began implementing a plan to replace the existing FWCS in both units with a digital, single-failure tolerant system to prevent recurrence of this type of event.

.3 November 4, 2002: Pressurizer Spray Valve Malfunction Results in a Reactor Trip

(LER 05000361/2002007-00, AR 021100192)

Description.

On November 4, 2002, one of two pressurizer spray valves began operating erratically and became stuck open while the reactor was operating at about 18 percent thermal power. Attempts to close the valve were unsuccessful, and the reactor was manually tripped when reactor coolant system pressure decreased below the minimum pressure allowed by the plant Technical Specifications.

Cause.

Initial tests indicated the cause of the valve behavior may have been internal binding. The valve will be disassembled and the exact cause determined during the next refueling outage, scheduled to begin in February 2004.

Corrective Action. Licensee staff isolated the malfunctioning spray valve by closing the inlet and outlet manual block valves. The second pressurizer spray valve was tested satisfactorily on November 5, 2002, and will be in use for the remainder of the operating cycle. The licensee will implement corrective actions after completion of the cause evaluation following valve disassembly.

.4 February 1, 2003: Incorrect Test Connection Results in Automatic Reactor Trip

(LER 05000361/2003001-00, AR 030200027)

Description.

On February 1, 2003, a technician performing outage work on Unit 3 attempted to establish a control power reference connection from the Unit 2 relay protection cabinet. The technician incorrectly connected the test equipment power source to the field suppression relay for the Unit 2 Main Generator, resulting in a generator/turbine trip and subsequent reactor trip on Unit 2.

Cause.

The licensee identified the root cause of the trip to be inadequate procedural guidance relative to the complexity and possible consequences of the task. Additional causes contributing to the trip were personnel errors in procedure use and job performance, inadequate staffing and resource management, and inadequate planning of maintenance activities.

Corrective Action. Immediate corrective actions implemented by the licensee for this trip were to install a physical barrier at the relay protection cabinets to prevent access, require supervisory approval for any connections between units, and hold a site-wide brief on reducing human performance errors. Long-term corrective actions for the root cause were to: develop a new procedure for main turbine relaying and metering preventive maintenance activities, install a separate patch panel to serve as a source of control power, develop a clear definition of high consequence activity and review all routine maintenance orders and preventive maintenance items for such activities, and revise the maintenance governing procedure to require a higher level of detail for work plans involving high risk activities. Additional corrective actions included: enhancing the work authorization request checklist by adding a line to verify whether connections to the other unit are required; revising the maintenance governing procedure to require opposite unit component identification to be listed on maintenance work orders when applicable; implementing a plan to qualify additional test technicians; conducting training on the identification of high consequence activities and error likely situations; developing a clear definition of critical component and reviewing equipment classifications against the same; and revising the Maintenance Activity Component Checking and Independent Verification Flowchart to address high consequence and error likely situations.

b. Duration of issue and prior opportunities for identification The PI returned to the Green band in the second quarter of 2003. In their self-assessment evaluation, the licensee identified several opportunities for prior identification of performance issues related to equipment reliability and human performance.

Although the self-assessment evaluation considered trips and plant transients dating back to January 2000, the licensee identified a trip on Unit 3 in February 2001 as the first explicit indication of significant problems in equipment reliability. In January 2002, the licensee performed an analysis to compare the equipment monitoring and maintenance programs at SONGS with accepted industry practices, which also identified weaknesses in equipment oversight. In response to these weaknesses, the licensee formed a site team to improve equipment reliability. This team has since assumed responsibility for implementing the corrective actions relating to equipment reliability from the licensees self-assessment evaluation for the White PI.

A self-assessment in March 2001 identified weaknesses in the use of human performance error prevention tools by workers and supervisors. Subsequent assessments in January and April 2002 indicated continued weaknesses in these areas.

In November 2002, the licensee initiated AR 021100235 to develop an integrated site solution to improve human performance. Since this AR was still open when the PI became White, corrective actions from the AR were incorporated with the corrective action for the White PI self-assessment.

c. Risk consequences and compliance issues The licensee performed a quantitative probabilistic risk assessment for three of the four plant trips to determine the risk associated with each of these trips. The trip on November 2, 2002, was determined to have the highest conditional core damage probability of 5 x 10 -6 and a maximum conditional large early release probability of 2.2 x10 -7. The licensee performed a qualitative risk analysis based on the Updated Final Safety Analysis Report for the June 30, 2002, trip and determined it was also of very low safety significance.

Two compliance issues associated with the above trips are addressed in previous NRC inspection reports. A Green finding for an inadequate procedure relating to the trip on June 30, 2002, is discussed in NRC Inspection Report 05000361, 362/2002005 dated October 17, 2002. A second Green finding for an inadequate procedure associated with the February 1, 2003, trip is contained in NRC Inspection Report 05000361, 362/2003002 dated April 28, 2003. The inspector identified no additional findings of significance during this inspection.

02.02 Root Cause and Extent of Condition Evaluation a. Evaluation of method used to identify root causes and contributing causes The inspector reviewed the licensees root cause evaluation for the White PI and determined that the licensee staff appropriately followed their procedural guidance for performing root cause evaluations.

The inspector determined that the licensees process was adequate to provide accurate root causes and to evaluate the extent of condition for the White PI.

b. Level of detail of the root cause evaluation The licensees root cause evaluation for the White PI was thorough and identified five repetitive inappropriate actions (RIA) and associated apparent causes and one root cause for the performance issues. The causes identified cover the areas of human performance; procedural adequacy; corrective action effectiveness; and equipment design, monitoring, and maintenance. The RIAs and their associated apparent causes are as follows:

RIA 1: Plant design did not include equipment failure tolerance to avoid power transients. This was caused by an inadequate engineering program procedure which did not completely ingrain the need for power transient avoidance.

RIA 2: Equipment monitoring did not identify or resolve equipment problems prior to plant impact. This was caused by an inadequate program procedure for online equipment monitoring for components important to reliability.

RIA 3: Preventive maintenance did not maintain equipment and components in a condition to prevent plant impact. This was caused by inadequate program management resulting in a lack of commitment to ongoing maintenance.

RIA 4: Inadequate defense in depth in work plans and in job performance. This was caused by a lack of resource management and inadequate knowledge levels, such that neither supervisors nor workers understood defense in depth and error precursors.

RIA 5: Corrective actions associated with plant problems did not prevent repeat events.

This was caused by inadequate management and supervisory job direction leading to insufficient extent of condition and extent of cause evaluations.

The licensee identified an overarching root cause for the above RIAs as inadequate resource management in that managers and supervisors lacked the skills needed to critically evaluate organizational effectiveness.

Based upon a review of the root cause evaluation, action requests associated with the trips, and discussions with plant personnel, the inspector determined that the evaluation to establish the root and contributing causes was adequate and accurate.

c. Consideration of prior occurrences of the problem and knowledge of prior operating experience The licensees evaluation included a review of causes and actions for approximately 20 human performance and 20 equipment reliability issues at other industry sites.

Based on the review of industry data, the licensee concluded that the scope of their root cause evaluation and planned corrective actions were adequate to address the performance issues associated with the White PI.

The inspector found that the licensee appropriately considered operational and industry experience in their root cause evaluation.

d. Consideration of potential common causes and extent of condition of the problem The licensees evaluation considered the potential for common cause and the extent of condition associated with the performance issues. In addition to the previously discussed inappropriate actions and apparent/root causes, the root cause evaluation identified two generic site issues.

The first generic issue identified a need to perform self-critical evaluations in areas other than equipment oversight and human performance. The second generic issue was a failure to recognize important program procedures (e.g., equipment reliability) as Site Programs, which would give these procedures higher management attention and expectations.

The inspector found that the licensee adequately considered and evaluated potential common causes and extent of condition of the performance issues associated with the White PI.

02.03 Corrective Actions a. Appropriateness of corrective actions To address the root cause of the performance issue, the licensee participated in industry evaluations and considered industry best practices in the areas of equipment reliability and human performance. Additionally, the licensee assigned the Equipment Reliability Improvement Project (ERIP) Team to review and improve reliability standards for equipment and the Human Performance Improvement Team (HPIT) to improve overall human performance. Both the ERIP and HPIT procedures were established as site programs to elevate the standards and management expectations for the teams.

The RIAs identified in Section 02.02 are repeated below with the associated corrective actions initiated by the licensee for each.

RIA 1: Plant design did not include equipment failure tolerance to avoid power transients. The licensee implemented a Three Cycle Plan to upgrade and replace fault intolerant equipment. Also, the ERIP Team was assigned to identify equipment not previously designated as safety-related, the failure of which could cause a plant trip or transient. Once identified, this equipment was classified as important to reliability (ITR)and given increased maintenance focus per RIAs 2 and 3, below.

RIA 2: Equipment monitoring did not identify or resolve equipment problems prior to plant impact. The ERIP Team established equipment classifications in the site routine/corrective maintenance database program to identify equipment that was important to reliable operation of the plant. The ERIP Team was also assigned to develop a long-term strategy for maintenance and use of ITR classifications and to establish an equipment monitoring program for critical ITR equipment.

RIA 3: Preventive maintenance did not maintain equipment and components in a condition to prevent plant impact. The ERIP Team was assigned to reevaluate and update the routine/corrective maintenance program and to establish a preventive maintenance program for critical ITR equipment.

RIA 4: Inadequate defense in depth in work plans and in job performance. The HPIT initiated site-wide human performance training and human performance leadership training for managers and supervisors. The purpose of the training was to provide management and the work force with the knowledge and skills to plan and perform error-free work.

RIA 5: Corrective actions associated with plant problems did not prevent repeat events.

The licensee established a Corrective Action Review Board (CARB) to challenge the breadth, adequacy, and timeliness of all cause evaluations. Additionally, the licensee revised the corrective action program governing procedure to clarify requirements for assigning cause evaluators and developed plans for a qualification standard to train and qualify cause evaluators.

As corrective action for the generic issues discussed in Section 02.02.d, the licensee implemented plans to expand the self-assessment program and to correlate their current list of site programs with those recommended by an accepted industry model.

Based upon a review of the root cause evaluation, discussions with plant personnel, review of plant procedures, direct observation of human performance training, and a meeting of the CARB, the inspector determined that the corrective actions were appropriate.

b. Prioritization of corrective actions Immediate corrective actions by the licensee involved industry benchmarking to establish a baseline for areas requiring improvement in human performance and equipment reliability. Based on these results, the licensee initiated a phased approach to implementing the above listed long-term corrective actions to improve overall performance. The CARB was initiated to improve the corrective action process, and the ERIP Team and the HPIT established implementation schedules with appropriate consideration of risk significance.

The inspector reviewed the prioritization of corrective actions and determined that the licensee properly prioritized these actions.

c. Establishment of schedule for implementing and completing corrective actions The inspector found that the licensee established appropriate schedules for implementing and completing corrective actions. Actions completed to date were completed as scheduled. The licensee appropriately considered risk in the scheduling of corrective actions. Several corrective actions involve permanent plant modifications and are scheduled for upcoming Units 2 and 3 refueling outages.

d. Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence The licensees corrective actions are intended to prevent a reactor trip resulting from equipment failure or from a human performance error. Additional corrective actions are intended to improve existing maintenance procedures and practices and the corrective action process as a whole. The licensee plans to measure the effectiveness of their corrective actions by conducting self-assessments and performing additional industry benchmarking. The licensee also revised the Self-Assessment Steering Committee charter to be more aggressive in assigning and reviewing division and site self-assessments.

The inspector considered the measures to determine the effectiveness of corrective actions to be adequate.

MANAGEMENT MEETINGS

Exit Meeting Summary

The results of the supplemental inspection were presented to Mr. J. Wambold, Vice President, Nuclear Generation, and other members of licensee management and staff on October 10, 2003. The inspector confirmed that proprietary information was not provided or examined during the inspection.

OTHER Event Followup

(Closed) LER 05000361/2002006-00. Main FW Controller Fault Causes Loss of Main FW and Reactor Trip on Low Steam Generator Level On November 2, 2002, a Main FW controller in Unit 2 failed, causing the associated regulating valve (2FV1111) to close, resulting in a reactor trip on low steam generator level. The licensee replaced the controller cards for both main FW trains and developed and implemented a plan to replace the FW control system with a digital single-failure tolerant system. The LER was reviewed by the inspector and no findings of significance were identified. The licensee documented the failed equipment in AR 021100079. This LER is closed.

(Closed) LER 05000361/2003001-00. Incorrect Test Connection Results in Automatic Reactor Trip On February 1, 2003, a technician performing an equipment test on Unit 3 made an incorrect terminal connection, which actuated the Unit 2 Field Suppression Relay, resulting in a generator/turbine trip and subsequent reactor trip on Unit 2. As corrective action, the licensee implemented procedural changes and equipment modifications. A Green finding for an inadequate procedure associated with this trip is documented in NRC Inspection Report 05000361, 362/2003002 dated April 28, 2003. No new findings were identified in the inspectors review. The licensee entered the finding in their corrective action program as AR 030200027. This LER is closed.

3. (Discussed) LER 05000361/2002007-00. Pressurizer Spray Valve Malfunction Results in

a Reactor Trip On November 4, 2002, one of two pressurizer spray valves began operating erratically and became stuck open while the reactor was operating at about 18 percent thermal power. Attempts to close the valve were unsuccessful, and the reactor was manually tripped when reactor coolant system pressure decreased below the minimum pressure allowed by the plant Technical Specifications. Initial tests indicated the cause of the valve behavior may be internal binding. The valve will be disassembled and the exact cause determined during the next scheduled refueling outage. The LER was reviewed by the inspector and no findings of significance were identified. The licensee documented the failed equipment in AR 021100192. This LER remains open pending review of the final cause determination during the next refueling outage.

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

T. Adler, Work Control Manager

R. Clark, Manager, Performance Assessment

R. George, Consulting Engineer

K. Johnson, Manager, Electrical and Instrumentation Control Systems

C. McAndrews, Manager, Nuclear Oversight and Assessment

M. McBrearty, Engineer, Nuclear Regulatory Affairs

J. Osborne, Corrective Action Program Lead

M. Short, Systems Engineering Manager

T. Vogt, Operations Manager

R. Waldo, Station Manager

C. Williams, Supervisor, Regulatory Compliance

NRC Personnel

C. Osterholtz, Senior Resident Inspector

M. Sitek, Resident Inspector

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

NONE

Closed

05000361/2002006-00 LER Main Feedwater Controller Fault Causes Loss of Main Feedwater and Reactor Trip on Low Steam Generator Level

05000361/2003001-00 LER Incorrect Test Connection Results in Automatic Reactor Trip

Discussed

05000361/2002007-00 LER Pressurizer Spray Valve Malfunction Results in a Reactor Trip (Section 02.01)

Attachment