Text

Results of Level I Investigation at Zion Station,970221 Reactivity Mgt Investigation
	ML20140G527
Person / Time
Site:	Zion File:ZionSolutions icon.png
Issue date:	04/18/1997
From:	; COMMONWEALTH EDISON CO.
To:	;
Shared Package
ML20140G524	List: ML20140G519; ML20140G527;
References
	NUDOCS 9705120008
	Download: ML20140G527 (50)
	v • d • e

._

i 2

3, l

RESULTS OF LEVEL I INVESTIGATION AT ZION STATION FEERUARY 21,1997 REACTIVITY MANAGEMENT INVESTIGATION Issued April 18,1997

[0"$000!0500 i

Zion Level 11nvestigation Report Table of Contents i N T R O D U C TI O N..................................................................................

E X E C U TIV E S U M M A RY...................................................................................... 2 1

CHRONOLOGY...................................................................................................3 i

1 1

AN ALYSIS OF CAU SAL FACTO RS................................................................... 9 Int roduetion & Eveni Significance............................................

......................... 9 Analysis..............................................................................................10 Plannine a Briefine...

.10 Communications..

. 11 Egndamental Knowledceffraininc..

. 14 Manacement Oversicht..

.16 Organizational Behaviors.

. 17 Corrective Actions for Previous Events / Industry Experienceffrends.

,.19 SOV/ISEG Oversicht..

.20 s

1 C O N C L U S l O N S..........................................................................................

R E C O M M E N D ATl O N S.................................................................................. 2 3 K EY L E SSO N S TO B E L EAFA A................................................................... 25 NF APPENDICES..................................................................................................27 Appadix A: Investigation Team Chaner & Investigation Team Membership and Investigation Observers Appendix B: Cause and Effect Analysis Appendix C: Common Causal and Contributing Factors of the ISAT and the Level I Investigation Appendix D: AdditionalIssues Appendix E: Zion Reactivity Event Safety Significance ii

Zion LevelIInvestigation Report INTRODUCTION l

This report details the results of the Level I Investigation chartered to provide an independent investigation of actions taken during shutdown of Zion Station Unit I on February 21,1997. Zion j

Nuclear Power Station is owned and operated by Commonwealth Edison Company (Comed). The Station l

consists of two,1085 Mwe, Westinghouse-designed, four loop reactor plants. The purpose of this investigation and report is to share the Investigation Team's understanding of the actions taken during the Unit I shutdown and to assist Zion Station and Comed's Nuclear Operating Division in its efforts to improve safety and operations.

j i

Following the Executive Summary, this report is divided into the following major sections: Chronology (a narrative of the issues principle to the shutdown), Analysis of Causal Factors (including event significance and an analysis of key causal factors), a Conclusion, Recommendations, and Key Lessons Learned (the important experiences to be gained).

Appendices follow with information that the Investigation Team used to support its analysis of the investigation. Also relegated to an appendix is a description of observations that, while not central to cither the Investigation Team's Charter or the reactivity management issues, were noteworthy issues to be shared for Station review and possible corrective action. This list of" Additional Issues" appears in Appendix D. The Investigation Team Charter and Membership are included as Appendix A to this report.

l 1

Zion LevelIhivestigation Report EXECUTIVE

SUMMARY

During a Technical Specification required plant shutdown, a reactor operator attempted to reduce power to the point of adding heat (POAH) by continuously inserting the control rods until the power li. vel specified in the shutdown procedure was reached. When the operator stopped rod insertion, power continued to decrease below the POAH. This rod motion resulted in an undetected mode change to Mode 3 (Hot Shutdown). The operator then withdrew rods approximately 80 steps over the next two minutes in an attempt to correct the power undershoot. Initial rod insertion resulted in taking the reactor significantly suberitical, and then the subsequent extensive rod withdrawal constituted an unrecognized and un-approved attempted reactor startup and mode change.

While the technical safety consequence of the reactivity manipulations that occurred during the shutdown was minimal, as there was no challenge to any safety or design limitations, it is the Investigation Team's determination tiet the failure of personnel to properly operate or oversee reactor controls is a significant safety failing. Behaviors exhibited raise concerns of great importance.

i The Investigation Team identified several causal factors related to this event, including: (1) planning and briefing; (2) communications; (3) a lack of fundamental knowleds applied during the event and training; (4) management oversight; (5) organizational behaviors; (6) corrective actions; and (7) independent oversight from the Site Quality Verification / Independent Safety Engineering Group organization. The investigation team also assessed circumstances surrounding the event in light of the areas for improvement identified by the Independent Safety Assessment Team (ISAT) in February of this year and found great similaritics between the ISAT causal factors and the causes attributed to this event.

The Investigation Team recognizes that corrective actions are currently being developed / implemented for the recently issued ISAT Report. However, as the ISAT report is written at a broad level of detail, the Team was not certain that a comprehensive response to the ISAT would necessarily include correction of some of the specific issues identified in this event. Consequently, while the Investigation Team expects that the corrective actions for the ISAT should bound the corrective actions for this event, several items are presented for consideration as the Station finalizes its plan in response to the ISAT. Key lessons to be learned are included to sharc important learnings from this event.

2

Zion Level 1 Investigation Report CHRONOLOGY This event centers around inappropriate control rod manipulation by a reactor operator during a Technical Specification required shutdown of Unit 1 on February 21,1997. The shutdown was required because of the failure of tae diesel engine driven Containment Spray Pump to start in the required time. This investigation ii focused on the actions taken beginning on February 21 as part of the shutdown and accepts the irability to return the Containment Spray Pump to senice as a "given" initiating condition.

While this chanology begins with testing of the pump on February 19th for completeness, the troubleshooting and repair process that occurred from the 19th through the 21st was not fully explored other than to identify significant evolutions that took place during that period.

On Wednesday, February 19,1997, at 10:20 hours, the IC Containment Spray (CS) pump was aligned for a routine performance test (PT), thus technically making the pump inoperable. This realignment was inadvertently recorded in the Unit Operator's Log as occurring at 10:40 hours. At 12:03 hours, the IC CS Pump, a dicsci engine driven pump, was started for the PT. The engine start time did not meet the 5 second acceptance criteria of the Zion Operability Determination Manual (ZODM). Accordingly, the pump was declared inoperable and the 48 hour5.555556e-4 days 0.0133 hours 7.936508e-5 weeks 1.8264e-5 months Limiting Condition for Operation (LCO) clock was entered, however the LCO clock was assigned an inappropriate expiration time of 10:40 hours on February 21,1997. The importance of this twenty minute time differential (10:20 vice 10:40) will become apparent later. Diagnosis and troubleshooting commenced and the pump was taken out-of-senice (OOS) at 18:30 hours.

On Thursday, February 20,1997, the problem with the CS Pump engine was diagnosed as a dicsci engine fuel system problem. The fuel system was flushed, strainers cleaned, and filters replaced. An engine start and one-hour maintenance run were successfully conducted from 21:35 to 22:35 hours on February 20, 1997.

On Friday, February 21,1997, at 04:40 hours the pump engine did not start for approximately 20 seconds after the start switch was pressed during the PT. From 05:00 to 07:00 hours, an additional engine start and troubleshootag were conducted to determine the problems. The Unit 1 (Ul) Operations Manager, UI Plant Manager, and Plant General Manager became aware of the failed surveillance between 05:00 and 05:30 hours.

At 07:00 hours on February 21,1997, the operations day shift crew completed their turnover. The start-of-shiR briefing did not focus on the potential or planning for a Unit shutdown. Unit I was at 42% power.

At approximately 09:00, the IC CS Pump was taken OOS.

Discussions during the 08:00 plan of the day (POD) meeting reported that Unit I should be prepared for shutdown, an action plan for the IC CS Pump should be prepared, and the Qualified Nuclear Engineers (QNEs) should meet with shin management at 09:30 to prepare for the sbutdown. The shutdown briefing for the operating crew was scheduled for 11:00.

Between 09:00 and 13:00 hours, System Engineering and Electrical Maintenance conducted troubleshooting and repair of the engine start circuitry. The problem was narrowed to a faulty starting circuit relay or faulty solenoids. Two solenoids and a relay were replaced.

Between 08:00 and 11:00 hours the Shin Engineer (SE) and UI Unit Supenisor (US) were involved in numerous activitics, including the following:

The SE and US determined the testing requirements for returning the CS Pump to operable status.

They respended to several telephone calls concerning the CS Pump status.

3

Zion LevelIInvestigation Report There were three turnovers of Nuclear Station Operators (NSOs or reactor operators) on the Unit I secondary panel because of organization unitization limitations (apparently imposed by union understandings) and shift time limitations.

The Auxiliary Steam supply had to be realigned from Unit 1 Main Steam to the Auxiliary Boiler.

Preparations were being made for a Lake Discharge Tank release.

Work was in progress on radiation monitors.

Maintenance was troubleshooting electrical grounds.

Work was in progress on the common penetration pressuri7ation air compressor.

SE/US authorization was needed to remove a reactor building escape hatch security barrier.

Troubleshooting was in progress on switchyard circuit breaker OCB 1718.

e Emergency Core Cooling System chemistry samples were being taken.

The SE coached Unit 2 NSOs on control panel attentiveness.

The US requested the Instrument Mechanics to be present in the control room during the shutdown to assist with expected control rod position indication (RPI) alarms.

As a result of the SE and US involvement in these activitics, they did not have sufficient time to prepare for the shutdown briefing and did not meet with the QNEs at 09:30 to prepare for shutdown (as suggested at the POD). Activities including the turnover of the Secondary NSO prevented the Ul Primary NSO (the NSO at the reactor controls) from being relieved in time for him to resiew the shutdown procedures (GOP-3 and GOP-4) before being directed to perform those procedures.

At 10:40 hours, the CS Pump 48-hour LCO time expired and the Technical Specifications (TS) required the Unit to be in llot Shutdown within 4 hours4.62963e-5 days 0.00111 hours 6.613757e-6 weeks 1.522e-6 months (14:40 hours).

At approximately 11:00 hours, a discussion was conducted regarding the shutdown between the Unit 1 operating crew and plant management (U1 Shift Operations Supenisor (SOS), Ul Operations Manager, U1 Plant Manager and Site Vice President). The Site Vice President expressed his expectation that the crew should focus on a safe Unit shutdown. While present for this discussion, the Shift Engineer may have been distracted by other control room dutics as he did not recall being present for this part of the discussion.

The shutdown briefing was held in the control room from approximately 11:10 to 12:00 hours, while testing and other Unit I activities continued. The briefing was conducted in the control room because sufficient relief NSOs could not be obtained for all the Unit i NSOs, however the Ul Primary NSO was relieved so that he could focus on the briefing. The Site Vice President, the Ul Plant Manager, the Ul Operations Manager, and the U1 SOS were present in the Main Control Room for most of the crew briefing.

Attendecs recall that they discussed the intent to start the Unit shutdown at 12:00 hours, and be shutdown by 14:40 hours. At approximately 15% power, the status of CS Pump repairs and testing would be assessed. If CS Pump testing was not close to completion, the pump testing would be suspended. If the CS Pump was returned to ser ice, the Unit shutdown would be terminated. The two QNEs (a primary and a secondary QNE) recommended that reactor power be reduced using control rods if there was an intent to return to power Following the briefing, an extra Licensed Shift Supenisor (LSS) and NSOs were assigned to conduct the CS Pump testing.

Also following the briefing, the NSOs, US, and the QNEs continued a discussion of the effects of control rod movement on core axial flux that was deferred during the briefing. Power control at the Point of Adding IIcat (in the intermediate range of nuclear instrumentation) was not resiewed during this discussion. The SE, US, and QNEs were also involved in other discussions in preparation for the shutdown. At approximately 12:07 hours, the Ul SOS reminded the crew that it was time to begin the Unit power reduction.

4

l Zion LevelIkn estigation Report At 12:09 hours, the power reduction began at a ramp rate of 1/4% per minute. The US notified the Load Dispatcher of the power reduction and impending shutdown. There were more than 20 people in the l

control room during this period including operators, maintenance staff, senior managers, independent Safety Engineering Group (ISEG) observers, and NRC. Consistent with plant practices, two extra NSOs were assigned to the secondary panel to assist with controlling steam generator levels and the Secondary NSO was assigned to the turbine control panel. Steam generator water level during shutdown would be complicated because the 1D feedwater regulating bypass valve was isolated. Numerous alarms occurred as a result of the power reduction including feedwater heater Icvel and RPI alarms. Auxiliary boiler alarms were also received. It should be noted that the extra NSOs on the secondary panel had recently completed training that changes the way control room operators announce alarms, but the rest of the crew had not yet received the training (called "ZIoPR Training"). Although the old expectation was to ant' ounce unexpected alarms, the new training required announcement of each alarm, even if the alarm was expected. This new method of announcing alarms for expected conditions diverted the US's attention from other control room activities.

At 12:14 hours, the initial Event Notification System (ENS) phone call was made to the NRC to notify them of the plant shutdown.

At approximately 12:40 hours, an ISEG cngineer discussed the LCO expiration time with the SE. The ISEG cngineer had reviewed the original PT failure package and determined that the clock should have started when the 1C CS Pump was aligned for testing in preparation for the PT at 10:20 hours im February 19,1997. This resulted in the expiration of the forty-eight hour LCO twenty mir.utes earlier than anticipated, at 10.20 rather than 10:40, and expiration of the four hour action statemen' for shutdown at 14:20 and not 14:40.

Also at approximately 12:40 hours, the shutdown notification was transmitted to the NRC by facsimile indicating a 14:40 time for required shutdown. At 12:50 hours, the OOS on the 1C CS Pump was cleared.

At 12:55 hours, the SE notified the US that the LCO had expired at 10:20 rather than 10:40 hours, and that the shutdown clock would expire at 14:20 (rather than 14:40 hours). At 12:59 hours, the new LCO expiration time was logged and the power reduction ramp rate was increased to 1/2% per minute. At 13:00 hours, an ENS phone call was made to the NRC to notify them of the new LCO expiration time.

At this point in the shutdown, an extra NSO experienced difficulty controlling feedwater regulating valves. Feeding the D Steam Generator was made more difficult by the unavailability of the feedwater regulating bypass valve.

At approximately 13:05 to 13:10 hours, the SE and US discussed chemistry sample requirements for the Auxiliary Feedwater (AFW) Pump. Procedure GOP-4 requires sampling the pump prior to pump start for contaminants. The US and SE discussed this requirement because they were concerned for the safety of the operator who would take the sample from the pump. The SE and US were concerned that the operator could be subjected to a spray of high pressure water if the AFW Pump should start automatically. There was an increased probability for an automatic start of the AFW Pump due to steam generator level fluctuations with the feedwater regulating bypass valve isolated. They ultimately decided not to take the required sample for these safety reasons.

From 13:17 to 13:58 hours, the testing of the IC CS Pump was conducted successfully. The CS system still had to be realigned from its test configuration to its normal operating configuration to declare it operable and exit from the shutdown clock.

As the shutdown continued, the Primary QNE remained in the vicinity of the process computer available for the Primary NSO and US. The secondary QNE was asked by the LSS to stand back by the control room center desk area, well away from the reactor controls.

5

Zion LevelIinvestigation Report At 14:05 hours, the IB AFW Pump was started.

At approximately 14:05 hours, the SE decided that the reactor should be kept critical in Mode 1 (2%

power or greater) following the turbine trip as the SE was aware that inoperable equipment would prevent immediately returning to Mode 1 if powcr was reduced to Mode 2; it is not apparent whether the US or Primary NSO understood this. Holding critical in Mode 1 is a condition that is not explicitly captured in either GOP-3 and GOP-4, but the SE's instructians to the US were not clear. The SE instructed the US to keep the reactor critical following the trip of the turbine. Procedure GOP-4 directs a turbine trip followed by control rod insertion to stabilize reactor power at 2.5 E-2%, which is the POAll. [ Note: this would mean the reactor would be in Mode 2 (less than or equal to 2% power but keff greater than or equal to 1).]

The US thought that the SE was directing him to follow the shutdown procedure instructions. The Primary QNE understood the SE's intent to keep the reactor critical (kefT= 1), but the QNE did not understand the power level the SE intended to maintain. Note that there is no meter for kefror that shows the mode of the reactor.

To follow the SE's instructions as the US understood them, the US read step 5.21.f of GOP-4 to the Primary NSO. The step says, " Hold #363 ROD MOTION CONTROL switch IN to minimize dumping steam and establish power at or less than the Point of Adding IIcat (2.5 x 10-2 % IR)." The NSO questioned whether he should hold the rod motion switch in. The US read the step again and the NSO did not question him further.

At 14:07:33 hours, the turbine was manually tripped and at 14:07:35 the Primary NSO began inserting control rods continuously by holding the rod motion switch in. Following the turbine trip, there were numerous secondary system alarms that the NSOs announced and the US repeated back. Three NSOs were calling out secondary plant alarms to one US. According to one observer, there were 15 people in the Unit 1 "horseshoc" area at this time.

At 14:09.04 hours4.62963e-5 days 0.00111 hours 6.613757e-6 weeks 1.522e-6 months , during continuous control rod insertion, RPI alarms were received. The Primary QNE responded by going from the NSO desk to the nuclear instrumentation system panci. Also at 14:09:04 hours, the control bank C Demand Deviation alarm was received.

By 14:09:45 hours, the NSO had announced ' Control Rod Bank Limit Low' and the ' Control Bank Limit Low-Low' alarms, however, it is unclear w hether the US acknowledged these communications.

The Primary QNE saw Bank C inserting and told the Primary NSO that he was uncomfortable with the current reactor condition, however the exact nature of this concern was not made clear. The Primary QNE went to the process computer to check RPI alarms, then he checked indications on the nuclear instrumentation panel.

At 14:11:22 hours, Intermediate Range nuclear instruments indicated 2.5 E-2 %, and the Primary NSO stopped control rod insertion with Control Bank C at step 104.

From 14:11 to 14:13 hours, a volume control tank (VCT) automatic makeup occurred. The Primary NSO j

matched make-up boron concentration with reactor coolant system (RCS) flow. It was later determined that the additional negative reactivity from boron in the RCS makeup was relatively insignificant, and that the control rod insertion alone resulted in the reactor being in Mode 3 (less than or equal to -0.01 Delta k per k or -1,000 percent millitho or pcm).

During this period, power had decayed to approximately 1.0 E-2 %. The Primary NSO announced he was withdrawing control rods to stabilize power at 2.5 E-2 %. The US acknowledged the notification.

At 14:12:21 hours, the Primary NSO began control rod withdrawal in an attempt to return to the POAli.

This could be characterized as an atternpt to re-establish criticality from a substantially suberitical condition.

6

l Zion Level 1 Investigation Report During control rod withdrawal, the Primary QNE recalled telling the Primary NSO, "I don't like this,"

and the Primary NSO replied, "I don't like this either." Despite his expressed misgivings, the Primary QNE did not tell the NSO to stop rod withdrawal. The Primary QNE did not express his concern about control rod withdrawal to the US, but told the SE that he did not like the "in and out" activity.

At about this same time, the SE determined that the CS Pump would not be returned to operability prior to expiration of the shutdown clock. The SE did not understand the QNE's comment, but the SE had already decided to trip the reactor to ensure compliance with the LCO time clock. The SE directed the US to trip the reactor. At 14:14:06 hours, the Primary NSO stopped control rod withdrawal. The US l

instructed the Primary NSO to trip the reactor immediately. The reactor was manually tripped at 14:15:05 hours.

The position indicators for four control rods did not have all the indications that they were fully inserted.

Accordingly, the crew entered procedure AOP 2.2, " Emergency Boration."

At approximately 15:00 hours, the two QNEs discussed their concern about control rod continuous insertion and withdrawal with the SE. The QNEs decided to write a Problem Identification Form (PIF),

and agreed with the SE to discuss the concern with the US and NSO. The PIF, as written, adequately characterizes the rod motion involved and the sustained rod withdrawal from a substantially subcritical condition. It makes no reference to procedural adequacy.

The QNEs wrote the PIF and notified the SOS at 15:51 hours. At approximate' 17:30 hours, the Ul SOS notified the U1 Operations Manager of the QNEs' concern. There was a meeting between the QNEs, the U1 SOS and the UI Operations Manager. A root cause investigator was called in ta begin intenicws.

At about 18:30, the Ul SOS and the root cause investigator decided to recreate the event the following day on the simulator with the crew members involved. On Friday night, the Ul SOS decided to remove the US and the Primary NSO from licensed dutics, and they were taken off the watch list for Saturday, February 22nd.

The Ul Operations Manager initially notified the UI Plant Manager that diflicultics were experienced during shutdown, and the Ul Operations Manager provided the U1 Plant Manager with an update at approximately 22:30. It is not clear why these two discussions did not adequately characterize the reactivity management concerns in this event.

At about 06:00 on Saturday, February 22,1997, the U1 Operation Manager and the UI Plant Manager discussed the reactivity management aspects of the event. At about 07:00, the Ul Plant Manager notified the Plant General Manager of the event and the simulator recreation. The UI Plant Manager also notified the Site Vice President.

The event was re-created on the simulator with the SE, US, Primary NSO, and Primary QNE from about 09.00 until 11:00 hours. The UI Operations Manager made an initial notification to the NRC around 09:00 hours that there were problems observed during the U1 shutdown. The Site Vice President arrived at the simulator around 10:00 hours. Following his review of the event, the Site Vice President directed that both Units should be placed in Cold Shutdown. Midday Saturday, the Ul SOS related the decision to remove the Shift Engineer from licensed dutics, however, it is unclear precisely who made this decision.

During the course of the day, on Saturday, the Site Engineering Manager suspended the Primary QNE from QNE dutics.

The Site Vice President notified the Executive Vice President for Nuclear Operations of this event around noon, and the UI Plant Manager contacted the NRC Resident inspector around 12:30 hours. The Ul Plant Manager discussed the event with NRC Region 111 and the NRC Senior Resident Inspector late in the aflernoon of Saturday, February 22,1997. The Station Duty Officer became aware of the event this 7

.m...__.-_._._

... ~. _ _.. _ - -._. -.

l Zion LeveliInvestigation Report V

I same day. The Nuclear Operations Division Duty Officer was not aware of the event until the followmg Monday, February 24th.

At approximately 17:00 on Saturday, February 22,1997, the U1 Plant Manager, the UI Operations l

Manager and the Ul SOS discussed returning the licensed individuals involved with this event to shift.

l The Ul Plant Manager deferred to the UI Operations Manager and the Ul SOS. The UI Operations -

Manager and the Ul SOS d-ided that the operating crew should remain on shift.

l On Sunday, February 2.

.t 07:00 hours, the SE, US and Primary NSO involved in the shutdown resumed shift duties. At oo:.w ' sours, the US initiated a PIF concerning GOP-4, step 5.21.f. At a 10.00 a.m. meeting between the Site Vice President, the Plant General Manager, the U1 Plant Manager and the Operations Manager, the Site Vice President directed the UI Operations Manager to remove from licensed duties the crew personnel who were on shift for the shutdown. At 12:45 hours, the SE, US, and NSO were formally suspended from licensed duties.

t On Monday, February 24,1997, the Plant General Manager established a senior manager control room l

overview program. This was formalized by release of Standing Order 97-05, Revision 0, " Senior j

Management Presence Required for Shift Briefings," on Wednesday, February 26th.

1 l

l l

t 4

f i

8

Zion Level 1 Investigation Report ANALYSIS OF CAUSAL FACTORS Introduction & Event Significance The Level I Im estigetion Team (Im estigation Team or Team) was assembled to resiew the event at Zion on February 24,1997. The Investigation Team gathered the facts and developed a chronology for the events related to this shutdown by performing inteniews of the personnel involved, and by reviewing the operator logs, procedures, test documents, problem identification forms and Station policies relevant to this shutdown. The Team recognizes that, despite its best efforts to assure that all of the information presented here is complete and accurate, the Team experienced some difTiculty reconstructing details surrounding this event. Those difficultics include the passage of time (the Team started its inteniews the Monday following the event), intervening events (the crew recreated the event on the simulator, potentially distorting their recollection), and the need for others to conduct inteniews first (the NRC's Augmented Inspection Team directed that they should inteniew some personnel in advance of the Investigation Team). Furthermore, the operating crew did not submit detailed written statements immediately following this event. Even so, the Team believes that its understanding of the event, analysis, and conclusions are correct.

The consequences of the actual reactivity excursion in this event is minimal, however the actions taken during the event call into question the knowledge, judgment and ability of the operations and support personnel involved in this event. These later factors make the event hignly significant from the perspective of needed actions. Additionally, these actions call into question the commitment of the individuals involved to conservative decision making and the understanding of their obligation to assure maintenance of reactor safety.

As summarized in Appendix E (" Zion Reactivity Event Significance"), the actions taken during this event did not challenge the safety limits for the reactor or the nuclear fuel. No barriers to containment of fission products uere challenged by the actions taken. The power 1r :1 changes that occurred as a result of operation of control rods throughout this event remained within design limits for the fuel and the reactor at all times. Projections made even if operator action was not taken to manually shutdown the reactor clearly demonstrate that any subsequent power excursion would not have ciellenged any reactor or fuel safety limits. Accordingly, this Investigation Team has determined that this cet, in and ofitself, did not pose a risk to the health and safety of the public or workers at Zion Station.

The actual safety significance aside, the Investigation Team understands that the operator manipulation of the reactivity controls of the Unit I reactor was not in keeping with the most basic expectations for nuclear plant operations. The investigation Team recognizes that failure to perform reactivity manipulations in a safe, controlled manner, and in accordance with approved procedures, can result in the violation of reactor safety limits, the breach of fission product barriers, and an off-site release. It is for these reasons that the safe control and oversight of the nuclear reactor is the preeminent obligation of the people licensed to operate the reactor and the personnel who support them.

To understand the issues that are relevant to this event and their relationship, a cause and efTect analysis of the problems was performed using stream analysis techniques. The diagnostic chart, a theme chart and a summary of the results of this cause and efTect analysis appears as Appendix B to this report. The Investigation Team further analyzed the significant problem areas for the central and underlying event symptoms, causes or effects. The results are summarized below in the section titled " Analysis."

On February 18,1997, Zion Station received the results of an Independent Safety Assessment Team (ISAT) inspection. That inspection was an intrusive assessment of the condition of the Management.

Operations, Training, Engineering, Maintenance, and Plant Support organizations at Zion Station. After 9

Zion LevelIinvestigation Report the Investigati)n Team gathered the facts, developed the chronology, and developed a list of potential causes for this, event, the Investigation Team reviewed the ISAT inspection report.

The ISAT report is a comprehensive comparison of Zion's performance with the best nuclear plants in the industry. While the ISAT inspection broadly addressed issues in all major functional areas at Zion Station, the Investigation Team, applying a similar standard, focused on the reactivity management aspects of the shutdowTi along with the issues and organizations central to the apparent causes of this event. Those causal and contributing factors common to both the ISAT and this event are summarized in Appendix C to this report Recognizing that broad corrective actions are being developed for the issues identified by the ISAT, this Team did not recommend corrective actions that address a broad range of deficiencies, but rather has provided a list of specific issues that should also be addressed.

The Investigation Team's analysis was grouped in the following areas: planning and briefmg; communications; fundamental knowledge / training; management oversight; organizational behaviors; corrective actions; and SQV/ISEG oversight. These analysis of causal factors are intentionally highly self-critical and are based on a retrospective resiew of this event. These results are discussed further in the conclusion of this report.

Analysis Planninn & Briefine Planning: the Prior Ship and Support Organi:ations did not Create an Executable Plan e

This event disclosed several areas for improvement with respect to planning in support of plant operations at various levels of the organization. The following are examples of the Team's obsenations:

On Febru ry 21,1997, the IC CS Pump failed its operability PT surveillance at 04:40, only six hours before the expiration ofits 48 hour5.555556e-4 days 0.0133 hours 7.936508e-5 weeks 1.8264e-5 months LCO. At 07:00, the night shift operating crew turned over to the day shift crew. Although the start of the action statement shutdown clock was less than 4 hours4.62963e-5 days 0.00111 hours 6.613757e-6 weeks 1.522e-6 months away, no plan was yet developed, by either Operating or Work Planning, of the actions required for an orderly Unit shutdown or the return of the IC CS Pump.

The Unit 1 (Ul) Operations Manager, Ul Plant Manager, and the Plant General Manager were made aware of the failed surveillance between 05:00 and 05:30. No actions were put in place to ensure success of the Operations crew. At the 08:00 Plan of the Day (POD) meeting it was reponed that Unit I should be prepared for shutdown, an action plan developed for the IC CS Pump, and that the Qualified Nuclear Enginects (QNEs) should meet with the operating crew at 09:30 to prepare for the shutdown. Responsibility for these actions was not assigned to the plant support groups, and the responsibility fell to the Operations shift personnel by default. As a result, the operating crew supervisors were unable to meet with the QNEs and were unprepared for the Unit shutdown.

It appears that the CS pump repair efforts would continue until the expiration of the action statement. During the 11:10 to 12:00 shutdown briefing, a discussion was held to start the shutdown at 12:00 and to be shutdown by 14:40. The crew was advised that at approximately 15% power, the status of the CS Pump repairs and testing would be assessed. If the CS Pump was not "close to completion," the pump testing would be suspended. It was not defined what was meant by "close to completion." At 14:05,just fificen minutes before the expiration of the shutdown action statement, the SE decided that the Unit should remain in Mode I because the CS Pump testing appeared to be "close to completion."

10

Zion LevelIhn'estigation Report Crew focus on continued CS Pump restoration, ongoing issues with plant equipment, determining CS Pump testing requirements, and personnel manning needs, prevented operating shin management from meeting with the QNEs to discuss and plan reactivity issues during the expected shutdown. In addition, the NSOs and control room supenisors were not able to resiew the shutdown operating procedures. Consequently, the AFW lines were not sampled prior to starting the AFW Pump, and a question ofinserting control rods to the POAH was never properly reviewed and addressed.

Briefing This event revealed several areas for improvement with respect to briefing. The following are examples of the Team's observations:

The pre-brief, conducted with operating crew management and Senior Management, left the operating crew with mixed signals. Senior Management expressed the importance of a safe, error-free shutdown. There was also discussion of continuing the CS Pump restoration.

Ilowever, it would have been beneficial had a time been set beyond which the shutdown would continue to completion regardless of pump status.

The brief was not conducted until 11:10 hours. This was after the four hour action statement for shutdown clock had been entered. The crew had not resiewed the procedure for shutdown and the crew management was not prepared to conduct the brief. Since the brief was started aner the four hour clock was running, the briefing was rushed and less than thorough. Consequently, a thorough review of the shutdown procedure was not performed. No checklist was prepared for issues to discuss and no attendance sheet was created. Senior management prompting was required to get the Unit Supenisor to include the shutdown procedure precautions and limitations during his brief. The roles of the personnel in the Main Control Room were not reviewed and clearly defined, particularly the role of the QNE during the shutdown. No criteria were presented for aborting the pump restoration efforts, a point beyond which the shutdown would continue regardless of CS Pump status or the final state or mode of the reactor.

Additionally, the brief was ended with unresolved issues, specifically Delta 1 concerns, to be resolved by the NSO and the QNE aner the brief.

The actions of Senior Management and the operating crew early in the shift were clearly dedicated to the return of the CS Pump to operable status. Senior management focused on the efforts to repair and test the CS Pump to avert the shutdown. Arrangements had been made for the QNE to discuss shutdown plans with the Shin Engineer (SE). The SE was unable to hold the meeting because he was too busy with the on-going activitics, including the CS Pump work.

Such a meeting might have helped provide greater clarity concerning reactivity management, the use of control rods vice boration, and Delta I control during shutdown.

Communications This event disclosed several areas for improvement with respect to communications at various levels of the organization. The following are examples of the Team's observations:

e Written Communications There is no general requirement for written communications from line management. Performance might have been improved had there been written communications with respect to the time for the Unit to be in Mode 3. In hindsight, a time certain for shutdown, other than the expiration of the I.CO action statement timcciock, would have allowed the operating crew to turn their attention froni other actnitics and focus on 11

Zion LevelIInvestigation Report the shutdown. Given adcquate time, Shif1 managers could have discussed reactivity management before briefing the crew, reviewed the shutdown procedurcs, held a proper brief, discussed how the feedwater regulating valve would be managed, and determined how to address Main Control Room distractions and general milestones during the shutdown.

Verbal Communications The Team made the following observations with respect to verbal communications:

A.

Site Vice President / Operations Afanagement on prioritiesfor shutdown. At a discussionjust prior to the Unit shutdown briefing at 11:10 a.m., the Site Vice President indicated that the crew's priority was to be a safe shutdown and that the Containment Spray Pump was not to get in the way. Several levels of management were in attendance for this discussion, including the Unit 1 Plant Manager, Unit i Operations Manager, Shift Operations Supenisor, Shift Engineer and Unit Supenisor.

B.

Unit 1 Operations Afanager/ Unit 1 Plant Afanager on event reporting. On Friday evening, February 21,1997, the Unit 1 Operations Manager contacted the Unit 1 Plant Manager on two occasions. The diflicultics with the shutdown were discussed, including the need to recreate the event on Saturday at the simulator. There was no communication to the Plant General Manager or the Site Vice President on Friday night.

C.

Senior Afanagement and Operations Afanagement communications on removal ofpersonnelfrom ship. On Friday, February 21st, after discussion of the event with the root cause investigator, the Ul SOS concluded that the Unit Supenisor, and Primary NSO should be relieved of shift duties. Midday I

Saturday, the U1 SOS also directed that the Shift Engineer should be removed from licensed dutics. However, the UI Plant Manager, in a meeting with several department head level personnel, communicated the decision to take 1

Unit I to Cold Shutdown and to " remove 2 NSOs, I LSS [ Licensed Shill Supenisorl. and the QNE from duty until a root cause evaluation was completed." Later in the day on Saturday, the U1 SOS and Ul Operations Manager discussed their thoughts with the Unit 1 Plant Manager on allowing the involved personnel to remain on shift Sunday. The Ul Plant Manager deferred the decision on whether to remove personnel from shift to the U1 Operations Manager and SOS and the personnel were allowed to remain on shift. The involved personnel were ultimately relieved of shift duty at 12:45 on Sunday, February 23,1997, after specific direction was given by the Site Vice President. These communications reflect weaknesses in understanding and being understood at various levels in the organization. Specific diflicultics were encountered conveying the event significance, understanding the actions of the shift personnel during the event, and sharing what was understood between individual members of the management team. These difficultics affected management's ability to take appropriate timely actions following this event.

D.

Operations management did not make timely reportsfor a subsequent event. The nced for quick and formal reporting of plant events was not reinforced following the shutdown. Consequently, when a subsequent event occurred (an Residual licat Removal (RHR) lineup change-induced Pressurizer level drop) it was not expeditiously reported to the Senior Managers.

12

Zion LevelIInvestigation Report Clear Directionfor Reactor Operation The Shift Engineer directed the Unit Supenisor to trip the turbine and keep the reactor critical.

Although that is what the Shift Engineer told the Unit Supenisor to do, the Shift Engineer's intent was to remain in Mode 1 (greater than 2% power). The Unit Supenisor, in turn, directed the Secondary NSO to trip the turbine and then directed the Primary NSO to implement step 5.21.f of GOP-4 which requires insertion of rods to reach the Point of Adding Heat (defined in the procedure step as less than or equal to.025% power). The NSO questioned the step and the Unit Supenisor re-read the step to him. The NSO did not pursue his concern any further. Subsequently, the NSO continuously inserted rods, drove the reactor significantly subentical, then withdrew rods continuously in an attempt to reestablish.025% power. While the Shift Engineer intended to remain in Mode 1, the Unit Supenisor directed actions which would put the reactor in operational Mode 2, and the NSO actually took the reactor to Mode 3 prior to attempting a restart. Further, the decision to remain entical was not briefed to the entire control room or even to the crew operating Unit 1. This limited the ability of those obsening the evolution to understand the actions being taken or to recognize inappropriate operator actions as several observers did not hear that the reactor was to remain critical.

Interface ofthe Quahjied Nuclear Engineer (QNE) with the Operating Crew The QNE had the understanding that the Unit was to remain critical and, when he saw rods being continuously driven in, he became uncomfortable with the situation. While reports vary about the discussions between the QNE and NSO during rod insertion and withdrawal, it is apparent that the discussions, if any, were not effective in halting the rod movement. Subsequently, during rod withdrawal, the QNE told the NSO that he did not like the NSO actions; the NSO agreed, however, j

nothing was done to stop rod withdrawal. The QNE then approached the Shift Engineer and told him i

that he did not like the "in and out stuir' but did not effectively express his concern about rod withdrawal. The Shift Engineer's direction to trip the reactor was based on the impending expiration of the LCO timeclock and not the communication from the QNE. Additionally, it should be noted that the Zion Operations Department Standards and Expectations, dated January 18,1997, defines the role of the QNE as follows: "The NSO shall have sole responsibility for reactivity manipulations

]

from the main control board at all times, the Unit SRO is supenision and oversight, and the QNE shall function in an advisory and coordinating role only."

Procedural Guidance Zion Stm a maintains an administrative procedure on Reactivity Management. This event identified the need to revise the procedure to reflect present organizational titics and to improve the information available to operators facing circumstances similar to this event. The reactivity management administrative procedure is maintained in the control room. There is also a policy statement on reactivity management dated September 25,1996. An improvement would be to also maintain this policy statement in the control room. It provides a specific level of guidance that could have helped prevent this event. For example, this policy statement requires communications between the US and NSO that should have occurred many times during the power reduction (starting at 12:09).

GOP-4, the procedure used by the Primary NSO to move control rods to the POAH, has two operator activities in a single step. Specifically, step 5.21.f directs the operator to " Hold ROD MOTION CONTROL switch IN to minimize dumping steam pnd establish power at or less than the Point of Adding Heat (2.5 x 10'2 % IR)." (Emphasis added.) As a human factors matter, two activitics in a single procedurai step is undesirable because of the potential for confusion on the part of the person performing the step, but this was not identified by any of the operators as a reason for confusion 13

Zion LevelIInvestigation Report during the February 21st shutdow n. In the Team's view, although this step could be improved, the step, as written, provides adequate guidance for a licensed operator to perform the intended actions.

Fundamental Knowle.d_gerfrainine This event disclosed several areas for improvement with respect to knowledge of reactor fundamentals.

Issues which the Team considered include:

0 What was the knowledge of reactor fundamentals possessed by the crew and how able were they to apply that knowledge?

0 How well was the team prepared to perform the step ia GOP-4 which allows the reactor to be maintained critical at the POAH following a plant shutdown?

O How well are standards established in training carried over to and enforced in the plant?

O How well have the lessons learned within the industry been communicated to the operating and site organization?

Knowledge and Application ofReactor Fundamentals e

After an initial resiew of the chronology of events surrounding the shutdown, Investigation Team members felt that operator knowledge of fundamentals needed to be assessed. Accordingly, a list of questions addressing pertinent nuclear concepts such as the Point of Adding Heat (POAH), delayed neutrons, and startup rate (SUR) were developed to assess operatcr knowledge of fundamentals applicable to this event. Addi% nal questions were developed which measured their ability to relate these fundamentals to s. cessful completion of GOP-4 step 5.21.f.

This exam was given orally to 8 NSOs,5 LSS/SEs,6 Training Instructors, and 2 management oversight individuals (SQV/QNE). The individuals examined included all licensed operators (RO/SRO) who were present on Unit I during the event and other individuals.

The following are suggested by the results of this informal exam:

+ The knowledge and application ability of the two individuals closest to this event was inadequate.

+ The knowledge and application ability of the other individuals present on Unit I should be improved but was adequate.

+ The level of knowledge of the tested individuals not present on Unit I was mixed.

+ The knowledge of the instructors was toward the higher end of the scale and of sufficient level to provide cITective training.

Individual performance of the operators during this event did not demonstrate an understanding for the basic tenets of reactivity control or an adequate respect for those controls. The Investigation Team's resiew of training materials suggests that several important lessons critical to safe operations were not applied by the operator at the controls the dav nrthe event. The Team identified several lesson plans where conservative decision making is discussed along with reactivity control. Fundamental to carrying out that principle is that the operator understands what he or she is doing, w hat will happen, and what the effect of operation of a switch or control will be before that manipulation takes place. The operator at the reactor controls, especially w hen he continuously withdrew control rods to restore reactor power, did not apply these principles.

Other training guidance discusses maintaining a questioning attitude, challenging actions that arc inconsistent with the fundamental requirement to operate the reactor safely at all times. Training must 14

Zion LevelIinvestigation Report continue to emphasize the need to understand procedure steps before executing them, the absolute need to stop when conditions are questionable, and finally to put reactor safety before all other prioritics.

Preparation ofthe Crew to Maintain the Reactor Criticalin Mode 2 Step 5.21.f of GOP-4 provides the necessary guidance to trip the main turbine and drive control rods to stabilize reactor power at the Point of Adding Heat (POAH). This procedure also permits the operator to take the reactor to Mode 3 following tripping the turbinc. Actual plant experience shows that reactor power was stabilized at some intermediate power during reactor shutdown or reactor startup multiple times in the previous 3 years. This appears to be a frequently conducted step. The Team concluded that requalification training should contain scenarios which provide operating crews the hands-on experience to maintain proficiency for this evolution.

Standards Training andin-Plant Performance e

During this event a number of comments were made concerning the adherence of the crew to Operating Standards, specifically threc-way communications, response to annunciator alarms, and procedurc adherence.

A review of simulator training records and crew notebooks indicate that adherence to standards has been a priority during scenarios over the past year. This is esidenced by crew specific self assessments and critiques. There have been crew specific improvement plans to better standards performance. In addition, a focus has been placed on standards during ZioPR Training this year, training the on-shift crew was scheduled to receive on March 4th. Contrary to past training efforts, performance was not satisfactory on 4

February 21. The crew's performance indicates a lack of proficiency at maintaining conformance with cven the old standards to focus on reactor control during a shutdown.

The published standard for communications within the Operating Department is three-way communications. Control room observers indicate that some communications during the plant shutdown were incomplete (one or two way).

The standard for annunciator response requires alarms to be announced by the operators and repeated back by supenisors. This can be waived during transients for expected alarms because the myriad of alarms produced would make this requirement impractical. On February 21, when the turbine was tripped, cach of the alarms which came in were called out by the operators on the secondary panels. As was discussed above, this was the result of certain non-crew members who had received ZIoPR training operating the secondary panels. Calling out numerous expected alarms is reported to have taken the Unit Supenisor's attention away from the reactor panel.

The reactor operator failed to meet the intent of GOP-4 when he erroncously interpreted step 21 and continuously inserted control rods until power reached 2.5 E-2 % as opposed to inserting rods to establish a negative startup rate to reach that power level.

Application ofPrior Industry Experience e

In 1996, the Engineering Manager identified past reactivity events at Zion Station and brought them to the attention of the Operations Manager. Additional training was developed and implemented based upon the past events identified by the Engineering Manager. Such training should be more broadly developed.

The concerns raised by the Engineering Manager were covered as a topic in recurring training with several other topics. The significance of the reactivity issues may have lost some importance when presented in this way. INPO has issued Significant Operating Experienced Reports (SOER) with recommendations. The training on specific recommendations in the SOERs were provided in discretc scenarios. Future training should include an oveniew of the entire SOER.

15

Zion LevelIinvestigation Report l

i I

Manacement Oversicht l

Through its investigation, the Team identified the following opportunitics for improvements in management oversight:

Maintaining a High Standards Environment Management should improve its efforts to establish high performance standards consistent with excellence for nuclear safety and to reinforce existing standards. The following are examples of areas for improvement.

l l

+ The Unit started its shutdown after the 48-hour LCO clock had expired, and about two hours l

into the four hour shutdown clock. It would have been more conservative to have started shutting down earlier.

+ Senior managers were in the Main Control Room during the shutdown and observed the large number of observers, on-going CS pump testing, and alarm announcements and repeat-backs. A Senior Manager directed that a few operators be removed from the control room.

+ Despite the presence of Senior Managers in the Main Control Room, it is the Shift Enginecr's principal obligation to manage the shift The Shift Engineer must keep a broad oveniew ofimportant activitics and must not get so involved in the details of operating a Unit where he is unable to provide his crew with the necessary oversight and direction.

+ Following the event, Management allowed the crew involved in the event to continue license l

duties and did not promptly direct their suspension.

l l

e Production Culture l

It is important for Management to provide clear and concise expectations concerning handling of j

circumstances like the 48 hour5.555556e-4 days 0.0133 hours 7.936508e-5 weeks 1.8264e-5 months limiting condition for operation (LCO) time clock for the CS Pump. Forty-two hours into the LCO, w hen a new prob!cm (faulty component starting circuit) was encountered, it was not known if the Station could meet the 48 hour5.555556e-4 days 0.0133 hours 7.936508e-5 weeks 1.8264e-5 months LCO limitation. Therefore, it was necessary for Operations to prepare for an orderly shutdown of Unit 1. Following the expiration of the 48 hour5.555556e-4 days 0.0133 hours 7.936508e-5 weeks 1.8264e-5 months clock and entry into the four hour shutdown period, the crew procccded with the CS Pump repair efTorts up to the expiration of the four hour clock. Thus, the crew was primarily focused on supporting the repair efforts so that, if the pump was repaired, the Unit could be kept on line.

Senior Managers stated they expected a safe, uneventful shutdown. Several Senior Managers present in the control room focused on the status of pump repairs while the shutdown event occurred. These managers did not direct Shift Management to stop pump repairs at any time before expiration of the time clock. The Shift Engineer directed that the reactor should remain critical until the last minutes before time clock expiration while pump repairs continued. A manual scram was initiated with five minutes remaining before a shutdown was required. In hindsight, a more conservative approach toward LCO shutdown action statements with clear guidance for the performance of a safe unit shutdown should be j

provided.

l l

16

Zion LevelIInvestigation Report The Ship Engineer Did Not Rely on Specific Procedural Guidance When Directing the Unit e

Supervisor to Alamtain the Reactor Critical After the Shift Engineer was informed that the last performance test of the CS Pump was completed satisfactorily, he directed the Unit Supenisor to " keep the reactor critical." Not only did this ambiguous direction result in a misunderstanding of the power level at which the reactor was to be maintained, but the Shift Engineer did not direct the Unit Supenisor to any particular step in the procedure where that condition was permitted. Had the Unit Supenisor done precisely what the Shift Engineer had intended -

stop the shutdown and hover above two percent power - the Unit Supenisor would have been operating outside the guidelines of the shutdown procedure.

The Unit Supervisor Did Not Oversee the Reactor Operator The Zion Operating Department Standards and Expectations (Jan. 18,1997) places the responsibility for reactivity manipulations with the NSO while the " Unit SRO [ Senior Reactor Operator, here the Unit Supenisor] is supenision and oversight" for those manipulations. Once the Unit Supenisor received the order from the Shift Engineer to keep the reactor critical, the Unit Supenisor read step 5.21.f of GOP-4 to the NSO. During the next seven minutes of control rod manipulation, control rod position alarms were an opportunity to re-focus attention on the NSO's actions.

Afanagement Corrective Actions Two days after the shutdown, the operators involved were ultimately removed from their shift dutics (the Primary QNE having been removed from QNE dutics the day after the event). No other immediate corrective actions were established until February 24th, some three days after the shutdown.

On February 24th, a Senior Manager on Shift Program was instituted to establish oversight in the Main Control Room. This corrective action should have provided more direction for the assigned managers or Operating crew to follow and should have been more rigorous in order to more rapidly alter the behavior of shift personnel. Subsequently, on February 25th, a Residual Heat Removal (RHR) lincup change-induced Pressurizer level drop event occurred. The Pressurizer level dropped about four percent while lining up the RHR system for reactor cooling. The immediate corrective actions did not prevent this RHR mis-operation. On February 26th, Management formalized the Senior Manager on Shift Program through issuance of Standing Order No. 97-05, Resision 0, " Senior Management Presence Required for Shift Briefings."

On February 27th, the Standing Order was revised to permit only a select few managers at Zion Station to have oversight responsibilitics. This direction was promulgated in Standing Order No. 97-05, Resision 1,

" Senior Management Oversight." Subsequently, management limited Unit activities to two " planned" evolutions per Unit per shift.

Orcanizatiorial Behaviors Afain ControlRoom L)emeanor and Distractions To properly ensure that a unit shutdown can be performed efficiently and safely, the Main Control Room decorum and standards must be maintained. Any distractions from activitics at the reactor control panel must be kept at an absolute minimum. The Team made the following observations with respect to contral room demeanor and distractions.

The Secondary NSO assigned to Unit I called in for a vacation day. A " Unit 2 Training Weck NSO" was assigned the dutics of the Secondary NSO. Another NSO informed the Unit Supenisor that this 17

Zion Level 1 Investigation Report assignment violated an agreement for the unitization of NSOs. Thus, another " Training Week NSO" was assigned to the position. This NSO was not scheduled to work a 12 hour1.388889e-4 days 0.00333 hours 1.984127e-5 weeks 4.566e-6 months shift, so a third NSO was called out to fill the Unit i Secondary NSO position. Discontinuity in coverage of the Secondary NSO position provided the Primary NSO with no consistent support from the Secondary NSO position and added needless confusion to a hectic emironment.

There were 24 people in the Main Control Room from 14:00 to 14:20 and 35 others who entered and exited the Main Control Room envelope during this period. A large number of people in the Main Control Room limits the physical space for operators and increases noise levels to an unacceptable level.

Senior Managers were present in the control room for various reasons including to observe progress with CS Pump repairs and the reactor shutdown. The Shift Engineer perceived that Senior Managers were in the Main Control Room to watch the crew save the Unit before the LCO action statement clock expiration.

In hindsight, Senior Managers must ensure that their presence does not convey the wrong signals and must actually remain aware and engaged when they are in the Main Control Room or in the plant.

The ID FW Regulating Bypass Valve was isolated because its closure could not be verified. Even so, Zion practices had two extra NSOs assigned to control Steam Generator levels on the Unit shutdown. The control of the Steam Generator levels was a constant concern for the Unit Supenisor and the crew The extra NSOs were focused on the Steam Generator Level Control so they were not available to the crew for backup assistance.

Annunciator communications after the turbine trip drew the Unit Supenisors atternion away from the NSO on the Reactor Panel. The NSOs on the FW panel had completed ZloPR Training, whereas the NSO on the reactor had not received this training. The training was at least partially responsible for the FW NSOs being more aggressive in communicating their annunciators. Aner the turbine trip, many of these annunciators should have been filtered by the extra NSOs and Secondary NSO to minimize the distraction. The NSO on the reactor announced two key annunciators (" Control Rod Bank Limit Low" and " Low-Low" alarms) but did not ensure his message was communicated by insisting on an acknowledgment from the Unit Supenisor.

Another event which distracted the crew from focusing on the Unit shutdown was radiation monitor testing. The radiation monitor testing was not necessary to bring the Unit to a safe, shutdown condition.

Therefore, this testing was an evolution that could have been stopped to prevent an unnecessary distraction to the operators.

Although pre-job briefings are customarily conducted outside the Main Control Room to minimize distractions, due to the lack of available manpower the crew was unable to be relieved and the briefing was conducted in the Main Control Room. The Primary NSO was relieved of panel duty by another NSO to allow him to devote his full attention to the brief. The NSO who relieved the Primary NSO was a member of the operating team for the evolution assigned to control steam generator levels.

Finally, the CS Pump inoperability was incorrectly identified as 10:40 on February 19,1997, instead of 10:20. Due to this error in the CS Pump LCO inoperability time clock, Shift Management originally thought the Unit had to be in Mode 3 at 14:40 on February 21. At 12:55 on February 21, the Shift Engineer, after discussions with ISEG, informed the crew that the CS Pump LCO action statement expired at 14:20 instead of 14:40 due to the incorrect inoperability identification on Febmary 19th. After notifiction of the resised CS Pump LCO expiration, the crew increased the ramp rate on the Unit shutdown.

18

Zion LeveH hive.stigation Report e

Teamwork The Investigation Team determined that some teamwork attributes expected of an operating crew need to be improved.

During the events leading up to the control rod insertion and withdrawal, there were several instances when teamwork between operating crew members could have been better. The Shift Engineer and Unit Supenisor could have better prioritized emergent work, and eliminated non-essential tasks in the time leading up to the event. They were heavily involved in the 1C CS Pump troubleshooting, actual testing, and determination of testing requirements required to exit the LCO. They were also actively involved in miscellaneous activitics: annunciators, phone calls, and numerous personnel in the MCR.

The crew members were generally experienced control room operators. No crew member adequately resiewed the GOPs prior to commencing the Unit shutdown. The SE or US could have provided a better briefing for the crew with regard to the Unit shutdown and its sequence of events.

Much of what faced the Shift Engineer and Unit Supenisor on the morning of February 21,1997 could have been better prepared and planned by Station support organizations or previous shifts. The crew had to act in a very compressed time frame and under significant pressure. This need for teamwork by the Zion support organizations cannot be over-emphasized.

The QNE should have been utilized as a "rcactivity management consultant" in order to provide the crew the expertise necessary to properly accomplish the establishment of core power at the POAH.

Corrective Actions for Previous Events / industry Exocrienec/Frends Examples of presious events, industry experience, and problem trend analysis considered by the Team include the following:

Previous Events On January 30,1996, uhile operators were performing a performance t.'st (PT) on Unit 2 Auxiliary Fecdwater Pumps (AFW), reactor power exceeded 2 percent which is a transition from Mode 2 to Mode 1. These AFW Pumps were in a Technical Specification 1 CO during the mode change, so a mode change in those conditions was contrary to requirements of Technical Specifications. Contributors to this event included weaknesses in the pre-job briefing, unclear understanding of test abort criteria, inadequate resiew of test conditions, poor communications between operators and management, weak management oversight, and workarounds. Corrective actions included:

counseling the Shift Engineer, Unit Supenisor, and Nuclear Station Operators (NSOs);

resiew of the event with Operations and Engineering during continuing training; and resiew of the administrative procedure on "Use of Procedures" in continuing training for Operations, Maintenance, and Engineering.

On September 16,1996, while Unit I was in startup, the Nuclear Station Operator continued to withdraw control rods with a rod position indication deviation greater than twelve steps, which was contrary to requirements of Technical Specifications, General Operating Procedure GOP-2

(" Unit Startup"), and management expectations. Contributors to the event included ineffective briefing, weak command and control, long-standing rod position indication problems, and lack of sensitivity to reactivity addition with control rod indication alignment problems. Corrective actions included:

a modification to the rod position indication system; 19

~

Zion Level 1 Investigation Report development and implementation of the 1997 Operational Plan; and i

interim actions pending implementation of the 1997 Operational Plan.

}

i Industry Experience e

The Investigation Team reviewed the Station response to recent SOERs on reactivity events.

Some training was conducted in response to these SOERs. The Team concluded that more training is required in order to improve operating practices at Zion.

i Problem Trend Analysis e

The Zion Engineering Manager had identified an adverse trend based on reactivity events during the three year period ending in 1996. He reported to the Operations Manager that the frequency and severity of events had increased. Specific training was developed and provided to operating i

crews as a result of this communication.

SOV/ISEG Oversicht i

SQV/ISEG provides control room oversight during major evolutions and monitored events. Such oversight can add value when it results with constructive input for the operating team. This function provides an additional barrier to recognize events, help prevent unplanned transients, inappropriate 1

operations, and unacceptable results.

On the day of the Unit I shutdown, the oversight for this evolution was a formerly NRC licensed individual with extensive Zion operating experience. While reviewing the test data for the CS Pump the ISEG representative recognized the 20 minute error in the LCO clock and pointed the error out to the operating crew. This prevented a Technical Specification violation for not meeting the required shutdown time.

The ISEG representative in the control room focused his attention on the operation of the secondary systems after the turbine was tripped instead of the reactor.

20

Zion LevelIInvestigation Report CONCLUSIONS The Team assembled to resiew this event believes that this event demonstrated several weaknesses at Zion Station. A first look at this event focuses on the operator at the controls of the reactor, his Unit Supenisor, and the Shift Engineer. While the Investigation Team has not tried to assign individual responsibility for this event, the Team's review has made cicar that the responsibility for the event, the errors that permitted it to occur, and the failings subsequent to the event belong with not only the individual operator, but with his supenisors and support personnel. The Investigation Team recognizes that its findings in this regard cre consistent with the issues identified by the Independent Safety Assessment Team (ISAT) at Zion Station in their resiew which wasjust completed in late February,1997.

The Investigation Team has identified and analyzed specific causal factors with regard to what it perceived as ineffective planning by support organizations which placed responsibility for operational success for the day's activitics with the operating crew. The operating crew then was unable to plan, prepare for, or adequately brief in advance of the required shutdown. The Team has noted incidents of ineffective communications were observed throughout this event by all levels of the organization.

The individual at the reactor controls and his immediate supenisor did not operate consistent with the intent of the procedure and did not obtain a clear understanding of that intent before manipulating the reactor controls. Further these individuals did not apply their training or experience when operating the reactor controls during the last minutes of the reactor shutdown. The operating crew clearly demonstrated a failure to apply their knowledge to operation of the reactor. Operations training did not reinforce 4

fundamentals at a level that assured application of this knowledge by certain operators during plant operations.

It is the Team's conclusion that there is a clear line of accountability for the three licensed individuals on shift during the reactor shutdown. The Team recognizes that had these individuals properly exercised their responsibilities, that alone could have precluded the event. However, it should also be clear that a line of accountability exists for line management, for it is management that is responsible for planning and creating the emironment in the control room where licensed operators can be successful. Had management better exercised its authority, certain conditions contributing to this event might have been precluded. On the day of the event, conditions in the Main Control Room were such that they distracted supenision from and interfered with communications regarding the most important piece of equipment on site - the reactor. Furthermore, tolerance of these conditions in the control room appeared to reinforce the crew's acceptance of low standards for performance.

The control room crew demonstrated weaknesses in teamwork. Additionally, the Investigation Team determined the crew performed as individuals and did not support cach other during performance of critical plant evolutions, and therefore individual crew members were not helping other crew members to prevent errors. Support personnel brought in to provide technical insight with reactor control, specifically the Qualified Nuclear Engineers, were not part of the control room crew. Additionally, the QNEs have not been taught to assertively intercede with operations supenision when they question the condition of the reactor.

Finally, the Investigation Team found other conditions that could have reduced the likelihood of this event. Those include accepting only timely and comprehensive corrective actions in response to Station and industry events that fully address the causes of events. For the issues influencing this particular event, the Team found both Zion specific and industry experience that was relevant. If corrective actions in response to precursor events at Zion Station and in the industry had been effective, such actions could have served to correct some underlying weaknesses identified through this event. Additionally, the SQV/ISEG organization should be encouraged to be more intrusive and therefore should have additional opportunitics to influence constructive management actions for improved performance.

21

..._ _ __.._ _ _ _ _ _.. _ _. _ _ _._ _ _. - _ _..m._ _. _.. _ _ _. _.

i Zion LevelIinvestigation Report l

i The analysis of causal factors in this report are intended to assist Zion Station and Comed's Nuclear Operating Division in its efforts to improve safety and operations. Accordingly, the findings and recommendations are based upon the best practices that presently exist in the nuclear industry, rather than i

minimal or acceptable operations standards. The Team has not assessed or reached any conclusions concerning the prudence of Comed's actions as these analyscs of causd factors were developed in hindsight, after the event had occurred. In summary, this report focuses On worthwhile needs for -

improvement in Station operation and does not describe or explain the good practices or laudable conduct j

that was demonstrated by sonic operations and plant support personnel during and subsequent to this event as this report's aim is to identify areas for Station improvement.

The Investigation Team recognizes that there is a certain amount of subjectivity in the identification of causal factors from reported behaviors and their effects. Readers should recognize that there are multipic ways to prevent undesirable results and that the reader's own professional opinion deserves consideration comparable to that of our Team. The Investigation Team is confident, however, that its analysis of causal.

i factors are well grounded, and if corrective actions are conscientiously implemented they will reduce the probability and severity of future similar and related events.

The Team hopes that its work will provide the basis for case study exercises that will help others identify potential causal factors before comparable consequential events. If this is so, the Team will have succeeded.

l l

I I

i 22

1 Zion LevelIInvestigation Report RECOMMENDATIONS l

l The Independent Safety Assessment Team (ISAT) has very recently produced its critical assessment of Zion Station. That report has identified a number of causal factors that will form the basis for corrective

{

actions and an overall improvement plan to support unit restart. As the ISAT report is written at a broad l

level of detail, the following listing is provided to ensure that the Station consider including the below-listed specific recommendations with those broader corrective actions now under development:

The roic of the QNE as an integral support for the control room crew during significant reactivity changes needs to be reviewed.

The reactivity management Zion Administrative Procedure and Operations Policy should be re-e examined to ensure that clear, consistent guidance is provided and that unrealistic expectations are not being imposed.

The expectation for three way communications being taught in ZIoPR training should be reviewcd to ensure appropriateness during transient conditions. During this event, this became a distraction and led to a degradation of teamwork among the crew at the control panels when a number of expected

{

alarms were communicated.

The methodology for tracking of LCOs in the control room should be improved to ensure awareness e

of the operating crew for all LCOs in efTect on the Unit and to ensure that the time of LCO entry is recorded when equipment is made inoperable for periodic or surveillance testing.

The process for developing " night orders" for the operating shift should be re-reviewed given the e

climination of the Operating Engineer position to ensure adequate content and proper direction.

Several operator workarounds were evident during the course of this event that were not included in the operator workaround list. The threshold for determining and documenting a workaround requires attention.

j l

Weaknesses were identified in the Temporary Procedure process that allow more than one change to

=

exist to a procedure at a time and also to allow changes to exist for extended periods of time that should be addressed. (See Appendix D, AdditionalIssues, for more details about this issue.)

Unitization issues complicated the replacement of a vacancy on shift to the extent that three turnovers e

were required until a Secondary NSO was finally in place. This and similar possible distractions from control room dutics should be resolved in a manner that minimizes the impact on operations.

The control room crew became overloaded prior to the event with emerging tasks, determination of e

testing of the pump, preplanning of the shutdown, and other issues. This planning and coordination burden should be removed from the control room crew to free them to execute the plan.

The fundamental knowledge weaknesses identified in this event require changes to be made to the licensed operator requalification program to include not only reactor fundamentals but also I

simulator practice of normal shutdowns and operation at or near the Point of Adding Heat.

Oversight expectations for Main Control Room demeanor and allocation of operator work load should be emphasized. Such clear oversight expectations then need to be enforced.

More effective ISEG and SQV oversight needs to be established.

I 23

Zion LevelIinvestigation Report Pump start criteria should be added to the Containment Spray Pump performance test (PT 6C).

e Additionally, a calibrated stopwatch should be required by procedure and used during surveillances which time events that establish equipment operability. (See Appendix D, Additional Issues, for more details about this issue.)

The two actions directed by GOP-4 in step 5.21.f should be divided into two distinct steps for human factors reasons.

Procedure adherence needs to be revisited in terms of standards, guidance and oversight. This applies e

for actions related to the unauthorized reactor start-up and deficiencies noted in the Containment Spray Pump performance test.

Additionally, the Team reviewed existing requirements for Fitness For Duty testing following an cvent of this nature. The Director of Corporate Nuclear Security has committed to review the threshold at u hich testing would be a required consideration, as directed by Nuclear Security Guideline 207. (See Appendix D, Additional Issues, for more details about this issue.)

As is noted in the Cause and Effect Analysis, Appendix B to this report, the full scope of the problems identified in this report, their causes and efTects should be considered when developing and implementing the corrective actions. If the symptoms are addressed by corrective actions, but the causes not addressed, the symptoms will recur.

On a Division-wide level, the Investigation Team also recommends that this event be presented at the l

other Comed sites to further the understanding of the issues and to help the other sites to internalize the lessons to be learned from this experience. Additionally, it is recommended that a case study of this event be developed for inclusion in the Hop Howlett Coursc presently being conducted for Operations personnel.

24

l Zion LevelIhivestigation Report KEY LESSONS TO BE1 EARNED l

1 Licensed operators' understanding ofreactorfundamentals needs to be improved. This event e

disclosed significant knowledge weaknesses among certain licensed reactor operators and senior reactor operators in basic principles of reactor theory that were unexpected. It is the Team's view in i

hindsight that Licensed Operator Requalification training needs to improve its coverage and reinforcement of fundamentals that were originally included in initial licensing. Further, simulator training should include evolutions such as controlling the reactor at the point of adding heat.

Organizations " hear" management 's actions louder than their words and try to deliver what they e

think management wants. The presence of senior management in the control room was perceived as

.,i condoning the priority that had been placed on the repair of the containment spray pump and as i

implicit approval of the actions being taken. The Site Vice President's caution that "the priority was safe shutdown of the Unit and not to allow the testing of the containment spray pump to get in the 4

way" was not adequately heeded by everyone nor were the actions undenvay to return the pump to senice challenged by any of those overseeing the control room actisity.

Any control room entry (including Management overviews or tours) must be conducted in a business-like manner and have a specife purpose. Managers in the control room must understand their

]

responsibility to enforce rigorous standards and procedural adherence. Senior Managers cannot simply become bystanders. Therefore, if managers are present, they must be prepared to proside active oversight and exercise their accountability.

The number ofpersonnelin the control room, especially during startups, shutdowns, or other sigmpcant evolutions, should be minimized to promote the proper environment. (This includes i

j individuals with " unlimited access. ") Limiting the control room attendance to those with a need to l

be there would have reduced the background noise level and imparted a formality to the room. Those with " unlimited access" should use that privilege judiciously and recognize that multiple layers of oversight usually adds little value and sometimes may intimidate someone from taking an action they othenvise might have taken if they felt total responsibility.

7here should be clear criteriafor any limiting conditionfor operation at which point a shutdown will e

be initiated including a determination ofthe point where the shutdown will not be interruptedfor any reason. This event was driven by the belief that the pump would be returned to senice and the shutdown could be averted. This thinking resulted in lack of adequate preplanning for the shutdown, 2

waiting to begin the shutdown until well into the four hour clock and was still evident with only miuutes remaining when the order was given to trip the turbine but keep the reactor critical. More clear and conservative decision points should have been established and communicated during the briefing.

l The role ofthe Quahfed Nuclear Engineer aspart ofthe control room crewfor planned reactivity e

?

changes needs to be re-evaluated to integrate the Quah)ed Nuclear Engineer 's knowledge into the l

command and control ofthe crew. Comed has had events in which the NSO and Unit Supenisor had been overly dependent on the QNE and had, in essence, relinquished their responsibility to him.

Zion appears to have overcompensated for this concern and has relegated the role of the QNE to that of an observer. The QNE must have a place on the " team", be included in communications, and be in i

a posit on to contribute his knowledge of the core and its response. Further, although this evolution began as a shutdown which would typically not have required QNE involvement, it was clear from the beginning that there was an expectation of return to power. As such, the QNE should have been fully engaged in monitoring the core and not consumed with RPI alarms, etc. which served to distract i

j him.

1 25

Zion LevelIhnesligation Report 7he Operatsons organi:ation (and its interface with the rest ofthe Station) must shield ihe control e

roomfrom distractions and requests that can be handled otherwise. Numcrous requests were made of the Unit Supervisor in the first four hours of his shift that prevented him from preparing for the shutdown. Either the presious shift or non-shift operations support personnel should have researched the Containment Spray Pump testing and coordinated the preplanning for the shutdown. The other Unit Supenisor could have accommodated the common senices that needed attention (e.g., radwaste discharge). Numerous calls were made to the control room for a status of the Containment Spray Pump. The replacement of the absent Secondary NSO required multiple turnovers and became another distraction for the Unit Supervisor. Additionally, the organization appeared insensitive to the fact that this was the first day back from regular days off for this crew and they received no help coming up to speed.

The extent to which the Zion organi:ation experiencedpersonnel changes over the pastfew months e

appeared to have had a significant impact on the organi:ation 's ability to effectivelyfunction and communicate with each other. This is based on the Team's view of the cvent, perceptions while conducting the investigation, side discussions with site personnel, and other formal and informal inputs. It is appropriate to recognize that, for the Operations organization, new and long-tenured Zion managers meet at the juncture of the Unit Plant Manager and the Unit Operations Manager.

Further, the Unit Operations Managers on both Units, although they have been at Zion for some time, are both new to Operations and have not come up through the Operations ranks. As a result, they

]

may not be well respected within the Operations Department as having an adequate understanding of

)

the issues the department is undergoing. When any organization is undergoing dynamic change, additional emphasis on teambuilding, teamwork, proper communications, and building effective working relationships under the new conditions must be a part of the process. Change management considerations contributed to probicms with communications about important issues in this event.

l 26

App:ndix A Zion Level I Investigation Team Charter, Team Membership and Investigation Observers The Zion Level I Investigation Team's Charter is to:

Perform an independent resiew and investigation of the reactivity management event that occurred during operations within a unit shutdown action statement on Friday, February 21,1997.

In the course of pursuing this action statement control rods were inserted approximately 180 steps from the critical condition and immediately thereafter the rods were withdrawn continuously approximately 80 steps from the then existing suberitical condition. The focus of the resiew and investigation shall be on the level of knowledge / qualifications of the operators then on watch and all levels of supenision engaged in the event to understand and recognize the event. If knowledge deficiencies are identified the Team should review the training program (limited to the scope of the identified deficiencies) including the ability of the instructional staff. To the extent that the operators and supenisors were able to understand and recognize the event, the Team should determine why the operators and supemsors were unable to apply their knowledge, communicate the adverse condition and permit command, control or other management initiatives to occur in a timely manner.

Based upon its conclusions from resiew of the crew and supenisors engaged in the event, the Team should investigate the degree to which the same or similar conditions exist in other operating crews and supenision.

The Team shall produce a root cause evaluation of knowledge deficiencies which it determines to be extensive. To the extent that the Team identifics behavior inconsistent with actions intended by senior Station management to shutdown Unit 1, a root cause analysis of that behavior shall be provided. In addition, the Team shall determine the potential safety consequences of the event.

The Team should note other conditions uncovered in its review that are adverse to quality operations but beyond the areas of focus. The Site Vice President, Zion Station, will direct the Team to conduct further review of any conditionno identified as he deems appropriate.

The core members of the Investigation Team and their respective Team responsibilitics are:

Member Team Responsibility Station / Oreanization Jack Brons Investigation Director NOD / VP Nuclear Support Kevin G. Bartes Team Leader Braidwood / Office of SVP Joe Eenigenburg Assistant Team Leader NOD / Plant Support Mgr.

Duane A. Karjala Root Cause Analyst NOD / Generating Support Steven P. Barrett Team. Member Dresden / Oflice of SVP Douglas E. Cooper Team Member NOD Corrective Actions Mgr.

James L. Grzemski Team Member Braidwood / Root Cause Team Hak-Soo Kim Team Member NOD / Nuclear Fuel Senices Michael J. Prospero Team Member Braidwood / Op. Shift Mgr.

Daniel Stewart Team Member Zion / SQV -ISEG Jerry E. Ballard Subject Matter Expert Zion / System Engineering Peter C. LeBlond Subject Matter Expert NOD / Reg. Process Chief Eng.

Daniel R. Redden Subject Matter Expert NOD / Nuclear Fuel Senices George E. Rowe Jr.

Subject Matter Expert Zion / Root Cause Team Observers Titic/Denartment Orcani7ation Bob Ciminel Senior Project Manager, Events INPO Analysis & Info. Exchange Louis Cortopassi Senior Evaluator, Training INPO Support 27

Appendix B Cause and Effect Analysis The Investigation Team first developed the chronology of the even. The Team determined that there were significant similarities between the errors initially identified during the Zion Unit I shutdown and events captured in an INPO SOER. The Team then sorted the preliminary causes, causal factors and effects identified in the chronology into the experience report topical areas listed. This list was continuously refined during the investigation. The Investigation Team's goal was to identify the significant causal factors and link them in an appropriate cause and effect listing. The original list was refined and the grouping was changed throughout the process. Additionally, the Team found that some issues should be treated within more than one group.

The causal factor problems and contributors identified during the investigation into the event were j

grouped into issues, each with subjective characteristics identified by the Team. The description of those characteristics is embodied in the referenced pages of the analysis of this report, as follows:

Management: See pages 16 to 17 (Management Oversight).

Planning (& Briefing): See pages 10 to 1I (Planning & Briefing).

Control Room Demeanor and Distractions: See pages 17 to 18.

Control Room Crew Teamwork: See page 19.

I Production Culturr: See page 16.

Communications: See pages 11 to 14.

1 Fundamental Knowledge / Training: See pages 14 to 15.

l Corrective Actions: See pages 19 to 20.

i Nuclear Oversight: Sec page 20.

These issues were analyzed to determine w hether there were cause and effect relationships among them.

Some of the issues were not included in the analysis because they were not direct contributors to the event (e.g., Corrective Actions, and Nuclear Oversight). Each of the issues was placed into a box on the diagnostic chart. The boxes were grouped by function into categories and category names were selected to best fit the issues. The Team then examined the relationship between cach pair of boxes to determine if

one was a cause and the other an efTect. Artcws were drawn from the cause to the effect.

i' The simplified results are desctd in Figure 1, Diagnostic Chan. The causes are those boxes with the fewest arrows going into them and the most arrows going out. The effects (or symptoms) are those boxes with the fewest arrows going out and the most going in.

The analysis was then further simplified in the form of a theme chart (Figme 2). The theme chart places the causes at the top and the effects at the bottom, and can be followed from symptoms to causes (bottom to top)as follows:

Communications (02) challenges were caused by control room crew teamwork (C2), control room demeanor and distractions (Cl), and production culture (01).

Control room crew teamwork (C2) was caused by control room demeanor and distractions (Cl),

and planning (M2).

2t

i Appendix B 4

Cause and Effect analysis e

I Control room demeanor and distractions (Cl) were caused by planning (M2), a production culture (01), and management (MI).

}

(

Planning (M2) was caused by management (MI). Production culture (01) is also caused by

?

management (MI).

1

.This analysis should be considered when developing and implementing the corrective actions. If the l

l symptoms at the bottom of the theme chart (Figure 2) are addressed by corrective actions, but the causes at l

the top of the chart are not addressed, the symptoms will recur. For example, corrective actions taken to improve control room crew teamwork (C2) may be only temporarily effective if actions are not also implemented to correct control room demeanor and distractions (Cl); and so on up the chart.

I I

Fundamental knowledge / training (II) behaviors were not addressed in the theme chart. It is clearly recognized that fundamental knowledge / training weaknesses have a causal link to both control room 1

teamwork and communications, however, the Investigation Team did not determine that one of the other j _'

causal factors directly lead to or was a symptom of these fundamental knowledge / training weaknesses.

i 4

2

'l I

1 4

t if 5

29

Appendix B Cause and Effect Analysis Figure 1 DIAGNOSHCCHARf l

MHLaut Catrol Roarn Ggaunitional Individtal OdEViT BdEvia schivia ediavia I

T Mll C1l 01l 11l Rhwiui y

Gutrd Rxxn 4

lhxistian Fudrintal 11rrunraxi Ohne Krxmkx4/Trairy D.tratkin Jf b

al liarrity j

II lf lf aI eI

$1, c=uimie=

Teanuk k

30

l Appendix B Cause and Effect Analysis Figure 2 THEME CHART: CAUSE AND EFFECT Mt Management If If 1

M2 or Planning Production Culture

)

1f Cl ContmlRoom Demeanorand Distractions l

C2 Control Room Crew Teamwork 1f 1

c2 Communications c

+

i

-1 l

l 31

__ =

Appendix C Common Causal and Contributing Factors of the ISAT and Level I Investigation The Investigation Team performed a review of the causal and contributing factors developed by the ISAT, in its report dated February 18,1997, and compared those with the list of significant issues and potential causes for the errors during the February 21,1997 shutdown. The Investigation Team found that all of the major issues identified by the Investigation Team were previously identified by the ISAT. Listed below in bold are the causal and contributing factors previously identified by the ISAT that the Investigation Team determined to be common to those observed in the shutdown. Below each section of ISAT causal factors are the major issues identified by the Investigative Team, which contain specific examples of the general weaknesses and causal factors.

Management and Organization Commitment to Excellence a) Cause - Comed has not consistently maintained focus on:

1.

the vision of world class performance 2.

a commitment to the principle that strong economic performance must be drisen by excellence in nuclear operations, and e

and uncompromising safety e

b) Fundamental and contributing causes 1.

The organization has not continually pursued a safety culture ahead of production and budgets; and 2.

production and budget incentives appear to drive the organization.

(The Level 1 Investigation Team identified similar factors, identified in the Team's report as:

=> Management Oversight.)

l Leadership a) Cause - Senior management has not consistently provided the leadership to achieve excellence in nuclear operations and safety b) Contributing causes 1.

Leadership has not always fostered an environment that promoted:

high standards, and e

conservative decision making.

e 2.

Training has not been continuously evaluated and updated.

(The Level 1 Investigation Team identified similar factors w hich are identified in the Team's report as:

=> Management Oversight

=> Planning and Briefs

=> Communications

=> Fundamental Knowledge / Training Deficiency

=> Organizational Behaviors)

Standards a) Cause -

1.

Senior NOD and Zion management have not established consistently high standards of performance 2.

standards accepted result in a tolerance of deficient conditions and non-conservative decisions b) Contributing causes I

32

Appendix C Common Causal and Contributing Factors of the j

ISAT and Level I Investigation 1

i 1.

Corrective actions, to resolve important process, materiel condition and 1

configuration management issues, at times have been:

slow, e

narrowly focused, e

deferred, and i

e incorrectly prioritized.

e 2.

Engineering has not routinely been considered as an essential function for the safe and irliable operation of the Station.

3.

Operations has not fully assumed primary leadership in assuring the safe operation of Zion.

(The Level 1 Investigation Team identified similar factors, identified in the Team's report as:

=> Management Oversight

=> Corrective Actions From Previous Events / Industry Experienceffrends.)

Management Skills a) Cause -

1.

The nuclear organization did not have the required management skills:

to improve substandard performance,

)

e to monitor and continue improvement efforts, and e

to implement sound oversight programs.

e 2.

Senior management did not have a good understanding of the significance and depth ofissues at Zion.

b) Contributing causes 1.

Nuclear oversight organizations:

have not effectively evaluated the available information on plant e

performance; and have not been forceful in influencing constructive management actions for e

improved performance.

2.

Lessons learned were frequently not communicated, implemented and monitored within the nuclear organization.

3.

Numerous personnel changes have cortributed to instability and distrust, w hich has hindered upward communication and delayed resolution ofimportant issues.

As a result, personnel changes have often not resulted in performance improvement.

4.

Change management has been more like a trial and error process rather than a process w hich is planned, implemented, evaluated and adjusted, as necessary.

5.

There has been a lack of understanding of what it takes to effect change, including the buy in of personnel.

(The Level 1 Investigation Team identified similar factors, identified in the Team's report as:

=> Management Oversight

=> Organizational Behaviors

=> Corrective Actions From Previous Events / industry Experience / Trends

=> SQV/ISEG Oversight.)

Operating and Training Conservative Operating Philosophy -

a) Lack of appreciation for conservative decision making (CDM) and role in Defense in Depth.

33

1 Appendix C Common Causal and Contributing Factors of the ISAT and Level I Investigation i

b) Operations does not act as a standards bearer, c) Causal factors:

1.

no clear definition for CDM / no role model, 2.

production and schedule overriding consideration day to day decisions, and 3.

simulator not used to reinforce CDM.

4 t

j (The Level 1 Investigation Team identified similar factors, identified in the Team's report as:

=> ManagementOversight

=> Planning and Briefings

=> Fundamental Knowledge / Training)

Tolerant of Deficient Conditions:

a) affected plant performance, b) operators have not taken a lead role in improving performance, and c) standards too low - not consistent with excellence philosophy.

I (The Level 1 Investigation Team identified similar factors, identified in the Team's report as:

=> Management Oversight j

=> Organizational Behaviors i

=> Corrective Actions From Previous Events / Industry Experiencerfrends)

Line Mgmt. ownership of training a) not utilized to:

1.

deliver / reinforce expectations.

l 2.

communicate standards, 3.

prepare operators for change, and 4.

drive change to improve pedormance.

(The Level 1 Investigation Team identified similar factors, identified in the Team's report as:

=> Management Oversight

=> Fundamental Knowledge / Training

=> Organizational Behaviors) l Operations Human Pedormance a) Numerous occasions where human performance has been deficient.

b) Operator human performance is a significant issue at Zion.

c) Management has accepted low standards.

d) Causal factors:

1.

expectations are not clear and lead to confusing standards on human performance; j

2.

long-standing reluctance to confmnt substandard hu nan performance - previous j

management has indirectly promoted a culture ofinaction on human pedormance issues, and 3.

lack of feedback and coaching on human pedormance issues as a result of a lack of supenisory skills, and i

e a lack of Mgmt. support for tough decisions that address human performance e

(The Level 1 Imtstigation Team identified similar factors, identified in the Team's report as:

l

=> Management Oversight

=> Planning and Briefings

=> Communications 34

i Appendix C Common Causal and Contributing Factors of the ISAT and Level I Investigation Fundamental Knowledge / Training j

=>

=:> Organizational Behaviors Corrective Actions From Previous Events / Industry Experience / Trends) o f

Operational Excellence a) Few individuals did a reasonable job of demonstrating attributes of operational excellence.

b) Causal factors:

1.

Individuals did not understand that the best-pe forming plants were invariably the most cost competitive; and i

2.

operations management does not communicate consistent expectations as to w hat constitutes excellence.

(The Level 1 Investigation Team identified similar factors, identified in the Team's report as:

=> Management Oversight

=> Communications Fundamental Knowledge / Training

=>

=> Events / industry Experienceffrends) 1 4

9 1

5 4

f i

i 35

Appendix D Additional Issues The following is a description of observations that, while not central to either the Investigation Team's Charter or the reactivity management issues, were noteworthy issues to be shared for Station review and possible corrective action.

IC Containment Sprav Pump Testine 2/12/97 - 2/21/97 Ilackground in January,1996, the NRC issued Violation 295(304)/93023-01 for inadequate tests performed on the IC CS pump. The NRC identified that,

"..on November 24,1993, the procedure chosen to monitor the degraded IC containment spray pump starting times (PT-6C-ST " Containment Spray C Pump System Tests and Checks") did not contain any requirements or acceptance criteria to evaluate the acceptability of starting time delays. Furthermore, again as identified on November 24,1993, the test results for the July 28,1993 test of the IC contair. ment spray pump, were not evaluated to ensure that starting time requirements were met before declaring the pump operable."

Corrective actions implemented to prevent recurrence for this event included revising the ZODM to include appropriate guidance regarding diesel driven CS pump start sequencing, ensuring that the Zion CS syston Design Basis Document (DBD) (which was in draft form as of 2/18/94) would include additional information un the starting sequence and times for the CS dicscis, and a resiew of a sample of accident analyses to ensui? that nominal plant data (pump starting times etc.) are coruistent with inputs utilized in the accident analysis had to assess the availability of the data for use by plant personnel.

In addition, the Station's response to the NRC also states that: "The System Engineering personnel and members of the On-Site Review committees involved in the IC CS diesel degraded starting issue were counseled regarding the importance of utilizing a questioning attitude, and a rigorous approach in preparation for, and execution of, all On-Site Resicw (OSR) activitics. Particular emphasis was placed on the potential consequences of an inadequate OSR whenever operability issues or evaluation of degraded equipment is involved." The document also states that consideration will be given for any OSR issue if a need exists for a knowledgeable design engineer to participate.

Observations The ZODM was changed and (as of 2/12/97') includes as one of the listed failures applicable to the ZODM cvaluation,

" Pump fails to achieve full running speed within 5 seconds during performance of any PT-6C series procedure."

The ZODM also stated,

" During the performance of any PT-6C scrics procedure, the pump shall be demonstrated operable ifit achieves full running speed within 5 seconds. Any start time greater than 5 seconds shall be subject to System Engineering resiew to determine operability. The Zion Containment Pressure Analysis of record shall be used in the evaluation."

Although the ZODM was changed, the CS performance test procedure PT-6C-ST was iever changed to reflect this acceptance criteria. The CS performance test procedure PT-6C-ST-RT does contain instructions for the taking of response time data; however, it also does not contain any acceptance criteria fci the CS pump start time.

The reference to Irr-6C series was changed to PT-6C-ST-RT on 2/13/97.

I 36 1

l

1 i

Appendix D Additional Issues During a resiew of the Design Basis Document, no specific pump start criteria was foimd. The CS-DBD is l

currently resision A, dated 8/10/93.

On 2/12/97, during the performance of N-6C-ST, operations identified a delay of approximately 5 seconds during the start of the IC Containment Spray (CS) pump. Operations determined that the pump attempted to start from battery bank #2, then automatically ratcheted over to battery bank #1 as designed. The CS pump started normally on battery bank #1. Operations determined that battery bank #2 had apparently discharged during the preceding one day maintenance outage on the IC CS pump. The pump's stating circuitry is aligned to one ofits two battery banks. Upon sensing an under voltage condition, the circuits automatically transfer to the remaining battery bank and locks out the degraded battery bank. The lock out can be manually reset using a reset push-button. In this cvent, the starting circuit apparently sensed a degraded voltage condition on the #2 battery bank and automatically transferred to the #1 battery bank. This automatic evolution took approximately 5 seconds to accomplish.

Following this pump run, the operators apparently reset the locked out battery bank because a second start attempt had similar results. The PT-6C procedures do not provide guidance on the purpose of the reset mechanism. Based on discussion with the System Engineer, the procedures had in past resisions directed the operators to depress the reset button following a pump run. The action by the operator to reset the starting circuit following the slow start would imply a basic lack of knowledge regarding the purpose of the reset and the consequences of their action.

(The 5 second delay times were only approxanate, no stopwatch or recorder was used.)

Following discussion with the CS System Engineer regarding the observed start delays and the ZODM requirements, the Shift Engineer declared the IC CS pump operable as all N-6C-ST acceptance criteria was satisfied, and closed out the M-14 (Inoperable Equipment Surveillance Tests procedure) that was opened at the start of the maintenance outage (#97000103). The acceptable starting time was apparently determined from the moment that the starting circuitry switched to the #1 battery bank. The #2 battery bank was declared inoperable and entered on the Degraded Equipment Log. Operating initiated a PIF to identify some OOS problems, and in particular, the problem with the CS pump Out of Senice that de-energized the pump's battery chargers. No 1C CS pump operability concerns were identified on the PIF.

On 2/13/97, the system engineer wrote a procedure change request to ZODM-CS to change the referenced procedure series from PT-6C to PT-6C-ST-RT, as procedurc N-6C-ST-RT is the only procedure where pump start times are taken. (A resiew of procedure PT-6C-ST-RT verified that a recorder is used to obtain start times for response timing; however, the pump start time is not recorded in the procedure, and there is no acceptance criteria for the pump start time. The start time data is normally used in the Response Time Testing procedures.)

On 2/14/97, the system engineer determined that an operability question may exist due to the possible 5 second delay if #2 battery bank was selected by the starting circuitry as the primary battery bank. This operability assessment, which was completed on the same day, determined that the IC CS pump was operable based upon presious response time testing that showed a >l5 second margin. No engineering calculations were included in the operability assessment document, nor were any specific calculations referenced. Step 4.b stated that an operability issue did not exist. The operability assessment was resiewed by a System Engineering Supenisor and a Shift Enginect, but the assessment did not receive on-site resiew. The assessment was not resiewed by a design engineer. Discussion with the System Engineer determined that the referenced >l5 second margin came from an Engineering calculation (22S-B-006M-080) performed for an 8.5 second start of the 2C CS pump on 12/3/96. The LOCA analysis calculation indicated a margin of 18.3 seconds, and the MSLB analysis indicated a margin of 5.6 seconds. The engineer advised that the >l5 second margin was used based on his engineering judgment that the LOCA was the most limiting condition.

On 2/16/97, the Engineering department determined that a potential operability issue did exist and changed the operability assessment. An Appendix B was initiated as required and identified the degraded battery bank B and the required action plan to return it to senice.

37

Appendix D Additional Issues On 2/19/97, the IC CS pump experienced a slow start (approximately 62 seconds). The failure was identified to be the result of fuct oil problems. Following maintenance, the pump was restarted on 2/21/97 and experienced a slow start time of approximately 20 seconds. This failure was determined to be the result of problems in the electrical starting circuit relays or solenoids. Following maintenance, the pump was started, on 2/21/97, per procedure PT-6C-ST-RT, to verify that the maintenance work performed had repaired the slow start time, and to restore the pump to operability.

Pretcquisite 3.12 of FT-6C-ST-RT calls for a calibrated data recorder with 2 channels. Instruction step 5.0.1 requires the logging of the Data Recorder's Identification number and the Cal / Cert Due Date. This step was marked N/A. The procedure indicates that a recorder was used in the procedure for the start of the pump on the #2 battery. The procedure requires two pump starts, one from each battery bank, but the recorder is not required for the second pump start. Step 5.1.33 requires listing the date, chart speed, equipment name and the system enginecr's initials on the chart paper; this step was marked N/A and initialed. Step 5.1.34 requires that the recorder data be retained for analysis; this step was also marked N/A and initialed. A comment on the completed procedure's cover sheet (page 3) states that acceptance criteria 3,5,6, and 8 were met in the PT with no pump start delay. The actual pump start time, as required for operability in the ZODM, is not recorded in this procedure.

Discussion with the System Engineer revealed that an electronic recorder (which is currently used for all response time testing) was used during this test. The System Engineer advised that the start times were both < 5 seconds.

Conclusions The corrective actions initiated as a result of the January 1994 NOV were ineffective at preventing a recurrence of similar problems.

The C train CS pumps are still being tested without start time criteria in the performance tests. The 1C pump was declared operable on 2/12/97 and 2/21/97 without adequate documentation, and/or engineering resiew as required in the ZODM. (The System Engineer initiated a PIF regarding the lack of start time acceptance criteria in the PT on 3/7/97.)

The 5 second stan times entered in the ZODM for the C train CS pumps was apparently based on acceptable sequence start times for electrically driven pumps, and does not reflect aaual 9perating experiences or design conditions.

The operators resetting the starting circuit lockout following the slow start on 2/12/97 indicates a lack of procedural guidance, and also indicates a lack of operator knowledge regarding the purpose of the reset mechanism and the consequences of their actions.

The CS-DBD does not contain specific CS pump start information as was identified in the 1994 response to the NRC NOV.

The operability assessment for the 2/12/97 IC CS pump slow start was not On-Site Resiewed and no resiew was performed by Design Engineering.

The test recorder requirements specified in IYT-6C-ST-RT was not adhered to during the 1C CS pump performance tests on 2/21/97. A recorder different from the indicated recorder type was used during the test, and no Cal / Cert information was documented for the recorder which was used. No pump start time data was recorded in the procedure verifying that the 5 second start time requirement that is specified in the ZODM was met.

Even though start time was critical to CS Pump being declared operable, based on criteria in the ZODM, no stopwatch was used to time the pump start.

Note: This issue was documented and turned over to Zion Station in Problem Identification Form (PIF) #

971235, dated March 11,1997.

38

I App 2ndix D Additional Issues Responsibility for Enterine and Exitine Limitine Conditions for Operations The responsibility for entering LCOs is the Operating Crews. LCO entries must be clearly defined and should be logged in the Unit NSOs and Shift Engineer's Logbook. LCO entries should not be tracked in lYrs only without also being tracked in the Unit NSOs and Shift Engineer's Logbook. If LCOs are not tracked in the Unit NSOs and Shift Engineer's Logbook, then the Operating Crew will not clearly understand when the LCOs are entered or exited. The IC CS Pump was declared Inoperable 2/19/97 at 1040 due a failure of PT-6C-ST Testing with a 48 LCO expiring 2/21/97 at 1040. The IC CS Pump would then require the Unit to be placed in Mode 3 on 2/21/97 at 1440. On 2/21/97, at 1255, after further resiew, the Shift Engineer with the aid of SQV determined that the Unit was required to be in Mode 3 on j

2/21/97 at 1420 instead of 1440. This error was due to the alignment of the IC CS Pump for testing and may have been avoided if a log entry had been made in either the NSOs or Shift Engineer's Logbook when the Pump was realigned and thus inoperable.

The Shift Engineer was aware that the U2 Hydrogen Recombiner being inoperable affected UI. The Shift Engineer knew that he could not increase operational modes with the U2 Hydrogen Recombiner inoperable and continue operating. While this was part of his thought process when he directed the turbine trip during the shutdown, he did not verbalize this. It is not apparent, by their actions, that the 1

Unit Supervisor or the NSO were aware of this technical specification situation as they subsequently l

changed operational modes by lowering reactor power.

The final responsibility for exiting LCOs is also the operating crews. However, it is not the operating j

crews responsibility to do all the research to determine which NWRs, surveillances, OOSs, lineups, or the i

testing requirements are required to exit a LCO. The operating crew'sjob is to operate the plant in a safe and conservative manner and the crew should not be diverted to determine administrative requirements.

S/G PORV and Steam Dump Response Reauirine Control Rod Insertion followine a Turbine Trin Following a Turbine Trip per GOP-4 Section 5.0 Step 21d, the Operators are directed to " Hold #363 Rod Motion Control Switch in to minimize dumping steam and establish power at or less than the Point Of i

Adding Heat (2.5 x 10-2% IR)" per GOP-4 Section 5.0 Step 21f. The intent of the step is to establish a negative Startup Rate to establish reactor Power at or less than the Point of Adding Heat and to reduce Tm j

to prevent opening the S/G PORVs.

Operations shuts down with the Steam Dumps in the Tm Control Mode and normally approaches the point of tripping the Turbine with Tm lower than Tar. With Tm lower than Tmr the control circuit is saturated in the close direction. The S/G PORVs have a rate sensitive control circuit which will result in the S/G PORVs opening prior to the Steam Dumps on the pressure spike following the Turbine Trip.

Plant personnel are sensitive to opening the S/G PORVs as indicated in GOP-4 Section 4.0 Step 29 " Zion city officials should be notified prior to using the S/G Atmosphere relief valves for extended periods of time (872-8000). An announcement over the plant paging sy stem should be made before opening S/G Atmospheric Relief valves to prevent hearing loss to personnel around the plant site."

Control Rod Insertion should not be required to prevent the S/G PORVs from lifting following a Turbine Trip. The Steam Dumps should respond quick enough in Tm Mode or the Operating Department should operate the Steam Dumps in the Steam Pressure Mode, if appropriate, to prevent lifting the S/G PORVs following a Turbine Trip. If the Station chooses to continue to operate the Steam Dumps in the T.

Mode, then the Steam Dump Response Circuitry must be modified to allow the Steam Dumps to open prior to the S/G PORVs. Another option may be to crack open the Steam Dumps in the Steam Pressure 39

Appendix D Additional Issues Mode by reducing the setpoint prior to the Turbine Trip to prevent the S/G PORVs from opening. This technique was tried on the simulator with good results.

Consideration of FFD Testine for Event Participants No consideration was given following this event to require Fitness For Duty (FFD) testing of the event participants. Although there was no clear esidence to suggest that FFD issues were involved, it is apparent to the Investigation Team, in retrospect, that it would have been desirable to have confirmed that the Shift Engineer, Unit Supenisor and Nuclear Station Operator were fit for duty at the time of the event.

Corporate Security was contacted to understand the current Company policy on the need to perform FFD testing following a significant event or human performance issue. Corporate Nuclear Security Guideline No. 207 requires for-cause testing "upon an actual or potential substantial degradation of the level of safety of the plant if there is reasonable suspicion that the worker's behavior contributed to the event (A substantial degradation is defined as any event meeting criteria for a " Site Emergency")." The parenthetical clarification is the current Company application of the threshold for testing, based on the use of similar language in other parts of NRC regulations, while the rest of the quotation is directly out of 10 CFR 26.24.

This event did not border on a " Site Emergency" but this may be too narrow of an interpretation to cuum that for-cause testing is performed at appropriate times. This issue has been referred to Corporate Security for resolution.

1B Auxiliary Feed Water Pump start on 2/21/97 Prior to starting an Auxiliary Fced Water (AFW) pump, procedures direct operations to flush and then sample the water in the AFW Pump for magnesium concentration. The purpose of these sampling requirements is to i

minimire the introduction oflake water into the steam generators. Magnesium is used as an indicator oflake water in-leakage into the AFW Pump. The procedures require the magnesium content to be less than or equal to 20 ppb. The normal concentration of magnesium in condensate is less than 2 ppb, and is around 11 ppm in lake j

water. These sampling requirements have been in the procedures for at least 4 years.

At least six PIFs have been written during the last two years identify recurring high values of magnesium in these samples and resulting delays in running the pumps or in performing pump tests. An operability concern was recently identified regarding reduced pump flows that would result if a pump was required for operation with an ongoing pump flush through the open pump vent. Engineering determined that although the pumps currently have sufTicient capacity to meet design flow requirements with the vent open, pump degradation over time could result in the pumps not meeting design P sw requirements in this configuration. The System Engineering and Chemistry departments are currently attempting to determine the source of the magnesium and are considering possible lake water intrusion during stroke tests of the senice water to AFW isolation valves, or leeching of magnesium from system components. The Senice Water System is isolated from the AFW system by two isolation valves in each piping connection. All of these isolation valves are normally closed. A "tell tale" drain valve is located between each pair of these isolation valves and is normally open. These normally open drain valves prevent senice water from leaking through the isolation valves into the AFW suction lines. The isolation valves were replaced during the dual unit senice water outage in 1993, and now have little to no leakage.

Step 5.0.1 of GOP-4 directs the preparation for an AFW Pump start by directing the local operator to align the AFW Pump for startup per SOI-10B (or SOI-10C) and directing the Chemistry department to sample the AFW Pump per SOI 10B.(or SOI-10C). At 1308 on 2/21/97, the Unit Supervisor and the Shift Engineer discussed not performing 1% AFW sampling due to personnel safety concerns for the operator if the AFW Pump was started 40

Appendix D Additional Issues l

during the sampling process. At 1405, the IB Auxiliary Feed Water (AFW) pump was started in support of the j

Unit I shutdown, and in accordance with GOP4, step 5.0.17. Based on system valve alignment which prevents j

gross in-leakage of senice water into the AFW Pump intake, the conscquence of not performing the flush and sampling evolution appear very minor. The sampling could have been performed if the shutdown was planned properly, and procedure GOP-4 was reviewed prior to the shutdown evolution.

Conducting Plant Shutdown with the Feedwater Regulatine Bioass Valve Isolated Prior to the plant shutdown the decision was made to continue operation with the ID (IFW $30 A) feedwater regulating bypass valve isolated. The decision to operate in this configuration was made after a temporary repair to a steam leak was made in November 1996. This decision was made because of a i

concern by the plant staff that the repair would impair the ability of the valve to close after receiving an j

ESF signal. A 50.59 cvaluation was performed which considered the impact of this configuration. This evaluation is part of the work package. While most items were evaluated, the additional impact of operator burden was not addressed from the perspective of possible contingencies or appropriate operator training. GOP-4 permits this configuration but this is an atypical lincup w hich requires the operators to control valve position and main feed pump speed (pump discharge pressure) to control steam generatur t

level. The concern over this configuration was voiced by the operators during the post-event inteniews.

i 1

Unitization Zion has recently unitized certain portions of the plant organization. This process created new management positions which were filled by either new employees brought in from outside Commonwealth Edison or with promotions from within. In the last six months, a significant number of people have been brought in from outside Commonwealth Edison and have been promoted from within to fill key managerial positions. Other line positions created by the unitization process have been filled by promotion, temporary promotion, or remain vacant.

?

t This has resulted in:

a significant number of peopic in kcy management positions who are unfamiliar with the e

Commonwealth Edison structure and policies, an even larger number of people in unfamiliar roles, e

an entire staff unfamiliar with the " unitized" organizational structure, e

an organizational " bottleneck" created with two Operating management organizations e

coming together at the Shift Engineer, and i

e reduced population of personnel available to fill vacant shift positions as positions are filled on a unitized basis.

As a result of the unitized organization changes, the responsibilitics for some dutics are unclear, of particular note is the responsibility for Night Orders. The Investigation Team resiewed the Night Orders for the period surrounding the Unit shutdown. Those Night Orders provided only brief and vague guidance on the plan for the CS Pump, and provided no help to the on-coming day shift the day of the shutdown. When the Investigation Team inteniewed operations managers, these managers said that the Night Orders are written by a variety of people but it is not clear who has the responsibility. Furthermore, the authors of the Night Orders do not seek topical guidance from Senior Management or remain consistent with content among themselves.

41 i

Appendix D Additional Issues Temporary Procedure Channes At the time of the event, there were two temporary procedure changes outstanding against the procedure being utilized to shutdown the Unit (GOP-4). These temporary procedures did not contribute to the initiation, exacerbate the consequences of the event, or inhibit the mitigation of the event. However, the practice of allowing multiple temporary changes against a procedure, and not incorporating the changes into the procedure in a timely manner are not good practices.

Another example is ZAP 300-01, Conduct of Operations," which contains two temporary changes. The first lists no " effective date" but was approved 11/9/95. The second also lists no " effective date" and the copy provided to the Team does not have any of the required operations management signaturcs.

Notwithstanding, it was prepared on 10/2/96. Both changes affect on shift briefing requirements.

During a brief resiew of ZAP 300-01 several errors or out-of-date conditions were noted. One is particularly noteworthy.Section VII.D.5 contains a requirement for the SE or US to obtain the concurrence of the load dispatcher before making a voluntary entry into an LCO. This practice was dir, continued by direction of CNOO in September of 1996 and has recently been formalized in NOP-OP 9.

The Station Manager communicated this change to the Operating Department September 17,1996, but the procedure was not changed.

ZAP-110-02B," Temporary Procedure Change Control and Approval," requires in Section E, "1.

If the Expiration Date is left blank, then the temporary change will expire in 30 days. 2. If a calendar date is entered, the date shall be equal to or less than 6 months from the authorization date." This is foot noted as a response to commitment #295 294-90-0801. The intent of this commitment is being subverted by annotating the Expiration date with "At the next revision," resulting in some procedures with temporary changes greater than a year old against them. This practice should be corrected in a timely manner.

Roles and Ressmsibilities of the Oualified Nuclear Engineer (ONE)

Although the Reactivity Management ZAP (ZAP-300-OlB) specifics a QNE presence in the control room for the specific activities of physics tests, reactor startups, and core alterations, the ZAP does not address any QNE involvement for a reactor shutdown. There is currently no Station document which outlines the expectations for QNE communications with the Operations Crew (NSO and SRO) during Control Room evolutions. A September 1996 Operations Reactivity Manipulation Policy restricts direct communication between the QNE and NSO since it requires that the NSO receive the SRO's verbal concurrence prior to any reactivity manipulation. In addition, the latest cycle of simulator training of a physics testing scenario has reinforced that the QNE must not communicate directly to the NSO. The training establishes that the QNE can only provide technical guidance for reactivity management to a SRO who directs the NSO.

The lack of defined responsibilitics of a QNE and the current Operations communications policies have significantly diminished the QNE's role in all Main Control Room activities except those specific reactivity manipulations which procedurally require the QNE's presence. This diminished role and restricted communication was a significant contributor to the ability to detect rod movement errors. The QNE was not routinely included in the ongoing reactivity manipulation communications between the NSO and the SRO. This essentially removed the QNE from the Main Control Room team and relegated him to a distant observer role, especially when the SRO became extremely occupied with several other Main Control Room activitics. These conditions helped establish a barrier that prevented the QNE from effectively comprehending the actions of the NSO and taking appropriate intervention action.

42

Appendix D Additional Issues Maintenance And Troubleshootine of the 1C Containment Sorav Pump The Team viewed the scope of the investigation as beginning with the events occurring on 2-21-97. In doing so, we viewed the expiration of the 48 hour5.555556e-4 days 0.0133 hours 7.936508e-5 weeks 1.8264e-5 months timeclock as a "given" and did not fully probe the events surrounding its maintenance and troubleshooting to determine if inappropriate actions took place that caused the timeclock to expire. The Investigation Team probed the maintenance and troubleshooting of the IC Containment Spray Pump only to the point of understanding the sequence of events that occurred. The bulk of the input on the sequence of events was derived from the System Engineer and his notes / logs. Interviews of Electrical Maintenance, Mechanical Maintenance, Work Analysts, Work Control personnel to fully understand and evaluate their actions and inputs were not performed. The adequacy of the initial troubleshooting to discern w hether the c!cctrical problem was in txistence on 2 97, when the failure was determined to be fuel related, was not thoroughly evaluated by this Team.

Further, no autopsy or other inspection of the replaced relay and solenoids to determine their failure mode or mechanism was performed as part of this Team's activity. Additionally, the Team only performed a cursory evaluation of the failure of the IC Containment Spray Pump failure on 2-12-97 and its relation to the problems experienced on 2-19-97.

Operator Workaround List During the sequence of activities throughout this event, there were a number of material condition deficiencies that operators worked around; this created additional distractions for the operators. Only one of the issues was included in the Station's Operator Workaround list. The control rod position indication problems and resulting alarms is an issue included on the Operator Workaround list; those listed below

{

are not included on the list:

)

Steam generator power operated relief valve and steam dump response (see issue description above).

e IB Auxiliary Feedwater Pump chemistry sampling requirements (see issue description above).

e Feedwater regulating valve bypass valve temporary leak repair and isolation.

Auxiliary Boiler trip and alarms.

Feedwater heater level alarms.

Loose parts alarms.

e The Station should determine if these issues should be included on the Operator Workaround list. The Station should also determine whether the list is being adequately maintained to identify and include issues such as these that cause distractions for operators.

43

Appendix E Zion Reactivity Event Safety Significance The purpose of this section is to thoroughly evaluate both the actual and the potential core reactivity response during the February 21,1997 Unit I reactor shutdown at Zion Station. This analysis focuses on the actual or potential threat to exceed design tolerances rather than on the appropriateness of the operator's actions during the shutdown. However because the actions taken are inconsistent with the fundamental principles of conservative reactivity management, they are also addressed in the Significance section, below.

Nuclear Fuel Services (NFS) calculated the best estimate Unit 1 Cycle 15 core neutronics parameters using the ANC/ Phoenix three dimensional nuclear design computer code. This code was used to model the actual Cycle 15 core burnup conditions which existed on February 21,1997. The ANC/ Phoenix code was also used to calculate the Unit I core reactivity conditions based on the actual control rod positions and other key recorded plant parameters. These reactivity results arc discussed in the Description and the Consequences sections below. NFS then used the Unit I core neutronics parameters as input into a RETRAN02 ModOS transient computer code model of Zion in order to compare a postulated continued rod withdrawal event to the design basis Rod Withdrawal From Subcritical (RWFS) results. These comparison results are summarized in the Safety Significance section.

Description On February 21,1997 at 14:07 hours, the Unit I reactor was stabilized at about 7.25 % power (per IN-

35) in preparation for tripping the main turbine. Inunediately following the manual turbine trip, control rods were inseded to reduce the reactor power to the point of adding heat (POAH), using guidance in GOP-4. Rod insertion was halted at 14:11 hours with Control Bank D (CBD) fully inserted and Control Bank C (CBC) inserted to 104 steps. At this point the core was approximately -1050 pcm suberitical and reactor power was decreasing at the maximum negative startup rate. In order to stabilize the reactor power near the POAH per GOP-4, control rod withdrawal was then initiated at about 14:12 hours. CBC was withdrawn from 104 to 183 steps (CBD @ 5 steps), with rod motion halted at about 14:14 hours. The subsequent rod withdrawal added approximately 570 pcm positive reactivity though the core still remained suberitical by about -480 pcm. At 14:15 the Unit i reactor was tripped and reactor operation was terminated.

Conseguences The Unit I reactivity event did not result in any adverse consequences to the reactor or plant systems. The core remained subcritical throughout the event, and was still subcritical by about -480 pcm when the rod withdrawal was halted. There was no challenge to any Reactor Protection System or safety function. The fuel integrity was never challenged since the nuclear power level was below the POAH when the rods were withdrawn and it continued decreasing throughout the event as shown in Figure 1.

Sienificance The Zion Umt I reactivity event is significant and has been evaluated for two major reasons. First; the initial rod insertion resulted in a large negative reactivity addition which had unknowingly shut the reactor down into a Mode 3 condition. Second; the subsequent rod withdrawal then essentially constituted the inadvertent leaving of Mode 3, and the performance of a reactor startup outside the approved GOP-2

(" Unit Startup") procedural controls. This event is clearly in violation of the Zion Station and the Nuclear Operating Division's Reactivity Management Policy. If the rod withdrawal had continued, the reactor could have experienced a large positive startup rate and a rapid power increase w hich could have resulted in a challenge to the plant Reactor Protection and Control Systems. In addition, any rapid power increase has the potential to challenge the mechanical integrity of the fuel. Therefore, the following evaluation was performed to determine whether the continued rod withdrawal on Zion Unit I would have resulted in 44

=

Appendix E Zion Reactivity Event Safety Significance any reduction in the margin to safety as defined in the Zion Technical Specifications or the Final Safety Analysis Report.

Safety Significance Evaluation A comparative analysis has been performed for a postulated reactivity event which assumes that the operator continued to withdraw the control rods regardless of flux level or startup rate indications on February 21,1997. The safety significance of the Zion February 21,1997 reactivity manipulations was s

evaluated by comparing this postulated reactivity event to the bounding design basis Rod Withdrawal from Subcritical (RWFS) established in the Zion FSAR Section 15.4.1. The analysis results show that the postulated reactivity event was bounded by the design basis RWFS transient analyzed in the FSAR.

i i

The design basis RWFS transient assumes that some operator error or system failure causes the two highest worth banks of control rods to continuously withdraw at the maximum speed. The core is l

assumed to be at an initial flux level that is well below the POAH. The POAH is the fiux level (power) at which the nuclear heat generation rate begins to significantly increase the fuel temperature. An increase i

in fuel temperature provides a negative reactivity due to Doppler feedback which would tend to offset whatever original positive reactivity insertion had caused the initial power increase. Consequently, w hen a significant amount of positive reactivity is added below the POAH, an extremely large positive startup l

rate can be generated since the increasing flux level does not cause any offsetting negative Doppler feedback reactivity.

1 The design basis RWFS analysis assumes very conservative neutronics input parameters which are summarized in Table 1. These include a small Doppler Cocmcient value of-0.91. pcm/ F, and a very conservative positive Moderator Temperature Coeflicient value of 7.0 pcm/ F. These parameters

]

minimize the offsetting negative feedback as power increases. The design basis analysis also assumes a maximum positive reactivity addition rate of 75 pcm/sec which corresponds to the maximum design difTerential rod worth moving at the maximum rod speed (72 steps per minute). The only reactor

)

protection trip function credited in the design basis analysis is the Power Range Neutron Flux Level trip j

l (Low Setting) setpoint of 35% (25% including maximum uncertainties).

The RWFS analysis an m the core isjust critical (keff = 1.0 ) with an extremely low initial power level of 1.0 E-7 % power. ' b sitial conditions generate a conservatively large startup rate since the power must increase almost hve decades before the POAH is reached and any negative reactivity feedback begins.

The sequence of events for the design basis RWFS are listed in Table 2, while Table 3 provides a l

summary of the results. At time t = 0 seconds, positive reactivity addition begins. At t = 10.5 seconds, j

the power level has increased to the Power Range Neutron Flux Level trip (Low Setting) setpoint of 35%

The analysis assumes conservative electronics delays such that the control rods do not begin insening negative reactivity into the core until t = 12.9 seconds. During this period, the nuclear power increases to j

a peak value of 254.9 % at t = 10.7 seconds before the negative Doppler feedback offsets the power excursion. Since the fuel pellet design has an inherent thermal transport time of 2-3 seconds, the thermal power and fuel temperature lag behind the extremely rapid nuclear flux increase. The peak thermal flux of 47.6% and fuel temperature (1680 F) occur at t = 13.0 seconds.

)

To accurately evaluate the postulated reactivity event, the analysis uses the NFS calculated Zion Unit 1 Cycle 15 (ZlCIS) data listed in Table 1. The ZlCIS core has more negative reactivity feedback with a Doppler Temperature Coeflicient (DTC) of -1.5 pcm/ F and a Moderator Temperature Coeflicient (MTC) of-17.1 pcm/*F. The ZlC15 End of Life (EOL) delayed neutron fraction is Icss, which creates a 45

Appendix E Zion Reactivity Event Safety Significance slightly faster reactor period for the same reactivity insertion compared to the design basis RWFS which is analyzed at the Beginning of Life (BOL). However, this effect is more than offset and insignificant compared to the dominant negative reactivity feedback effects of EOL versus BOL conditions. In addition, the Unit I rod speed of 48 steps per minute (manual operation) and the maximum calculated differential rod worth of 12.5 pcm/ step result in a much slower reactivity insertion rate (10 pcm/sec) than the design basis RWFS transient.

The sequence of events and results summary for the postulated reactivity esent are compared in Table 2 and Table 3, respectively, with those of the design basis RWFS. The time t = 0 for the postulated event is defined as the time when the control rod withdrawal began adding positive reactivity to the ZlC15 core.

At t = 105 seconds, the positive reactivity addition made the reactor critical and flux began increasing.

Once the POAH was reached, the greater negative reactivity feedback and smaller positive reactivity insertion rate resulted in a much slower power increase compared to the design basis RWFS. The postulated event startup rate was slow enough to maintain thermal equilibrium between the nuclear flux and heat flux within the fuel pellet. The nuclear flux reached the peak value at the Power Range reactor trip setpoint of 35% power, w hich occurred at t = 260 seconds. The peak heat flux (33.7%) and fuel temperature (1430 *F ) occurred only a second later at t = 261 seconds. It should be noted that the 10%

Reactor Trip / Turbine Trip, the Intermediate Range Neutron Flux Level trip (25 %), the High Neutron Flux Rate Trip (+5% in 2 seconds), and the Pressurizer High Pressure Trip (2385 psig) were available during the postulated event, but were not credited for this evaluation.

The comparative evaluation demonstrates that the postulated Unit I rod withdrawal event generated nuclear power and heat flux results well below the design basis RWFS values and therefore, remained within the thermal design limits of the fuel.

In summary, it is the determination of NFS that the reactivity manipulations during the Zion Unit I shutdown on Febmary 21,1997 did not create any unreviewed safety auestion. and there was no reduction in the marnin to safety _as defined in the Zion Technical Soccifications or the Final Safety Analysis Report.

46

1 Appendix E Zion Reactivity Event Safety Significance Table 1. Comparison ofinput Parameters for the Safety Evaluation l

Parameters Design Basis RWFS Postulated Reactisity l

Event Moderator Temperature

+7.0

-17.184 Coefficient (pcm/F)

Doppler Temperature Coefficient

-0.91

-1.5 1

(pcm/F)

Delayed Neutron Fraction 0.0075 00516 Maximum Reactivity Insertion 75.0 10.0 Rate (pcm/sec)

Power Range Neutron Flux Level 35 35 Trip - Low Setting (%)

Table 2. Sequence of Events Times for the Safety Evaluation Event Design Basis RWFS Postulated Reactivny t = seconds Event, t = seconds Initiate Positive Reactivity 0.0 0.0 Insertion Reactor Critical, 0.0 105 keff = 1.0 l

l Reactor Trip @ 35%

10.5 260 1

Peak Nuclear Power 10.7 260

)

i Peak Fuel Average Temperature 13.0 261 Peak Heat Flux 13.0 261 Table 3. Comparison of Results for the Safety Evaluation i

Parameters Design Basis Postulated Reactivity RWFS Event Peak Nuclear Power 254.9 35

(% of nominal)

Peak Thermal Power 47.6 33.7

(% of nominal)

Peak Fuel Average Temperature 1680 1430

(*F)

Peak Fuct Centerline Temperature 2270 1980

(*F) 47 1

=

w

I

}

l r

I'

j

,j!

i ji l

6

'it l

i:

>e ) 8 E' M D.gx'm$?#

<@~

gSP w gs' 2hN2 e T'5 C

s AP wgi a

l ll j[l

\\

l j

i 6*

!.a' 6

A

)

f mcp a

(

y

)

t i

t t

v t

(

it r

7 c

e

{s l/

I 0

4 a

w e

4 I

o R

P e

r a

oC

' /

te N

A g t h

i a.

gt j

gt h

a.

6 7

8 9

0 1

2 3

4 5

6 0

0 0

0 4:

4 4:

4 4

4 4:

1 1

1 4

4 1

1 1

7 7

7 9

9 9

/

1 1

1 2

2 2

/

2 2

2 lP$

!

ML20140G527

Text

SUMMARY

Navigation menu

Search