IR 05000498/2001010

IR 05000498/2001-010, IR 05000499/2001-010, STP Nuclear Operating Company, South Texas Project Electric Generating Station, Units 1& 2, Special Team Inspection on 12/18/2001 - 12/21/2001. Corrective Actions, Event Response

ML020110200
Person / Time
Site:	South Texas
Issue date:	01/11/2002
From:	Graves D NRC/RGN-IV/DRP
To:	Cottle W South Texas
References
IR-01-010
Download: ML020110200 (19)

Text

January 11, 2002

SUBJECT:

SOUTH TEXAS PROJECT ELECTRIC GENERATING STATION - NRC SPECIAL TEAM INSPECTION REPORT 50-498/01-10; 50-499/01-10

Dear Mr. Cottle:

On December 21, 2001, the NRC completed a special team inspection at your South Texas Project Electric Generating Station, Units 1 and 2. The enclosed report documents the inspection findings which were discussed on December 20, 2001, with Mr. G. Parkey and other members of your staff.

Based on the results of the inspection, the team identified one issue of very low safety significance (Green). No violations of NRC requirements were identified.

In accordance with 10 CFR 2.790 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, will be made available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Should you have any questions concerning this inspection, we will be pleased to discuss them with you.

Sincerely,

/RA/

David N. Graves, Chief Project Branch A Division of Reactor Projects Dockets: 50-498 50-499 Licenses: NPF-76 NPF-80

STP Nuclear Operating Company-2-

Enclosure:

NRC Inspection Report 50-498/01-10; 50-499/01-10

REGION IV==

Dockets:

50-498 50-499 Licenses:

NPF-76 NPF-80 Report:

50-498/01-10 50-499/01-10 Licensee:

STP Nuclear Operating Company Facility:

South Texas Project Electric Generating Station, Units 1 and 2 Location:

FM 521 - 8 miles west of Wadsworth Wadsworth, Texas 77483 Dates:

December 18-21, 2001 Team Leader:

Inspector:

N. O'Keefe, Senior Resident Inspector W. Sifre, Reactor Inspector, Engineering Maintenance Branch Approved By:

David N. Graves, Chief, Project Branch A

SUMMARY OF FINDINGS South Texas Project Electric Generating Station, Units 1 and 2 NRC Inspection Report 50-498/01-10; 50-499/01-10 IR 05000498-01-10; IR 05000499-01-10; on 12/18-21/2001; STP Nuclear Operating Company; South Texas Project Electric Generating Station; Units 1& 2; Special Team Inspection Report.

Corrective actions, event response.

The inspection was conducted by two team members consisting of one resident inspector and a region based engineering and maintenance inspector. The inspection identified one Green issue. The significance of the issue is indicated by its color (Green, White, Yellow, Red) using IMC 0609, "Significance Determination Process."

Cornerstone: Mitigating Systems

Green. Following an event in which Train C of the Unit 1 essential cooling water system was rendered inoperable because the discharge strainer in Train 1C clogged with small fish, the licensee was slow to identify and implement actions which would have effectively prevented recurrence. Specifically, no effective barriers were identified which would have prevented an increase in the fish population in an idle train suction bay during a rapid drop in temperature in the essential cooling pond similar to that which occurred during the event. The initial corrective actions did not assure that the licensee would have effectively monitored for an increase in the fish population and did not include any specific steps to prevent a train from being rendered inoperable if an increase was detected. The licensee added actions to chlorinate idle intake bays at least daily, improve monitoring, and identify a response plan if a buildup of fish was detected. The inspectors concluded that this provided a reasonable barrier to fish population increases in the bays, even during a period of cold weather.

This issue was considered to be more than minor because it represented a potential for a repeat failure, which had a credible impact on safety, and could affect the operability, availability, reliability, and function of a train of accident mitigation equipment. This finding was determined to be of very low safety significance because two trains would remain available (Section 03).

Report Details SPECIAL INSPECTION ACTIVITIES

Inspection Scope The inspectors conducted a special inspection to better understand the circumstances surrounding the loss of function of one train of the Unit 1 essential cooling water (ECW)

system as a result of small fish plugging a strainer. The team reviewed the sequence of events, human performance in response to the event, the licensees root cause analysis and corrective actions, the licensing and design basis of the ECW system, and the risk significance of the event, including the potential for a common mode failure of other trains in one or both units. The inspectors used Inspection Procedure 93812, Special Inspection Procedure, to conduct the inspection. The team reviewed procedures, logs, and corrective action documents. Key personnel, including an expert on fish behavior, were interviewed. The inspectors observed the conditions of the essential cooling pond and each of the pump suction bays.

Special Inspection Areas 02.01 Overview and Sequence of Events On November 27, 2001, South Texas Project Nuclear Generating Station, Units 1 and 2, were operating at 100 percent reactor power. A cold front entered the region and caused the essential cooling pond temperature to drop 15
F over the following 2 days.

On the morning of December 1, the Unit 1 operators were preparing to start the ECW Bay 1C pump so that Train B could be secured for planned maintenance. The following sequence of events then occurred:

12/01/2001 8:28 a.m.

• Control room operators started the ECW Bay 1C pump.

• The reactor plant operator observed lower than normal seal water flow.

• The control room operators observed less than normal ECW flow (~7000 gpm vs 20,000 gpm normal).

• The ECW Bay 1C pump was secured based upon above indications.

• Operators and mechanical maintenance personnel suspected the discharge butterfly valve was not working properly.

• The discharge valve was stroked satisfactorily.

-2-

Live fish were observed when the discharge strainer was drained.

12/01/2001 9:08 a.m.

• Control room operators again started the ECW Bay 1C pump.

• Again, low seal water flow was observed.

• The control room received unexpected alarms for strainer differential pressure and low system flow.

• The ECW Bay 1C pump was again secured.

• Instruments were placed in the system to verify the installed instrumentation.

• The pump discharge valve was stroked while being observed with a boroscope. Live fish were observed in the vicinity of the discharge valve.

• The lube/seal water filters were drained. Fish were observed in the discharge.

• Engineers confirmed a likely strainer differential pressure concern.

12/01/2001 10:30 p.m.

• Control room operators started the ECW Bay 1C pump in the fill and vent mode.

• ECW system flow decreased during the approximately 2-minute run.

• The ECW Bay 1C pump was secured and the train was declared inoperable.

• The strainer was disassembled for inspection and cleaning.

• The discharge strainer was found to be extensively clogged with small (1-2 inch) fish.

• Approximately 2 gallons of fish were removed from the strainer during cleaning.

• ECW Intake Bay 1C was chlorinated and an air sparger was used to try and move the fish away from the pump intake.

-3-12/02/2001 12:21 a.m.

• Unit 2 control room operators ran the ECW Bay 2C pump (idle train) to assure operability.

12/03/2001 2:11 a.m.

• After reassembly, the ECW Bay 1C pump was again started.

• The operators secured the ECW Bay 1C pump again due to strainer clogging.

• The strainer was again disassembled and approximately 2 gallons of fish were again removed.

• The bay was then blocked off from the pond with a gate and approximately 135 gallons of fish were removed from the ECW Intake Bay 1C pump.

• Mechanics cleaned out ECW Intake Bay 1C and filled it with chlorine-treated firewater.

12/04/2001 10:17 p.m.

• After strainer reassembly and suction bay gate removal, the Unit 1 control room operators started the ECW Bay 1C pump to fill and vent and for postmaintenance testing.

12/4/2001 2:03 p.m.

• The ECW Bay 2C pump was started to verify operability. The licensee instituted a policy to continuously run all available ECW pumps until assurance was available that fish would not congregate in idle ECW bays.

12/05/2001 4:50 a.m.

• After a successful postmaintenance test, the ECW Bay 1C pump train was declared operable.

General Descriptions The ECW system is composed of three separate and identical trains in each unit. A deep-draft pump takes suction from an individual suction bay and supplies flow through a check valve, a motor-operated discharge valve, and a self-cleaning strainer to the loads cooled by the system. This safety-related system cools the associated train of standby diesel generator, component cooling water, and essential chilled water.

Component cooling water cools the residual heat removal system and reactor containment fan coolers. These components are required to support the emergency core cooling systems. The pump suction bays draw water from the essential cooling pond through a trash rack and traveling screens. The pump bays are arranged from north to south, 2A - 2C, 1A - 1C.

-4-The fish observed in the essential cooling pond and ECW intake bays were primarily sailfin molleys, a minnow indigenous to ponds in the region. The fish grow to be 4-5 inches in length, although most of those observed were 1-2 inches in length. These were small enough to get through the 3/8-inch mesh of the traveling screens but too large to pass through the 1/16-inch mesh of the selfcleaning strainer. The smallest heat exchanger tube in the loads supplied by the system is 3/8 inch.

02.03 Human Performance a.

Inspection Scope The inspectors interviewed operators, maintenance personnel, and managers involved in the initial response to the clogging of the ECW Bay 1C pump strainer. Log entries, corrective action documents, and maintenance records were reviewed and compared to regulatory requirements.

b.

Findings No significant human performance issues were identified.

Root Cause Analysis and Corrective Actions a.

Inspection Scope The team reviewed the licensee's root cause analysis documented in Condition Report 01-19410 and discussed the processes used to reach the conclusions with both event review team leaders. The inspectors also reviewed background documentation on the type of fish observed and discussed the behavior of the fish with the licensees consulting fish expert. The essential cooling pond and each of the suction bays were inspected to observe the amounts of algae and fish present. The inspectors held discussions with operators to determine the actual practices in effect prior to the event regarding equipment rotation and chlorination. Design and licensing basis documentation were reviewed and compared to the existing configuration and practices.

Data were reviewed for weather and associated pond conditions and for ECW train run and idle times.

b.

Findings Licensee Event Evaluation and Root Cause Determination The licensee determined that the root cause of fish plugging the discharge strainer in the ECW Bay 1C pump train was a combination of environmental conditions that caused a large accumulation of fish in that one suction bay.

The cold weather front that entered the area of the plant on November 27 caused a pond temperature drop of about 15
F. According to the licensee, the minnows would be expected to seek refuge in warm water as a result of this stimulus. The intake structure

-5-has six intake bays, with four pumps normally running and two in standby. The running bays would be at the same temperature as the pond, so they would not be attractive refuge spots. The water in the two idle bays was relatively unaffected by the temperature changes in the pond and would have appeared attractive as refuges.

ECW Bay 2C was located at the center of the structure and apparently had no significant buildup of fish. This was believed to be because the fish would avoid the higher flow velocities in crossing the running intakes. Also, ECW Bay 1C was located on the south side of the intake structure adjacent to the shallow shoreline, which was the normally preferred habitat for sailfin molleys and thus contributed to the higher fish population in ECW Bay 1C.

ECW Bay 1C was also observed to have a larger amount of algae growing on the walls than the other bays. Since sailfin molleys feed on algae, this would be expected to attract more fish than the other bays, but not necessarily when compared to the pond in general. However, once the fish entered ECW Bay 1C seeking refuge, there was sufficient food to sustain even a large mass of fish for days.

The licensee concluded that the removal of any one of these factors (pond temperature differential, food supply, and shoreline proximity) would have prevented the large accumulation of fish. This was primarily based on approximately 12 years of successful operation without clogging a strainer in this system. Contributing to this event, the licensee concluded that the self-cleaning feature of the strainer was ineffective for a short time following starting of a pump. The backwash water was supplied from downstream of the strainer. When the pump was started, the entrained fish immediately clogged the strainer elements and significantly reduced the effectiveness of the backwashing action.

Licensing and Design Basis The inspectors reviewed the design basis and licensing basis information for the ECW pond and system. There was very little information available relating to the expected biological conditions in the pond. The design features for keeping foreign material out of the ECW system were intended for general debris. The trash racks and traveling screens did not provide protection from fish of this size. The self-cleaning strainer was expected to protect the downstream part of the system provided the system was able to backwash the strainer as quickly as the minnows entered. It was the licensees position that the design envisioned that, although fish would exist in the pond, they had not been determined to be a realistic threat to system operation. The inspectors noted that no documentation could be located to support or refute this position. Further, the licensee concluded during the course of this inspection that this event revealed a nonconforming condition in that the Updated Final Safety Analysis report stated that the licensee would treat the ECW system to avoid fouling. The inspectors noted that the licensee had been slow to reach this conclusion, since such an assessment was requested by the senior resident inspector on December 3 because it was necessary in order to assess the operability of the systems in each unit per Generic Letter 91-18, Information to Licensees Regarding NRC Inspections Manual Section on Resolution of Degraded and Nonconforming Conditions.

-6-However, the inspectors concluded that a significant number of years of successful operation without fish-related problems tended to support the licensees position that fish were expected to populate the pond over time but had not been expected to pose a problem with system operation. Therefore, the licensee should not necessarily have been expected to design against this specific problem.

Immediate Corrective Actions The licensee cleaned the fish from the strainer and attempted to restart the system, which immediately clogged again with fish. It was then recognized that the suction bay contained a significant number of fish. An observation of the bay while the strainer was being cleaned the first time had revealed only a few dead fish. The licensee drained the bay, scraped the algae, and cleaned out more than two 55-gallon drums of fish. The licensee started to refill the bay from the pond, but numerous fish were being introduced back into the bay, so a clean fill source was used and heavily chlorinated. The pump was then successfully started and declared operable.

When the licensee realized the ECW Bay 1C strainer was clogged, operations management concluded that the idle pump in Unit 2 should be run to demonstrate operability. However, the inspectors noted that this was performed before the source of the strainer plugging was identified and before fish were known to be the cause. It was fortuitous that ECW Bay 2C did not contain a significant number of fish, because the licensee took no special precautions to assure a successful start of the ECW Bay 2C pump. Further, when the licensee later realized that fish were the cause, and when the number of fish in the ECW Bay 1C pump was subsequently discovered to be very large, the licensee did not go back and re-evaluate their determination that the Bay 2C train was operable while idle, as prescribed in Generic Letter 91-18, until prodded by the inspectors. This resulted in the licensee deciding to run all available ECW pumps to continuously demonstrate operability until the causes could be fully evaluated.

By December 19, the licensees actions to prevent the recurrence of this specific problem primarily involved a plan to monitor for a buildup of fish during cold weather.

The primary trigger for starting to monitor was entry into Procedure 0POP01-ZO-0004, Extreme Cold Weather Guidelines. Revision 12 of this procedure added steps to have chemistry personnel periodically inspect idle ECW bays for a buildup of fish or algae.

However, the inspectors identified that the entry conditions for this procedure were not bounding for the weather during this event, and the use of air temperature did not directly indicate what would stimulate the movement of fish. In response to this observation, the licensee issued Revision 67 to Procedure 0PSP03-ZQ-0028, Operator Logs, to trigger intake bay inspections when pond temperature dropped 4
F in consecutive 12-hour log readings. The inspectors confirmed with the licensees consulting fish expert that this was an appropriate and conservative indicator.

The licensee proceduralized the intake bay inspections in Procedure 0PCP01-ZQ-0004, Cooling Water System Inspection Guidelines, Revision 2. This inspection consisted of a visual inspection with a powerful flashlight, and using a dip net a minimum of 4-5 times to obtain a representative estimation of fish population.

-7-The inspectors had reservations about the licensees ability to effectively monitor the number of fish in the intake bays. They observed that the water was so murky that observations were limited to within a few inches of the surface with natural light or a powerful flashlight. Such observations were ineffective in estimating the fish population of a 14-foot deep bay. When a high-intensity underwater light was requested, the inspectors could see fish within a 1 foot radius, but the intensity of the light repelled the fish, making a quantitative estimate difficult. While the dip net was observed to catch fish, the licensee had not developed meaningful guidance on the threshold for concluding that an abnormal number of fish existed. More significantly, the licensee had developed no defined action to take in response to an indication of an abnormal number of fish in an intake bay and no correlation of how many fish could be expected to impact the operability of a train.

To deter fish from entering the ECW intake bays and reduce the available food supply, the licensee made minor improvements to the way the system was operated and chlorinated. Revision 18 to Procedure 0POP02-EW-0001, Essential Cooling Water Operations, and changes to operating logs, required running an idle train at least weekly and performing chlorination when switching a running train to a standby condition. This meant that running trains would be chlorinated for 20 minutes every 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> to a concentration of 0.6-1.0 ppm and that an idle train could go 7 days between chlorination injections. Through discussions with the licensees consulting fish expert, the inspectors determined that this chlorination scheme could be expected to keep the algae under control in the areas in contact with the chlorine flow (not the entire suction bay) but would not be expected to deter fish from entering a bay, especially under the stimulus of cold weather conditions. Even in an idle bay without any food, the cold weather would still provide enough stimulus for the fish to congregate over a 1-to 3-day period to a population that could match what the licensee experienced during this event.

As a result, the inspectors concluded that the licensees attempt to minimize the food supply, while beneficial on a routine basis, would provide little assurance that the event would not recur.

While it was recognized that the licensee was still developing corrective actions, after almost 3 weeks the licensee had not introduced a significant barrier that provided a high level of assurance that the incident would not recur under the same conditions. The plant continued to operate, the licensee continued to rely on all trains being operable, and the major stimulus of cold weather could reasonably recur. Partially in response to the inspectors expressing this concern, the licensee decided to chlorinate idle ECW bays at least daily as an interim measure while more permanent improvements were developed. The inspectors were satisfied that this level of chlorination would build up a substantial barrier to the entry of fish, even in cold weather, and was demonstrated to attain concentrations lethal to fish after three injections.

Long-Term Corrective Actions The licensee was still developing long-term corrective actions at the completion of the inspection. The licensee was considering such options as replacing the traveling

-8-screens with ones with a smaller mesh, using strobe lights and/or acoustic devices to repel fish, or running the traveling screens continuously.

The licensee also planned to evaluate and monitor the fish population in the pond.

c.

Conclusions The inspectors concluded that the licensees root cause assessment was reasonable.

However, based on discussions with the licensees consulting fish expert, the inspectors did not agree that removal of any one of the factors described would provide an equal probability of preventing recurrence of the event. Specifically, the presence of a rapid drop in pond temperature like that experienced prior to this event was judged to be sufficient to cause an accumulation of fish in an idle intake bay. Therefore, prompt corrective actions to prevent a large accumulation of fish in idle intake bays was called for since the weather pattern could recur. The licensees initial corrective actions did not provide any barrier to the entry of fish into the intake structure, did not assure the licensee would have effectively monitored for a buildup of fish, and did not include any specific steps to prevent a train from being rendered inoperable if a buildup were detected. Following further discussions between the inspectors and the licensee, the licensee added actions to chlorinate idle intake bays at least daily. The inspectors concluded that this provided a positive barrier to building up fish, even during a period of cold weather. This was not considered a violation because the licensee responded promptly to the NRCs safety concern and because the licensees corrective actions were not yet complete.

This issue was more than minor because it represented a potential for a repeat failure, which has a credible impact on safety and could credibly affect the operability, availability, reliability, and function of a train of accident mitigation equipment. This finding was determined to be of very low safety significance using the Significance Determination Process evaluation primarily because two trains would remain available.

02.02 Risk Significance of the Event

a.

Inspection Scope The inspectors reviewed weather and pond temperature data and operating logs to conservatively determine the length of time that the function could have been lost for the ECW Bay 1C train. The inspectors also reviewed fish literature, held discussions with the licensees consulting fish expert, and reviewed the licensees corrective action data base to find cooling water problems related to fish in order to determine the potential for common mode failure of multiple ECW trains.

b.

Findings The inspectors attempted to determine when a significant accumulation of fish in ECW Suction Bay 1C might have begun which had the potential to cause plugging of the strainer. Since the train was idle for about 15 days, fish had time to accumulate in the bay in sufficient quantity to cause a loss of function when the pump was called to

-9-start some time before it was actually recognized. Using part of the licensees root cause that concluded that the fish accumulation was triggered by the onset of cold weather, the inspectors reviewed weather and pond temperature data to determine a conservative start time. The assumptions and conclusions were discussed with the licensees fish expert, who agreed. Thus, the inspectors assumed that the ECW Bay 1C train had the potential to have been inoperable from about noon on November 28, when pond temperature was dropping rapidly, until the end of December 4, when the train was successfully returned to service.

The safety significance of this was determined to be very low. While this event rendered virtually an entire train of engineered safety features equipment inoperable, this conservative estimation of out-of-service time remained within the allowed outage time permitted in Technical Specifications (7 days), and no safety equipment from any other train was inoperable during this period.

Potential for Common Mode Failure As stated above, the licensee concluded that the running trains were not susceptible to clogging. The inspectors were unable to find evidence to refute this conclusion.

The inspectors evaluated the possibility that the pond could have recently experienced a large increase in the population of fish. Although the licensee had never monitored the fish population, they concluded that the population was stable because there had been no noticeable evidence of an increase in algae, numbers of dead fish, or predatory bird activity in the pond. The inspectors observed each suction bay and found that the bays contained similar, relatively small quantities of algae growing on the walls, with small populations of live minnows.

The live minnow population indicated that the fish continued to be free to enter and were not effectively deterred by the existing chlorination program. The algae was clearly under control and limited to areas that would be expected to get little chlorine contact when the system was in operation. The inspectors concluded that the number of fish under normal conditions should remain small enough to avoid impacting the system function. Further, the literature on fish behavior reviewed by the inspectors indicated that sailfin molleys would avoid the high flow velocities close to a running pump. The inspectors determined that a common mode failure under normal conditions was unlikely. Under station blackout conditions, power would be lost to all ECW pumps for up to 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />. Based on a discussion with the licensees consulting fish expert, the inspectors concluded that this was insufficient time, even with a cold weather stimulus, for the migration of significant numbers of fish into the bays that had previously been running. This migration would be expected to occur over a 1-3 day period following a significant pond temperature decrease.

As discussed under corrective actions, the inspectors identified potential problems in the licensees ability to detect and prevent a buildup of fish with an external stimulus. While this might allow a recurrence of this loss of a single train, possibly in both units, it was not expected to cause a common mode failure.

-10-

Meetings 04.01 Exit Meeting Summary The team presented the inspection results to Mr. G. Parkey, Vice President, Generation, and other members of licensee management at an exit meeting on December 20, 2001.

The licensee acknowledged the findings presented.

The team asked the licensee whether or not any materials examined during the inspection should be considered proprietary. No proprietary information was identified.

ATTACHMENT PARTIAL LIST OF PERSONS CONTACTED Licensee W. Baker, Manager, Ecological Resources, Reliant Energy (consulting)

T. Bowman, Unit 1 Operations Division Manager J. Cook, Supervisor, System Engineering J. Crenshaw, Manager, System Engineering R. Engen, Supervisor, Design Engineering R. Gangluff, Manager, Chemistry and Event Review Team Leader E. Halpin, Plant General Manager S. Head, Manager, Licensing G. Janak, Operations Shift Supervisor J. Johnson, Supervisor, Engineering Oversight D. Leazar, Manager Nuclear Fuel Analysis M. McBurnett, Director, Quality and Licensing G. Parkey, Vice President, Generation L. Peter, Unit 2 Operations Division Manager M. Sicard, Manager, I&C Maintenance G. Smith, Event Review Team Leader W. Stillwell, Supervisor, PRA Group NRC T. Pruitt, Senior Reactor Analyst, Region IV J. Tatum, Plant Support Branch, NRR ITEMS OPENED, CLOSED, AND DISCUSSED Opened None.

Closed None.

Discussed None.

DOCUMENTS REVIEWED The following documents were selected and reviewed by the inspectors to accomplish the objectives and scope of the inspection and to support any findings:

-12-Procedures:

0POP01-ZO-0004, Extreme Cold Weather Guidelines, Revision 12 0PSP03-ZQ-0028, Operator Logs, Revision 67 0PCP01-ZQ-0004, Cooling Water System Inspection Guidelines, Revision 2.

0POP02-EW-0001, Essential Cooling Water Operations, Revision 18 Condition Reports:

01-19410 01-19729 01-19799 01-19903 01-19919