ML030830151

Possible Common Mode Failure of Aux Feed Recirculation Lines, RCE00019l Draft

ML030830151
Person / Time
Site:	Point Beach
Issue date:	12/12/2002
From:	Bennett K, Bosacki W, Flessner R, Freels J, Hettick D, Peterson L, Schmidt E Nuclear Management Co
To:	Office of Nuclear Reactor Regulation
References
CAP029952, FOIA/PA-2003-0094 RCE000191, Draft
Download: ML030830151 (91)

Text

NI C--F'h c.o,"dto,,,*r*lea)

Point Beach Nuclear Plant Possible Common Mode Failure of Aux Feed Recirculation Lines RCE000191 (CAP029952)

Event Date: October 29, 2002 Report Date: December 12, 2002 Principal Investigators:

Richard Flessner - Team Lead Kevin Bennett William Bosacki Eric Schmidt Approved By:

Group Lead - Larry Peterson Date Issue Manager - James Freels Date CAP Manager - Dennis Hettick Date

Possible Commizon Mode Failure of RCEO00191 Aux Feediwater Recirculation Lines Table of Contents I.

Executive Summary..........................................................................

3 II. Event Narrative..................................................................................

6 III. Extent of Condition Assessment.....................................................

15 IV. Nuclear Safety Significance............................................................

16 V. Report to External Agencies..........................................................

17 VI. Data Analysis.........................................

17 Information & Fact Sources...................................................................

17 Evaluation Methodology & Analysis Techniques............ ;................... 44 Data Analysis Summary........................................................................

44 Failure Mode Identification...................................................................

53 VII.

Root Causes & Contributing Factors....................................

60 VIII.

Corrective Actions......................................................................

62 IX. References..............................................................................................

65 X. Attachments...........................................................................................

69 Attachment A - Event Timeline..........................................................

70 Attachment B - Barrier Analysis..........................................................

73 Attachment C - Stream Analysis..........................................................

85 Attachment D - Charter........................................................................

87 Attachment E - PBNP Orifices.............................................................

88

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines I.

Executive Summary

Purpose:

CAP029952 identified that a potential existed for a common mode failure where all AFW pump recirculation lines could have restricted flow rates (due to orifice plugging) resulting in eventual pump failure. The purpose of this investigation is to determine the root and contributing causes of why that potential existed and why the condition was not identified previously.

Event Synopsis:

Low AFW line recirculation flow was identified during performance of IT 10 following maintenance on AFW pump P-38A. During troubleshooting, the restricting orifice (RO) from the AFW recirculation line was removed and inspected. Corrosion product debris was found in almost half of the holes in the outer cylinder of the orifice. Additional discussions identified a concern that the limiting dimensions of the orifice were smaller than the service water strainers and a potential existed for the orifices to become plugged when AFW was aligned to its safety-related source (service water) during accident response. The AFW ROs had been installed as modifications between November 2000 and September 2002.

==

Conclusions:==

The design engineer for the initial RO modifications for the Motor Driven Auxiliary Feedwater Pumps failed to consider service water strainer size as an input to the design; however the potential plugging issue was discovered during the safety evaluation process.

The design engineer decided to disposition the plugging issue in the safety evaluation instead of revising the design. This inappropriately moved evaluation of the potential plugging issue out of the design process and into a licensing process. The initial safety evaluation concluded that the ROs would not plug based on a line of reasoning that was not verified, and based on misapplied vendor information. The mindset that the ROs would not plug resulted in key information regarding important design functions being omitted from the safety evaluation. During the MSS review, the conclusion that the ROs would not plug was challenged and one MSS member suggested that the AFW recirculation line wasn't needed because the AOV fails closed on loss of instrument air.

The safety evaluation was revised without technically determining if the ROs would plug and without recognizing that important design functions would be lost as a result of the plugging. Because the plugging issue was dispositioned via a safety evaluation, no other design output documents contained the (erroneous) conclusion that AFW recirc lines were not needed under accident conditions. The same deficiencies were repeated for the design of the ROs for the Turbine Driven Auxiliary Feedwater Pumps, primarily because the modifications were viewed as repeat activities.

Missed opportunities occurred during the evaluation of CAP013812 and development of MR 02-029. CAP013812 was initiated in April 2001 due to recirculation flow concerns associated with AFW pump P-38A. The CAP attributed the problem to FME concerns associated with the drilling of additional holes in the flow orifice in March 2001. An action item was created to evaluate the FME practices for that work. The CA concluded 3

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines that FME controls were adequate. The CAP was closed without identifying the cause of the flow concerns.

MR 02-029, which upgraded the AFW recirc AOV open function, failed to evaluate the AFW recirc line RO. Time and schedule pressure resulting from inadequate planning (management imposed actions and due date) combined with an evolution that was infrequently performed (safety function upgrade) to create an error-likely situation (design review). Error prevention methods, such as pre-job briefs, were not taken to ensure that the task was properly conducted. This resulted in an inadequate design review of the AFW recirculation line components because of mindsets and wrong assumptions.

Nuclear Safety Significance:

The safety significance of this has not yet been determined. The significance is dependent on the actual probabilities of plugging. Evaluations are in progress to determine the actual plugging probabilities. The on-line risk after taking compensatory measures was "Yellow". Actions were initiated to provide additional oversight of work while in this condition.

Root Cause:

The root cause of this event was the failure to properly evaluate the potential for orifice plugging within the design process. The evaluation of the plugging issue via a safety evaluation resulted from inadequate program management.

"* The design engineer decided to disposition the plugging issue in the safety evaluation instead of revising the design. This inappropriately moved evaluation of the potential plugging issue out of the design process and into a licensing process.

"* The supervisor condoned resolution of the issue via a safety evaluation in lieu of pursuing a formal technical evaluation.

Significant contributing causes included:

The use of unverified information in the safety evaluation for modification MR 99 029*A/B and the omission of key design function information from the safety evaluation resulted in an erroneous conclusion. That omission reinforced knowledge deficiencies that existed with MSS members resulting in the conclusion that the AFW recirculation line was not needed during accidents.

"* The vendor information concerning its design to preclude plugging was misapplied and not verified

"* Information on the design functions of the Appendix R flowpath and throttling of AFW flow during emergency operations was omitted from the safety evaluation Inadequate knowledge of AFW recirculation line design functions existed

• Inadequate independent verification resulted from inadequate program management

• Engineering management involvement and oversight (at the manager and supervisor levels) was inadequate and contributed to high human error rates 4

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines Corrective Action Synopsis:

"* Implement periodic reviews of Engineering's products by the Quality Review Team to identify and address human performance related issues. CATPR

"* Increase engineering management involvement in the approval and oversight of modifications. CATPR

"* Present lessons learned from this event to Engineering personnel stressing the importance of following the design process. CATPR

"* Revise training materials to accurately describe the AFW recirculation line design functions.

"* Review licensing basis documents to ensure that they accurately describe the AFW recirculation line design functions during accident conditions.

• Implement use of Human Performance Tools in Engineering (ENG-10).

• Initiate a supplemental record to the modification file documenting the acceptability of the additional holes drilled in ROs for MR 99-029*A/B.

• Revise the modification procedure (or provide supplemental guidance) to include an impact review on other work in progress and to clarify what materials are distributed to which design team members.

"* Evaluate adequacy of actions related to closure of CAP013812 (AFW RO FME issue from April 2001).

"* Evaluate adequacy of MSS member qualifications for performing 50.59 reviews of modifications.

"* Evaluate why OSRC had no comments on SE 2000-0055.

"* Revise MSS procedure to specify that the final design must be approved prior to the review of safety evaluations for modifications.

"* Revise MSS practice of reviewing only the safety evaluation for modifications to include a review of the final design description.

5

Possible Common Mode Failure of RCEO00191 Aux Feedivater Recirculation Lines II.

Event Narrative

Background

On May 5, 1988 NRC issued Bulletin 88-04, Potential Safety-Related Pump Loss. This bulletin requested licensees to investigate and correct as appropriate two mini-flow design concerns. The first concern was the potential for deadheading one or more pumps that have a common mini-flow line. The second concern was whether or not the installed mini-flow capacity is adequate to prevent damage to safety related pumps. In a response dated June 28, 1988, we acknowledged that each of the pumps in the Auxiliary Feedwater (AFW) System have their own recirculation lines with an AOV isolation valve and an orifice upstream of the common return line to the condensate storage tank (CST). We discussed the logic of the recirculation valves to open or shut dependent on AFW system forward flow. We also acknowledged that the flow orifices for the pumps needed to be replaced with higher flow orifices to ensure sufficient flow for indefinite pump cooling via the recirculation lines. The installed ROs provided a recirculation flow of 30 gpm based solely on pumped fluid temperature rise considerations only. New flow rates that addressed hydraulic instability and pumped fluid temperature rise considerations were being established by the pump manufacturer and were estimated to be 120 gpm for the Motor Driven Auxiliary Feedwater Pumps (MDAFPs) and 175 gpm for the Turbine Driven Auxiliary Feedwater Pumps (TDAFPs).

On July 7, 1988 modifications MR 88-099*A-C were initiated to increase the recirculation line flows to prevent pump degradation due to hydraulic instability. The minimum pump flow prior to this MR was 30 gpm. The MR increased the minimum flow to 70 gpm for the MDAFPs and 100 gpm for the TDAFPs, based on vendor recommendations in a letter dated August 7, 1989. The replacement orifices were installed during 1991.

The vendor letter, dated August 7, 1989, stated that the data was analyzed in accordance with minimum flow requirements as outlined in NRC Bulletin 88-04. It went on to say that the following guidelines should be followed in order to avoid damage due to operation at low flow rates:

AFW Pump Accumulated Time Min. Flow

% Best Efficiency (Hours/Year)

(GPM)

Point TDAFPs 1500 210 42 "64 60-1500 130 26 60 100 20 MDAFPs 1500 105 28

""t 60-1500 75 20 1

60 70 19 Operation at 30 GPM should be avoided for both (style) pumps (The minimum flow values for operational periods up to 60 hours6.944444e-4 days <br />0.0167 hours <br />9.920635e-5 weeks <br />2.283e-5 months <br /> were subsequently lowered to 50 gpm (MDAFPs) and 75 gpm (TDAFPs) based on a vendor letter dated March 2, 2001.)

6

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines During 1996 and 1997 several condition reports were written regarding high noise levels associated with operation of the AFW pumps while on mini-recirc. Engineering Work Request EWR 99-031 was initiated on December 8, 1998 to determine if the orifices were incurring damage from cavitation and needed replacement. During 1998 and 1999 weld cracks were discovered in the recirculation piping associated with the MDAFW pumps. On June 30, 1999 the Engineering Advisory Committee (EAC) reviewed three options proposed to address EWR 99-031. The options were to install a multi-stage restricting orifice (RO) of the same type as currently installed but with 10 stages instead of 2 stages used in the present design, to install a new design involving a pressure reducing flow element in a valve body, or to install a multi-stage pressure-reducing trim in the recirculation line AOVs. The EAC did not select an option (contrary to the EAC procedure), but did recommend implementing modifications for the MDAFW pumps during 2000 and evaluate performance to determine if TDAFW pumps should be modified during 2001. Modifications MR 99-029*A-D were initiated during 1999 to replace the AFW recirculation line restricting orifices. RCE 99-081 evaluated the socket weld failures in the AFW recirculation lines and concluded that vibrations induced by the RO cavitation caused the cracking. RCE 99-081 was approved on June 18, 1999 and presented to CARB on January 18,2000. The RO modification scope was expanded to address weld and pipe replacement and installation of oversized sockets.

The option to install a new design involving a pressure-reducing flow element in a valve body was chosen by Design Engineer 1, independently reviewed by Design Engineer 2, and approved by the Mechanical Design Supervisor during May 2000. The new orifices were installed in the recirculation lines for the MDAFW pumps during November 2000.

Some difficulty was encountered in achieving the desired flow rates through the orifices and it was necessary to drill additional holes in the trim. The design was changed for the TDAFW pump orifices to include a movable plug that could be adjusted to vary the flow.

The Unit 2 TDAFW pump orifice was replaced in April 2002 and the Unit 1 TDAFW pump orifice was replaced in September 2002.

Event On October 23, 2002 at 0400, Operations removed MDAFW pump P-38A from service for scheduled maintenance. During the day shift, Maintenance personnel conducted scheduled corrective maintenance on P-38A. Work orders 9945610, 0205651 and 9949098 were performed. These WOs consisted of changing the operating diaphragm on AFW discharge control valve, AF-4012, repacking the inboard and outboard stuffing boxes (pump seals) on P-38A, and replacing service water drain valve AF-38.

While repacking the pump, Maintenance found only six packing rings on both the inboard and outboard sides of the pump instead of the expected seven rings.

Maintenance put 7 rings in each seal and documented these findings on CAP029923.

The physical work with replacing AF-38 involved cutting the socket weld, manually removing the valve, inspecting the piping visually, prepping the area for reinstallation, and then installing the new valve with a socket weld.

7

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines The work on AF-4012 involved replacing the valve's operating diaphragm. Following completion of the physical work, I&C performed a drop test on the AOV diaphragm.

The drop test failed and day shift turned this work over to night shift for completion.

On night shift, Maintenance changed the stem diaphragm gasket on the AOV early in the shift. Re-performance of the drop test was successful, I&C completed testing on the instrumentation, and the pump was turned over to Operations for post-maintenance testing.

Operations filled and vented the pump suction line, casing, and discharge line per 01 62A, Motor-Driven Auxiliary Feedwater System (P-38A & P-38B). The venting procedure requires the suction line to be vented first until a solid stream of water is observed, followed by the casing vent and finally the discharge vent.

On October 24, 2002 at 0159 hours0.00184 days <br />0.0442 hours <br />2.628968e-4 weeks <br />6.04995e-5 months <br />, Operations started P-38A for post maintenance testing per IT 10, Test of Electrically Driven Auxiliary Feed Pumps and Valves (Quarterly). When the pump started, a Maintenance supervisor observed that the suction Srelief valve (AF-4028) momentarily lifted and reseated, but continued to drip water. The relief valve has WO 9948850 associated with it. The relief valve was also observed to not be sucking air. This was determined by the Maintenance supervisor holding his hand over the discharge tailpiece to feel for suction.

Maintenance personnel proceeded to adjust the pump packing as expected. It was observed that the recirc flow was 64.5 gpm, which was less than the 70 gpm acceptance criteria. The shift manager was informed of the discrepancy. The Shift Manager directed the crew to first vent the flow indicator. This resulted in no improvement in flow. He then had the crew monitor the pump casing for overheating. The pump was reported to be cool to the touch. The shift manager then directed the crew to secure the pump. The pump was secured at 0212 hours0.00245 days <br />0.0589 hours <br />3.505291e-4 weeks <br />8.0666e-5 months <br />. The crew observed a normal coast down of the pump.

The Shift Manager stated he felt assured that the pump was not degraded or damaged from low recirc flow based upon the short run time (13 minutes) and the above listed parameters. IT 10 requires 70 gpm recirc flow.

After the pump was secured, the Shift Manager discovered that I&C had calibrated the flow indicator the previous day. He requested that I&C perform a calibration check to verify proper operation. This was performed with different MTE than the original calibration. The calibration check was satisfactory (-0440).

The Shift Manager also requested that I&C review the instrument venting procedure with the auxiliary operator who performed the earlier vent. The auxiliary operator and I&C technician concluded that the transmitter was vented correctly. The crew also re-vented the pump suction piping, casing, and discharge piping. During each of the vents, solid streams of water were observed with no air. An auxiliary operator verified the valve lineup was correct.

8

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines CAP029908 was initiated by Operations at 0323 to document the inadequate flow obtained during performance of IT 10.

The AFW System Engineer was notified at 0400 on October 24, at which time an investigation was commenced. Between 0500 and 0700 hours0.0081 days <br />0.194 hours <br />0.00116 weeks <br />2.6635e-4 months <br />, the potential causes were defined in order of likelihood:

1. Restriction in the piping.

a. Recirculation orifice AF-4008

b. Check valve degradation: AF-112 or AF-115

c. Manual valve problem: AF-39 or AF-27

d. Recirc AOV stem-to-disc separation: AF-4007

e. FE-4050A degradation

2. Instrumentation: FIT-4050A.

3. Degradation of the pump rotating assembly.

4. Valve leakage to the discharge path.

5. Pump P-38A was not adequately vented.

At about 0603, the BOPINSSS Mechanical Systems Engineering Supervisor was established as the overall lead for Engineering for this issue.

At 0830 hours0.00961 days <br />0.231 hours <br />0.00137 weeks <br />3.15815e-4 months <br /> on October 24, 2002, a meeting was held to discuss the issue. The BOP/NSSS Mechanical Systems Engineering Supervisor led the meeting, with representatives from Engineering, Maintenance, and Operations in attendance. It was decided to perform the following actions:

1. I&C - Re-vent the transmitter once again (Engineering to observe) and verify the position of the equalizing valve.

2. Operations - Vent all lines and the pump casing with Engineering observing.

3. Engineering - Use a UT flow transmitter to verify installed indication.

4. Engineering to take vibration data.

5. Monitor discharge pressure.

6. In parallel, prepare a work plan to open RO-4008 if adequate flow cannot be achieved.

The venting was performed by Operations in accordance with 01 62A, Section 7.1. No air was noted during the vent; there was a solid stream of water from all three vents. The UT flow device was attached and the flow transmitter was vented. A pre-job brief was held by Operations with the control operator, auxiliary operator, and supervision. The brief included the direction that if recirc flow was not >70 gpm, then the pump would be secured immediately.

At 1137, Operations started P-38A. Equipment and components responded as expected.

Indicated flow on the recirc transmitter went to about 64 gpm. The UT device flow stabilized at 60 gpm. The on-scene SRO observed these indications and directed that the pump be secured after running for less than one minute. During the run, Engineering 9

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines took vibration data, which indicated normal. P-38A was then danger tagged and drained to allow removal and inspection of the flow orifice.

Maintenance removed the orifice. The Mechanic noted several of the holes on the outside sleeve were plugged. There was no evidence of debris in the pipe (a boroscope was used to verify this). The orifice was taken to the maintenance shop where Engineering mapped the plugged holes and took photographs of the orifice. Corrosion product accumulation was noted in 24 of the 54 outer holes. Maintenance milled off the spot welds on the retaining pins to facilitate removal of the sleeves. The pins were driven out with a punch. The nested sleeves had to be tapped out using a block and hammer.

The mechanic-electrician who disassembled the orifice stated that all of the sleeves except the outermost sleeve were "spotless" with no evidence of debris. Material did fall out of and off the orifice during disassembly. Four small particles were collected for further analysis.

Following cleaning and inspection, the nested sleeves were reassembled and reinstalled using a new bonnet gasket (spiral wound) and the old seat gasket (spiral wound but already compressed because a new gasket was not available). Installation was completed by 1800 hours0.0208 days <br />0.5 hours <br />0.00298 weeks <br />6.849e-4 months <br />.

During night shift on October 24, Operations cleared tags, and filled and vented the system per 01 62A. Maintenance, Operations and Engineering met to discuss the next test run. The Maintenance Supervisor requested that the high points be re-vented prior to the run. Operations re-vented the suction (solid stream) and discharge (slight amount of air) piping. Operations then started P-38A pump per 01 62A at 2115 hours0.0245 days <br />0.588 hours <br />0.0035 weeks <br />8.047575e-4 months <br />. Recirc flow was observed to be within specification at 75 gpm. Following satisfactory observance of key parameters, the pump was secured.

Earlier in the shift, at approximately 1700 to 1800 hours0.0208 days <br />0.5 hours <br />0.00298 weeks <br />6.849e-4 months <br />, the Shift Manager directed Chemistry to sample all four AFW pump suctions and both condensate storage tanks.

The purpose of these samples was to determine the water quality as it related to possible plugging of the recirc orifices. The communication received by the chemistry technician was to obtain and analyze samples from Unit 1 and Unit 2 auxiliary feedwater and from the Unit 1 and Unit 2 condensate storage tanks.

The Chemistry Supervisor stated that the Unit 1 and Unit 2 auxiliary feedwater suction samples were taken at 1 C26 and 2C26, feedwater sample control panels. These were aligned to the 1&2 P29 turbine-driven auxiliary feedwater pump suctions, and that is where the samples were drawn from. The chemistry technician also took samples from both condensate storage tanks (CSTs) at their normal sample points on El. 26'. The CST sample points tap into the CST are, 6" to V' above the elevation that auxiliary feed suction piping exits the CSTs. The samples were analyzed for total suspended solids (TSS), chlorides, fluorides, and sulfates. The Chemistry Supervisor stated that the results showed nothing abnormal.

10

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines The Shift Manager then made preparations to perform IT 10, Test of Electrically Driven Auxiliary Feed Pumps and Valves (Quarterly), for both the post-maintenance return to service test and the scheduled quarterly run on both motor-driven pumps. A Unit 2 power reduction to meet IT 10 requirements of 98% power was completed at 2200 hours0.0255 days <br />0.611 hours <br />0.00364 weeks <br />8.371e-4 months <br />.

The Shift Manager decided to turn over IT 10 to the mid shift.

Testing of P-38A was completed satisfactorily at 0216 hours0.0025 days <br />0.06 hours <br />3.571429e-4 weeks <br />8.2188e-5 months <br /> on October 25, 2002; the recirc flow indicated 75.2 gpm on FIT-4050A, mini-recirc flow, at which time AFW pump P-38A was returned to service and the TSAC was exited. Testing of P-38B was completed satisfactorily at 0450 hours0.00521 days <br />0.125 hours <br />7.440476e-4 weeks <br />1.71225e-4 months <br /> on October 25; the recirc flow indicated 74 gpm on FIT 4050B, mini-recirc flow. A partial performance of IT 8A to test 1P29 recirc flow was performed and IT 9A was performed to verify 2P29 operation. All testing was completed and all four pumps were back in service by 1206 hours0.014 days <br />0.335 hours <br />0.00199 weeks <br />4.58883e-4 months <br /> on October 25.

ACE0O 1023 was initiated on October 25 at 1030 to evaluate the inadequate recirculation flow on P-38A.

On October 25, the Operations Manager and the Assistant Operations Manager discussed the event. The Assistant Operations Manager stated that we needed a short-term follow up action plan to address the orifice plugging issue. He wanted to know if contingencies needed to put in place, such as: increase our testing frequencies; do call-ups to run the pumps more often; perform the Service Water flush more often; institute a sampling plan; etc. The Operations Manager stated that the conversation focused on the additional actions needed to finalize extent of condition/cause for the P-38A recirc line plugging.

Late in the afternoon they paged the BOP/NSSS Mechanical Systems Engineering Supervisor who called back from home. The BOP/NSSS Mechanical Systems Engineering Supervisor, Operations Manager, and Assistant Operations Manager had a conference call. They acknowledged that an Apparent Cause Evaluation had already been assigned to System Engineering to look into the cause of the plugging. Because ACEs have 30 day due dates, the Operations Manager and Assistant Operations Manager wanted System Engineering to develop any recommended short-term actions more quickly than 30 days. They mutually agreed to a one week due date (November 1, 2002) for Engineering to recommend what, if any, short-term actions should be taken. During the conversation the BOP/NSSS Mechanical Systems Engineering Supervisor asked if Operations was questioning operability, and they said they were not. The BOP/NSSS Mechanical Systems Engineering Supervisor stated that he did not have any operability concerns at this point in time. The purpose of the action plan was to find the source of the foreign material and to determine what other testing or flushing would be required to assure that future plugging does not occur. The Operations Manager stated that they also discussed the fact that we needed to look at service water to ensure we didn't have a potential to plug the orifice from service water.

The BOP/NSSS Mechanical Systems Engineering Supervisor later stated that since neither he nor the Operations Manager had any concerns with AFW operability, no immediate response was required and that over the weekend (October 26-27) he considered the actions that might be required and formulated potential questions that II

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines needed to be answered that would be discussed at the meeting he would hold on October

28. The Operations Manger has also stated that he didn't ask for weekend evaluation of the issue because he didn't think it was realistic that the recently designed and installed orifices would be susceptible to plugging from service water.

On October 28, 2002 at 1100 hours0.0127 days <br />0.306 hours <br />0.00182 weeks <br />4.1855e-4 months <br />, Engineering held a meeting to discuss the event.

The meeting included the AFW System Engineer, the Engineering Programs Supervisor, the Engineering Analysis Supervisor, the PRA Supervisor, the BOP/NSSS Mechanical Systems Engineering Supervisor, a Maintenance Support Engineer, and the SW System Engineer. The outcome of this meeting identified a need to research the following items:

"* What is the material?

"* What is its origin?

"* What are the tolerances in the pump?

"* What is the size of holes in the main SW Zurn strainer?

"* Can we perform an operability determination on the SW issue from a realistic approach?

"* Explore the silt study (AFW SW samples analysis to support AFW pump operability concern due to zebra mussel issue raised by the NRC in IR 99-013).

"* Get detailed information on the recirc orifice.

Early on October 29, 2002, the BOP/NSSS Mechanical Systems Engineering Supervisor convened a meeting to review the issue in detail with other members of Engineering. He stated that his purpose in convening the meeting was to communicate his conclusions that he had a reasonable degree of certainty that the issue of potential orifice clogging from service water debris was valid. The meeting was convened at 0900 hours0.0104 days <br />0.25 hours <br />0.00149 weeks <br />3.4245e-4 months <br />. Personnel in attendance were the Director of Engineering, the BOP/NSSS Mechanical Systems Engineering Supervisor, the PRA Supervisor, the Auxiliary Feedwater System Engineer, the Service Water System Engineer, and the Engineering Analysis Supervisor. The Engineering Director inquired whether there was a concern regarding the ability of the system to perform its design functions. Most, if not all persons in attendance stated that they had the concern that in a situation where AFW was required to take suction from the SW system, the running pumps' recirculation orifices would likely become plugged and non-functional. The Regulatory Affairs Manager was then briefed on the issue. The BOP/NSSS Mechanical Systems Engineering Supervisor immediately informed Operations of the concern.

On October 29,2002 at 0945 hours0.0109 days <br />0.263 hours <br />0.00156 weeks <br />3.595725e-4 months <br />, a meeting with Operations and Engineering personnel was held to discuss the internal tolerances of the RO with respect to potential plugging. Attendees at the meeting were the Operations Manager, a Shift Manager, the Operations Procedure Coordinator, the Acting Plant Manager, the Regulatory Affairs Manager, the CVCS and RCS System Engineer, the Service Water System Engineer, the Auxiliary Feedwater System Engineer, the BOPINSSS Mechanical Systems Engineering Supervisor, the PRA Supervisor, the Senior PRA Engineer, and two Operating Supervisors. The presentation basically expressed a concern with the recirc orifice plugging if SW flow was initiated to the pumps. The Operations Manager stated that the 12

Possible Connmon Mode Failure of RCEO00 191 Aux Feedwater Recirculation Lines discussion centered on the fact that the service water basket strainer mesh is 1/8" and the orifice is much finer (- 15mil x 90 mil). He further stated that they had extensive discussion on the fact that the AFW safety related suction source is service water. They also talked about whether we could call AFW operable from the Condensate Storage Tanks (CSTs). Most people in the room were confident that the CSTs were acceptable as a suction source, however two engineers questioned that. The Operations Manager stated that he felt that the two engineer's questions were based on uncertainty. The attendees discussed whether we could do an evaluation (OPR) on the service water suction and engineering personnel concluded that we could not in the short term. The Operations Manager stated that since service water is the safety related suction source, and we had a condition where the recirc line could fail from potentially poor water quality, and there was no indication of recirc flow in the control room, we did not have reasonable confidence that AFW could perform its safety function under all accident conditions and subsequently declared all AFW pumps out of service.

At 1027 hours0.0119 days <br />0.285 hours <br />0.0017 weeks <br />3.907735e-4 months <br /> on October 29, 2002, all four auxiliary feedwater pumps were declared out of service due to AFW recirc orifice flow issues. Upon declaring AFW out of service, the attendees discussed options for return to service. Per Technical Specification 3.7.5, when all AFW is out of service, we maintain stable plant conditions until we return a pump to service. We then take actions as directed by the other TSACs. The attendees also discussed compensatory measures and concluded that we needed a combination of administrative controls, briefs and training to essentially eliminate the need for the recirc line.

The Operations Manager stated that the short-term actions were crew briefings and temporary information tags in the control room at the flow indicators to maintain 50/75 gpm forward flow for motor/turbine driven pumps. Once the briefings and temporary information tag placement were complete, AFW could be declared operable.

Briefing of the on-shift crew on the AFW recirc flow orifice issue was completed at 1100. Posting of temporary information tags for all 4 AFW pumps with requirements to secure AFW if minimum flow is not maintained was completed at 1210.

At 1241 a CAP was initiated by the Systems Engineering Supervisor on the concern for a potential common mode failure for the AFW pumps due to plugging of the recirculation line orifices.

All 4 AFW pumps were returned to service at 1305, based on the compensatory actions taken.

At 1410, the Engineering Director established a root cause team comprised of two Engineers from the Process Engineering Group, the AFW System Engineer, and a Design Engineer.

13

Possible Common Mode Failure of RCEO00191 Aux Feedwvater Recirculation Lines At 1525 the Engineering Director discussed the need for an operability determination with the Regulatory Affairs Manager and the Operations Manager because of the possible operable but nonconforming condition of the AFW pumps.

Simulator runs were made from about 1530 to 1630 to evaluate the risk impact of the crmpensatory measures taken.

At 1700, the Operations Manager accepted Safety Screening SCR 2002-0458 for the procedure changes needed due to the potential for AFW orifice plugging.

The 8-hour NRC notification required by 10 CFR 50.72(b)(3)(v)(D) was made using the Emergency Notification System (ENS) telephone at 1711. Event number EN 39330 was assigned to this notification.

The Operations Manager requested an OPR for CAP029952 concerning the common mode failure of the AFW pumps from Engineering at 1830.

At 2030, the Safety Monitor transitioned to Yellow (from the normal Green condition) for both units based on PRA Group input.

On October 30 at 0700, the initial meeting of the Root Cause Team was held to discuss scope and responsibilities.

At 1030, the Engineering Director suspended the qualifications for design work for three individuals involved with the AFW orifice modifications.

At 1100 the site was notified that an NRC Special Inspection Team would be responding to the Event Notification.

The Engineering Director approved the charter for the Root Cause Team at 1250.

OPROO0031 was completed and approved at 1850. OPROO0031 concluded that the AFW pumps were operable but nonconforming because the AFW pump recirculation paths described in the FSAR were not available. Compensatory actions specified included shift briefings and placement of temporary information tags. An additional action specified was to implement operating procedure changes. The condition where compensatory actions could be terminated was stated as an engineering evaluation or modification to restore the function of the recirculation lines.

On October 31, the Site Vice President established an Event Resolution Team to provide management oversight to drive resolution of the issue involving the potential to have a common mode failure due to orifice plugging in the AFW recirculation lines. The organization consisted of:

"* Team Lead - Nuclear Oversight Manager (temporarily relieved of job responsibilities)

"* Incident Investigation - Radiation Protection Manager 14

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines

"* On-Line Risk Management - Production Planning Manager

"* Interim Corrective Actions - Operations Manager

"* Issue Resolution Team and Root Cause Evaluation - Engineering Director III.

Extent of Condition Assessment Following the October 2002 event with the RO-04008 flow orifice associated with the P 38B MDAFW Pump, PBNP conducted a review of all flow-restricting devices that perform a QA-scope function. Attributes considered during this review were component classification, orifice style (passage size), and process medium. The AFW flow restrictors use a multi-stage anti-cavitation trim package installed in the body of a globe valve to limit flow. This type of flow restrictor uses very small holes in each stage along with a torturous path to limit flow and reduce the affects of cavitation.

A printout of the CHAMPS database was used as a starting point to determine the population of QA-scope flow restrictors installed in the plant. Next, a review of the plants P&IDs was conducted to determine if any orifices were installed in Safety-Related systems that did not have CHAMPS IDs. The CHAMPS and EMPAC databases were searched for parts associations and manufacturer codes to find additional flow restrictors similar to RO-04008. Finally, a limited plant walk-down and interviews with plant personnel were conducted to come up with a total population of 61 components that function as flow restriction devices in QA-scope applications (see Attachment E). Forty eight of these are in safety-related applications.

Attachment E also includes five components (1/2MS-279/283 and ORO-SA-001) that are non-QA-scope. The MS components are actually valves; however, they have a similar trim design to that used in the AFW pump recirc flow restrictors. The ORO-SA-001 components are of a similar design as the AFW Pump recirc piping flow restrictors.

These five flow restrictors are installed in non-QA-scope applications. While these components, primarily the MS-279/283, have experienced some plugging problems in the past, the fact that they perform no safety-related function results in these problems not challenging the plant's nuclear safety. In addition, the plugging that has occurred has historically been gradual, as the associated MS-279/283 flow restrictors are manual valves they can be adjusted to compensate for gradual plugging and allow time to schedule maintenance on the flow restrictors at the plant's convenience. The ORO-SA 001 flow restrictor has been modified from its original design because of plugging concerns. This has resulted in major reductions in ORO-SA-001 flow restrictor plugging concern. This design is acceptable for these non-QA-scope designs.

The Safety Injection System (HHSI, RHR, and CS) pumps all have flow restriction devices installed in their recirc lines; however, none are similar to that installed in the AFW pumps recirc lines. The HHSI pumps have multi-staged off-center orifice plates installed in their recirc lines; whereas, the RHR and CS pumps have venturi tube type (extended length small bore pipe) flow restrictors. These types of flow restrictors are not as susceptible to clogging as the anti-cavitation trim packages used in the AFW system because of larger flow passages. In addition, they pass primary system water as opposed 15

Possible Conmnon Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines to the Condensate Storage Tank and Service Water seen in the AFW system.

As can be seen in Attachment E, PBNP also utilizes standard orifice plates in a large number of safety-related applications. These flow restrictors are not susceptible to small particle clogging like the trim sets installed in the AFW lines because of larger size flow passages. The same is true for the large bore venturi tube style flow restrictors used at the PBNP. In addition, no venturi type flow restrictors were found installed in any safety-related applications that were not controlling primary system water flow.

The review of PBNP's flow restriction devices found only four applications (PB 1/2 RO 04003, RO-04008, and RO-04015) where anti-cavitation trim packages were used in safety-related application. These are the flow restrictors installed in the recirc piping associated with the four AFW pumps (PB 1/2 P-29 and P-38A/B). None of the other safety-related flow restriction devices were found to have this same susceptibility to small particle clogging due to their larger flow passages.

IV.

Nuclear Safety Significance The nuclear safety significance of this event has not been determined yet. The significance is dependent on the actual plugging probabilities for the AFW restricting orifices due to service water debris. Analytical testing of service water samples and sediment are currently in progress by Performance Improvement International (PH1) to assist in determining the plugging probabilities.

Immediate actions were taken to mitigate the consequences for potential blockage of the AFW recirculation orifices. These actions included posting temporary information tags, changes in procedures that provide operator direction to ensure minimum flow to the steam generator or secure the affected AFW pump(s). Training was performed with all crews to ensure understanding of the issue and simulator training was given to provide hands on use of the procedures that were affected. The total affect of these changes was to lower the on-line plant risk from a "High-Yellow" to a "Low-Yellow" Safety Monitor color (note: this does not equate to the SDP colors).

CA026958 was initiated to ensure compliance with procedure NP 10.3.7, "On-line Safety Assessment," while at elevated risk due to the potential for common mode plugging of AFW recirc orifices. Focus would continue with daily updates to the Event Resolution Team Leader until interim corrective actions had restored on-line risk to a "Green" condition. It was determined that a permanent modification would be needed to restore on-line risk to a "Green "condition. During the period of time when the AFW team was performing in-plant activities, daily updates on station risk management activities were provided. The weekly schedule was the starting point, with updates provided as items affect the planned work for the week. With those activities completed, it was concluded that CA026958 could be closed with recognition that the normal risk management process (NP 10.3.7) is managing station activities, as we would expect.

16

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines CA026900 has been initiated to track expeditious determination of the risk significance of this potential common mode failure related to RO plugging.

CA02691 I has been initiated to evaluate the AFW hydraulic system response to determine if the orifices can become plugged.

V.

Report to External Agencies This event was determined to reportable to the NRC in accordance with 10 CFR 50.72(b)(3)(v) as a condition that could have prevented the fulfillment of the safety function of structures or systems that are needed to: (D) Mitigate the consequences of an accident. This is an eight-hour non-emergency notification. The NRC notification was made using the Emergency Notification System (ENS) telephone at 1711 on October 291h. Event number EN 39330 was assigned to this notification.

A Licensee Event Report (LER 266/2002-003-00) will be submitted within 60 days of this event as required by 10 CFR 50.73.

CA026899 has been initiated to track issuance of the LER.

CA026903 has been initiated to issue an operating experience notice to the industry if appropriate.

CA026896 was initiated to notify KNPP of this event.

VI.

Data Analysis Information & Fact Sources Document Review Results EWR 99-031 Engineering Work Request EWR 99-031 was initiated on December 8, 1998 to address Control Room noise resulting from operation of the AFW pumps in the recirculation mode. The EWR was to determine if restricting orifices in the recirculation lines should be replaced with multi-stage restricting orifices. An action item was created on April 15, 1999 and assigned to Mechanical Design Engineering. In an update to this action item made on June 4, 1999, Design Engineer 1 concluded that most of the noise was due to cavitation in the restricting orifice, and that the orifice did not have an adequate number of stages. It was recommended that an EAC evaluation be performed to look at three options to resolve the issue. The options were installation of a multi-stage pressure reducing orifice, replacement of the present orifice with a pressure-reducing flow element in a valve body, and retrofitting the AOV control valve with a multi-stage pressure reducing trim.

On July 9, 1999 the Engineering Advisory Committee reviewed three options proposed to address EWR 99-031. The EAC did not select an option, but did recommend 17

Possible Common Mode Failure of RCEO000!9 Aux Feedwater Recirculation Lines implementing modifications for the MDAFW pumps during 2000 and then evaluate performance to determine if TDAFW pumps should be modified during 2001.

Modification requests MR 99-029*A-D were initiated to replace the AFW recirculation line restricting orifices and the EWR action item was closed on July 29, 1999.

Evaluator Observations:

"* Design Engineer I did not provide enough technical information to EAC regarding orifice size such that the plugging conceni could be identified.

"* EAC did not select which option should be pursued to achieve the desire modification, which is contrary to NP 7.1.6.

OD CR 99-1391 Rev. 0 of this OD was issued on May 21, 1999 and dealt with three pinhole leaks that had developed on the MDAFW pump recirculation lines in the past year. (This CR was the basis for performing RCE 99-081.) The safety function for the recirculation line is described as designed to ensure a minimum flow through the AF pumps to protect from adverse effects of hydraulic instability at low flow rates. The minimum flow rate for the MDAFW pumps (P-38A and P-38B) is 70 gpm, and for the TDAFW pumps (1P-29 and 2P-29) is 100 gpm, based on vendor recommendations.

In the section of the OD that discusses the basis for declaring the SSC Operable But Degraded or Nonconforming, reference is made to the Westinghouse LONF/LOAC Analysis and that it requires the AFW recirc AOV to close within 60 seconds after the flow setpoint has been reached. DBD-01 describes the delay in reaching full capacity for MDAFW pumps as being 35 seconds and for the TDAFW pumps it is 39 seconds. It is then concluded that the recirculation line AOV will not be open longer than 100 seconds.

The discussion then describes that various PBNP EOPs require the use of AF to maintain SG levels, and that this may require operation at low flows such that the recirculation AOV could be open. Part II of the OD specifies that implementation of MR 99-029*A-D will replace the existing orifices and that build up of the socket welds will occur during installation of those modifications.

The OD was revised on January 7, 1900 to combine it with an OD from CR 99-1844, which dealt with the cause of the weld failures. The OD was revised again on March 6, 2001 to reflect completion of MR 99-029*A&B for the MDAFW pumps, which restored them to a fully operable condition. The OD was closed on October 22, 2002 after MR 99-029*C and D were completed.

Evaluator Observations:

• The OD clearly recognizes the need for having AFW recirculation capability during an accident.

MR 99-029*A/B (MDAFW Pump ROs)

The purpose of the proposed modifications was to minimize piping line noise and vibration when operating the MDAFW pumps in the recirculation mode. This would also eliminate the socket weld failures caused by cycle fatigue created by this vibration.

18

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines The scope of the modifications was to replace the existing restricting orifices with new pressure reducing orifices. In addition, a portion of piping was being replaced which included oversized weld sockets.

Installation and testing were completed during November 2000. MR 99-029"B was accepted during November 2000; MR 99-029*A was accepted in March 2001, following drilling of additional holes in the innermost stage of the orifice to achieve the desired flow rate. The acceptance of MR 99-029*A was delayed because required flow rates could only be achieved with the use of clamp on ultrasonic flow meters, which was not considered acceptable for long-term operations.

A review of design documents identified the following relevant information:

Final Design

Description:

• DBD-01, Auxiliary Feedwater System, is listed as a design input

• The replacement RO will have the same function as the existing orifice RO-4008 (RO-4014), which is to provide pressure reduction and act as a pressure boundary for the AF system piping.

• It (Bechtel Specification No. 6118-M-6 Rev. 3 dated October 28, 1968) specifies that, "Each pump shall be furnished with a pressure reducing orifice to be used in conjunction with the on-off control valve in the pump recirculation piping."

• The design of the new RO is different than the presently installed orifice.

The replacement RO is heavier than (the) existing one.

• None of the above changes is introducing new, unknown equipment to PBNP.

• The proposed modification is located in the Control Building on Elevation 8', and adheres to the requirements of the Fire Protection Evaluation Report. A fire protection analysis, for the affected area, has been performed and the Fire Protection Conformance Checklist, PBF-2060 has been completed and approved.

Design Input Checklist (PBF-1584 Rev. 6, March 31, 1999):

"* Item A.6, Incorporate new types/models of equipment not presently used at PBNP? - APPLIES TO DESIGN? - NO

"* Item A. 10, Consider failure effects on structures, systems, and components:

(Failure analysis is only required for maintenance rule systems. Contact the NSA-PSA group for guidance and scope.) - APPLIES TO DESIGN? - NO F.2.b, Affect fire protection requirements? - APPLIES TO DESIGN - YES F.2.f, Based on Tables 6.7-1 to 6.7-4 and Figures 6.6-1 to 6.6-8b and 6.9-1 to 6.9 2d, will the change add to, delete from, or revise the listed systems and components? - APPLIES TO DESIGN - YES o The supporting Fire Protection Conformance Checklist stated "This orifice is in an Appendix R flow path and when it is placed back into service it will not affect the systems ability to perform."

Modification Request Checklist (PBF-1606 Rev. 5, July 24, 1998):

Item B.4, Component Instruction Manuals - N/A 19

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines Item B.14, EPIX Update - N/A

• Item B. 18, PSA Models and Documentation - N/A

• Item C.5, Spare parts stocking and scrapping inputs into CHAMPS - N/A

• Item D.1, Abnormal Operating, Normal Operating, and Refueling Procedures N/A Item D.5, EOPs, ECAs, CSPs - N/A

• Item E.4, Preventive Maintenance - initiate/revise CHAMPS callups - N/A Section I, ECRs - blank The MR 99-029*A package has an addendum with a revised IWP to drill 6 additional holes in the innermost stage of the orifice. The MR 99-029*B package utilized the existing IWP and revised it to drill 4 additional holes in the innermost stage of the orifice.

There is a pen and ink change to the IWP revising the number of holes from 4 to 6.

These changes to the IWPs were necessary due to low flow results during testing.

Evaluator Observations:

• The final design description does not discuss the potential for orifice plugging from service water

• The final design description does not discuss service water strainer size

• DBD-01 describes a safety-related function of the orifice to provide sufficient flow to prevent low-flow instabilities and excessive fluid temperature rise in the AFW pumps - this is not discussed in the final design description (Note that at this point in time the DBD has been determined to be in error and a change is pending to delete this open safety function)

• The basis for concluding that the RO is not new or unknown equipment does not address the new orifice size being less than the service water strainer

• The basis for not performing a FMEA is not provided (AFW is a maintenance rule system and the design input checklist requires a failure analysis)

• The Fire Protection Conformance Checklist answer to question 5.8, "Will the revised shutdown component continue to perform its function required by AOP IOA, AOP 1OB, AOP 10C, and AOP 1CD?" is marked YES with no basis provided.

• The drilling of additional holes was not controlled via ECRs.

SE 2000-0055 (MR 99-029*A/B)

The safety evaluation screening for MR 99-029*A/B concluded that there was a difference in the design of the replacement RO, so the proposed activity would constitute a change to the facility as presently described in the current CLB.

The approved safety evaluation contains the following relevant information:

"* The ROs do function to support operation of the AF system pumps P-38A and P 38B and these pumps are involved in accidents (Section 2.A.1)

"* The changes that will be implemented by the proposed modifications will not affect the overall performance of the AF System and operation or function of the AF pumps P-38A and P-38B to perform their intended function (Section 2.A.1) 20

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines During accident conditions, the safety related functions of each AF pump recirculation line orifice is as follows: (in part) These ROs must provide adequate flow to prevent low-flow instabilities and excessive fluid temperature rise in the AF system pumps (Section 2.A.2)

"* The components are passive in nature when the system is operational and will be designed, installed and tested in accordance with the existing procedures and controls. Therefore, they do not introduce any new failure mechanisms not already considered for the area. (Section 2.A.2)

"* The modified recirculation lines will function identically to the currently installed recirculation lines (Section 2.A.4)

"* The oversized socket welds and replacement ROs do not change the function, method(s) of operation, or introduce any new credible failure mechanisms to the AF pumps P-38A and P-38B and their recirculation lines (Section 2.A.5)

"* The flow passage area of replacement ROs could possibly lead to reduced pump recirculation flow during operation of the pumps with SW since particles/debris in the SW could be filtered by the RO's trim. To preclude this, the RO's design directs flow through the outside of the trim. The outside cartridge of the trim contains the smallest size flow passage area. The flow passage area then becomes progressively larger. Therefore, the smallest flow passage areas are located at the zone of highest differential pressure. This design feature reduces the potential of debris accumulation on the RO's trim. (Section 2.A.5)

"* The recirculation line AOV automatically closes approximately 45 seconds after the pump discharge flow is approximately 95 gpm and increasing. Failure to pass flow through the recirculation orifice during the 45 seconds would be conservative since flow to the SGs would be delivered sooner. The recirculation line AOV is also designed as a failed closed valve to ensure that recirculation flow is not diverted from the SG in the event of a loss of instrument air. (Section 2.A.5)

"* The recirculation line flow path is not required to support AF system in its response to the design basis accidents since the pumps' discharge valves will automatically open fully in response to the accident and provide a flow path for the pump. Failure to pass flow through the recirculation orifice due to potential of SW debris accumulation on the ROs trim would be conservative since flow to the SGs would be delivered sooner. (Section 2.B)

The Manager's Supervisory Staff reviewed the safety evaluation at a meeting on April 18, 2000. The version of the safety evaluation presented to MSS had different wording that generated some discussion. The MSS Chairman questioned a phrase in Section 2.A.5 that stated "The AF pumps have 9 stages with 0.009" to 0.014" diametrical clearances and a minimum 0.4375" impeller vane path. Since the pumps have multiple stages and small clearances, they will reduce larger particles size contained in SW to less than 0.015"." The MSS Chairman did not feel the statement was entirely true because he felt that particles could go through the vane paths. The AFW System Engineer said the pump had 9 stages and the particles would have to clear all 9 clearances in order to pass through, and this was unlikely to occur.

21

Possible Common Mode Failure oj RCEO00191 Aux Feedwater Recirculation Lines The Mechanical Design Supervisor said the discussion that could be included in that section is that of plant configuration and the fact that SW is used. He also did not like the quantitative discussion and recommended discussing the recirculation line flow path.

The Design Engineer subsequently removed the phrase from the safety evaluation and included a different discussion including the phrase "The recirculation line flow path is not required to support this function since the pump discharge valves will automatically open fully in response to the accident and provide a flowpath for the pump. The recirculation line AOV automatically closes at approximately 95 gpm increasing. Failure to pass flow through the recirculation orifice during the 45 seconds would be conservative since flow to the SGs would be delivered sooner. The recirculation line AOV is also designed as a failed closed valve to ensure that recirculation flow is not diverted from the SG in the event of a loss of instrument air."

With those changes made, MSS recommended approval of the SE. The Operations Manager approved the document in the absence of the Plant Manager.

The OSRC 50.59 Subcommittee reviewed and approved the SE without comment at its meeting held on June 27, 2000.

In an earlier, undated version of the draft safety evaluation, the following information was provided:

0 In summary, based on the large differential pressure across the ROs and the effect of pump discharge clearances on SW sample particle size, the replacement ROs design will not result in acute reduction in AF pumps recirculation flow, therefore, causing the pumps to fail. In addition, the body of the replacement ROs contain a bolted blind flange that can be removed to facilitate cleaning of the trim in the event of long-term debris accumulation. This long-term accumulation can be identified in advance by noticing a changing pattern of recirculation flow values that are recorded during IT-10/1OA/IOB.

Evaluator Observations:

"* The safety evaluation does not discuss the Appendix Rfunction (part of the CLB) of the recirculation line as a safe shutdown flow path.

"* The safety evaluation only discusses the recirculation line function during AFW pump start and does not address pump shutdown or the need for operation on recirc during the other stages of accident response.

"* The safety evaluation states that the modified recirculation lines willfunction identically to the existing lines even though it is assumed that the orifices may plug.

"* The safety evaluation does not consider that loss of the recirculation line due to loss of instrument air is recoverable via actions specified in AOP-5B.

"* The safety evaluation discussion concerning the orifice design that precludes plugging is extracted from the vendor manual, where the information is discussing why the internal throttle plug (on a different design orifice) will not stick or gall.

22

Possible Common Mode Failure of RCEO00191 Af x Feedwater Recirculation Lines 0 Neither MSS nor OSRC raised the aforementioned issues.

MR 99-029*C/D (TDAFW Pump ROs)

The purpose of the proposed modifications was to minimize piping line noise and vibration when operating the TDAFW pumps in the recirculation mode. This would also eliminate the socket weld failures caused by cycle fatigue created by this vibration.

The scope of the modifications was to replace the existing restricting orifices with new pressure reducing orifices that had a movable plug. In addition, a portion of piping was being replaced which included oversized weld sockets.

Installation and testing of the Unit I TDAFW pump RO was completed during September 2002, and the modification was accepted during October 2002. Installation and testing of the Unit 2 TDAFW pump RO was completed during April/May 2002, and the modification was accepted during May 2002.

A review of design documents identified the following relevant information:

Final Design

Description:

• DBD-01, Auxiliary Feedwater System, is listed as a design input (*D only)

• The replacement RO will have the same function as the existing orifice, which is to provide pressure reduction and act as a pressure boundary for the AF system piping.

• It (Bechtel Specification No. 6118-M-6 Rev. 3 dated October 28, 1968) specifies that, "Each pump shall be furnished with a pressure reducing orifice to be used in conjunction with the on-off control valve in the pump recirculation piping."

• The design of the new RO is different than the presently installed orifice.

The replacement RO is heavier than (the) existing one.

• None of the above changes is introducing new, unknown equipment to PBNP.

• A Fire Protection Conformance Checklist, PBF-2060 is listed as a design output

(*C only).

Design Input Checklist (PBF-1584 Rev. 8, June 8, 2001 for *D and Rev. 9, November 5, 2001 for *C):

"* Item A.6, Incorporate new types/models of equipment not presently used at PBNP? - APPLIES TO DESIGN? - NO

"* Item A.10, Consider failure effects on structures, systems, and components:

(Failure analysis is only required for maintenance rule systems. Contact the NSA-PSA group for guidance and scope.) - APPLIES TO DESIGN? - NO

• Item F.2.b, Affect fire protection requirements? - APPLIES TO DESIGN - NO

• Item F.2.e, Based on Section 2 and Appendix A of the SSAR, will the change add to, delete from, or affect the performance of safe shutdown systems or equipment?

- APPLIES TO DESIGN - YES (for *C) and NO (for *D)

Modification Request Checklist (PBF-1606 Rev. 5, July 24, 1998 for *D and Rev. 6, October 2, 2001 for *C):

23

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines PSA Models and Documentation - N/A

• Spare parts stocking and scrapping inputs into CHAMPS - N/A

• Abnormal Operating, Normal Operating, and Refueling Procedures - N/A

• EOPs, ECAs, CSPs - N/A

• Preventive Maintenance - initiate/revise CHAMPS call-ups - N/A

• ECRs - blank Evaluator Observations:

• The final design description does not discuss the potential for orifice plugging from service water

• The final design description does not discuss service water strainer size

• DBD-OJ describes a safety-related function of the orifice to provide sufficient flow to prevent low-flow instabilities and excessive fluid temperature rise in the AFW pumps - this is not discussed in the final design description

• The basis for concluding that the RO is not new or unknown equipment does not address the new orifice size being less than the service water strainer

• The basis for not performing a FMEA is not provided (AFW is a maintenance rule system and the design input checklist requires a failure analysis)

• Applicability of affect on Fire Protection is not addressed correctly for the *D modification.

• For the *C modification, the Fire Protection Conformance Checklist answer to question 5.6, "Does the modification affect operation of a system relied upon for post-fire safe shutdown (e.g. changes is system flow rate, change in normal position, etc.?" is marked NO with a statement that "Operation of the auxiliary feedwater system will not be affected. The replacement RO will be set to the same flow rate as the current RO." This is not consistent with assumptions from the previous modifications to the MDAFW pump ROs that the recirculation line may not be available due to plugging.

SCR 2001-0981 (MR 99-029*C/D)

The safety evaluation screening for MR 99-029*C/D was performed under the new 10 CFR 50.59 rule and concluded that the activity did not adversely affect the design function of an SSC credited in safety analysis, and did not adversely affect the method of performing or controlling the design function of an SSC credited in safety analysis.

Supporting information included:

"* The restricting orifices design functions affected by the modifications are: (in part) They ensure adequate flow and pressure drop through the AFW pumps when they are operated in the recirculation mode, thus preventing low flow instabilities and excessive fluid temperatures. (Section 1.2 and I)

"* These orifices are not explicitly required in an accident to be able to pass service water, since the recirculation control valve would be closed when the pump is aligned to the steam generator. However, it is possible that when the pump is aligned to the service water system supply after the condensate storage tanks have been drained, service water could be pumped through the recirculation lines.

(Section 111. 1) 24

Possible Common Mode Failure of RCEO00191 Aux Feediwater Recirculation Lines To preclude the chance of clogging the orifice trim, the flow is directed from the outside of the stages inward. The holes in the outer stage are the smallest and they get progressively larger in the inner stages. This causes the largest differential pressure to exist at the outer stages at locations with the smallest hole, which will reduce the potential for debris accumulation inside the orifice.

(Section Ill. 1)

The MSS and JOSRC did not review this information because it was determined to only be a screening under the new rule.

Evaluator Observations:

"* The safety screening does not discuss the Appendix Rfunction (credited in the safety analysis) of the recirculation line as a safe shutdown flow path.

"* The safety screening does not fully evaluate the recirculation line function while supplying service water to the SGs. The ability to start and stop the AFW pumps while on service water and to throttle back AFWflow to the SG is not addressed.

"* The safety screening raises the issue of the potential for the recirculation orifices to accumulate debris from SW but does not provide any definitive basis for the conclusions drawn regarding its impact on AFW pump operation.

"* The safety screening discussion concerning the orifice design that precludes plugging is extracted from the vendor manual, where the information is discussing why the internal throttle plug will not stick or gall.

"* There is no discussion explaining the change in conclusions drawn from SE 2000 0055 regarding not needing the recirculation line.

MR 02-029 (Upgrade safety function of AFW recirculation AOVs and line)

The purpose of this modification was to upgrade AFW minimum flow recirculation AOVs to have a safety related function to open.

The scope of the modification included removal of the internals for check valve AF-l 17 to prevent a common mode active failure of all AFW pumps due to isolation of the mini recirculation line. The AF-1 17 valve is non-QA, non-Seismic, and non-ASME. The modification documented the upgraded design basis of the AFW recirculation line AOVs and piping to support the safety-related function to provide a flow path for the AFW pumps to prevent overheating and hydraulic instabilities.

There was not EAC review of this modification because it was scoped as a Level of Effort Minor Plant Change.

This modification was initiated on August 20,2002, and was installed, tested and accepted on September 12,2002.

A review of design documents identified the following relevant information:

25

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines Final Design

Description:

"* The minimum recirculation flow AOVs have a safety-related function to close to ensure adequate flow to the steam generators during several events.

"* FSAR Section 10.2 also discusses the effects of a failure of a mini-recirc AOV to close and gives the flow that is diverted from the steam generators through the recirculation lines as limited by the flow restricting orifices.

"* These min-flow recirc AOVs have never been classified as having a safety related function to open to prevent pump damage. This has been described as a non-safety related function only, since the AFW pumps will always have forward flow to the steam generators on auto-start.

"* These recirculation lines AOVs have an augmented quality function to be opened for Appendix R fires to support AFW pump operation, per SSAR 2.3.1.4.

"* Letter NRC 2002-0068 dated August 12, 2002 states that PBNP will classify the open function for the AFW pump minimum flow recirculation valves as safety related. The letter also states that because not all of the recirculation flow path is safety-related, operability of the AFW pumps will not be dependent on the availability of that flow path. However, it has been conservatively decided to tie AFW pump operability to this recirculation line. Therefore, even though the recirculation line downstream of the orifices is not safety-related, it is required to be in-service to consider the AFW pumps fully operable per TS 3.7.5.

"* The current safety-related boundary for the recirculation lines is at the flow restricting orifices.

"* This line can be credited to support a safety-function while not being classified safety-related because failure of the piping would be conservative in terms of AFW pump protection.

"* The only credible failure of the piping that would cause AFW pump damage would be if check valve AF-1 17 failed to open (an active failure). Therefore, the internals for this check valve will be removed.

"* All other non-conservative failure modes for the recirc line are passive in nature.

Several manual valves exist in the recirculation lines, and all of these valves are currently red-locked open. Mispositioning is not credible due to procedural controls in place (red locks), and a disk separation failure of a manual valve is considered passive.

"* (Installation) Both units may be in any condition, but the common recirculation line must be isolated to remove the AF-1 17 internals. While this line is out of service, the AFW pumps will be considered fully operable. Manual operator action will be credited to prevent pump damage by stationing a level 3 dedicated operator at the AF-4035 relief valve. If the minimum flow recirculation line is needed, and the relief does not open automatically while the line is isolated, then the dedicated operator will notify the control room that the recirculation flow path is not available.

"* This action for the dedicated operator will be required only when the control operator has taken action to reduce AFW flow.

26

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines Design Input Checklist (PBF-1584 Rev. 10, August 19, 2002):

"* Item A.10, Consider failure effects on structures, systems, and components.

APPLIES TO DESIGN - YES.

"* Item A.10.a, The design discusses those events/accidents, which the system/components are to withstand? - APPLIES TO DESIGN - YES. The AFW recirculation line is required to support AFW pump operation during events where AFW is required to provide reactor heat removal. Most of the recirculation piping is non-Seismic, as are the CSTs, but a failure would be conservative in terms of recirc flow.

"* Item A.10.b, The failure effect of the system/components: - APPLIES TO DESIGN - YES. AF-1 17 internals are being removed to prevent a single active failure from making all AFW pumps inoperable.

"* Item F.2.e, Based on Section 2 and Appendix A of the SSAR, will the change add to, delete from, or affect the performance of safe shutdown systems or equipment? - APPLIES TO DESIGN - YES. The AFW system is a safe shutdown system credited for Appendix R. The mini-recirc AOVs are credited to be opened in this scenario to provide AFW pump cooling. Upgrading the safety function of the AOVs and removing the AF-117 internals improves this capability.

"* Item G. 1, Installation requirements/plant conditions have been determined?

APPLIES TO DESIGN - YES. Additionally, a level 3 dedicated operator will be present to monitor the relief valve, and notify the control room if it fails to open.

Modification Request Checklist (PBF-1606 Rev. 6, October 2, 2001):

• Item B.3, Component Instruction Manuals - N/A

• Item D.5, Spare parts stocking and scrapping inputs to CHAMPS - N/A Item J, ECRs - blank Evaluator Observations:

"* Design evaluation does not evaluate impact of safety function upgrade on all components in the AFW recirculation line (did not evaluate ROs or electrical circuit for AOVs)

"* Design evaluation does not evaluate impact of safety function upgrade on AFW modifications in progress (MR 99-029*C)

SCR 2002-0359 (MR 02-029)

The safety evaluation screening for MR 02-029 was performed under the new 10 CFR 50.59 rule and concluded that the activity did not adversely affect the design function of an SSC credited in safety analysis, and did not adversely affect the method of performing or controlling the design function of an SSC credited in safety analysis. Supporting information included:

The scope of the screening involves: (in part) State in the FSAR (FSAR 10.2) and Technical Specification Basis (B.3.7.5) that (1) the open safety function for all AFW pump mini-recirc valves is safety-related, and (2) the recirculation line downstream of the flow restricting orifices has a safety function and is required 27

Possible Comnmon Mode Failure of RCEO00191 Au v Feedwater Recirculation Lines for AFW operability, but the line is not safety-related since failure of the line is conservative. (Section I. 1)

"* The AFW system has the following functions described in the licensing basis (in part):

"o To provide sufficient feedwater to remove decay heat from both units for one hour during a station blackout (SBO) event (TDAFP only)

"o To provide sufficient flow to the steam generators to remove decay heat to achieve cold shutdown within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> following a plant fire (Appendix R)

"o To provide flow to the steam generators during plant startup and shutdown, and during hot shutdown or hot standby conditions for chemical additions and when operation of the main feedwater and condensate systems is not warranted. (Section 1.2)

"* The AF-1 17 check valve has an implicit function to open to allow minimum recirculation flow from the AFW pumps to return to the condensate storage tank(s). (Section II.1)

"* The safety-related flow restricting orifices in the recirc line from each AFW pump limit the flow and pressure from each pump. (Section 11.l.b)

"* The minimum recirculation valves for the AFW pumps have the design functions to isolate the minimum recirculation line to ensure that the AFW pumps deliver the required flow to the steam generators as needed to support the mitigation of accidents or events. (Section 1.2)

"* The only function identified for the AF-17 check valve was to open to allow mini-recirculation water from AFW pumps to return to the condensate storage tanks. It has no design function to close. (Section I1.1)

"* The change in designation of the open function of the mini-recirculation valves (AOVs) as safety-related has no adverse affect on the valve's function to open or close.

Evaluator Observations:

"* The description of design functions in Section HIfocuses on selected components and does not discuss the overallfunction of the line.

"* The discussion of the ROfunction in Section ll.l.b does not include its need to pass flow for pump cooling.

The potential for plugging of the restricting orifices is not discussed.

The discussions of specific design functions in Section II and III do not include the Appendix Rfunction.

SCR 2002-0377 (AFW Operability During MR 02-029)

The safety evaluation screening for AFW operability was performed under the new 10 CFR 50.59 rule and concluded that the activity did not adversely affect the design function of an SSC credited in safety analysis, and did not adversely affect the method of performing or controlling the design function of an SSC credited in safety analysis.

Supporting information included:

• The Level 3 Dedicated Operator will remain in constant radio communications with the control room, and the operator's only function is to monitor actuation of 28

Possible Commnon Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines AF-4035, and to notify the control room if the valve fails to open after AFW pumps start. If AF-4035 fails to relieve (open), the dedicated operator will notify the control room immediately while the pumps are still being cooled by forward flow, and the control room operators will know that when they reduce AFW flow to control steam generator level they will have to maintain the minimum forward flow in AFW pumps or secure pumps as necessary as directed by the EOPs as discussed above. (Section I.1)

The minimum recirculation lines for the auxiliary feedwater pumps and the recirculation header for the AFW pumps have a function of providing recirculation flow paths from the AFW pumps to prevent hydraulic instabilities and to dissipate pump heat. Hydraulic instabilities are prevented by the presence of flow restricting orifices in the individual AFW pump recirculation lines, so the function of maintaining individual pump cooling and the impact of diverted CST water are the concerns. (Section II)

Evaluator Observations:

The responsibilities for the dedicated operator in the safety screening only address AF-4035 monitoring and notification associated with AFW pump starts; there is no discussion of monitoring when flow is reduced to less than required minimum flow values.

a The potential for plugging of the restricting orifices is not discussed.

DBD-01, Auxiliary Feedwater System The DBD section on AFW Pump Recirculation Line Orifices (Section 3.16 in the current Revision 3 of the DBD) describes the Safety-Related Functions as:

1. These orifices shall provide passive flow resistance in the recirculation line of each AFW pump; thereby establishing the required min-recirc flow and pressure drop from the AFW pump discharge pressure to CST pressure. These orifices must provide sufficient flow to prevent low-flow instabilities and excessive fluid temperature rise in the AFW pumps [REF 9.5.117].

2. These orifices shall limit the recirculation flow in the event that the recirculation control valve fails to close during the AFW operation response to an accident

[REF 9.5.117].

3. These orifices shall passively maintain the AFW system pressure boundary integrity.

These functions have remained essentially the same since the original DBD was issued in 1994. However, as part of CA003702 from Root Cause Evaluation 01-069, the description of the AFW recirculation line in the FSAR, DBD-01, and the IST program was reviewed for consistency and accuracy, and required changes were initiated. One change that was identified was that the function regarding providing sufficient flow (Item I above) was not a Safety-Related Function because the recirculation AOVs did not have an open safety function. The original DBD-01 did identify the recirculation AOVs as having an open safety function based on the same reference provided for the ROs, REF 9.5.117, which was MR 88-099*A-C for increasing the recirculation line flow rate in response to NRC Bulletin 88-04. CR 97-3363 evaluated the open safety function of the 29

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines recirculation AOVs and concluded the DBD needed to be revised to reflect that there was no open safety function. That change was made in Revision I of DBD-0l on March 21, 2000.

As a result of CA003702, a marked-up draft of the DBD was created that showed moving the RO function for providing sufficient flow from the Safety-Related Function section to the Non-Safety-Related/Non-QA Function section. CA003702 was completed on August 13, 2002.

While this change was pending, the decision was made to upgrade the open function of the recirculation AOVs to safety-related. As part of the DBD-01 changes needed to support MR 02-029, the proposed revision to the RO safety function for providing sufficient flow was abandoned.

A related issue was identified regarding the statement in the DBD regarding the RO function to limit flow. This statement was also attributed to MR 88-099*A-C.

CAP029983 was initiated on November 2, 2002 to resolve the actual flow limit values for the ROs.

Evaluator Observations:

After it was recognized that the RO only had a Non-Safety-Related function to provide sufficient flow, there was no additional evaluation performed to change that function to Safety-Related, to support MR 02-029.

NP 1.6.5, Manager's Supervisory Staff and Qualified Reviewer Revision 3 (November 8, 1999) of NP 1.6.5 was reviewed to see what guidance was in effect when MSS reviewed SE 2000-0055 on April 18, 2000. Section 3.2.4 lists the responsibilities of an MSS Member as:

"* Review all proposed changes or modifications to plant systems or equipment where changes affect nuclear safety.

"* This includes 10 CFR 50.59/72.48 evaluation reviews and a review of the

- modification request design scope.

Qualified Reviewer responsibilities are listed as:

"* Review procedures or changes thereto, which affect nuclear safety as designated by the Plant Manager.

"* Review temporary changes to procedures described in Attachment B within two weeks.

In general, the MSS Members must meet the ANSI N18.1-1971 qualifications for their discipline. There are also requirements for a Qualified Reviewer to either have 10 CFR 50.59 Safety Evaluation qualifications or be designated by the plant Manger based on plant experience.

Revision 6 (November 14, 2001) of NP 1.6.5 was also reviewed. MSS Procedure requirements are basically the same as Rev.3. However, the 10 CFR 50.59 New Rule 30

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines that went into effect during March 2001 changes the basis for determining if a safety evaluation is needed. The MSS procedure still requires review of modifications that affect nuclear safety, but the results of the 10 CFR 50.59 screening no longer answer that question.

Evaluator Observations:

"* There is no requirement for MSS Members that review modification safety evaluations to be 10 CFR 50.59 qualified.

"* The procedure specifies MSS Members are to review the modification request design scope, but the practice is to only distribute the safety evaluation document for review.

"° The decision of whether a modification requires a 50.59 evaluation versus a screening is no longer an appropriate threshold for determining if the changes affect nuclear safety due to the new 50.59 rule.

NP 7.2.1, Plant Modifications Revision 6 (July 21, 1999) of NP 7.2.1 was reviewed to see what procedural controls were in place when MR 99-029*A/B was initiated. The procedure required that a justification be provided if the conceptual design was determined to be not applicable (Step 4.5.3). The procedure adequately raised FMEA question for consideration (in PBF 1584). The procedure directed review of the modification package by all team members (Step 4.7.7.b), although some documents can be included by reference only (Step 4.7.5.g). The procedure directed safety evaluations go to MSS for approval following the steps for final design review and independent technical review (Step 4.8). The procedure described FDGH design approval as addressing the question "Have all reasonable modes of failure been identified and addressed in the design?" (Attachment 1, step 4.7.6.c)

Revision 10 (June 12, 2002) of NP 7.2.1 was also reviewed to see what guidance existed regarding safety function upgrades related to MR 02-029. The procedure does not have guidance for evaluating a modifications effect on other work in progress.

Evaluator Observations:

"* The modification procedure does not specify what design documents are to be evaluated by each design team member and it allows incorporation of documents by reference, which does not ensure that design team members see those documents.

"* The procedure does not contain guidance on evaluating the impact of a modification on other work in progress.

NP 7.7.10, Q-List Nuclear Safety Classification for Structures, Systems, and Components Revision 1 (October 11, 2001) of NP 7.7.10 was reviewed to see what procedural controls were in place when MR 02-029 was initiated. This procedure provides the directions for revising and controlling Q-List classifications of SSCs. It does not provide direction on how to classify an SSC or how to evaluate a component's ability to support a safety function. It does describe interfaces with other programs to enable consistent 31

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines classifications and analysis. Those programs are EQ, FP, Appendix R, Seismic, ASME Section XI, Maintenance Rule, Procurement, Electrical components QA classification, and computer software QA.

Recirculation Flowpath Potential Failure Modes Review On November 6, 2002 a review of the AFW PRA Notebook, associated with the AFW pump recirculation flowpaths, was performed as part of the RCE000191 charter. An experienced KNPP Design Engineer who possessed no previous PRA knowledge or experience in the development of PRA assumptions or bases performed this review. The intent of this technical review was to identify any additional potential component failures and/or postulated failure modes not already addressed in the Point Beach AFW PRA model.

The Point Beach Senior PRA Engineer provided a brief overview of the AFW PRA Notebook structure and use of the PRA Notebook drawings and section tables.

I The applicable AFW Notebook drawings and section table failure modes, associated with the AFW recirculation system, were reviewed against applicable portions of the following documents:

• AFW system P&ID drawings - BECH 6118 M-217, Sheets 1, 1A, and 2.

• Fire Water P&ID drawings - BECH 6118 M-208, Sheets 1 and 2.

• Service Water P&ID drawing - BECH 6118 M-207, Sheet IA.

• DBD-01, Auxiliary Feedwater

• DBD-12, Service Water System, Sections 3.15, 3.20, and 3.20.2.

• FSAR, Sections 10.2,10.3, and 14.

• CR 99-1391, Operability Determination, Rev. 0, dated May 21, 1999.

• CHAMPS data for valves 1-AF-04002, 2-AF-04002, AF-04014, AF-04007, 1 AF-00 114, 2-AF-00 114, AF-00 115, AF-00 116 and orifice flow elements 1-FE 4049, 2-FE-4049, FE-4050A, and FE-4050B.

• Design Changes MR 99-029*A, B, C, D and MR 02-029.

The review consisted of reviewing the PRA data and assumptions against the data and assumptions located in the above listed documents. This was performed on a component by-component basis for the components associated with the AFW pump recirculation flowpaths. This included review of the main air/backup air supply circuits for the recirculation line AOVs and cooling water supply to the AFW pump bearings.

A summary of findings and items to be addressed were identified in Point Beach CAP030045 initiated on November 7, 2002. No significant Failure Modes and Effects issues were identified under this review.

Training Material Training documents were reviewed to determine what information on Auxiliary Feedwater and its Appendix R function was presented during engineering systems training.

32

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines Lesson plan TRCR 25.4, Plant Systems and Components (ESI-04), Rev. 1, November 9, 2001 and ESI-04-LPAFW, Auxiliary Feedwater System, Rev. 0, March 13, 2002 contained the following information. Service water is the backup water supply and provides cooling to the pump; however, it does not make any statement as to the service water strainer size and the ability of the system to pass flow. It also states that the recirculation valves open on pump start and when minimum flow through the pumps drops to below a preset value (<I 10gpm for P29 and <75gpm for P38). The lesson plan does not state that the recirculation valve or line has an Appendix R function for safe shutdown of the unit.

Lesson plans LP2413 ESP 95-4, Appendix R, Rev. 0, August 9, 1995 and ESI-02-LP015, Appendix R Overview, Rev. 0, May 5, 2002 were reviewed to determine the information presented to new Engineering employees. Lesson Plans PL2413 ESP 95-4 states as one of its learning objectives that the student is to be able list the major systems used for hot and cold shutdown. The lesson plan states that, "instruction provided is to be presented by an Appendix R system expert who will provide answers to questions verbally. As such, information contained in this lesson will be brief." The information prepared by Training included a Power Point presentation. This presentation is uncontrolled and is not retrievable. It is therefore impossible to determine the exact information presented during this training.

ESI-02-LPO15, Appendix R Overview states that one train is needed free of fire damage to maintain Hot Shutdown for greater that 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />. It does not explicitly state that this train is Auxiliary Feedwater; however, this is the only system that can maintain this condition. The lesson plan does not specifically identify the recirculation line as an Appendix R requirement.

Evaluator Observations:

The training materials do not provide a clear and accurate description of the design functions of the AFW recirculation line (and its components)

Interview Results Civil/Structural Design Engineer The engineer recalled that a meeting was held to discuss making the recirculation line safety related (modification MR 02-029). This meeting was held approximately 1.5 to 2 weeks prior to the installation of modification MR 99-029*C which, installed the last restricting orifice in the aux feedwater pump recirculation line. It was believed that Design Analysis arranged this meeting. This meeting was held after the decision was made to make the lines safety related but prior to the issuance of a letter to the NRC stating this position. It is believed that representatives from System Engineering, Mechanical Design, Design Analysis, and Design Drawings participated in this meeting.

No meeting notes were taken to document the discussion.

The purpose of the meeting was to discuss the relief and check valves. The discussion focused on the common portion of the line because the individual recirc lines were 33

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines already safety related. A review of the recirculation lines in their entirety for the ability to provide flow was not performed nor were the orifices discussed. The discussions included questioning if forward flow could always be credited and therefore preclude inoperability of the pumps if recirculation flow was lost. No resolution of this question was stated during the interview.

It was stated that the attendees were apprehensive about declaring a safety function for these lines since it was not designed or installed to meet this qualification. A discussion as to whether it was necessary to remove the relief valve or any other component that may cause problems in the future also took place. No components were identified during this discussion. All participants had reservations about the check valve modification (MR 02-029) and giving the recirculation line a safety function. He stated that the participants felt that the decision to make the line safety related would eventually "bite" US.

Civil/Structural Engineering stated that Engineering felt left out of the decision process to "do the right thing" to get past the red finding (NOV EA-02-03 1, for inadequate operator instructions while controlling AFW recirculation flow). Engineering felt that senior management dictated an action without proper input.

Design Engineer 1 The modification engineer for MR 99-029*A/B no longer works at PBNP and was contacted by telephone for this interview. The engineer was asked why this particular design of orifice was chosen. His reply was that this orifice was selected because it was the cheapest available design that would fit in the space available. The engineer stated adamantly that he knew that the orifice would plug instantly if exposed to service water.

He also stated that this was not a problem because flow would always be present to the steam generators because it would not be throttled to lower values. He stated that this was the accepted function of the line at the time (no safety function/not needed). He also stated that the system engineer agreed with the conclusion that the recirculation line did not have a safety function and was not required for the pump to be operable.

When asked why the design description did not include a discussion of orifice plugging, the engineer stated that he felt placing the discussion in the safety evaluation was acceptable because the safety evaluation was a part of the design package and it didn't matter where plugging was discussed.

Design Engineer 2 The engineer was asked to describe his involvement and knowledge of the process used to upgrade the safety function of the AFW Recirc AOVs. He replied that he was not involved in the discussions related to that decision, the information was sent to him after the fact. He stated that the modification to install a pneumatic back-up supply for the MDAFW recirc valves was always going to be safety related. The TDAFW system was a whole new back-up system and that the modification began as safety related. During the mod process (December 2001-January 2002 time frame), a decision was made to change the modification to augmented quality. Second hand information stated that the 34

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines Engineering Director made this decision. The modification was already installed on the MDAFW pump AOVs and the components for the TDAFW pump AOVs were purchased as safety related, therefore the effect of this decision was minimal. During installation of the modification for the TDAFW AOVs (April 2002) a decision was made to make the modification safety related. This occurred just prior to the NRC meeting, which was scheduled for April 29, 2002. Memo NPM 2002-0228 was issued by the Site Vice President explaining the upgrade decision In August 2002, a decision was made to add a safety function for the AFW AOVs to open. The engineer stated that he was not involved in this decision. The Acting Engineering Director was involved in this decision. The engineer stated that he had an approximately one minute conversation in the hallway with the acting director regarding the topic. He stated that the site's response to the NOV (NRC 2002-0068) dated August 12, 2002, stated that the line had an open safety function, however the recirculation line doesn't have to be available. This position did not make sense to the engineers. The engineer stated that he was told the modification for the open function had to be completed prior to the NRC inspection that was scheduled for September 23, 2002.

Operations raised a concern about not declaring the AFW system inoperable when the recirculation line was out of service. It was the decided to link AFW operability to the availability of the recirculation line. The engineer stated that he was not involved in any discussions until after the NOV response dated August 12, 2002.

The engineer understood that the modification involved both a physical change and a function change. He went on to state that if a physical change wasn't needed (removing the internals of AF 117) the upgrade to safety related probably wouldn't have been done via a modification. The impact on AOV testing (IST Program) and the operability/availability decision required a change to the Tech Spec Basis. In retrospect, the engineer believes he should have considered additional items. He did not see a problem with the orifice at that time. He felt "under the gun" to get the mod done and had to develop a contingency mod for the reactor head inspection. He also stated that the Site Vice President told him that the AFW modification was more important than the other work he had.

When asked what direction he received from his supervisor, he stated that his supervisor said that this was a lot to do and if it couldn't be accomplished to let him know and he would get someone to help. The engineer stated that this was the supervisor's management style and it was okay with his group.

The engineer was asked what his recollection was for modification MR 99-029*C, the last orifice to be installed. He stated that this was a copycat modification. He was the reviewer on the other orifice modifications and that he co-signed as the preparer of the safety evaluation with the original design engineer because he wasn't qualified. The engineer also stated that the modification was released after the NRC red finding became known. When asked if he made any connection with the upgrade of the recirculation line and the orifice modification, he said, no. The modification had already been done 3 times

- it was a "no brainer". He was aware of the NRC findings and its implications. He 35

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines however, did not relate it to the SCR for the orifice modification; he wrote the screening and moved on to other work.

When asked to relate any discussions on the orifice plugging issue, the engineer stated that the safety evaluation performed stated that the flow path made it difficult to plug the orifice. This information was taken from the vendor information on the orifice. He was involved with the MSSM discussion for the orifice evaluation. What he retained was that the orifice could pass service water without plugging. He stated that the wording in the safety evaluation and the screening does not mean that the line or orifice has a safety function, but that it has a design function. The DBD was wrong, and a marked up copy was produced to change the function of the orifice to a non-safety function. This change was never made because the decision to upgrade the AFW recirculation AOVs to have an open safety function changed what the orifice needed to do.

When asked if he thought it was appropriate to classify the upgrade modification as a minor plant change the engineer stated that the paperwork is the same. The only difference is that a minor plant change doesn't go to EAC. He also stated that EAC isn't being used much. The engineer stated that this was another example of a modification where he had to drop everything and do it. Most of the modifications he has worked on during the last year have been that way, e.g. AFW, SI, and reactor head modifications.

The engineer was asked if he had any concerns with the level of independence for the orifice modifications in light of being the preparer of one modification and the reviewer on the others. The reply that he was a member of a small group - independence is a luxury. Being the technical reviewer made him the best person to prepare the last orifice modification. The engineer stated that everything is hurried and sporadic in nature, which is difficult to work with. He believes that if he had more time there would be a higher probability that this may have been caught. He stated that he was uncomfortable with this. The engineer stated that he talked to other engineers about this but never raised it as an official concern with his supervisor.

Design Engineer 3 The mechanical design engineer interviewed stated that his involvement with the MR 99 029* modifications started with performing the technical review for the last modification being installed (MR 99-029*C). The design review for this modification was after the root cause was performed and before the regulatory conference for the red finding associated with auxiliary feedwater. He also aided with the installation of MR 99 029*A/B when additional holes had to be added in the trim assembly to obtain the required flow in the recirculation lines for the P-38A and B auxiliary feedwater pumps.

The engineer stated that the issue of plugging was that if the orifice would plug it would be good because more water was supplied to the steam generator. This was the accepted site position on the issue at this time. He also stated that he believed from a legal stand point the site was better off if the AOV didn't open because the plant was designed for forward flow only and recirculation flow only lessened the flow that could be supplied to the steam generators. It was stated that the Engineering Director at the time MR 99-36

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines 029*AID was being installed absolutely refused to consider the need for the recirculation line. The engineer also stated that there was an old letter (actually dated July 30, 1992) to the NRC stating that even though we tested the AOV in the recirculation line in both the open and closed direction the valve had no safety function in the open direction.

The mechanical design engineer stated that he had no direct involvement in modification MR 02-029. He stated that he felt that inadequate rigor was applied to the modification, that the time line was too short. He felt that the message was "Pull the check valve out and call the line safety related".

The design engineer stated that the recirculation flow was not an acceptance criterion for pump operability. (This was later determined to be inaccurate by reviewing the procedure revisions for IT 10. The recirculation flow did not have an acceptance criterion for IST purposes until September 9, 2002. However, the procedure has had a statement to check mini-recirculation flow > 70 gpm on the FIT since 1991, which operators monitored during testing.) He felt that the site upgraded parts/pieces but not the whole system when deciding to assign a safety function to the recirculation lines. As part of the upgrade process, the backup pneumatic supply to the AOVs was upgraded to safety related. The engineer stated that he originally wanted to install this modification (MR 02-001) as safety related and that someone above the Design Manager would not approve this proposal. This occurred in January 2002. Though the engineer disagreed with this position it was consistent with our understanding of the regulatory requirements and he could not argue the point.

Design Engineer 4 A fourth design engineer was interviewed about his involvement in the Aux Feedwater modifications. He stated that he started with NMC approximately one year ago; the NRC red finding had just occurred. He stated that his selection for work on MR 02-029 was based on his not being involved in any of the preceding modifications to the Auxiliary Feedwater System.

The engineer stated that he was modification qualified; however, his experience was limited to performing a review for a Safety Injection System vent modification. His knowledge of the Auxiliary Feedwater system was limited to the training he received in system fundamental training as part of ESP. When performing the MR 02-029 technical review, he stated that he spoke with other design engineers, reviewed piping and instrumentation diagrams, and walked down the recirculation portion of the piping. No mentoring was provided by supervision.

He stated that he was focused on what the modification was doing and now feels that the focus was too narrow. He stated that his focus was on mechanical components: the removal of the check valve internals. He stated that he did not consider that the portion of the line containing the AOV and orifice was safety-related when he prepared the modification. He also stated that this modification was not a classification change to the piping, it was a function change.

37

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines The engineer stated that he feels that there was time pressure to complete the modification; that it was "pushed through too fast." He stated that normally all groups listed on installation check sheet are talked to about their individual areas of responsibility; however, in this case this did not occur. He stated his concern as, "How often does Mechanical Design get involved in a circuit change?" (The reference to a circuit change was because a contact in the control logic for the AFW recirc AOV was found to be non-safety related - CAP030040.)

Mechanical Design Supervisor When asked about the safety evaluation for MR 99-029*A/B, the supervisor stated he believed that the wear ring would not grind up the particles, as did the MSS Chairman.

Therefore, the evaluation was changed to a nuclear safety oriented approach. This was based on not needing the recirculation line because of the AOV failure mode. He stated that the safety screening done for MR 99-029*C/D had similar thoughts but was stated somewhat differently. He did not think the orifices would plug and not requiring the recirculation line was an additional conservative position.

For the MR 02-029, he said there were a lot of other discussions on upgrading the line that he wasn't involved in. He was given the modification and had to rationalize how to do it. He recalled that the modification to install the backup pneumatics was originally going to be safety related, but he was told not to make it safety related. Later that was changed to be safety related to support the open safety function. He was given verbal directions for MR 02-029, and then had internal Engineering discussions with the other design groups to clarify the design specifics. He said there was nothing in the licensing basis review that pointed them to look at past decisions.

When asked if there was anything in the modification process that initiated a review of a modification that had been open for a long time pending installation, he said there wasn't.

(This question was asked because the final design for MR 99-029*C was approved on December 4, 2001, but was not installed until the end of September 2002.) He said the 50.59 for old open modifications were reviewed with operations (on an honor system) to see if anything had changed.

AFW System Engineer 1 The engineer that had AFW system responsibilities at the time of MR 99-029*A/B was designed was interviewed. The system engineer stated that the issue of plugging and Appendix R requirements were not considered when selecting the orifice for the recirculation line. The basis for selecting the orifice was that it was a "quick and dirty fix". A multistage plate type orifice was too long to fit in the available space that would result in an isolation valve being moved; that would involve a freeze seal. He wasn't sure how the choice was made between the remaining options. He thinks it may have been because the channel stream orifice was more likely to knock down the noise problem.

He does not remember discussing orifice selection with supervision. When the issue of service water flow through the orifice was raised, the engineer's reaction was one of dismay. The thought at that time was, "what can we do to salvage what we have and 38

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines make it work?" The engineer stated that it was felt that it was necessary to make the modification work despite a bad design instead of going back and correcting the design deficiency. That is why the plugging issue was addressed in the safety evaluation, not in the design detail.

The engineer stated that he does not remember seeing a version of the safety evaluation without a discussion of service water plugging. He stated that in the first version of the safety evaluation they had included the results of samples of service water taken from the AFW suction and compared it to the orifice size. He stated that he, not the design engineer who prepared the safety evaluation, added the analysis describing the effect on the particles as they passed through the AFW pump to the safety evaluation. Staff (MSS) didn't like that and the Operations Manager said it doesn't make a difference because the AFW recirc AOV fails closed. He said he believed that any build up on the orifice would be gradual, not acute, but staff wanted a yes/no answer. The position that the line was not needed seemed conservative and he had no problem with the change to the safety evaluation.

He stated that he did not make a connection between the outcome of the AFW Red Finding (importance of proper recirculation line operation) and the last RO modification being installed.

The engineer was involved in a later modification, MR 00-0077 which replaced an AFW discharge flow control valve trim. This modification package specifically stated that the flow holes in the valve trim were larger than the 1/8" mesh size of the service water strainer. When asked why this information was included in this package, he replied that it was a lesson learned from involvement with modification MR 99-029*A/B.

AFW System Engineer 2 System Engineer 2 became the Auxiliary Feedwater system engineer in April 2002. The engineer stated that he had no involvement in the MR 99-029*A-D modifications; consequently, he had no knowledge of the design of the orifices that were installed. The engineer's direct involvement in MR 02-029 only included the performance of the review of the safety screening for the modification. He stated that he attended several meetings dealing with upgrading the recirculation line. One he attended discussed the cost of upgrading the Auxiliary Feedwater recirculation line to safety related and the impact on the In Service Testing Program of such a modification. Another meeting discussed what was to be performed and by whom to support MR 02-029. There was no discussion as to the status of the electrical support systems for the recirculation line. It was believed that since the valves are already safety related that there was no need to evaluate their ability to fulfill an open safety function. The RO was also discussed, however, this discussion only dealt with whether or not the safety related portion of the line would include the orifice. The discussion did not include whether or not the orifice could support a safety related function to pass flow. The engineer stated that he felt that there was time pressure to complete the modification (approximately three weeks from inception to installation).

39

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines Fire Protection Engineers Two engineers with fire protection responsibilities were interviewed. One of the individuals participated on the modification design team for MR 99-029*A/B and the other participated on the design team for MR 99-029*C/D.

The engineer on the design team for MR 99-029*A/B stated that there was little if any discussion of orifice plugging associated with their design. He personally felt that plugging was not an issue with the proposed modification. He stated that the installation of the orifices was a non-outage work package fire protection issue and not an Appendix R operability issue.

He also stated that he never saw the safety screening or evaluation for the modification.

The only information he would review was the information in the detailed design description and the fire protection checklist contained in the modification package.

The engineer stated that the orifice was an integral part of the piping system. To his knowledge, the change in orifice design did not change the fit/form/function of the component. To his knowledge the orifice remained a passive component in the system and did not require a QA code for Appendix R or fire protection.

Discussions with the second fire protection engineer (MR 99-029*C/D) revealed the following. It was assumed that the orifice was a piece of pipe. He had no idea that it could plug and nothing in the design description told him other wise. "The orifice worked yesterday and it will work tomorrow unless someone tells me different and no body did."

The air accumulators used to open the recirculation line AOVs are installed to buy the site time to manually gag open the valves and make AOP I0A work, that is why there is a QA code 20 associated with the valves and air tanks. We do not need to manipulate the orifice therefore there is no QA 20 code associated with them.

MSSM Chairman When interviewed, the MSS Chairman stated that the purpose of the MSS was to review safety evaluations and to either concur or disagree with the conclusions reached. It was not their responsibility to review the design package. The individual stated that he has not (at least recently) read the procedure governing the functions of the MSS. He stated that he is not qualified to perform safety evaluations and does not know if anyone on MSS is qualified to perform safety evaluations. It was stated that the MSS does not take a vote on the safety evaluation but a consensus is reached. If any member of the MSS has a strong dissenting opinion, the individual has the right to bring it to the attention of the Plant Manager. He stated that this has happened in the past.

The safety evaluation (SE 2000-0055) for modifications MR 99-029*A/B stated the orifice might plug if service water was used as the suction source. The staff did not believe that the pump was capable of grinding any sand/silt into small enough particles to allow them to flow through the orifice. The staff member stated that staff recommended 40

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines that service water be sampled to determine the size of the grit entrained in it. [It should be noted that this was performed prior to the safety evaluation being presented to MSS.] He believes that Engineering's opinion was that the orifice would not plug but did not present a conclusive argument to back the opinion.

Staff stated: there was no safety function for the recirculation line; they were aware that instrument air was needed for the recirc line AOV to work and instrument air is not safety related. Therefore, there was no guarantee that the AOV in the recirculation line would open. Consequently, since you could not rely on the AOV to open to provide a recirculation flow path, it did not matter if the orifice plugs and prevents recirculation flow. It was also stated that MSS has been criticized in the past for getting too involved in the details associated with safety evaluations; therefore, they didn't require this issue to be resolved operationally, nor did they assess the installation of a "filter" in the line.

Operations Representative An interview was conducted with the Operations Department Auxiliary Feed Water System owner. He stated that he has been the owner of this system for in excess of four years.

When questioned about his involvement with modification MR 99-029*A-D, he made the following statements. He was unaware of any consideration as to the size of the service water strainer mesh when selecting the restricting orifice. He also stated that he was not aware of the size of the flow channels in the restricting orifice until November 2002. He does not remember seeing the safety evaluation or screenings associated with the implementation of this modification.

The Operations system owner was aware that an operability determination (OD 99-1391) had been written on the Auxiliary Feedwater System when weld cracking occurred on the recirculation lines. However, there were no compensatory actions required by this OD.

There is no routine call up for review of operability determinations; therefore, the issue of the need for the recirculation line to be operable when throttling flow to the steam generators was lost. The system owner was also aware that there was a 10CFR50 App. R function for the air-operated valve in the recirculation line. He however, did not associate this requirement with the need for the recirculation line to be operable for the pump to be able to perform its safety function.

It was stated that per procedure, any time that the indicated recirculation flow did not meet the minimum requirements stated in IT 10 the Auxiliary Feedwater Pump was inoperable. This requirement was instituted when the flow meters were installed on the recirculation lines in the early 1990s. (As stated above in the Design Engineer 3's interview, a review of IT 10 procedure revisions shows that recirculation flow was not an IST acceptance criteria until September 2002.)

Modification MR 02-029 was initiated to remove the internals of check valve AF-l 17 and give the recirculation a safety related function. Operations involvement was to identify and implement any necessary procedure changes to implement this modification.

41

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines It is his understanding that no one performed a comprehensive review of the Auxiliary Feedwater system or the modifications that had taken place to that system as part of design review of MR 02-029.

Thermal/Hydraulic Design Supervisor He recalled that the issue about upgrading the AOV and recirc line came up during the NRC SSDI time frame (July 22-August 9, 2002), and he was only briefly involved. He participated in a brainstorming session on what would have to be done to upgrade the recirc AOVs to have an open safety function. He believed that upper management made the decision.

On about August 19, 2002, he was assigned oversight of the upgrade modification. He said the schedule was very pressured with several aspects of the normal design development process being circumvented. The design engineer had some reservations with the modification scope and direction. This is contrary to normal practice where the design engineer evaluates various options and chooses an approach. He recalled that NMC Corporate Staff were brought in to perform an independent critique of the proposed changes out of concern for the rushed and extraordinary circumstances. During the course of the modification, at least one major scope change was made from simply upgrading the open function of the AOV and removing a possible common mode failure (AF-117), to upgrading the line to pass flow.

Acting Engineering Director The Acting Engineering Director was filling the position of Design Engineering Manager at the time EWR 99-031 was presented to EAC and MR 99-029*A & B were being installed. Her only recollections of those activities were that a new RO having additional stages wouldn't fit in the line without extensive changes, and that she had a brief conversation with the Mechanical Design Supervisor as a result of the failure of the factory tests of the new ROs, around August 2000. At that time the Mechanical Design Supervisor described the orifice design as three concentric cans with small holes and expressed his opinion that they were not susceptible to service water plugging based on vendor information.

When asked how the decision was made regarding the upgrade of the AOV safety function, she recalled the following facts. Prior to the NRC SSDI (began on July 22, 2002) the Site Vice President asked what it would take to upgrade the safety function.

Program Engineering held a meeting and evaluated the impact due to EQ and IST program changes, which was a small impact. A brief conversation with Design Engineer 2 revealed a concern about having only one common return line to the CSTs and that this may not be adequate. She developed a rough estimate of $250K based on this information and provided it the Site Vice President. She also asked a design engineer to do a walk down of the AFW recirc line back to the CSTs (during the last week of the SSDI -August 5, 2002).

She stated that the decision was wanted by the Site Vice President and by Hudson personnel and that it had to be done by the time of the NRC follow-up inspection for the 42

Possible Common Alode Failure of RCEO00191 Aux Feedwater Recirculation Lines AFW Red Finding. The commitment to perform the upgrade was included in the NOV response dated August 12, 2002. Oversight of the modification was assigned to the Thermal/Hydraulic Design Supervisor around August 19, 2002, because he was on vacation during the week following the SSDI (week of August 12, 2002). She said she was worried about single failures. There was an assist visit performed by NMC personnel that looked at the modification during the week of August 19, 2002. At this point the responsibility for PBNP Engineering was transitioned to the new Engineering Director.

Industry and Station Operating Experience CAP013812 (CR 01-1259) Foreign Material Exclusion (FME) for Aux Feed Work CAP013812 was originally issued as CR 01-1259 on April 13, 2001. This CR describes a low flow condition for the P-38A Auxiliary Feedwater Pump. This condition was discovered during PMT for close out of work order WO 9934148. This work order involved drilling 6 holes in the trim for RO-4008 to increase the flow through the orifice from 67 gpm to75 gpm.

The flow increase after this work was performed did not meet expectations. The calibration of the flow instrument was checked and found satisfactory. CAP013812 generated Corrective Action (CA) 006836. This CA recommended that it be determined if Foreign Material Exclusion (FME) practices were adequate for the work performed.

Maintenance staff interviewed the workers who performed this task. The workers stated that they followed established FME practices and that there was no debris in the orifice when it was reassembled. They also pointed out that there was a QC hold-point for cleanliness prior to assembling the orifice.

The CA was closed with no further action required on August 28, 2001. The scope as defined by the CAP and associated CA were narrow, in that the problem was defined as an FME issue and not a low flow condition. Since foreign material introduction to the system due to work practices was ruled out as the cause of the low flow condition, another mechanism had to be the cause. No corrective action or condition evaluation was initiated to determine why flow was low. This was a missed opportunity to identify a potential common mode failure for plugging the orifices.

The INPO and NUTRK databases were searched for similar events. There are no events that discuss the potential blockage of an AFW system orifice. The only AFW system orifice related event found was the one submitted by Point Beach (OE13861) that discussed weld failures due to cavitation downstream of the recirc orifice. (See CAP013812)

EPIX A search of the INPO EPIX and NPRDS databases did not identify any previous failures of the same model restricting orifices. Other utilities that use the same style ROs were contacted, but none reported using them in a similar application.

43

Possible Common Mode Failure of RCEO00091 Atux Feedwater Recirculation Lines Evaluation Methodology & Analysis Techniques The primary analytical techniques used in this investigation were interviewing, document review, barrier analysis, and stream analysis. A timeline of the event is presented in Attachment A, the barrier analysis is presented in Attachment B, and the stream analysis is presented in Attachment C. The Charter for this RCE is included as Attachment D.

Data Analysis Summary MR 99-029*AIB - New ROs for the MDAFW Pumps The need for a new AFW recirculation line restricting orifice design was created by the noise and vibration resulting from cavitation in the orifices installed under MR 88 099*A-C. RCE 99-081 later identified the causes of that design problem. EWR 99-031 was initiated to provide a new design. Three options were developed by Design Engineer I (DEl) and presented to EAC on June 30, 1999. The information presented did not go into specific details of the design of the new orifices. No dimensional information was discussed that would have alerted EAC members to the potential for orifice plugging due to service water debris. The description provided to EAC was: "The pressure-reducing flow element consists of a set of multi-stage trim for energy dissipation installed in a valve body, capped with a blind bonnet flange." This description did not evoke questions about potential plugging concerns (Inadequate Communications Within an Organization). EAC recommends performing modification but does not select specific option to be pursued as specified by the EAC procedure (NP 7.1.6, Rev. 0) (Inadequate Tracking).

During initiation of the modifications, the Mechanical Design Supervisor (MDS) waived the requirement for a conceptual design; however, no justification was provided as required by the modification procedure (NP 7.2.1, Rev. 6) (Inattention).

Based on document reviews and interviews, it is concluded that DE1 did not consider service water strainer size and the potential for plugging during the design development; the FMEA portion of the design input checklist was marked as not being applicable to the design and the Fire Protection evaluation concluded that the Appendix R flowpath would not be affected by the new RO design (Tunnel Vision). [DE1 was an experienced design engineer (20+ years) that was augmenting the Plant's engineering staff.]

The new ROs were ordered during December 1999. During development of the safety evaluation around February 2000, it was realized that the new orifices had dimensions that could make the ROs susceptible to plugging. DE1 decided to disposition the issue via the safety evaluation instead of re-evaluating the design (Fear of Failure/Shortcuts Taken). This approach was condoned by the MDS (Inadequate Program Management). At this point, the resolution of the plugging issue is moved from the design process into the licensing process. The process controls for ensuring an adequate design are bypassed, including performance of an adequate FMEA and independent verification of information used in the safety evaluation (normally would have been a design input).

44

Possible Common Mode Failure of RCEO00191 Aux Feediwater Recirculation Lines A rationale was used that the AFW pump would grind up service water debris to a size that would pass through the orifice (Flawed Analytical Model). Information was taken from a section of the RO vendor manual on dirt and debris, and was used to fortify the argument that the orifice would not plug. That information was taken from a section where the discussion was describing why the movable plug (not used in this model) would not stick or gall (Misdiagnosis).

Draft design documents for MR 99-029*A & B were distributed to the Design Team for review on February 16, 2000; however, the distribution memos stated that not all documents were included in everyone's package. The modification procedure (NP 7.2.1, Rev. 6) states some documents can be included by reference only (Methods Not Clearly Described). A safety evaluation is listed in the memos as one of the design documents.

Fire Protection Engineer 1 and the Operations member of the Design Team did not recall seeing the safety evaluation that had the information on plugging (Inadequate Communications Among Organizations).

The proposed safety evaluation correctly listed the FPER Auxiliary Feedwater System flow diagram as a CLB reference, but did not describe the Appendix R recirc line function any further in the text (Shortcuts Taken). Other design documents clearly indicated that this function was known. The safety evaluation discussion of the AFW recirc line function described its role during AFW pump starts, but did not address pump shutdown or the use of the line when AFW flow is throttled (Inadequate Knowledge of Standards). Design Engineers 1 & 2 signed as preparers of the proposed safety evaluation on April 10, 2000. Design Engineer 2 (DE2) co-signed because DE1 was not qualified to perform safety evaluations. AFW System Engineer 1 (AFW SE1) reviewed the proposed safety evaluation on April 13, 2000 and failed to ensure that all of the design functions were addressed. He believed that the ROs would not plug and that these related functions were not impacted (Wrong Assumptions). He also did not detect that the vendor information was being misapplied (Inadequate Verification).

Safety evaluation SE 2000-0055 was presented to MSS for review on April 18, 2002. At this point, the final designs for MR 99-029*A & B had not been completed, verified and approved (Inattention). The safety evaluation procedure (NP 10.3.1, Rev. 12) states that the safety evaluation shall be performed on the final design of the facility change (Actions not Tied to Another Process When Necessary). The MSS procedure (NP 1.6.5, Rev. 3) states members review all proposed changes or modifications to plant systems or equipment where changes affect nuclear safety. This includes 10 CFR 50.59 evaluation reviews and a review of the modification request design scope. Contrary to this, the actual practice is to distribute just the 50.59 safety evaluation to members for review (Methods Not Cleary Defined). At the MSS meeting, the Chairman questioned the rationale used about the AFW pumps grinding up debris to less than the orifice size and subsequent discussion suggested a more qualitative approach be taken. The Operations Manager stated that the recirculation line isn't needed anyway since the recirc AOV fails closed (on loss of instrument air), so it didn't matter if the RO plugged. This argument failed to consider the Appendix R function of the line and its role later in 45

Possible Common Mode Failure of RCEO00191 Aux Feediwater Recirculation Lines accidents when AFW flow is throttled and that AOP-5B had instructions for manually opening the valve (Inadequate Knowledge of Standards). The Operations Manger had been previously licensed at a different PWR plant.

The safety evaluation was revised by DEI/DE2 to incorporate the qualitative argument that the line wasn't needed. The preparers failed to consider that the loss of the AFW recirc line due to loss of instrument air was a recoverable failure via AOP-5B whereas loss of the line due to plugging was not recoverable, and that the Appendix R function would be lost (Inadequate Knowledge of Standards). The safety evaluation reviewer (AFW SE 1) also failed to recognize this (Inadequate Verification). A statement was left in the safety evaluation that the modified line would function identically to the existing line (Inattention). MSS approved the safety evaluation after revision to incorporate the qualitative argument that plugging didn't matter because the line was not needed. The Operations Manger signed the safety evaluation for the Plant Manger on May 8, 2000.

The RCE Team initiated CAP029997 on November 4, 2002 to evaluate the concerns discussed above regarding the adequacy of SE 2000-0055.

The Design Team members for MR 99-029*A & B signed off on the final design between May 17 and 22, 2000 after the safety evaluation had been approved (Inattention).

Independent verification of the final design for MR 99-029*A & B was performed by DE2 on May 31, 2000, which was after the safety evaluation had been approved (Inattention). The Design Control procedure (NP 7.2.2, Rev. 5) states the design verification be performed by a competent individual other than the originator or any other person with direct responsibility for that design. DE2 was a co-preparer of the safety evaluation and lacked independence. This was attributed to the small size of the Mechanical Design Group, but alternative approaches were not pursued (Inadequate Program Management). Based on the MSS conclusion that the recirc line wasn't needed and his own belief that the line would not plug, the reviewer agreed with the design input checklist that indicated FMEA wasn't applicable and that the Appendix R flowpath would not be affected (Mfindset).

Based on the MSS conclusion that the recirc line wasn't needed and his own belief that the line would not plug, the Mechanical Design Supervisor approved the final designs for MR 99-029*A & B on June 2, 2002. The modification procedure identifies one of the responsibilities of the Final Design Group Head design approval as addressing the question "Have all reasonable modes of failure been identified and addressed in the design?" The Supervisor failed to ensure that an adequate FMEA was performed as required by the modification procedure (Mindset). A definitive resolution of the plugging issue was not pursued (Inadequate Emerging Issues Management). The final design was approved after the safety evaluation had been approved (Inattention).

46

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines The 10 CFR 50.59 Off-Site Review Subcommittee met on June 27, 2000 and reviewed SE 2000-0055 without comment. The subcommittee members failed to identify the flawed reasoning associated with not needing the AFW recirc lines (Inadequate Knowledge of Standards).

The new RO for P-38A was installed and tested on November 2, 2000. Some difficulty was encountered in obtaining the desired flow rate. Indicated flow was 66-67 gpm, so clamp on ultra-sonic flow elements were used. Flow rates of 72-75 gpm were obtained via that method, so the test was accepted. The IWP for MR 99-029*B was modified to allow drilling of additional holes in the innermost orifice stage of the orifice if desired flow rates could not be obtained. The new RO for P-38B was installed and tested on November 10, 2000. Unacceptable flow rates (64-70 gpm) were obtained,during the test, so Attachment B to the IWP was executed, which resulted in the drilling of 6 additional holes in the innermost orifice stage. The vendor had provided drawings for this and a vendor representative was present for the drilling evolution. A field change was made to the vendor drawing to increase the number of holes required from 4 to 6. No ECR was generated to document the acceptability of this change. DE2 stated that this was part of meeting the initial design criteria (Mindset). Flow rates of 75-81 gpm were then obtained.

Additional holes (6) were drilled in the orifice for P-38A on March 14, 2001 under an addendum to IWP 99-029*A. No ECR was generated to document the acceptability of this change (Mindset). Flow rates of 70-73 gpm were then obtained. The flow rate change obtained was less than expected. CAP013812 was initiated on April 13, 2001 raising a concern with potential FME issues associated with that work. CAP013812 generated Corrective Action (CA) 006836. This CA recommended that it be determined if Foreign Material Exclusion (FME) practices were adequate for the work performed.

Maintenance staff interviewed the workers who performed this task. The workers stated that they followed established FME practices and that there was no debris in the orifice when it was reassembled. They also pointed out that there was a QC hold-point for cleanliness prior to assembling the orifice. The CA was closed with no further action required on August 28, 2001. The scope as defined by the CAP and associated CA were too narrow, in that the problem was defined as an FME issue and not a low flow condition (Wrong Assumptions). Since foreign material introduction to the system due to work practices was ruled out as the cause of the low flow condition, another mechanism had to be the cause. No corrective action or condition evaluation was initiated to determine why flow was low and the CAP was closed (Inadequate Verification). This was a missed opportunity to identify a potential common mode failure for plugging the orifices.

MR 99-029*C/D - New ROs for the TDAFW Pumps The design of the ROs for the TDAFW pumps was changed to a similar model RO as used on the MDAFW pumps but with a movable center plug that allowed flow to be adjusted if needed. This would eliminate the need for drilling additional holes in the orifice if desired flows were not obtained. The stem for the movable plug could be lock wired in place once desired flows were set.

47

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Line.s MR 99-029*D was in progress to install the new RO for TDAFW pump 2P-29. On August 2, 2001, DE I distributed a design package to Design Team members for review, which included a reference to a safety evaluation. It is not known if the safety evaluation was distributed with the package to every member, selected members, or incorporated by reference only. Discussions with Fire Protection Engineer 2 and the Operations member indicated that they did not recall seeing a safety evaluation or screening that had information on plugging (Inadequate Communications Among Organizations). DE I left the PBNP Engineering organization during August 2001. A new Project Manager was not assigned to the modification (Inadequate Program Management). DE2 facilitated changes to the design package documents and obtained an e-mail approval from DEl on December 4, 2001. No changes were made to design documents to include a discussion of the plugging issue or its resolution (Shortcuts Taken).

The wording of questions in the design input checklist related to fire protection had been revised. The new question F.2.e asked "Based on Section 2 and Appendix A of the SSAR, will the change add to, delete from, or affect the performance of safe shutdown systems or equipment?" This was marked as not applying to the design (Inadequate Knowledge of Standards/Mindset).

Some of the same inappropriate actions taken in MR 99-029*A"B were repeated in this modification:

"* MDS waived conceptual design without providing justification (Inattention)

"* FMEA was marked not applying in the design input checklist (Mindset)

(DE1/DE2)

"* Independent Reviewer (DE2) was preparer of the safety screening and helped revise design documents, thereby lacking independence (Inadequate Program Management)

"* Independent Reviewer (DE2) agreed that FMEA and Appendix R did not apply to the design (Mindset)

"* MDS approved the design and failed to ensure an adequate FMEA was performed (Mindset)

Modification MR 99-029*D was installed and tested satisfactorily on May 11, 2002.

The new 50.59 rule went into effect during March 2001, so the 50.59 review of this modification resulted in a screening instead of a full evaluation. DE2 converted the draft safety evaluation from the design package into a safety screening, SCR 2001-0981. The SCR identified the AFW Section of the FPER as part of the CLB, but did not list the Appendix R flowpath as a design function (Shortcuts Taken). The SCR stated replacement of the RO would not adversely affect its design function and that the RO would allow flow to maintain TDAFW pump operability when in the recirculation mode.

The SCR also stated that service water could be pumped through the recirc lines. The SCR raised the issue of the potential for the recirculation orifices to accumulate debris from SW but did not provide any definitive basis for the conclusions drawn regarding its impact on AFW pump operation. There is no discussion explaining the change in 48

Possible Common Mode Failure of RCEO00191 Attx Feedwater Recirculation Lines conclusions drawn from SE 2000-0055 regarding not needing the recirculation line (Mindset). The SCR utilized vendor information on why the RO design precluded plugging (Misdiagnosis).

AFW SEI performed the review of SCR 2001-0981 for MR 99-029*C/D. He did not ensure that all of the AFW design functions were addressed (Wrong Assumptions); he did not ensure that the SCR content was technically correct because the vendor information regarding the RO design precluding plugging was used (Inadequate Verification); and he did not ensure that the conclusions were supported (Mindset).

The RCE Team initiated CAP030212 on November 21, 2002 to evaluate the above concerns regarding the adequacy of SCR 2001-0981.

MSS did not review the modification because a safety evaluation was not required. The MSS procedure requirements were basically the same as when SE 2000-0055 was processed. However, since then the 10 CFR 50.59 New Rule changed the basis for determining if a safety evaluation is needed. The MSS procedure still required review of modifications that affect nuclear safety and used the threshold of performing a safety evaluation as a trigger, but the results of the 10 CFR 50.59 screening no longer adequately answer that question (Actions Conflict With Another Process).

A nuclear safety engineer initiated CAP030174 on November 18, 2002 to address this concern.

JOSRC did not review SCR 2001-0981 because they do not review safety screenings, only safety evaluations.

MR 99-029*C was in progress to install the new RO for TDAFW pump 1P-29. DE2 was assigned as the modification engineer on February 18, 2002. This change was designated as a minor plant change per NP 7.2.6, Engineering Change Process, which" bypasses EAC Review and the conceptual design review.

The design description is essentially the same as that inMR 99-029*D. No changes were made to describe the potential for plugging (Shortcuts Taken). The design input checklist was changed from MR 99-029*D so that fire protection considerations were applicable. The RO is recognized as being in a safe shutdown flow path. The potential for plugging is not discussed. PBF-2060 is completed and concludes there is no adverse impact (Mindset). The design input checklist item for FMEA is still marked as not applicable to the design (Shortcuts Taken).

No record could be found of design package distribution to team members. Fire Protection Engineering and Operations members did not recall seeing safety screening/evaluation that had information on plugging (Inadequate Communications Among Organizations). Design Team members signed-off on the final design description between March 8 and March 14, 2002.

49

Possible Common Mode Failure of RCEO00191 Aux Feediwater Recirculation Lines Design Engineer 3 was the Independent Reviewer and agreed on March 14, 2002 that FMEA did not apply to the design and that the Appendix R function was not affected (Mindset).

The MDS approved the design on March 15, 2002 and failed to ensure an adequate FMEA was performed (Mindset).

The modification installation began on September 30, 2002 and was completed and tested satisfactorily on October 14, 2002.

MR 02-029 - Upgrade Recirc AOV Open Function to Safety-Related PBNP received NRC Notice of Violation (NOV) EA-02-031 on July 12, 2002 regarding inadequate procedures addressing AFW pump operation under loss of instrument air conditions. In response to the NOV, PBNP management began considering additional actions needed to address the violation.

Engineering held a meeting on July 17, 2002 to discuss the upgrade of the AFW recirc AOV open function to safety-related. Participants included the Engineering Support Programs Supervisor, a Q-List Engineer, AFW System Engineer 2, and an IST Engineer.

The discussion was focused on the impact of the function change on the IST program.

The Engineering Programs Manager requested a cost estimate for the upgrade by the next day. No design engineering personnel were involved. At this point engineering management believed the function upgrade could be accomplished outside of the modification process. The cost estimate information was provided to the Acting Engineering Director.

The Acting Engineering Director developed a rough estimate of $250K based on this information and provided it the Site Vice President. She also asked a design engineer to do a walk down of the AFW recirc line back to the CSTs (during the last week of the SSDI -August 5, 2002).

PBNP issued a response to the NRC NOV on August 12, 2002 that committed to classifying the AFW pump recirculation flow control valves' open function as safety related. The stated purpose was to further improve the future effectiveness of the AFW system by providing additional pump protection against low flow, and that it will provide a redundant method of providing for minimum AFW pump flow and consequently, pump cooling. It was also stated that because not all of the recirculation flow path is safety related, operability of the AFW pumps would not be dependent upon the availability of that flow path.

On August 19, 2002, the Thermal/Hydraulic Design Supervisor (THDS) was tasked with overseeing the modification to upgrade the AFW Recirc AOV open function; this was delayed from the week of August 12, 2002 because he was on vacation. He recalled being marginally involved in a meeting a few weeks earlier to brainstorm ideas on what would have to be done to upgrade the valves to have an open safety function.

50

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines There was an assist visit performed by NMC personnel that looked at the modification during the week of August 19, 2002. They identified that there was different understanding of the purpose of the modification within the organization. that the background and reason for the design basis change was not clearly communicated, and that there was no detailed plan for actions needed for successful development completion of the modification. The THDS developed an implementation plan on August 20, 2002, along with responses to the other issues identified. In the implementation plan it is stated that the AFW AOVs were already classified as Safety-Related so no physical changes were required (Wrong Assumptions).

Engineering management provided the initial design scope verbally to the MDS and he had internal Engineering discussions to clarify the design specifics. The scope was later revised to include the recirc line function to pass flow. The design needed to be developed and installed in a 3-week period (Inadequate Planning).

The plant modification procedure (NP 7.2.1) evaluates a proposed design against existing design requirements and functions. Because the RO plugging issue was addressed previously in licensing space via a safety evaluation versus design space, this did not result in any design document changes to state that the recirc line was not required to pass flow (although that was incorrect). Therefore, when a safety function change was proposed for the AFW recirculation AOV and line to pass flow, the conflict with the line not being capable of providing flow was not detectable via the design change process because that design feature was not in a design basis document. The safety evaluation was in conflict with the current licensing basis (Actions Conflict with Another Process).

DE2 was assigned as the modification engineer for MR 02-029 on August 20, 2002. No evaluation of the recirc AOV was performed because it was already classified as safety related (Wrong Assumptions). No evaluation of the overall design of the recirc line and components was performed to verify that it could support the added safety function.

Focus was on selected mechanical components (Unfamiliar or Infrequent Task/Time

& Schedule Pressure). No evaluation was performed of the change of safety function impact on other components and/or systems, or other work in progress (Actions Not Specified). DE2 did not relate potential plugging issue for ROs (MR 99-029*A-D) to recirc line function to pass flow or the need for the AFW recirc line to support AFW pump operability being in conflict with the SE and SCR for MR 99-029*A-D (Lapse of Memory/ Time & Schedule Pressure).

Design Team members approved the final design on September 5, 2002 and accepted the assumption that the recirc AOV already being safety-related would support the added safety function to open, and the narrow focus of the evaluation (Inadequate Verification).

Design Engineer 4 performed the Independent Review on September 5, 2002 and accepted the assumption that the recirc AOV already being safety-related would support 51

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines the added safety function to open, and the narrow focus of the evaluation (Inadequate VerificationlUnfamiliar or Infrequent Task/Time & Schedule Pressure).

The MDS approved the design on September 6, 2002 and accepted the assumption that the recirc AOV already being safety-related would support the added safety function to open, and the narrow focus of the evaluation (Inadequate Verification).

Nuclear Safety Analysis Engineer I (NSA 1) volunteered to perform the safety review of MR 02-029 because of the workload on the other design engineers. He prepared safety screening SCR 2002-0359 on September 5, 2002. The description of design functions in Section II focused on selected components and did not discuss the overall function of the line (Time & Schedule Pressure/Mindset). The discussion of the RO function in Section II. 1.b did not include its need to pass flow for pump cooling (Mindset). The potential for plugging of the restricting orifices was not discussed. He recalled reviewing the safety evaluation and screening for the RO modifications, but did not identify the plugging concern (Wrong Assumptions/Misdiagnosis). The discussions of specific design functions in Section II and III did not include the Appendix R function (Mindset).

DE2 performed the review of SCR 2002-0359 on September 5, 2002. The description of design functions in Section II focused on selected components and did not discuss the overall function of the line (Unfamiliar or Infrequent Task/Time & Schedule Pressure). The discussion of the RO function in Section II. 1.b did not include its need to pass flow for pump cooling (Mindset). The potential for plugging of the restricting orifices was not discussed (Lapse of Memory/Time & Schedule Pressure). The discussions of specific design functions in Section II and IH did not include the Appendix R function (Mindset).

MSS and JOSRC did not review SCR 2002-0359 for the same reasons as previously discussed for SCR 2001-0981.

During development of the modification, Operations raised concerns about taking the recirc lines out of service and not declaring the AFW pumps inoperable. It was decided that relief valve AF-4035 would be credited as providing a flow path for pump cooling while the common recirc line was isolated to facilitate removal of the internals from check valve AF-117.

NSA1 prepared the safety review for the installation work associated with MR 02-029.

Safety screening SCR 2002-0377 was prepared on September 8, 2002. The responsibilities for the dedicated operator described in the safety screening only address AF-4035 monitoring and notification associated with AFW pump starts; there was no discussion of monitoring when flow was reduced to less than required minimum flow values. This would be a condition where AF-4035 operated properly on AFW pump start, but failed when AFW flow was throttled; NSA1 Stated he did not consider this (Mindset). Actual operator instructions were broader than described in the SCR and no real concern existed. The potential for plugging of the restricting orifices was not discussed (Wrong Assumptions/Misdiagnosis). If RO plugging had occurred, the 52

Possible Comnmon Mode Failure of Aux Feedwater Recirculation Lines RCEO00191 dedicated operator would have detected AF-4035 not opening and he would have notified the control room immediately.

DE2 reviewed SCR 2002-0377 on September 10, 2002. The responsibilities for the dedicated operator in the safety screening only address AF-4035 monitoring and notification associated with AFW pump starts; there was no discussion of monitoring when flow was reduced to less than required minimum flow values (Mindset). The potential for plugging of the restricting orifices was not discussed (Lapse of Memory/Time & Schedule Pressure).

Failure Mode Identification Human Performance Failure Modes Attentional Issues Al I Inattention - Not paying attention to the task requirements.

"* Final design review performed after SE approved by MSS. Procedure adherence.

(MR 99-029*A/B) (DEl)

"* Independent review performed after SE approved by MSS. Procedure adherence.

(MR 99-029*A/B) (DE2)

"* Conceptual design not required -justification not provided. Procedure adherence.

(MR 99-029*A/B) (MDS)

"* Independent review approval performed after SE approved by MSS. Procedure adherence. (MR 99-029*A/B) (MDS)

"* Statement remained in SE that recirc line functioned identically to existing line.

(AFW SEI)

"* Conceptual design not required -justification not provided. Procedure adherence.

(MR 99-029*D) (MDS)

A7 ILapse of Memory - Loss of memory regarding information previously A7learned and known.

"* Did not relate potential plugging issue for ROs (MR 99-029*A/D) to recirc line function to pass flow (MR 02-029). (DE2)

"* The potential for plugging of the restricting orifices is not discussed. (SCR 2002 0359) (DE2)

"* The potential for plugging of the restricting orifices is not discussed. (SCR 2002 0377) (DE2)

A8 Inadequate Tracking - Method used to maintain control of necessary requirements was not properly used.

" EAC recommends performing modification but does not select specific option to be pursued. (EAC) 53

Possible Conmnon Mode Failure of Aux Feedivater Recirculation Lines RCEO00191 S

A9 Time & Schedule Pressure - Urgency or excessive pace required to perform task.

" No evaluation of overall design of recirc line and components is performed to verify that it can support added safety function. Focus is on selected mechanical components (MR02-029). (DE2)

" Did not relate potential plugging issue for ROs (MR 99-029*A/D) to recirc line function to pass flow (MR 02-029). (DE2)

" Independent Reviewer (MR 02-029) accepted assumption that recirc AOV already being safety-related would support the added safety function to open, and narrow focus of evaluation. (DE4)

" The description of design functions in Section II focuses on selected components and does not discuss the overall function of the line. (SCR 2002-0359) (NSA1)

" The description of design functions in Section II focuses on selected components and does not discuss the overall function of the line. (SCR 2002-0359) (DE2)

" The potential for plugging of the restricting orifices is not discussed. (SCR 2002 0359) (DE2)

" The potential for plugging of the restricting orifices is not discussed. (SCR 2002 0377) (DE2)

Fear of Failure - Apprehension regarding potential adverse consequences if the individual fails to perform at a high level, resulting in undesirable behaviors.

After plugging issue was identified, dispositioned issue in the safety evaluation.

(DEI)

Judgment Issues J3 Mindset/Preconceived Idea - The tendency of an individual to make a judgment based on a preconceived mental model or preconditioned bias that is not based upon the current information, conditions or indications.

" MSS had decided recirc line not needed. Independent Reviewer agreed that FMEA did not apply to design and that Appendix R flowpath would not be affected. (MR 99 029*A/B) (DE2)

" MSS had decided recirc line not needed. Approved design. Failed to ensure adequate FMEA performed. (MR 99-029*A/B) (MDS)

" Design input checklist was changed (for MR 99-029*D) so that fire protection considerations were not applicable. (DEl)

" Design input checklist for FMEA marked as not applying to design. (MR 99 029*A/B) (DE1/DE2)

" Independent Reviewer (for MR 99-029*D) agreed that FMEA and Appendix R did not apply to design. (DE2)

" Approved design (for MR 99-029*D). Failed to ensure adequate FMEA performed.

(MDS)

" The SCR raises the issue of the potential for the recirculation orifices to accumulate debris from SW but does not provide any definitive basis for the conclusions drawn regarding its impact on AFW pump operation. There is no discussion explaining the 54

Possible Common Mode Failtre of RCEO00191 Aux Feediwater Recirculation Lines change in conclusions drawn from SE 2000-0055 regarding not needing the recirculation line. (DE2)

"* Reviewer did not ensure that the conclusions were supported - The SCR raises the issue of the potential for the recirculation orifices to accumulate debris from SW but does not provide any definitive basis for the conclusions drawn regarding its impact on AFW pump operation. There is no discussion explaining the change in conclusions drawn from SE 2000-0055 regarding not needing the recirculation line.

(AFW SEI)

"* Design input checklist was changed from MR 99-029*D so that fire protection considerations were applicable. The RO is recognized as being in a safe shutdown flow path. Potential for plugging is not discussed. PBF-2060 is completed and concludes no adverse impact. (DE2)

"* Independent Reviewer (for MR 99-029*C) agreed that FMEA did not apply to design and Appendix R function was not affected. (DE3)

"* Approved design (for MR 99-029*C). Failed to ensure adequate FMEA performed.

(MDS)

"* The description of design functions in Section II focuses on selected components and does not discuss the overall function of the line. (SCR 2002-0359) (NSA 1)

"* The discussion of the RO function in Section II..b does not include its need to pass flow for pump cooling. (SCR 2002-0359) (NSAI)

"* The discussions of specific design functions in Section II and III do not include the Appendix R function. (SCR 2002-0359) (NSAI)

"* The discussion of the RO function in Section IL. 1.b does not include its need to pass flow for pump cooling. (SCR 2002-0359) (DE2)

"* The discussions of specific design functions in Section II and III do not include the Appendix R function. (SCR 2002-0359) (DE2)

"* The responsibilities for the dedicated operator in the safety screening only address AF-4035 monitoring and notification associated with AFW pump starts; there is no discussion of monitoring when flow is reduced to less than required minimum flow values. (SCR 2002-0377) (NSA1)

"* The responsibilities for the dedicated operator in the safety screening only address AF-4035 monitoring and notification associated with AFW pump starts; there is no discussion of monitoring when flow is reduced to less than required minimum flow values. (SCR 2002-0377) (DE2)

"* ECRs were not initiated to evaluate the drilling of additional holes in the innermost orifice stages. (DE1/DE2)

J4 Wrong Assumptions - Judgments are made without verification of the facts and is usually based upon the individual's perception of recent experience or events.

Reviewer did not ensure SE addressed all licensing basis functions: Appendix R function and AFW recirc line operation during accidents not addressed. (AEW SE1)

" Reviewer did not ensure SCR addressed all licensing basis functions: Appendix R function and AFW recirc line operation during accidents not adequately addressed.

(AFW SE1) 55

Possible Common Mode Failure of R CEO00191 Aux Feedwater Recirculation Lines

"* No evaluation of recirc AOV is performed because it is already classified as safety related (MR 02-029). (DE2)

"* The potential for plugging of the restricting orifices is not discussed. (SCR 2002 0359) (NSAI)

" The potential for plugging of the restricting orifices is not discussed. (SCR 2002 0377) (NSAI)

"* Actions from CAPO 13812 and CA006836 focused narrowly on FME issue only.

(Performance Assessment)

J5 Inadequate Verification - Insufficient verification of the facts, and is usually I based upon inaccurate information or a lack of information.

I

" Reviewer did not ensure SE content was technically adequate: vendor information was misapplied. (AFW SE1)

" After incorporating MSS comments, the SE did not consider that loss of the recirc line due to loss of instrument air is recoverable via actions specified in AOP-5B, whereas loss of the line due to plugging is not recoverable. (AFW SEI)

" Reviewer did not ensure SCR content was technically adequate: vendor information was misapplied. (AFW SEI)

Accepted assumption that recirc AOV already being safety-related would support the added safety function to open, and narrow focus of evaluation. (MR 02-029) (AFW SE2)

" Independent Reviewer accepted assumption that recirc AOV already being safety related would support the added safety function to open, and narrow focus of evaluation. (MR 02-029) (DE4)

" Approved design. Accepted assumption that recirc AOV already being safety-related would support the added safety function to open, and narrow focus of evaluation.

(MR 02-029) (MDS)

"* CAP013812 on AFW FME issue closed without identifying cause of low flow.

(Performance Assessment)

J7 IShortcuts Taken - Actions to allow the job to go "easier" or faster, contrary

"* After plugging issue was identified, dispositioned issue in the safety evaluation.

(DEl)

"* The SE identified the AFW section of the FPER as part of the CLB but the recirc line Appendix R function not described in SE. (DE1/DE2)

"* DEl distributed a design package for review to the design team on August 2, 2001. A safety evaluation is listed as one of the documents. DE1 had left PBNP by the time this modification was being finalized. No new PM was assigned. DE2 facilitated changes to design description and related documents and obtained approval from DEl via e-mail. Final design description and SCR are dated December 14, 2001. No changes were made to describe plugging potential. (MR 99-029*D) (DE2)

"* The SCR identified the AFW Section of the FPER as part of the CLB, but did not list the Appendix R flowpath as a design function. (DE2)

"* The design description (for MR 99-029*C) is essentially the same as that in MR 99 029*D. No changes were made to describe plugging potential. (DE2) 56

Possible Common Mode Failure of Aux Feedwater Recirculation Lines RCEO00191

• Design input checklist item for FMEA (for MR 99-029*C) is still marked as not applicable to the design. (DE2)

Knowledge Issues K2 Unfamiliar or Infrequent Task - Tasks that have not been performed before or are performed infrequently.

No evaluation of overall design of recirc line and components is performed to verify that it can support added safety function. Focus is on selected mechanical components. (MR 02-029) (DE2)

Independent Reviewer (MR 02-029) accepted assumption that recirc AOV already being safety-related would support the added safety function to open, and narrow focus of evaluation. (DE4)

The description of design functions in Section II focuses on selected components and does not discuss the overall function of the line. (SCR 2002-0359) (DE2)

F K3 Misdiagnosis - Decisions made with accurate information that is usually used or interpreted incorrectly when making a decision.

"* SE utilized vendor information on why design precluded plugging. (DE1I/DE2)

"* SCR utilized vendor information on why design precluded plugging. (DE2)

"* The potential for plugging of the restricting orifices is not discussed. (SCR 2002 0359) (NSAI)

" The potential for plugging of the restricting orifices is not discussed. (SCR 2002 0377) (NSAI)

K4 Tunnel Vision - Decisions are made without considering all the available K options or information needed to adequately assess the situation.

I

" No information in final design description on potential to plug RO; no discussion of orifice size being less than SW strainer size. Concluded that Appendix R flowpath would not be affected by new RO. Failed to apply FMEA to design. Inadequate design. (DEl)

K6 Inadequate Knowledge of Standards - Insufficient knowledge of codes, standards, design basis, licensing basis, regulations, etc. needed to perform the task.

"* SE only discusses the recirc line function during AFW pump start and does not address pump shutdown or the need for operation on recirc during the other stages of accident response. (DEI/DE2)

" MSS members offered qualitative resolution of plugging issue based on loss of recirc line due to AOV failing closed, without considering other recirc line functions.

(MSS)

" After incorporating MSS comments, the SE did not consider that loss of the recirc line due to loss of instrument air is recoverable via actions specified in AOP-5B, whereas loss of the line due to plugging is not recoverable. (DE1/DE2)

" OSRC members had no comments on SE 2000-0055. (OSRC) 57

Possible Common Mode Failure of Aux Feedwater Recirculation Lines RCEO00191

"* Design input checklist (for MR 99-029*D) was changed so that fire protection considerations were not applicable. (DE2)

EK7-7 Flawed Analytical Process or Model - Decisions based upon a flawed K7 analysis, such as using qualitative versus quantitative data.

Initial SE had technical discussion on why SW particles would not plug RO, but approved SE assumed RO might plug, yet stated that the modified recirc line would function identically to existing line. (DEI/DE2)

Organizational & Management Failure Modes Functional Issues Fl Inadequate Communications Within an Organization - A breakdown in communication (verbal or written) within one organization or work group.

Often leads to important issues not being addressed.

Specific details of RO not presented to EAC. Plugging not discussed. (DEl)

F2 Inadequate Communication Among Organizations - A breakdown in communication (verbal or written) among two or more organizations or work groups. Often leads to a breakdown in processes that require several groups to participate.

" Not all material is distributed to every team member. Fire Protection Engineering and Operations members did not recall seeing safety evaluation that had information on plugging. (MR 99-029*A/B)

" It is not clear what material was provided to the design team for review. Fire Protection Engineering and Operations members did not recall seeing safety screening or evaluation that had information on plugging. (MR 99-029*D)

" No record could be found of design package distribution to team members. Fire Protection Engineering and Operations members did not recall seeing safety screening/evaluation that had information on plugging. (MR 99-029*C) 7 F4 Inadequate Planning - Deficiencies in what work must be done, by whom, F4 Iwhen, and how long it will take.

" Initial design scope provided by engineering management. Scope revised to include recirc line function to pass flow. Design needed to be developed and installed in a 3 week period. (MR 02-029)

F5.

Inadequate Emerging Issues Management - Deficiencies in determining how

' to deal effectively with unexpected issues.

" Definitive resolution of plugging issue not pursued. (MR 99-029*A/B) (MDS) 58

Possible Common Mode Failure of Aux Feedwater Recirculation Lines RCEO00191 F6 Inadequate Program Management - Inadequate oversight of critical work processes to ensure they function smoothly and effectively; often results in program degradation over time or increased problems within those processes

"* Independent reviewer was a co-preparer of the safety evaluation. Lack of independence. (MR 99-029*A/B) (DE2)

"* Independent reviewer was the preparer of the safety screening and helped revise the design documents. Lack of independence. (MR 99-029*D) (DE2)

"* A new PM was not assigned after DEl left PBNP. (MDS)

"* Failed to ensure independent reviewer had no responsibilities for the design. (MR 99 029*D) (MDS)

"* Approximately 80% of the inappropriate actions identified during this evaluation were tied to human performance failure modes.

Process Related Failure Modes Roles & Responsibilities Related Issues RR1 Actions Not Specified - The action(s) that an individual or group must perform to accomplish a task are not contained in the documentation or instruction.

"* The modification process procedure does not have guidance for evaluating the effect of a modification on other work in progress. (NP 7.2.1)

RR4 Actions Conflict with Another Process - The action(s) that an individual or group must perform to accomplish a task conflict or contradict the actions specified by another document or instruction.

" The conclusions reached by SE 2000-0055 that the-AFW RO is not needed during accidents are contradictory to the current licensing basis.

" MSS Procedure requirements are basically the same as Rev.3. However, the 10 CFR 50.59 New Rule changes the basis for determining if a safety evaluation is needed.

The MSS procedure still requires review of modifications that affect nuclear safety, but the results of the 10 CFR 50.59 screening no longer answer that question. (NP 1.6.5)

RR5 Actions Not Tied to Another Process When Necessary - The action(s) contained in one document or instruction do not reference supporting documents or instructions when necessary.

" MSS reviewed SE prior to final design approval. Procedure adherence. (NP 1.6.5)

RR6 Methods Not Clearly Described - Action(s) are required by the document or instruction, but the method to accomplish the actions is not clearly specified by the document or instruction.

" Procedure directed review of the modification package by all team members (Step 4.7.7.b), although some documents can be included by referenced only (Step 4.7.5.g).

(NP 7.2.1) 59

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines MSS received only the SE for review and not the modification request design scope.

Procedure adherence. (NP 1.6.5)

Vii.

Root Causes & Contributing Factors Conclusions Design Engineer 1 failed to consider service water strainer size as a design input for MR 99-029*A/B; however, the issue was later identified during preparation of the safety evaluation. When the potential for RO plugging was discovered, it was dispositioned via a safety evaluation instead of incorporating the information into the design and having a technical evaluation performed. That decision moved evaluation of the plugging issue out of the design process into a licensing process. The safety evaluation did not adequately evaluate the impact of RO plugging on all of the required design functions; throttling AFW flow during accidents and the required Appendix R flowpath were not addressed. MSS members suggested that the AFW recirc line wasn't needed because the AOV fails closed on loss of instrument air. Again, emergency operations and the Appendix R function were not considered. The design was subsequently approved for installation. The end product was the installation of a restricting orifice that was possibly susceptible to plugging in a recirculation line that was viewed as not being needed during accidents. No design output documents, other than the safety evaluation, contained the (erroneous) conclusion that the AFW recirc line was not required during accident response.

The inadequate design resulted from a safety evaluation that reached erroneous conclusions based on unverified information. The vendor information concerning design features that precluded plugging was misapplied. The belief that the RO would not plug was never technically proven and a formal evaluation was not initiated. This mindset resulted in key information regarding important design functions being omitted from the safety evaluation. The RO plugging concern was deemed an acceptable condition based on a belief that the ROs would not really plug (that was also founded on unverified information) and that the AFW recirc line was not needed due to the AOV failing closed.

Personnel external to engineering were influenced primarily by inadequate information about the potential for plugging (the Operations design team member did not see the safety evaluation) and by a lack of knowledge of the AFW recirculation line design functions (Operations Manager, MSS Chairman, and OSRC) that was reinforced by the omission of key design function information related to the Appendix R flowpath and throttling of AFW flow during accidents.

The same design failure was replicated for MR 99-029*C/D. The belief that the ROs would not really plug prevailed, along with a mindset that this was just a repeat modification.

Inadequate program management resulted in Design Engineer 2 co-preparing the safety evaluation for MR 99-029*A/B and then performing the independent design verification.

60

Possible Common Mode Failure of RCEO00191 Aux Feed water Recirculation Lihes By the time the independent verification was performed, a mindset had already developed regarding the ROs not plugging. When MR 99-029*D was in development, Design Engineer I had left the site to work on another project and Design Engineer 2 was left in a position of updating the final design documents. He again was put in the position to perform the independent verification of the design; an opportunity for a truly independent review was lost.

Engineering's management involvement was limited mostly to that of the Mechanical Design Supervisor. Early review of the RO replacement by the EAC was hindered by a lack of technical information provided by the Design Engineer. A brief conversation between the Mechanical Design Supervisor and the Engineering Design Manager occurred around August 2000 when the factory flow test of the MDAFW pump ROs failed. At that time the Mechanical Design Supervisor described the orifice design as three concentric cans with small holes and expressed his opinion that they were not susceptible to plugging.

Approximately 80% of the inappropriate actions identified during this evaluation were tied to human performance failure modes. This is indicative of a larger problem (in Engineering) related to inadequate program management - inadequate oversight of critical work processes to ensure they function smoothly and effectively; often results in program degradation over time or increased problems within those processes.

Missed opportunities occurred during the evaluation of CAP013812 and development of MR 02-029. CAP013812 was initiated in April 2001due to recirculation flow concerns associated with AFW pump P-38A. The CAP attributed the problem to FME concerns associated with the drilling of additional holes in the flow orifice during March 2001. An action item was created to evaluate the FME practices for that work. The CA concluded that FME controls were adequate. The CAP was closed without identifying the cause of the flow concerns.

MR 02-029, which upgraded the AFW recirc AOV open function, failed to evaluate the AFW recirc line RO. Time and schedule pressure resulting from inadequate planning (management imposed actions and due date) combined with an evolution that was infrequently performed (safety function upgrade) to create an error-likely situation (design review). Error prevention methods, such as pre-job briefs, were not taken to ensure that the task was properly conducted. This resulted in an inadequate design review of the AFW recirculation line components because of mindsets and wrong assumptions.

Root Cause The root cause of this event was the failure to properly evaluate the potential for orifice plugging within the design process. The evaluation of the plugging issue via a safety evaluation resulted from inadequate program management.

• The design engineer decided to disposition the plugging issue in the safety evaluation instead of revising the design. This inappropriately moved evaluation 61

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Liues of the potential plugging issue out of the design process and into a licensing process.

The Mechanical Design Supervisor condoned resolution of the issue via a safety evaluation in lieu of pursuing a formal technical evaluation.

Contributing Causes The use of unverified information in the safety evaluation for modification MR 99 029*A/B and the omission of key design function information from the safety evaluation resulted in an erroneous conclusion. That omission reinforced knowledge deficiencies that existed with MSS members resulting in the conclusion that the AFW recirculation line was not needed during accidents.

Significant contributing causes included:

"* The vendor information concerning its design to preclude plugging was misapplied and not verified

"* Information on the design functions of the Appendix R flowpath and throttling of AFW flow during emergency operations was omitted from the safety evaluation

• Inadequate knowledge of AFW recirculation line design functions existed

• Inadequate independent verification resulted from inadequate program management

• Engineering management involvement and oversight (at the manager and supervisor levels) was inadequate and contributed to high human error rates Other contributing causes included:

• Inadequate planning for MR 02-029 created time and schedule pressure

• Inadequate closure of CAP013812 (AFW RO FME issue)

VIII. Corrective Actions Interim Corrective Actions (mitigation)

"* CA #1 OPROO003 1: OPR requested to strengthen basis of Aux Feed Recirculation Lines. System Engineering, Priority 2, Completed: 10/31/02.

"* CA #2 CA026897: Write and hang temp info tags on the pump hand switches, requiring that a minimum forward aux feed flow of greater than 50 gpm (P-38A/B) or 75 gpm (1/2 - P29) to be maintained, if not, stop affected pump; brief the Ops crews on the issue. Operations, Priority 2, Completed: 11/4/02 (Temporary information tags written and placed next to AFW flow indicators on 10/30/02.)

"* CA #3 CA026898: Make changes to affected AOPs, EOPs, and other critical procedures to ensure that minimum forward flow is maintained, or the affected pump is stopped. Operations, Priority 2, Completed: 1117/02 62

Possible Conmnion Mode Failure of RCEO00191 Aux Feedivater Recirculation Lines "CA #4 CA026908: Complete the interim administrative controls. Operator training and EOP and AOP changes to establish appropriate operator guidance for all accident sequences of interest. Operations, Priority 2, Completed: 11/19/02.

CA #5 CA026909: Independently evaluate the procedure changes being made to the EOPs and AOPs and assess their adequacy to provide adequate operator guidance.

Additionally, evaluated the adequacy of the temporary information tags and objectively evaluate our decision to declare the AFW pumps operable after taking interim actions to address this issue on Tuesday, October 29, 2002. Operations, Priority 2, Completed: 11/19/02.

CA #6 CA026910: Independently evaluate the briefings and training conducted or planned for operating crews. We would like an objective look to ensure that the training was adequate and well documented. Operations, Priority 2, Completed:

11/19/02.

CA #7 CA026917: Identify all other applications of this type of orifice in the Point Beach units and determine applicable operating history. Equipment Engineering Valves, Priority 2, Completed: 11/6/02.

CA #8 CA026958: Ensure compliance with procedure NP 10.3.7, "On-line Safety Assessment," while at elevated risk due to the potential for common mode plugging of AFW recirc orifices. Focus should continue with daily updates to Resolution Team Leader until interim corrective actions have restored on-line risk to a "green" condition. Production Planning, Priority 2, Completed: 11/18/02.

CA #9 CA026962: Perform an incident investigation. Scope is to conduct an investigation to collect facts surrounding the maintenance conducted on the P-38A motor driven AFW pump, the post maintenance testing, the evaluation of the corrosion products discovered in the recirculation orifice and conclude with the decision to declare the AFW pumps inoperable on Tuesday, October 2002. Use applicable portions of NP 5.3.3, "Incident Investigation and Post-Trip Review," for guidance. Radiation Protection, Priority 2, Completed: 11/21/02.

"* CA #10: Institute a process whereby senior management will provide effective oversight of all modifications to the facility. Engineering, Priority 2, Completed: 11/8/02. (Implemented via NPM 2002-0594)

Corrective Actions to Prevent Recurrence (CATPRs)

CATPR #1 CAOOOOOO: Implement periodic reviews of Engineering's products by the Quality Review Team to identify and address human performance related issues.

Process Engineering, Priority 2, Completed: 11/26/02 (NP 7.1.7, Quality Review Team, was issued on 11/1/02; the first QRT meeting was held on 11/26/02) 63

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines "CATPR #2 CA000000: Increase engineering management involvement in the approval and oversight of modifications (This is a long-term corrective action to address the interim measures implemented via NPM 2002-0594). Design Engineering, Priority 2, Completion Due Date: 3/10/03.

" CATPR #3 CAOOOOOO: Present lessons learned from this event to all Engineering personnel stressing the importance of following the design process. Design Engineering, Priority 2, Completion Due Date: 3/10/03.

Corrective Actions to Restore (broke - fix)

"* CA #1 CA026900: Ensure that the PRA work to evaluate the risk significance of this potential common mode failure is completed expeditiously. PRA, Priority 2, Completion Due Date 1/29/03.

"* CA #2 CA026901: Review for impact on aux feed pump unavailability for NEI cornerstone indicator. Equipment Engineering - NSSS, Priority 3, Completion Due Date: 12/20/02.

"* CA #3 CA026902: Redesign the recirc line orifices to make use of an orifice design that has aperture size >1/8". Design Engineering - Mechanical, Priority 1, Completion Due Date: 11/22/03.

"* CA #4 CA0269 11: Evaluate hydraulic system response to determine if the orifices could become plugged. Design Engineering - Thermal, Priority 2, Completion Due Date: 1/13/03.

"* CA #5 CA026912: Develop a test plan to evaluated plugging of a spare orifice. Use the vendor utilized by Kewaunee to evaluate the AFW suction strainer issue as appropriate. Design Engineering - Thermal, Priority 2, Completion Due Date:

12/30/02.

"* CA #6 CA026914: Evaluate the auxiliary feedwater system to determine the sources and quality of potential corrosion products. Equipment Engineering - NSSS, Priority 2, Completion Due Date: 1/20/03.

"* CA #7 CA026915: Contact vendor of the orifice to obtain test data and other relevant information that could be used as input to the PRA work. Design Engineering Thermal, Priority 2, Completed: 10/31/02.

"* CA #8 CA026918, Ensure that modifications to the orifice design or a replacement orifice are pursued expeditiously to resolve any questions. Design Engineering Mechanical, Priority 2, Due Date 12/13/02.

64

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines

"* CA #9 CAOOOOOO: Revise training materials to accurately describe the AFW recirculation line design functions. Technical Training, Priority 3, Completion Due Date: 4/10/03.

"* CA #10 CAOOOOOO: Review licensing basis documents to ensure that they accurately describe the AFW recirculation line design functions during accident conditions.

Design Engineering, Priority 3, Completion Due Date: 4/10/03.

"* CA #11 CAOOOOOO: Implement use of Human Performance Tools in Engineering (ENG-10). Process Engineering, Priority 3, Completion Due Date: 4/10/03.

" CA #12 CAOOOOOO: Initiate a supplemental record to the modification file documenting the acceptability of the additional holes drilled in ROs for MR 99 029*A/B. Design Engineering, Priority 3, Completion Due Date: 4/10/03.

CA #13 CAOOOOOO: Revise the modification procedure (or issue supplemental guidance) to include an impact review on other work in progress and to clarify what materials are distributed to which design team members. Design Engineering, Priority 3, Completion Due Date: 4/10/03.

"* CA #14 CA000000: Evaluate adequacy of actions related to closure of CAP013812 (AFW RO FME issue from 4/01). Performance Assessment, Priority 3, Completion Due Date: 4/10/03.

" CA #15 CAOOOOOO: In conjunction with the 50.59 process owner, evaluate adequacy of qualifications of MSS members for performing 50.59 reviews of modifications.

Regulatory Compliance, Priority 3, Completion Due Date: 4/10/03.

"* CA #16 CAOOOOOO: Evaluate why OSRC had no comments on SE 2000-0055.

Performance Assessment, Priority 3, Completion Due Date: 4/10/03.

"* CA #17 CAOOOOOO: Revise MSS procedure to specify that the final design must be approved prior to the review of safety evaluations for modifications. Regulatory Compliance, Priority 3, Completion Due Date: 4/10/03.

"* CA #18 CAOOOOOO: Revise MSS practice of reviewing only the safety evaluation for modifications to include a review of the final design description. Regulatory Compliance, Priority 3, Completion Due Date: 4/10/03.

Ix.

References

1. IOCFR 50.59/72.48 Screening and Safety Evaluation, SE 97-085

2. CAP 001625 (CR 99-1391), SCAQ Action - Potential common Mode Failure Mechanism Affecting Welds in AFW Pumps

3. CAP 027667 (CR 99-1163), Water Spraying From a Cracked Weld On the Aux Feed Pump Recirc Line 65

Possible Comnmon Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines

4. CAP 030045, Discrepancies in AFW PRA Model and Notebook.

5. CAP024973 (CR 99-1368), Weld Crack in Motor-Drive Aux Feed Pump Recirc Line

6. CAP025033 (CR 99-1844), Socket Weld Survey of Aus Feedwater Pump Recirc Piping

7. CAP029261 Appendix R Classification Discrepancies in AFW Common Recirc Line

8. CAP029908 P-38A, Motor Driven AFW Pump has Inadequate Recirc Flow during IT-10

9. CAP029921 Incorrect tag out

10. CAP029928 No parts were identified or available to perform maintenance on RO 4008

11. CAP029952 Possible Common Mode Failure of Aux Feed Recirculation Lines

12. CAP029983 AFW DBD description of Recirc Orifice flow rated bases in error.

13. CAP029997 50.59 Safety Evaluation for MR 99-029*A/B discussed plugging of ROs

14. CAP029999 Training Needs Analysis of AFW mini-Recirc Line Plugging Recommendations

15. CAP030009 Required information missing on record procedure

16. CAP030011 Inconsistencies noted in annotating NA on procedural steps.

17. CAP030012 AFW pump recirc orifice debris sample fidelity

18. CAP030014 P-38A Aux Feed Pump low recirc flow

19. CAP030015 Five day Delay in Identifying Common Mode Failure of Auxiliary Feedwater Pumps

20. CAP030026 RCE 01-069 Extent of Condition Assessment Inadequate

21. CAP030032 P-38A Recirculation Orifice, RO-4015, Unable to be Maintained/Cleaned

22. CAP030040 Power supply to AFW pump recirc valves not Safety Related

23. CAP030048 Lack of Specific Detail in Response to On-Line Safety Assessment Changes

24. CAP030052 Unplanned YELLOW - Requires CAP Per NP 10.3.7

25. CAP030055 T-24A&B tank inspection findings

26. CR 01-3654, Development and Revision of DBE-01 (AFW Design Basis Document) Appears to Have Been a Missed Opportunity to Identify a Design Weakness in the AFW System

27. CR 96-317, FSAR Has Inaccurate AFW Flow Rate Assumptions

28. CR 96-574, Alarm Indication Audio Levels Do Not Meet DG-GO1

29. CR 97-0720, High Background Noise in Control Room

30. CR 97-3363, IST Program Design Basis for AFW Minimum Flow Recirculation Valves

31. CR 99-2460, Control Room Noise and Pump Wear During AFW Pump Use During Plant Shutdown

32. DBD-01, Auxiliary Feedwater System, Final Draft 12/17/92

33. DBD-01, Auxiliary Feedwater System, Revision 0

34. DBD-01, Auxiliary Feedwater System, Revision 1

35. DG-G06, Guidelines for System, Component and Part Classification, Revision 6 66

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines

36. EWR 99-031, AF Pump Recirculation Noise in Control Room

37. FAX to JP Schroeder from FLOWSERVE, dated 12/09/99

38. FAX to JP Schroeder from FLOWSERVE, dated 8/25/99

39. FCR 0 1-050

40. FCR 02-002 4 1. FCR 97-033, Final Safety Analysis Report Change Request.

42. FCR 98-068, Final Safety Analysis Report Change Request

43. FCR 99-049, Final Safety Analysis Report Change Request

44. In-service Testing Program Relief Request No. VRR-28, Revision 1

45. In-service Testing Program Relief Request No. VRR-28, Revision 2

46. In-service Testing Program Relief Request No. VRR-28, Revision 3

47. IWP 88-099*A, Revision 0

48. Kewaunee Nuclear Power Plant Calculation C 11197, Auxiliary Feedwater System Water Quality Following Seismic Event, Revision 1

49. Kewaunee Nuclear Power Plant RCE 01-003, Auxiliary Feedwater Pump Suction Strainer Configuration Not as Expected

50. Letter to P Gonzales, Flowserve Corporation from JP Schroeder, Auxiliary Feedwater Pump Operation With Service Water, 12/01/99

51. Memo to R Flessner from William Bosacki detailing discrepancies in PRA Notebook 5.9

52. MR 02-029 (Upgrade safety function of AFW recirculation AOVs and line

53. MR 97-038A/B, AFW Discharge Valves AF-4012/4019 Modification, Revision 0

54. MR 99-029*A/B (MDAFW Pump ROs)

55. MR 99-029*C/D (TDAFW Pump ROs)

56. MSS Subcommittee Meeting minutes, March 28,1991

57. NP 1.6.5, Manager's Supervisory Staff and Qualified Reviewer, Revision 3

58. NP 7.2.1, Plant Modifications, Revision 10

59. NP 7.2.1, Plant Modifications, Revision 6

60. NP 7.2.2 Design Control, Revision 5

61. NP 7.2.6, Engineering Change Process, Revision 2

62. NP 7.7.10. Q-List Nuclear Safety Classification for Structures, Systems, and Components. Revision 1.

63. NP 7.7.12, Safety Related and QA Scope Classification Upgrade or Downgrade Process, Revision 1

64. NPL 97-0186, Licensee Event Report 97-014-00, Auxiliary Feedwater System Inoperability Due to Loss of Instrument Air.

65. NPM 2002-0228, Designation of Backup Pneumatics for AFW Mini Recirculation Valves as Safety-Related

66. NPM 2002-0594, Oversight of Plant Modifications

67. NPM 2002-0599, AFW Recirculation Line Orifice Evaluation

68. NPM 93-0450, Auxiliary Feedwater Design Basis Document Review

69. NPM 99-0749, Engineering Advisory Committee Meeting Minutes of 6/30/99

70. NPM 99-1323, IR 99-016 Evaluation

71. NRC 2002-0068, Reply to a Notice of Violation

72. NRC Bulletin NO. 88-04 67

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines

73. NRC Information Notice 98-45, Cavitation Erosion of letdown Line Orifices Resulting in Fatigue Cracking of Pipe Welds

74. NRC Information Notice No. 90-65, Recent Orifice Plate Problems

75. NRC Inspection Report 50-266/99013

76. NRC Letter to PBNP, In-service Testing, Third 10-Year Program, Including Relief Requests (TAC Nos. M79386 and M79387), dated 4/17/92

77. NRC-30-130 (VPNPD-90-500), In-service Testing Program for Pump and Valves, 12/21/90

78. NRC-88-062, PBNP Response to NRC Bulletin 88-04

79. NRC-92-085, In-Service Testing Pump and Valve Program Third 10-Year Program Safety Evaluation Report

80. NRC-93-031 (VPNPD-93-054), Pump and Valve In-service Testing Program Third 10-Year Program Safety Evaluation Report 8 1. Nuclear Oversight Quarterly Assessment Report of Point Beach Nuclear Plant, October 30, 2001

82. NUREG-0737, Clarification of TMI Action Plan Requirements

83. NUTRK item IR 99-013

84. OD CR 99-1391

85. OM 3.12, Control of Equipment and Equipment Status, Revision 8

86. OM 3.26, Use of Dedicated Operators, Revision 8

87. Operability Determination, CR 1391/1844, Revision 1

88. Operability Determination, CR 1391/1844, Revision 2

89. Operability Determination, CR 99-1391, Revision 0

90. Operability Determination, OPR 000031

91. Operating Experience Event Number 455-010412-1

92. PBF-1551, Q-List Classification Document, Revision 6

93. PBM 92-0663, AFW Mini-Recirc Valve Gag Operation

94. PBNP FSAR Section 10.2 (06/01)

95. PBNP In-service Testing Background Valve Data Sheet, page 21 of 96, Revision 3

96. PBNP In-service Testing Background Valve Data Sheet, page 66 of 96, Revision 4

97. PBNP In-service Testing Background Valve Data Sheet, page 67 of 96, Revision 3

98. PBNP In-service Testing Background Valve Data Sheet, pages 11 and 21 of 95, Revision 4

99. Permanent Procedure and Procedure Revision Review and Approval for Revision 23 to IT 20, Auxiliary Feedwater System Check Valves and Flow Indicators 100. PRA 5.9 Auxiliary Feedwater System Notebook, Revision 0 101. QCR 99-0115, Code Testing Conflicts with the Aux Feedwater Mini-Flow Recirc Check Valves 102. RCE 01-069, Increased CDF in AFW PRA Model Due to Procedural Inadequacies Related to Loss of Instrument Air, Revision 1 103. RCE 99-08 1, Socket-Weld Failures in Auxiliary Feedwater Pump Recirculation Piping, Revision 1 104. Recirculation Line Potential Failure Modes Review 68

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines 105. Report From Laboratory to JP Schroeder, analytical results from Service Water Samples, dated, 10/19/99 106. SCR 2001-0981 (MR 99-029*C/D) 107. SCR 2002-0007, Revise FSAR Table 9.6-1, "Essential Service Water Loads", To List The SW Supply To the AFW Pump Suctions 108. SCR 2002-0359 (MR 02-029) 109. SCR 2002-0377 (AFW Operability During MR 02-029 110. SCR 98-1583, In-service Testing Program Third Ten-Year Interval; Revision 5 111. SCR 98-1732. In-service Testing Program Background Document 112. SE 2000-0055 (MR 99-029*A/B) 113. SE 2001-0059, Operation of the Auxiliary Feedwater Pumps Without Cooling Water 114. SER 91-025 Safety Evaluation for IWP 88-099*A 115. Unofficial Station Logs 10/23/02 Mid Shift 116. Unofficial Station Logs 10/24/02 Day Shift 117. Unofficial Station Logs 10/24/02 Mid Shift 118. Unofficial Station Logs 10/24/02 Swing Shift 119. Unofficial Station Logs 10/25/02 Day Shift 120. Unofficial Station Logs 10/25/02 Mid Shift 121. Unofficial Station Logs 10/29/02 Day Shift 122. Unofficial Station Logs 10/29/02 Swing Shift 123. Validation of Design Basis Attributes Auxiliary Feedwater System, Sargent and Lundy, April 1993 124. Work order 0205651 125. Work order 0212107 126. Work order 9934148 127. Work order 9934148 128. Work order 9945610 129. Work order 9949098 X.

Attachments Attachment A: Event Timeline Attachment B: Barrier Analysis Attachment C: Stream Analysis Attachment D: Charter Attachment E: PBNP Orifices 69

Possible Common Mode Failure of Aux Feedwater Recirculation Lines Attachment A - Event Timeline RCEO00191 Time Description

.10123/02 (Wed)

ý ýý

`Y11 A V,.

',

0400 P-38A OOS for scheduled maintenance. Enter TS LCO 3.7.5 for both units.

Day Shift Maintenance repaired pump seal per WO 0205651; replaced diaphragm on valve operator AF-4012-0 per WO 9945610 (diaphragm fails drop test); and replaced valve AF-38 per WO 9949098 Evening Shift Diaphragm replaced on AF-4012 and passes drop test; performed ICP 06.086A per WO 0202494; Operations filled and vented pump suction line, casing, and discharge line per OI-62A

10/24/02 (Thiffr)

-. 11 r7~ -1 0110 Commenced IT-10 (Quarterly Test of Electrically Driven Auxiliary Feed Pumps and Valves) for P-38A. P-38A is still OOS per IT-10 and TS LCO 3.7.5 is not met for both units.

0159 Started P-38A per IT-10; suction relief valve observed to lift momentarily and then continue to drip; the flow transmitter was vented with no improvement in flow 0212 Secured P-38A due to mini-recirc flow of 64.5 gpm. Required flow is 70 gpm.

-0215-0500 I&C contacted and flow transmitter was re-vented and recalibrated 0323 CAP029908 initiated for P-38A, MDAFW Pump having inadequate recirc flow during IT-10

-0400 Engineering contacted; Operations informed pump is OK and investigation will begin.

0440 Completed calibration of P-38A flow transmitter; instrument found to be in calibration

-0500-0700 Engineering postulated potential failure modes

-0630 BOP/NSSS Mechanical Systems Engineering Supervisor established as overall lead for the issue 0830 Meeting with Engineering, maintenance, and Shift Manager to establish an action plan for the investigation

-1030-1140 Completed venting of the flow transmitter and AFW pump and started P-38A per OI-62A; P-38A was run on recirc only for about one minute - flow remained at -65 gpm on installed FE and -60 gpm on ultrasonic flow meter. Vibration data was taken and was normal.

-1140-1800 Maintenance removes RO-4008 and finds rusty looking material in openings of outer cylinder (24 of 54 holes had some degree of blockage), disassembles, cleans and reassembles RO. Boroscope inspection performed of piping upstream of RO-4008 up to and including FE-4050A and found no material. Very small amount of debris retained.

70

RCEO00191 Possible Common Mode Failure of Aux Feediwater Recirculation Lines Time Description 1700 50.59 screening completed for one time temp change to IT-10 to lower acceptance criteria for data collection purposes only 1710 Engineering e-mails recovery plan for comment 1830 Meeting with Engineering, Maintenance, and Shift Manager to discuss plans associated with P-38A

-1900 - 2100 Ops clearing tags; Maintenance, Engineering and Operations met.

Ops reported tags cleared and system was filled and vented.

Maintenance requested high points to be re-vented prior to run. Ops re-vented suction (solid stream) and discharge (slight amount of air) 2115 Started P-38A per OI-62A 2117 Secured P-38A per OI-62A. Flow was 75 gpm. P-38A is now considered available for Safety Monitor and Maintenance Rule 2336 Commenced IT-10 Evening Samples taken of both CSTs and suction of AFW pumps looking for evidence of rust - results did not show evidence of rust.

-ý40/25ff2'(Fri) &~

0008 Started P-38A per IT-10.

0050 Stopped P-38A per IT-10 0129 P-38A is available for operation per IT-10 0155 Commence IT-10C for P-38A 0216 IT-10C is complete for P-38A only. P-38A RTS. TS LCO 3.7.5 now met. Exit TSAC entered at 0400 on 10/23/02 0309-0450 Test P-38B per IT-10 0512-0608 Test 1P-29 per 1T-08A 0756-1121 Test 2P-29 per IT-09A 1030 Initiated ACE001023 to evaluate inadequate recirc flow on P-38A 1132-1206 Test 1P-29 per IT-08B 1222-1255 Test P-38B per IT-1OD

~1600 Operations informs Engineering that short term action plan needs to be in their hands by Friday (11/1); participants confirmed that no AFW operability concern existed at this time.

Weekend BOP/NSSS Mechanical Systems Engineering Supervisor considered the actions that might be required and formulated potential questions that needed answering and that would be discussed at the meeting he would hold on 10/28. The Operations Manger didn't ask for weekend evaluation of the issue because he didn't think it was realistic that the recently designed and installed orifices would be susceptible to plugging from service water.

1100-1230 System Engineering Meeting to discuss AFW recirc orifice fouling issues. Potential for SW fouling was raised as a concern. Actual I plugging event was considered a one-time occurrence.

71

Possible Common Mode Failure of Aux Feedwater Recirculation Lines RCEO00191 Time Description 10/29/02 (Tue) 0900 Meeting with Engineering Director held to review issue. Consensus was that the system would become dysfunctional when taking suction from SW. Licensing was contacted and Operations was briefed.

0945 Management Meeting held to identify immediate actions needed 1015 EDT group began scoping out a modification to replace existing orifices 1027 All AFW pumps declared OOS due to recirc flow orifice issue 1110 Crew briefed on AFW recirc flow orifice issue 1210 Posted temporary information tags for all 4 AFW pumps for requirements to secure AFW if minimum flow is not maintained 1241 CAP029952 was initiated for Potential Common Mode Failure of AFW Recirculation Lines 1305 All 4 AFW pumps RTS based on compensatory measures taken to brief crew and post temporary information.

1410 Engineering Director forms RCE Team 1525 Engineering Director discussed need for operability determination with Licensing and Operations - concern is being in an operable but non-conforming condition 1530- 1630 Simulator runs made on comp measures to assist in risk determination 1700 SCR 2002-0458 for procedure changes accepted by Operations Manager 1711 NRC 8 hour9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> notification (EN#39330) made regarding AFW system 1830 OPR requested for CAP029952 concerning common mode failure for AFW pumps. OPR is due at 1500 on 10/30/02 2030 Safety Monitor transitioned to Yellow for both units based on PRA Group input 0700 RCE Team Meeting held to define roles and responsibilities 1030 Qualifications suspended for design work for 3 individuals 1100 Notified of NRC special inspection team 1250 RCE Charter approved 1530 RCE Team Meeting 1850 OPR completed on AFW Pumps Day Shift SVP forms event resolution team 72

Possible Common Mode Failure of Aux Feedwater Recirculation Lines Attachment B - Barrier Analysis RCEO00191 73 Energy/Hazard Barrier Assessment Target Modification Procedure -

Procedure required that a justification be provided if the conceptual design was Good AFW MR 99-029*A/B Plant determined to be not applicable (Step 4.5.3).

design - no Modifications Procedure adequately raised FMEA question for consideration (in PBF-1584).

unknown (NP 7.2.1, Procedure directed review of the modification package by all team members failures modes Rev. 6 (Step 4.7.7.b), although some documents can be included by referenced only 7/21/99)

(Step 4.7.5.g). Methods not clearly described - RR6.

Procedure directed SE go to MSS for approval following the steps for final design review and independent technical review (Step 4.8).

Procedure described FDGH design approval as addressing the question "Have all reasonable modes of failure been identified and addressed in the design?"

(Attachment 1, step 4.7.6.c)

Procedure -

Procedure required that design verification be performed by a competent Design individual other than the originator or any other person with direct responsibility Control for that design.

(NP 7.2.2, Rev. 5 9/30/98)

Procedure -

Procedure specified some of the EAC responsibilities as:

Engineering Review/Screening of all proposed plant projects for technical need and Advisory feasibility from a plant operating, maintenance, and engineering bias, Committee considering alternative methods and impact on the Station.

(NP 7.1.6,

° Assure adequate technical description and justification is provided.

Rev. 0 1/6/99) e EAC shall select and approve the final disposition for the project considering I

alternative solutions.

RCEO00191 Possible Common Mode Failure of Aux Feedwater Recirculation Lines Energy/Hazard Barrier Assessment Target Design No information in final design description on potential to plug RO; no discussion Engineer of orifice size being less than SW strainer size. Concluded that Appendix R (DEl) flowpath would not be affected by new RO. Failed to apply FMEA to design.

Inadequate design. Tunnel Vision - K4.

After plugging issue was identified, dispositioned issue in the safety evaluation.

Shortcuts Taken - J7/Fear of Failure - A10. This bypasses the design process and moves the plugging issue into a licensing process.

Design Team Final design review performed after SE approved by MSS. Procedure adherence.

(FPEI, OP)

Inattention - Al.

Not all material is distributed to every team member. Fire Protection Engineering and Operations members did not recall seeing safety evaluation that had' information on plugging. Inadequate communications among organizations F2.

Independent Independent review performed after SE approved by MSS. Procedure adherence.

Review Inattention - Al.

(DE2)

Independent reviewer was a co-preparer of the safety evaluation. Lack of independence. Inadequate Program Management - F6.

MSS had decided recirc line not needed. Reviewer agreed that FMEA did not apply to design and that Appendix R flowpath would not be affected. Mindset J3.

Supervisor Conceptual design not required -justification not provided. Procedure Approval adherence. Inattention - Al.

(MDS)

Condones use of safety evaluation for disposition of plugging issue. Inadequate Program Management - F6.

Independent review approval performed after SE approved by MSS. Procedure adherence. Inattention - Al.

MSS had decided recirc line not needed. Approved design. Failed to ensure adequate FMEA performed. Mindset - J3.

Definitive resolution of plugging issue not pursued. Inadequate Emerging Issues Management - FS.

74

Possible Common Mode Failure of Aux Feedwater Recirculation Lines 75 RCEO00191 Energy/Hazard Barrier Assessment Target EAC Specific details of RO not presented to EAC. Plugging not discussed.

Inadequate communications within an organization -- Fl. EAC recommends performing modification but does not select specific option to be pursued.

Inadequate Tracking - A8.

Safety Procedure -

The SE is normally prepared by the individual proposing the facility change, but No unreviewed Evaluation Authorization may be co-signed by another technically competent and qualified individual, safety question SE 2000-0055 of Changes, The SE shall be performed on the final design of the facility change.

Tests, and The 10 CR 50.59 criteria can be broken down into six separate questions, which Experiments are each to be answered and explained.

(NP 10.3.1, Normal review of the prepared SE is assigned to the appropriate qualified system Rev. 12 engineer.

7/14/99)

The reviewer is responsible for reviewing the licensing basis, the content and technical adequacy of the SE, and verifying whether an USQ exists.

The MSS shall review the SE to ensure that the requirements of 10 CFR 50.59 have been met.

Procedure -

MSS Members review all proposed changes or modifications to plant systems or Manager's equipment where changes affect nuclear safety. This includes 10 CFR 50.59 Supervisory evaluation reviews and a review of the modification request design scope.

Staff and MSS Members meet their ANSI N18.1-1971 discipline requirements.

Qualified Qualified Reviewers are only required to review certain procedures or procedure Reviewer changes, and not modifications.

(NP 1.6.5, Qualified Reviewers meet one or both of being qualified to perform 10 CFR Rev. 3 50.59 safety evaluations or based on plant experience being designated by Plant 11/8/99)

Manger.

Possible Common Mode Failure of Aux Feedwater Recirculation Lines 76 RCEO00191 Energy/Hazard Barrier Assessment Target Procedure -

ANSI 3.1-1987, Selection, Qualification, and Training for Nuclear Power Plants, Offsite provides guidance for member qualifications.

Review OSRC reviews safety evaluations.

Committee Written records of reviews and resolution of comments arc documented and Process retained.

(NP 5.1.6, Rev. 1 8/26/98)

Preparer Initial SE had technical discussion on why SW particles would not plug RO, but (DEI/DE2) approved SE assumed RO might plug, yet stated that the modified recirc line DE1 not 10 would function identically to existing line. Flawed analytical model - K7.

CFR 50.59 SE utilized vendor information on why design precluded plugging. Misdiagnosis qualified

- K3.

The SE identified the AFW section of the FPER as part of the CLB but the recirc line Appendix R function not described in SE. Shortcuts Taken - JV.

SE only discusses the recire line function during AFW pump start and does not address pump shutdown or the need for operation on recire during the other stages of accident response. Inadequate knowledge of standards - K6.

After incorporating MSS comments, the SE did not consider that loss of the recirc line due to loss of instrument air is recoverable via actions specified in AOP-5B, whereas loss of the line due to plugging is not recoverable. Inadequate knowledge of standards - K6.

Possible Common Mode Failure of Aux Feedwater Recirculation Lines RCE00191 77 Energy/Hazard Barrier Assessment Target Reviewer Reviewer did not ensure SE addressed all licensing basis functions: Appendix R (AFW SE1) function and AFW recirc line operation during accidents not addressed. Wrong Assumptions - J4.

Reviewer did not ensure SE content was technically adequate: vendor information was misapplied. Inadequate Verification - J5.

Statement remained in SE that recirc line functioned identically to existing line.

Inattention - Al.

After incorporating MSS comments, the SE did not consider that loss of the recirc line due to loss of instrument air is recoverable via actions specified in AOP-5B, whereas loss of the line due to plugging is not recoverable. Inadequate Verification - J5.

MSS MSS received only the SE for review and not the modification request design scope. Procedure adherence. Methods not clearly defined - RR6.

MSS reviewed SE prior to final design approval. Procedure adherence. Actions not tied to another process when necessary - RR5.

MSS members offered qualitative resolution of plugging issue based on loss of recirc line due to AOV failing closed, without considering other recirc line functions, Inadequate knowledge of standards - K6.

OSRC OSRC members had no comments on SE 2000-0055. Inadequate knowledge of standards - K6.

Modification Procedure Same comments as MR 99-029*A/B.

Good AFW MR 99-029*D (NP 7.2.1, design - no Rev. 6 unknown 17/21/99) 1 failures modes

RCEO00191 Possible Common Mode Failure of Aux Feedwater Recirculation Lines Energy/Hazard Barrier Assessment Target Design DE I distributed a design package for review to the design team on 8/2/0 1. A Engineer safety evaluation is listed as one of the documents. DE I had left PBNP by the (DEI) time this modification was being finalized. No new PM was assigncd. DE2 facilitated changes to design description and related documents and obtained approval from DE1 via e-mail. Final design description and SCR are dated 12/4/01. No changes were made to describe plugging potential. Shortcuts taken

- J7.

Design input checklist was changed so that fire protection considerations were not applicable. Inadequate knowledge of standards - K6/Mindset - J3.

Design Team It is not clear what material was provided to the design team for review. Team (FPE2, OP) members sign off the final design review between 11/26/01 and 12/4/01. Rev. 0 of the final design description and SCR 2001-0981 are dated 12/4/01.

Fire Protection Engineering and Operations members did not recall seeing safety screening or evaluation that had information on plugging. Inadequate communications among organizations - F2.

Independent Independent reviewer was the preparer of the safety screening and helped revise Review the design documents. Lack of independence. Inadequate Program (DE2)

Management - F6.

Reviewer agreed that FMEA and Appendix R did not apply to design. Mindset J3.

Supervisor Conceptual design not required -justification not provided. Procedure Approval adherence. Inattention - Al.

(MDS)

A new PM was not assigned after DE1 left PBNP. Inadequate Program Management - F6.

Failed to ensure independent reviewer had no responsibilities for the design.

Inadequate Program Management - F6.

Approved design. Failed to ensure adequate FMEA performed. Mindset - J3.

78

RCEO00191 Possible Common Mode Failure of Aux Feedwater Recirculation Lines Energy/Hazard Barrier Assessment Target Safety Screening Procedure -

New Rule is in effect.

No LAR SCR 2001-0981 10 CFR The SCR is normally prepared by the individual proposing the facility change, required 50.59/72.48 but may be co-signed by another technically competent and qualified individual.

Applicability, A qualified individual prepares the SCR.

Screening and The SCR is reviewed by a qualified individual having the required technical Evaluation expertise as well as being screening or evaluation qualified.

(NP 5.1.8, The reviewer is responsible for reviewing the licensing basis, the content and Rev. I technical adequacy of the SCR, and verifying whether an evaluation i§ required.

11/14/01)

MSS is responsible for reviewing evaluations to determine if prior NRC approval is required.

Procedure -

MSS Procedure requirements are basically the same as Rev.3. However, the 10 (NP 1.6.5, CFR 50.59 New Rule changes the basis for determining if a safety evaluation is Rev. 6 needed. The MSS procedure still requires review of modifications that affect 11/14/01) nuclear safety, but the results of the 10 CFR 50.59 screening no longer answer that question. Actions conflict with another process - RR4.

Preparer The SCR identified the AFW Section of the FPER as part of the CLB, but did not (DE2) list the Appendix R flowpath as a design function. Shortcuts Taken - JV.

The SCR stated replacement of the RO would not adversely affect its design function and that the RO would allow flow to maintain TDAFW pump operability when in the recirculation mode. The SCR also stated that service water could be pumped through the recire lines. The SCR raises the issue of the potential for the recirculation orifices to accumulate debris from SW but does not pr6vide any definitive basis for the conclusions drawn regarding its impact on AFW pump operation. There is no discussion explaining the change in conclusions drawn from SE 2000-0055 regarding not needing the recirculation line. Mindset - J3.

SCR utilized vendor information on why the RO design precluded plugging.

1 Misdiagnosis - K3.

79

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines Energy/Hazard Barrier Assessment Target Reviewer Reviewer did not ensure SCR addressed all licensing basis functions: Appendix (AFW SEI)

R function and AFW recirc line operation during accidents not adequately addressed. Wrong Assumptions - J4.

Revie~ver did not ensure SCR content was technically adequate: vendor information was misapplied. Inadequate Verification - J5.

Reviewer did not ensure that the conclusions were supported - The SCR raises the issue of the potential for the recirculation orifices to accumulate debris from SW but does not provide any definitive basis for the conclusions drawn regarding its impact on AFW pump operation. There is no discussion explaining the change in conclusions drawn from SE 2000-0055 regarding not needing the recirculation line. Mindset - J3.

MSS MSS review is not required based on results of 50.59 screening. Evaluation if modification affects nuclear safety is not performed.

OSRC OSRC is not required to review safety screenings.

Modification Procedure Changes from NP 7.2.1 Rev. 6 used for MR 99-029*A/B are:

Good AFW MR 99-029*C NP 7.2.1, Minor plant changes do not require a conceptual design.

design - no Rev. 8 Other comments from MR 99-029*A/B apply.

unknown 10/2/01 failures modes Design The design description is essentially the same as that in MR 99-029*D. No Engineer changes were made to describe plugging potential. Shortcuts Taken - J7.

(DE2)

Design input checklist was changed from MR 99-029*D so that fire protection considerations were applicable. The RO is recognized as being in a safe shutdown flow path. Potential for plugging is not discussed. PBF-2060 is completed and concludes no adverse impact. Mindset - J3.

Design input checklist item for FMEA is still marked as not applicable to the design. Shortcuts Taken - J7.

Design Team No record could be found of design package distribution to team members. Fite (FPE2, OP)

Protection Engineering and Operations members did not recall seeing safety screening/evaluation that had information on plugging. Inadequate communications among organizations - F2.

80 1

RCEO00191 Possible Common Mode Failure of A ux Feedwater Recirculation Lines Energy/Hazard Barrier Assessment Target Independent Reviewer agreed that FMEA did not apply to design and Appendix R function Review was not affected. Mindset - J3.

(DE3)

Supervisor Approved design. Failed to ensure adequate FMEA performed. Mindset - J3.

Approval (MDS)

Modification Procedure -

The plant modification procedure evaluates a proposed design against existing Good AFW MR 02-029 Plant design requirements and functions. Because the RO plugging issue was design - no Modifications addressed previously in licensing space via a safety evaluation versus design unknown (NP 7.2.1, space, this did not result in any design document changes to state that the recirc failures modes Rev. 10 line was not required to pass flow (although that was incorrect). Therefore, when 6/12/2002) a safety function change was proposed for the AFW recirculation AOV and line to pass flow, the conflict with the line not being capable of providing flow was not detectable via the design change process because that design feature was not in a design basis document. The safety evaluation was in conflict w'ith the current licensing basis Actions Conflict with Another Process - RR4.

This procedure does consider other work in progress. Actions not specified RRi.

Procedure -

This procedure provides the directions for revising and controlling Q-List Q-List classifications of SSCs. It does not provide direction on how to classify an SSC Nuclear or how to evaluate a component's ability to support a safety function. It does Safety describe interfaces with other programs to enable consistent classifications and Classification analysis. Those programs are EQ, FP, Appendix R, Seismic, ASME Section XI, for SSCs Maintenance Rule, Procurement, Electrical components QA classification, and (NP 7.7.10 computer software QA.

Rev. 1 10/18/01) 81

Possible Common Mode Failure of Aux Feedwater Recirculation Lines RCE000191 Energy/Hazard Barrier Assessment Target Procedure -

The guideline provides direction on how to classify equipment, and how to Guidelines for classify interfaces. Appendix B provides examples of specific functions for

System, parent components. This information would be good reference material in Component assessing the capability of a component to perform a safety function. There is no and Part direction provided on adding a safety function to a component.

Classification (DG-G06 Rev. 6 6/8/01)

Design Initial design scope provided by Engineering management. Scope revised to include recirc line function to pass flow. Design needed to be developed and installed in a 3-week period. Inadequate Planning - F4.

Design No evaluation of recirc AOV is performed because it is already classified as Engineer safety-related. Wrong Assumptions - J4.

(DE2)

No evaluation of overall design of recirc line and components is performed to verify that it can support added safety function. Focus is on selected mechanical components. Unfamiliar or Infrequent Task - K2/Time & Schedule Pressure

-A9.

No evaluation is performed of change of safety function affect on other components and/or systems, or other work in progress. Actions not specified RR1.

Did not relate potential plugging issue for ROs (MR 99-029*A/D) to recirc line function to pass flow. Lapse of Memory - A7/ Time & Schedule Pressure A9.

Design Team Accepted assumption that recirc AOV already being safety-related would support (AFW SE2) the added safety function to open, and narrow focus of evaluation. Inadequate Verification - J5.

82

RCEO00191 Possible Common Mode Failure of Aux Feedwater Recirculation Lines Energy/Hazard Barrier Assessment Target Independent Accepted assumption that recirc AOV already being safety-related would support Review the added safety function to open, and narrow focus of evaluation. Inadequate (DE4)

Verification - J5/ Unfamiliar or Infrequent Task - K2/Time & Schedule Pressure - A9.

Supervisor Approved design. Accepted assumption that recirc AOV already being safety Approval related would support the added safety function to open, and narrow focus of (MDS) evaluation. Inadequate Verification - J5.

Safety Screening Procedure Same comments as for SCR 2001-0981. Procedure essentially the same as Rev.

No LAR SCR 2002-0359 (NP 5.1.8,

1.

required Rev. 2 4/3/02)

Preparer The description of design functions in Section II focuses on selected components (NSAI) and does not discuss the overall function of the line. Time & Schedule Pressure

- A9/Mindset - J3.

The discussion of the RO function in Section 11. L.b does not include its need to pass flow for pump cooling. Mindset - J3.

The potential for plugging of the restricting orifices is not discussed. Wrong Assumptions - J4/Misdiagnosis - K3.

The discussions of specific design functions in Section II and III do not include the Appendix R function. Mindset - J3.

Reviewer The description of design functions in Section II focuses on selected components (DE2) and does not discuss the overall function of the line. Unfamiliar or Infrequent Task - K2/Time & Schedule Pressure - A9.

The discussion of the RO function in Section II..b does not include its need to pass flow for pump cooling. Mindset - J3.

The potential for plugging of the restricting orifices is not discussed. Lapse of Memory - A7/ Time & Schedule Pressure - A9.

The discussions of specific design functions in Section II and III do not include the Appendix R function. Mindset - J3.

MSS MSS review is not required based on results of 50.59 screening. Evaluation if modification affects nuclear safety is not performed.

83

Possible Common Mode Failure of Aux Feedwater Recirculation Lines RCEO00191 84 Energy/Hazard Barrier Assessment Target JOSRC OSRC is not required to review safety screenings.

Safety Screening Preparer The responsibilities for the dedicated operator in the safety screening only No LAR SCR 2002-0377 (NSAI) address AF-4035 monitoring and notification associated with AFW pump starts; required there is no discussion of monitoring when flow is reduced to less than required minimum flow values. Mindset - J3.

The potential for plugging of the restricting orifices is not discussed. Wrong Assumptions - J4/Misdiagnosis - K3.

Reviewer The responsibilities for the dedicated operator in the safety screening only (DE2) address AF-4035 monitoring and notification associated with AFW pump starts; there is no discussion of monitoring when flow is reduced to less than required minimum flow values. Mindset - J3.

The potential for plugging of the restricting orifices is not discussed. Lapse of Memory - A7/ Time & Schedule Pressure - A9.

Possible Common Mode Failure of Aux Feedwater Recircitlation Lines RCEO00191 Attachment C - Stream Analysis Failure modes with a high number of arrows out of their box are the drivers of the event.

Failure Modes for MR 99-029*A-D 85

Possible Common Mode Failure of Aux Feediwater Recirculation Lines RCEO00191 Failure Modes for MR 02-029 86

Possible Common Mode Failure of RCEO00191 Aux Feed water Recirculation Lines Attachment D - Charter Root Cause Investigation Charter CAP029952 RCE000191 Issue Manager:

Jim Freels Problem Statement:

Discovery during the evaluation of CAP029908 (P-38A, MDAFW Pump had inadequate recirc flow during IT-10) that the recirculation line restricting flow orifices had become plugged during plant operation causing a reduced flow (but above minimum required) and that a potential existed for a common mode failure where all AFW pump recirculation lines could have restricted flow rates resulting in eventual pump failure.

Investigation Scope:

Determine the following:

Timeline of key events The root and contributing causes of why the condition exists, including any potential human performance issues Why the problem was not identified previously Make recommendations for:

Correcting the problem, including any remedial actions Preventing recurrence of the problem Applicability of the root cause to other areas (extent of condition), including verification that a safety-related AFW recirculation flow path exists for the postulated failure modes Team Members:

Team Leader - Richard Flessner, Engineering Processes Team Member - Kevin Bennett, Engineering Processes Team Member - Eric Schmidt, System Engineering Team Member - William Bosacki, KNPP Design Engineering Milestones:

Status Update - 11/4/02 Status Update - 11/11/02 Draft Report - 11/18/02 Final Report - 11/26/02 Approved:

(original signed by J. Freels)

Date:

10/30/02 Jim Freels, PBNP Engineering Director 87

RCEO00191 Possible Common-Mode Failure of Aux Feedwater Recirculation Lines Attachment E - PBNP Orifices quiptie~

ty~~~

~

~

-~-ocaon~-~ QA~SR,;~v- ~Equlýrnnt ei~Sys.

1,J Manuf.--

?-SerlaI.

Tech-Equipment Type I~

~~'

~'>~~

I~A;:

~-umer:

Manual

.~~

~ ~ ~

~'.~-Numnber PB1 RO-Venturi (2) 684,741-3 (D8) uic Y

Y P-"B RCP #1 SEAL BYPASS ORIFICE cv 00o198B*

I dv PBi RO-Venturi (2) 684,1741-3 (1-17) 21/u1C/REGEN Y Y

LETDOWN ORIFICE A ORIFICE CV

_ 001 99A I

HX AM PB1 RO-Ventur (2) 684J7411-3 (ll6) 21/U1C/REGEN Y Y

LETDOWN ORIFICE B ORIFICE CV

_00199B HX RM PBInRO' Venturi (2) 684J741-3 (H6) 21/U1C/REGEN Y Y

LETDOWN ORIFICE C ORIFICE CV 001 99C HX RM RO-00458 Orifice Plate (2) 541F091-1 (B3)

UIC Y

Y RC LOOP A COLD LEG RTD FC....

P'NC MANIFOLD ORIFICE PB1 RO-00459 Orifice Plate (2) 54iF091-1 (D10)

U1C Y

Y RC LOOP B COLD LEG RTD RC PNC

__......._MANIFOLD ORIFICE PB1 RO-00487 Venturi (2) 541F091-2 (F5) 66/U1C Y

Y IR-1 RV HEAD VENT FLOW ORIFICE R C PB1 RO-00488 Ventur I(2) 541F091-2 (GS) 66/UIC Y

Y ' T-1 PZR VENT FLOW ORIFICE RC PB1 RO-Orifice Plate 1i0E017-3 (G5) 8/PAB Y

Y P-14A CONT SPRAY PUMP FLOW SI 100900A ORIFICE PB1 RD-Venturi (1) 110E017-3 (F6) 8/PAB IY Y

P-14A CONT SPRAY PUMP RECIRC SI

_______________0___900B______________ORIFICE PB1,0iRO" Orifice Plate Ii0E017-3 (C5) 8/PAB Y

Y P-14B CONT SPRAY PUMP FLOW Si i

P 10 00 Vnui()

1E073(6 V

V ORIFICE RP.-B Ventud (1) 110E017-3 (D6) 8/PAB Y

Y

-14B CONT SPRAY PUMP RECIRC SI 000)ORIFICE L

i00900D

'a..

,m

,,.n...

A 1PB1 jU-U4U03

'PB2 RO 100198A PB2 iRO I

!00198B IP 2 **R -

I

'00199A Y

I1-9

'A-" MINI R-¶I"l.,IL UORIFICE

+

-t-t



trr-t U2C/P-1B RCP Y CUB/MIDDLE

.LVL E "21/U2C/REGEN Y HX RM

'211U2CIRFGEN IY IHX RM Y

Y V

Y P-1A RCP #1 SEAL BYPASS ORIFICE P-lB RCP #1 SEAL BYPASS O0 LETDOWN ORIFICE A ORIFICE CV 1UFICE CV

-cv Venturi (2) 685J175-3 (D8)

V renturi (2) 1685J175-3 (H7)

Venturi (2) 685J175-3 (H6)

REhSTRITCTING RESTRICTING

'RESTRICTING RESTRICTING "FLOW ORIFICE FLOW ORIFICE RESTRICTING

RESTRICTING

'RESTRICTING RESTRICTING RESTRICTING RESTRICTING

,ORIFICE RESTRICTING RESTRICTING RESTRICTING RESTRICTING 1H AM-88 PB1 nO-684J741-3 (D6)

UIU Y

Venturi (2) 685J175-3 (06) u/*iIP, "M 1P-29 CUB T

21/U2C/WEST STAIRS SOUTH Y

IPB2 RO 019ggB SOUTH S[

I

=

mv.....

i P*-lA IlL~l #1 SEA:/L I5 T PASS*;

ORIFICE LIV Venturi (2)

ValVe/Irlm/Adl I M-217-1 (B611)

/M-I

) 1 ILE.TDOWN ORIFICE B ORIFICE_

Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines I'P, EQI*'ii~;I*;'*~b 1/2',*!'

P*15*D!

• • i*L(*atI* p*'otI 81* I*

Eq~pm~rnt NmpI*ef÷*,

I;Sy*

fl

• Manuf.'\\ I* *SerlaI;-y

• Tech,,I Equipment Type '

1-1 Orifice Plate (2) 541F445-1 (B3)

Onfice Plate (2) 541F445-1 (D8)

H'X 1U2./R-

-NIY HX RM U2C U2C Y

Y Y

Y LE-WI UU ORIF vIIIICE UCtlORIFIC RC LOOP A COLD LEG RTD MANIFOLD ORIFICE PC LOOP B COLD LEG RTD MtANIFOLD ORIFICE I

I MANIFOLD ORIFICE I

I I

I Y

R-1 RV HEAD VENT -LOW OHII-IU:

~

-"4 541 F445-2 (G5)

U2C/T-1 PZR CUB/UPPER LEVEL Y

Y T-1 PZR VENT FLOW ORIFICE MU M.

MC IPB2 RO-Oriflce Plate 110E035'-3G) 8/PAB IY Y

I P-14A CONT SPRAY PUMP FLOW SI OOR900A ORIFICE 110E035-3 (C5) ilEU035-J

.u3)

M-217-1 (06)

U/PAB~

8/CB/AFP RM 2P-29 CUB Y

Y Y

y Y

Y Y

P-14A CONI SPRAY PUMP MEUIMU ORIFICE P-14B CONT SPRAY PUMP FLOW ORIFICE P-14B CONT SPRAY PUMP RECIRC ORIFICE 2P-29 AFP MINI RECIRC ORIFICE Si AF "rBO 4RO-Orifice Plate M-227-1 (DI0) 50/DGB/G-03 Y

Y 0-03 EDO HX*265A RADIATOR Vt4NT DG

____6 AI 'r_

j

.103362A RADTRI RMORFC Y

Y G-04 EDG HX-265B MADIATIO VEN I ORIFICE iRSTICIN C

W......

O RIFICE PNC FLOW ORIFICE

' RESTRICTING RESTRICTING FLOWCO DANFPI DANFIE D440T-1-1 01708 RESTRICTING RESTRICTING

RESTRICTING

'RESTRICTING ORIFICE B0RO Orifice Plate IM-227-1 (H8) 150/DGB/G.03 Y

YG-03 EDG HX-266A LUBE OIL DG

[DANFPI 03363A 4RADTR RM COOLER VENT ORIFICE I

50/DGB/G-04 RADTR RM Y

Y G-04 EDG HX-266B LUBE OIL COOLER VENT ORIFICE

'DG DANFPI PBO0RO-

.Orifice Plate M-219-3 (G5) 128/DGB/G-03 IY IY IT-1 76A G-03 EDG DAY TANK INLET FO DANFPI 03989A

________ P DAY TK RM ORIFICE 28/DGB/G-04 FOTP/DAY TK RM Y

Y T-176B G-04 EDG DAY TANK INLET ORIFICE P0RO-04008 Valve/Trim M-217-1 (D6) 8/CB/AFP RM

[Y Y

P-38A AFP MINI RECIRC ORIFICE P-38A CUB L,...

8/CB/AFP RM P-38B CUB 26/PAB/CENTR AL

","fEA*f/ABEN l

AL Y

Y Y

Y N

N4 P-38B AFP MINI HECIiHU UiIIE-I" SW FROM HX-15A-D CONT VENT CLR MOV BYP ORIFICE PB0 RO-04015 PB1 IRO-02909 FO AF SW SW FROM HX-15A-D CONT VENT CLR jSW MOV BYP ORIFICE I

DANFPI FLOWCO F-LOWOO' DANORF 01708 01708

'RESTRICTING RESTRICTING ORIFICE

'ORIFICE 89 RO oa199C

'PB2 IPB2 PB2 PB2 RO-00458 RO-00459 RO-00487 RO-00488 Venturi (2*

Venturi (2) 541 F445-2 (F5) 66/U2C PB2 RO AAt3lnOR Venturi (1)

ROOO0c 1 10E035-3 (F6)

[PB2 P132 Onfice Plate 00900B 8/PAB RO 00900D RO-04003 Venti (l)

V/alvefrrim/Adj 8/PAB 00900i 4--------------------

PB0 RO-Orifice Plate M-227-2 (E10)

PB0 IRO 03363B Onfice Plate 50/DGB/G-04 RADTR R M-227-2 (H8)

PB0 RO 03989B Orifice Plate M-219-3 (ES)

Valve/Trim M-217-1 (E6)

Orifice Plate 010rlfice Plate '

!M-207-4 (G9)

M-22U7-2 (B7) jlý821,9 0-02909 r!'00199C

-ý I

-- Il--,--i 4.........

I '

t I

tCV

'Venturi (2)

'685J 175-3 (1-6)

PB2 I

t i

I

-,I Possible Common Mode Failure of RCEO00191 Aux Feedwater Recirculation Lines

,I*

• `at on~i n

S.

Manuf,.

ý:Serialh TeW quipment Type

]'~t'.

~

~Y>

I.~'~

'~-

I I

INumber Manual

~k~

~j

~ I I

~

Number Iý Inline Orifice 684J972-1 (G6)

Inline Orifice 684J972-1 (B6) 46/PAB/WG COMP RM 46/PABJWG COMP RM Y

N N

K-1A W(G COMP SI:AL WA11-H MAKEUP INBOARD FLOW ORIFICE K-I B WG COMP SEAL WATER MAKEUP INBOARD FLOW ORIFICE "WGv UULIt

" DOLE 1PB0 RO-Inline Orifice 6844J972-1 (G6) 46/PAB/WG Y

IN tK-1AWG COMP SEAL WATER WG rDOLE "01006A COMP RM IMAKEUP FLOW ORIFICE COMP RM

'Y

'N r'.-1 B Wu C.OMMr E

WATE I

MAKEUP FLOW ORIFICE UU413 00413 I'LUW F5LOW 100413 1FLOW 10 PB0RO-Orifice Plate 684J972-1 (G6) 46/PABIWG Y

N K-1A WG COMP SEAL WATER FLOW WG 01009A COMP RM ORIFICE PBO RO-Orifice Plate 684,J972-1 (B6) 46/PAB/WG Y

N K-1B WG COMP SEAL WATER FLOW WG.

• 01009B COMP RM ORIFICE PB0ORO Orifice Plate 1684J972-1 (G6) 46/PABJWG Y

N K-1AWG COMP SEAL WATER WG "01011

_______COMP RM MAKEUP FLOW ORIFICE PSBORO-Orifice Plate 684J972-1 (B6) 46/PABYWG Y

N K-1B WG COMP SEAL WATER WG j_90 _I_.11B I

ICIMP PM MAKEUP FLOW ORIFICE i

46/PAB/WG COMP PM Y

N K-1AWG COMP NITROGEN MAKEUP FLOW ORIFICE WG PBORe-Orfice Plate 1684J972-1 (C6) 146/PAB/WG IY IN K-1iB WG COMP NITROGEN MAKEUP WG 01014B COMP RM FLOW ORIFICE PB0 IRO-06306 PB1 jUn numbered PB1 nUn Inumbered 11OE018-1 (C8) 11oE018-1 (B8)

B/CB/AIR COMP RM Y

N 4- --

8/PAB/PIPEWA Y Y #2 B/PAB/PIPEWA Y Y #2 Y

Y T-33B/C IA RECEIVER CROSSTIE ORIFICE IP-1OA RHR Pump Recdr Orifice 1P-10B RHR Pump Recirt Orifice IA I*B2 On-Venturi (1) 110E029-i (CS) 8/PAB/PIPEWA Y Y

2P-1OA RHR Pump Reclrc Orifice

_numbered Y #3

,PB2 Un-Venturi (1) 1 10E029-1 (B8) 8/PAB/PIPEWA Y Y

2P-10B RHR Pump Reclr Orifice i numbered Y #3 PB1 Un-Staged Orif (1) 110E017-2 (F8) 8/PAB/SI PUMP Y Y

1P-15A HHSI Pump Recirc Orifice numbered AREA W PB1 Un-Staged Orif (1) 110E017-2 (ES) 8/PAB/SI PUMP Y Y

IP-15B HHSI Pump Reclrc Orifice 1numbered AREAW jPB2 IUn-Staged Orif (1)i 1"

I10E035-2"(E-8).... 8V*'/S"0IP

.... ;..... 2P-15A HHSI Pump Reclrc Orifice

_4.numbered

.AREA P

MP P82 Un-Staged Orif (1) 1 10E035-2 (D8) 8/PAB/SI PUMP Y Y

2P-15B HHSI Pump Recdrc Orifice Inumbered AREA W PBI MS-00279 Valve/Trim/Adj M-201-3 (F5) 26/U1F N

N HX-18A112 SGBD HXOUTTOT-26 I

I SGBD TANK IN THRTL PB2 MS-00279 26/U1F N

N HX-18B1/2 SGBD HX OUTLET TO T-26 SGBD TANK THRTL 0413 FLOW

RESTRICTING

RESTRICTING RESTRICTING RESTRICTING RESTRICTING

'RESTRICTING UNION 90 1PB 0Re I 01005A PB0 RO 01005B I-uU MU 01006B

'inlne unrice PB0 IRO

!01014A Odrifice Plate 684J972-1 (H6)

Coupling M-209-3 (F8)

Venturi (1)

Venturi (1)

Valve/Trim/Adj M-2201-3 (F6) 1WG.

T I E O

684J972-1(B6)

RCEO00191 Possible Common Mode Failure of Aux Feedwater Recirculation Lines 91