ML050060025

CR 70029861 Hope Creek B Recirculation Pump Excessive Seal Leakage Condition Report, 03/01/2003

ML050060025
Person / Time
Site:	Hope Creek
Issue date:	04/15/2003
From:	Morrison J, Nagy G, Walsh P - No Known Affiliation
To:	Office of Nuclear Reactor Regulation
References
TAC MC5111 CR 70029861
Download: ML050060025 (49)

Text

Root Cause Analysis of:

HC 'B' Recirculation Pump Excessive Seal Leakage Condition Report No: 70029861 Event Date: March, 2003 Investigators:

Lead Investigator:

Team Leader:

D. Boyle, E. Delp, G. Gardner, C. Johnson, J. Thompson I/k. ! tl~

Y

-13

( John W. Morrison e--ne Nagy Date Date Completed Approved by:

/112 4-t4_

Pat Walsh

B. Recirc. Pump Seal 70029861 EXECUTIVE

SUMMARY

The Hope Creek 'B' Recirculation Pump (B Pump) mechanical seals degraded over 16 months to a point where a maintenance outage was required to replace the seal package. The seal had been replaced in each of the last two refueling outages. The expected service life for this seal package is 6 years. The 'A' Recirculation Pump seal was last replaced in September 1999, and is expected to last until its scheduled replacement during RF12 in fall 2004. Clearly, the B Pump is not meeting its expected performance, and the A pump is.

The scope of the investigation was to identify the cause(s) of the most recent seal failure and the long-term unreliability of the B Pump seals. The root cause team used Technical Issue Resolution Process tools, fault tree analysis and change analysis to identify the causes.

The investigation identified no inappropriate actions associated with the assembly or installation of the B pump seal or with the operator actions associated with the filling and venting procedures.

The direct failure mechanism for the seal failures is a bowed pump shaft in combination with relatively large particles in the seal cooling / lubricating fluid. There are two possible sources of high particles: normal corrosion products in the RCS which get to the seal only if seal purge is lost, or particulate in the purge flow due to valve seat degradation, large particulate from CRD system, or a combination thereof. The team was unable to conclusively eliminate any of the above sources of particulate, so corrective actions are recommended for all.

Determining the exact condition of the pump shaft, and its root cause will require plant shutdown and disassembly of the pump seals and/or pump. A corrective action has been created to complete those aspects of the investigation.

The team identified four most probable root causes for large particulate getting to the seal surface:

Operation of the Reactor Recirculation System in accordance with procedures inadvertently caused the Reactor Recirculation purge line relief valve to lift and chatter causing damage to the disc. As a result, the purge flow was diverted out the damaged relief valve. Without purge flow to the seal, reactor coolant flowed through the seal causing damage to the seal faces. Corrective Actions include procedure revisions, changes to tagging orders, and establishing routine monitoring of purge flow / relief valve leakage. Some of the procedures have been revised multiple times without effectively resolving the problem. The engineer responsible for the most recent changes is no longer with PSEG, so further investigation into the organizational / programmatic, or human performance issues that caused the poor procedures has not been conducted.

Poor design of the interface between CRD System Pressure and Seal Purge Flow requirements causes the purge line relief valve to chatter open and closed during routine system operation. Corrective Actions include a Temporary Modification (Tmod) that gags the valves shut (already installed), and a modification to permanently address the design weaknesses.

Misapplication of design for purge flow Control Valves causes particulate debris in purge flow.

The valve design will be upgraded.

The present Control Rod Drive (CRD) inlet filter size permits particulates to enter the seal purge flow from CRD flow that are large enough to reduce seal life due to accelerated wear to seal faces.

4/14/2003 Page 2

B. Recirc. Pump Seal 70029861 The following contributing causes were identified:

Poor location of the purge flow indicator contributed to the loss of purge flow, resulting the relief valve leakage going undetected through most of the operating cycle. If the relief valve is eliminated, no action is required. Otherwise, periodic visual inspection of relief valve discharge will be implemented to detect leakage for both A and B pumps.

The organization did not display high sensitivity to purge flow rates. The system operating procedure will be revised to provide additional guidance about how long we should operate the pumps without purge flow.

SIGNIFICANCE Extent of Condition The degradation of both stages of the 'B' Recirculation pump mechanical seal caused a planned shutdown of the plant. The Hope Creek Recirculation Pump Seals are designed to prevent Reactor Coolant System affluent from passing from the pump to the Reactor Drywell. In March 2003 drywell floor drain flow (DWFD) exceeded the administrative limit of 1.5 gpm and approached the technical specification limit. The technical specification limit for UNIDENTIFIED LEAKAGE (includes DWFD flow) is 5 gpm (TS 3.4.3.2 b). Undesired leakage from the 'B' Reactor Recirculation Pump mechanical seal was one of the sources of the floor drain flow.

Management chose to shut the plant down before either limit was exceeded. Initial drywell walkdown identified two additional sources of DWFD flow: Leakage from RACS valve ED-V034 at approximately 1 gpm, and leakage from Chilled Water valve GB-V224 at approximately 1 gpm.

Multiple sources of unidentified leakage are discussed in *Other Issues, section.

The consequences in lost generation associated with replacement of the B pump seals was calculated at 178,000 MWhr. This condition did not result in the loss of a Maintenance Rule function is the Reactor Recirculation (BB) system or the Control Rod Drive (CRD) system. The BB system is not monitored for unavailability since it required to be 100% available, instead it is monitored on plant level performance indicators. This event did not result in an unplanned reactor SCRAM, transient or safety system actuation as defined for the Maintenance Rule and NRC performance indicator.

The degraded seal and resulting leakage did not adversely affect the function or material condition of any other machinery.

Generic Implications The potential consequences of a degraded Recirculation Pump mechanical seal would have been the same at a different time or under different circumstances. Excessive seal leakage can cause the plant to exceed the administrative limit of 1.5 gpm drywell floor drain flow and/or the Technical Specification limit of 5.0 gpm UNIDENTIFIED LEAKGE. A degraded seal will not adversely affect the function of any other plant equipment, but can allow contaminated water to spray on the immediate vicinity of the pump. However, a failed seal can adversely affect the safe operation of the plant. A catastrophic failure of the seal package will result in a LOCA which is one of the design base accidents addressed in the H.C. FSAR.

Common Mode Failure The Reactor Recirculation system consists of 2 loops, WA' and 'B', each using nominally identical pumps. The 'A' pump mechanical seal is susceptible to each of the root causes and contributing causes listed below for the 'B' pump mechanical seal failure.

4/14/2003 Page 3

B. Recirc. Pump Seal 70029861 Each pump has two 100% capacity seals; therefore the failure of one seal does not impose any safety significance. GE Licensing Topical Report, NEDO-24083 evaluated the failure of both seals on the pump and determined that there was minimal safety significance as the failure results in leak rates well within the make-up capacity of the plant and leak before break criteria apply.

Operating Experience The industry has a long history of Recirculation Pump seal degradation. Most plants have also experienced numerous seal purge relief valve failures. Several plants have attributed the seal degradation to particulate intrusion brought on by inadequate seal purge flow and thermal stresses caused by inadequate cooling water flow or seal purge flow. Several plants have implemented vent and purge procedure changes to improve relief valve performance. One plant contacted was planning to eliminate the relief valves and another was planning to increase the setpoint. Several plants have also upgraded the CRD system filter efficincies to reduce particulate carryover from the Condensate System into the seal purge lines.

Hope Creek is unique in using manual Kerotest full port globe valves for seal purge flow control.

Most plants are using a Circle Seal FCV flow control valve with success.

The vendors have focused recommendations on adequate maintenance practices, Recirculation Pump operation to minimize heat stresses and seal venting procedures.

SOER 83-4 contains a comprehensive listing of recommendations to improve seal life including that CRD suction filters be upgraded from 50 microns to 5 microns to improve the quality of water supplied to the seal purge. See Attachment 3 for OEF details.

4114/2003 Page 4

B. Recirc. Pump Seat 70029861 CAUSES AND CORRECTIVE ACTIONS The direct failure mechanism for the seal failures is a bowed pump shaft in combination with relatively large particles in the seal cooling / lubricating fluid. There are two possible sources of high particles: normal corrosion products in the Reactor Recirculation system which get to the seal only if seal purge is lost, or particulate in the purge flow due to valve seat degradation, large particulate from CRD system, or a combination thereof. The team was unable to conclusively eliminate any of the above sources of particulate, so corrective actions are recommended for all.

Root Cause(s)

1. There are indications that B Recirculation Pump shaft is bowed which caused excessive wear on seal parts and made the seal faces more susceptible to particle intrusion.

Basis:

The following indications of wear were observed upon removal of the 'B' Pump mechanical seal in March 2003.

a. Heavy wear on lock bolts, lock bolt bushings, and stationary face spring holder.

b. Wear band on rotating faces 164" (15 mils) wider than mating nose on stationary faces.

c.

Fretting on pump shaft and sleeve.

d.

Indication of rub between auxiliary impeller and stuffing box

e. Radial cuts on both stationary faces isolated to a single sector of the circumference.

A bowed pump shaft would result in unwanted radial and axial motion (wobble) of the shaft and shaft sleeve of the rotating seal face, which would cause the rotating seal face to wobble and could lead to the wear described above. The mechanisms that lead to the wear described in items (a) through (e) are described below. Items (a) through (d) support the theory that the shaft is bowed, and item (e) is the damage that the seal faces sustained as a result of the bowed shaft.

a. Excess radial motion of the shaft transmits unwanted forces and vibration to the stationary face lock bolts via the stationary face subassembly. The stationary face spring holder is also subjected to these forces. Our inspection showed that the springs had worn grooves into holders.

b. The radial motion of the rotating shaft and rotating seal face with respect to the stationary face causes a wear band on the rotating face, which is larger than the nose of the mating stationary seal carbon face.

c.

Fretting on the shaft sleeve as well as the shaft indicates contact between the sleeve and shaft. This contact was sufficient to cause relative motion between the sleeve and shaft, leading to the fretting on the shaft.

d. The radial motion of the shaft also causes eccentric motion of the centerline of the auxiliary impeller. The impeller rides inside of the stuffing box and forces water through the cooling coils. Inspection revealed wear around 100% of the auxiliary impeller circumference, and 80% of the stuffing box circumference at the same elevation. The wear on the auxiliary impeller was concentrated in one small area, indicating that this area had the most contact with the stuffing box due to the shaft bow. The nominal diametrical clearance between aux impeller and stuffing box is 0.030".

4/14/2003 Page 5

B. Recirc. Pump Seal 70029861 This motion has not been evident in our vibration data. This is due to the stiftness of the hydraulic bearing maintaining the bearing journal and the impeller centered at normal running speeds. This prevents any amplitude increase of any vibrations that are generated by the rotating assembly. This is due to significant percentage of the mass of the rotating assembly held in a balanced position. The result is an eccentricity in the shaft that is small and its center of mass is significantly less than the total assembly. Our vibration reading is 0.008" (8 mils). The rub on the stuffing box may have occurred at lower speeds, when the shaft is not held as rigidly.

e. The axial motion (wobble) of the rotating seal face due to the shaft bow causes cyclic loading of the stationary seal face, seal face holder and the seal face holder springs. As stated in (a), the stationary seal face holder was found to have grooves worn into it by the springs. The springs tend to hang up on the grooves, causing lower contact pressure on the faces. This allows water and debris to flow across the face and initiate radial cuts.

These cuts worsen over time as water flowed through them and cavitate (in the case of stage #1) and flashed to steam (in the case of stage #2). The inspection revealed that the first stage carbon face had 8 cuts and the second stage had 18-20 cuts of varying size. The largest cuts were approximately 3132" wide and 1/8" deep and were located in 4-6" circumferential band in one quadrant of each seal face.

Refer to Attachment 12, B Recirc Pump Failure Modes Matrix for summary of conditions present on the B Recirc Pump.

The Flowserve representative who was present during the seal autopsy states in his report that high runout of the shaft is one of the causes of the seal failure, and that this runout could be the result of imbalance in the shaft-impeller assembly or a bow in the shaft. (See Attachment 10 for vendor inspection report).

In 1999, maintenance attempted a balance correction in response to an increase in vibration. The initial attempt to balance the pump increased vibration levels and a second attempt had no effect on vibration levels. It was concluded that balance was not the cause of the vibration. Subsequently, mechanical runout of the coupling was investigated and the eccentricity in the coupling previously described was identified and the proximity probes were relocated.

Quad Cities experience with a pump with a bowed shaft provides some insights.

According to their Reactor Recirc System Manager, Quad Cities is operating with a Reactor Recirc pump with a known bowed shaft since 4 th Quarter 2001. The overall shaft runout is 0.008". Byron Jackson indicated runout for a shaft in service should be 0.004" maximum, Exelon internal pump experts approved 0.004" - 0.008" shaft runout for an operating pump. Vibration readings using the proximity probes read 0.008", taken at the coupling.. Hope Creek B recirc pump has vibration reading at the proximity probes of 8 mils at the motor.

Quad Cities has experienced seal reliability problems with the pump ever since the bowed shaft was installed. Pressure on the second stage seal decreased to a minimum of 360 psig during the cycle, then raised again and stabilized at approximately 420 psig.

The N7500 seal design was added in December 1998, and the seal purge was added in November 2002.

Based on the supporting evidence the team concludes that the B Recirc Pump shaft is bowed and recommends replacement during the next refueling outage, 4/14/2003 Page 6

B. Recirc. Pump Seal 70029861 RF12. The combination of the bowed shafted and relatively large particles in the seal cooling / lubricating fluid is identified as the cause of the most recent seal failure. With these indications of a bowed shaft, B Recirc Pump is likely to experience continued accelerated seal degradation.

Corrective Action:

Type/Status:

Corrective Action:

Type/Status:

Effectiveness:

Develop plan to replace B pump shaft in RF1 2. Present to ORB/PR. If funding is not approved, develop plan to replace seals every outage.

CRCA 0090 Due Date 07/17/2003 E-REROO Prevent Recurrence/Open Effectiveness: Extension of seal life to multiple cycles. If the shaft is removed, it's runout can be measured to confirm that it was indeed bowed.

Replace shaft in RF12.

CRCA 0100 Due Date 12/01/2004 E-RERO0 Prevent Recurrence/Open Extension of seal life to multiple cycles. If the shaft is removed, it's runout can be measured to confirm that It was indeed bowed.

4/14/2003 Page 7

B. Recirc. Pump Seal 70029861

2. Operation of the Reactor Recirculation seal purge system during pump shutdown evolutions inadvertently causes the Reactor Recirculation purge line relief valve 1 BFPSV-F025B (F025B) to lift and chatter causing damage to the disc. As a result, the purge flow was diverted out the damaged relief valve. Without purge flow to the seal, reactor coolant flowed through the seal causing damage to the seal faces Basis: Purge system is designed to prevent the reactor coolant from entering the seal.

Without adequate purge seal water flow, the Reactor Recirculation System enters through the shaft throttle bushing carrying impurities into the seal cavity. The impurities in the water are deposited on the seal faces and result in damage to the seal faces during operation. Observations that indicate inadequate purge seal flow include: particulate residue in the seal cavity and on the seal faces, the high radioactive contamination of the residue and the cuts on the seal surface, seal cavity temperature spike after pump is taken out of service.

Seal Purge System Diagram CRD Isol Orifice Throttle Flow Ind Relief Viv Cont.

Check Vlv IsOl Header Vlv D01 1A/B Vlv D02OA/B F025A/B Isol. Vlv V043/047 Vlv

& Filter V100/101 V804/805 HV-3800A/B V0421046 Seal purge flow is diverted out of the damaged relief valve. As-found leak tests confirmed that relief valves were sufficiently damaged to divert the 2.5 gpm purge flow from the seal. On 03/26/2003, a Relief Valve Technician, performed as as-found leak test and inspection on the F025A/B relief valves that were removed during the outage.

The result of the leak test at 200 psig for one minute was:

F025B = 12,890 ml/min or 3.4 gal/min F025A = 9430 ml/min or 2.5 gal/min.

Valve internals were severely eroded indicating long-term leakage and the disc had gouges in the seating surface. See Attachment 7 for As found test details.

B Pump experienced temperature spikes coincident with four downpowers or plant trips between December 2001 and September 2002. In each of these events, the B pump was taken out of service, but purge flow was not. The temperature spikes provide further evidence that the flow was diverted out of the F025B relief valve because even with purge flow in service the seal cooling function was not achieved. In June 2002, the plant was placed in single loop operation on B pump to support A motor generator repair. 'A' pump was taken out of service and did not experience the seal cavity temperature spike, indicating that purge flow was reaching the seal at that time.

The CRD Hydraulic System supplies a branch from the CRD Pump discharge header before the flow control station to feed cooling water to the A and B Recirc Pump seals.

The seal purge tap is normally at about 1400 psig and is routed to two separate manual globe valves (V804 and V805) used to throttle the flow rate, one for each pump. A flow restricting orifice (FO-DO11AIB) is installed upstream of the globe valve. The purpose of the FO is to limit flow from CRD should the throttle valve fail. The FO is sized for a 200 4/14/2003 Seal Page 8

B. Recirc. Pump Seal 70029861 psig drop with 4 gpm flow. The flow path continues past the throttle valve, through flow indicator 020A/B, containment isolation valve HV-3800A1B, check valve V043N047 and into the pump seal. The downstream line contains a pressure relief valve set at 1250 +1-50 psig near the HV-3800A/B valve. The seal pressure runs at about 1000 psig in normal operation.

The seal acts as a large orifice to the seal purge line. There is no known mechanism for the seal to suddenly restrict purge inlet flow. Design Engineering developed a simplified hydraulic model of the system. Essentially, the restricting orifice and throttle valve, all but about I psig, takes up the entire differential pressure between the CRD header and the seals. The model showed that the throttle valves are barely open (< 5%). Thus the pressure at the relief valve is nearly the same as that at the seals. Even increasing the CRD header pressure to account for supply pressure surges doesn't significantly impact the results. Since the CRD supply is before the system flow control valves, there is no significant change in seal flow when CRD system flow is adjusted during plant shutdown evolutions. The biggest change would occur during plant startup and shutdown when the seal flow is affected by the changes in first stage seal pressure. As this pressure increases, seal flow drops almost in half. However, Operations monitors the flow and manually adjusts the throttle valves to compensate.

The plant computer via a pressure transmitter connected to the purge inlet line monitors first stage seal pressure. Plant Historian records indicate that the first stage seal pressure follows reactor pressure and never exceeded approximately 1020 psig even when CRD header pressure increased to 1478 psig during pump swaps. This is consistent with the model and indicates that pressure spikes at the relief valve during normal operation or during speed changes is very unlikely.

Three cases involving pump shutdown evolutions were identified that have the potential to cause the relief valve to lift. Individually or in combination these are the most likely to cause the relief valve damage.

Case 1: Fill and Vent Hope Creek operations procedure HC.OP-SO.BB-0002, Reactor Recirculation System Operation, provides direction for recirculating pumps fill and vent. Revision 42 of the procedure had potential mechanisms to cause excess pressurization of the Seal Purge line both during isolation and restoration. The F025B relief valve could have lifted during the fill and vent following the B seal replacement during RF10 in November 2001. During this evolution the purge line was potentially over pressurized, the relief valve lifted and chattered for perhaps 20-30 minutes. The relief valve would sustain sufficient damage in a matter of minutes of this type of chattering to cause leakage through the relief valve.

Overtime, the leakage would have increased due to erosion such that essentially all purge flow was diverted through the relief valve. The as-found condition of the F025B internals was consistent with this conclusion.

Prior to the planned outage of March 2003, the procedure was reviewed. Section 5.6.9 had the operator close the V805(V804) throttle valves, then close the isolation valve HV-3800A(B). This alone would likely result in relief valve PSV-F025A(B) operation as the throttle valves are known to be leak prone. Discussions with Operations indicates that the normal practice is to supplement the procedure with a tagging order that includes closing the supply shutoff valve V100(V101) and opening the drain/vent valves V770N771 (V772N773) but the sequence of performing these operations may not preclude relief valve operation.

During Seal Purge restoration in HC.OP-SO.BB-0002 Rev. 42, sections 5.1.1 through 5.1.5, the procedure goes through a series of vent valve manipulations to vent each portion of the line until all series valves are open and the Recirc Pump Suction valve HV-4/14/2003 Page 9

B. Recirc. Pump Seal 7002986 1 F023A(B) is open. However, the procedure directs that the purge flow be set at 4 gpm and the vent valves V156(V773) be throttled. Pressure in the line is not monitored and it is possible that flow control is transferred from the V805 (V804) throttle valve to the vent valves that could result in pressurization of the line up to the relief valve setpoint. The drain/vent valves are remotely located from the flow control throttle valve so the actual flow rate may not be known during the time the vent valve is throttled. At 4 gpm, the flow orifice FO-DO 1A(B) is sized to drop 200 psi so at the normal 1400 psig supply pressure, the line could reach 1200 psig. Thus, the isolation procedure had potential mechanisms to cause excess pressurization of the seal purge supply header both during isolation and restoration.

NOTIF 20134758 was initiated to correct this issue before restoring the recirc pump and seal purge line following the seal replacement in March 2003 outage. Most of this recommendation was addressed in Revision 43 prior to fill and vent following the B pump seal replacement during March 2003 mini-outage. However, the step 5.1.3.c.8 request to throttle back on V804(V805) instead of VI 56(V773) was not incorporated.

NOTIF 20134719 requested addition of venting points V645(V652) on the pump suction sensing line. This was also incorporated in Revision 43.

There have been previous attempts to improve the venting procedure but these may have been piecemeal and without rigorous review of valve elevations and precautions for potential valve leakage or throttling that could result in high pressure at the relief valve, crud intrusion, or insufficient air purge through the seals. A rigorous documented review of the entire shutdown and venting evolutions should be performed by an Operations and Engineering team. Appropriate changes should be made to ensure that an adequate seal venting is accomplished without challenging the relief valve setpoint.

Also, the procedure should provide for adjusting the seal purge flow rate to compensate for flow diversion, seal pressure changes during startup/shutdown or increase in seal temperatures. Based on a crude model of the purge line, the flow rate should drop almost in half during seal pressurization from 0 to 1000 psig.

Case 2: Containment Isolation Valve Stroke Test (Drvwell Isolation)

Operations procedure HC.OP-IS.BF-0102 Control Rod Drive System Valves - COLD SHUTDOWN - INSERVICE TEST was reviewed. This test strokes the HV-3800A and B seal purge isolation valves. There is a precaution 3.1.3 that states " never open HV-3800A(B) with the Recirc Pump isolated". However, there is no precaution or direction to isolate the seal purge supply valves prior to closing the valve. If seal purge is in service, the relief valve PSV-F025A(B) would relieve when the HV-3800A(B) is closed. The procedure has the potential mechanisms to cause over pressurization of seal purge supply header. The procedure should be revised prior to RF1 1 to require that seal purge be isolated before closing the valve.

Case 3: Local Leak Rate Test Tagging Order The seal purge line is routinely removed from service during outages by means of a tagging order. The tagging order to perform the Local Leak Rate Test (LLR7) for valves V046(V047) and HV-3800A(B) contains a listing of components with no particular order of manipulation specified. The HV-3800A(B) control switch is the first device on the list to be tagged (closed) and the VI 00(V1 01) purge block valves are tagged later. The vent valves that could depressurize the line at the relief valve are not listed. Therefore, there is a real likelihood that the tagging sequence used in conjunction with the HC.OP-SO.BB-0002 can result in the relief valve being pressurized to its setpoint for significant time periods.

411412003 Page 10

B. Recirc. Pump Seal 70029861 The team concluded that relief valve is damaged during pump shutdown evolutions rather than during normal operations or pump speed changes based upon the following evidence:

As-found F025A(B) test and inspection results.

Three cases of operations evolution with mechanisms to cause over pressurization of purge supply header System design with no know mechanism to restrict purge inlet flow Plant historian data indicates 10 stage pressure follows reactor coolant pressure and pressure at relief valve is same as first stage pressure.

Corrective Action:

Type/Status:

Effectiveness:

Corrective Action:

Type/Status:

Effectiveness:

Corrective Action:

Type/Status:

Effectiveness:

Repaired and replaced the damaged F025A(B)valves Remedial/Complete Monitoring of the discharge piping for F025A(B) during routine operator rounds VERF 0170 Due Date: 09/01/2003 R-REP05 Revise HC.OP-SO.BB-0002 procedure prior to RF1 1 to protect against over pressurization of the purge supply header and inadvertent lifting of the F025A(B) relief valves during shutdown and isolation of the reactor recirculation pump and during fill and vent evolution.

CRCA 0080 Due Date: 4/15/2003 O-HTP05 Remedial/Complete Monitoring of the discharge piping for F025A(B) during routine operator rounds See VERF 0170 A rigorous documented review of the entire shutdown and venting evolutions should be performed by an Operations and Engineering team.

Appropriate changes should be made to ensure that an adequate seal venting is accomplished without challenging the relief valve setpoint.

The review should include:

a. Verify air is sequentially vented from high points and the seal is vented (Consult Flowserve).

b. Assume supply valves may leak and keep vents open at the relief valve during shutdown and pump isolation until another vent path is established.

c. Consider monitoring pressure at the relief valves during the procedure.

d. Provide for coordination such that the pump and piping are filled prior to opening the suction valve to prevent crud from entering the seal.

e. Determine the best flow rate for venting. 4-6 gpm may be too high.

If a flow control valve is installed, this flow rate may not be achievable.

f.

Eliminate throttling of vents that could result in pressurizing the relief valve.

g. Consider verification of flow rate with M&TE at the HV-3800 valves prior to pump start.

h. Review precautions and limitations

i.

Provide guidance about how long we should operate the pumps without purge flow./

CRCA01 10 Due Date 08/30/2003 O-HTEOP Prevent Recurrence Monitoring of the discharge piping for F025A(B) during routine operator rounds See VERF 01 4/14/2003 Page 1 1

B. Recirc. Pump Seal 70029861 Corrective Action:

Type/Status:

Effectiveness:

Corrective Action:

Type/Status:

Effectiveness:

Corrective Action:

Type/Status:

Effectiveness:

Revise HC.OP-IS.BF-0102 to add precaution or direction to isolate the seal purge supply valves prior to closing the HV-3800 to prevent over-pressurization of the purge supply header.

CRCA 0080 Due Date: 04/15/2003 O-HTEOP Prevent Recurrence/Open Monitoring of the discharge piping for F025A(B) during routine operator rounds See VERF 0170 Revise RF1 1 tagging orders 4092202 and 4092200 to perform the tagging in a specific sequence such that the supply isolation valves Vi 00(V1 01) are closed and the vent valves VI 55N1 56(V773N773) and associated caps are opened prior to closing HV3800A(B) to prevent over-pressurization of the purge supply header.

CRCA03 Due Date: 04/15/2003 O-HTEOP Prevent Recurrence/Open Monitoring of the discharge piping for F025A(B) during routine operator rounds See VERF 0170 Establish routine monitoring to ensure purge flow is not diverted through the relief valve. Initiate a Temporary Log for operators to observe the relief valve discharge piping into the radwaste drains for the next cycle for both A and B relief lines to monitor valve leakage. The "B" relief line will be cut as part of design changes 80059696 and 80059697.

CRCA 0120 Due Date: 05/15/2003 E-REP05 Prevent Recurrence/Open None. This corrective action established monitoring for previous corrective actions and long-term reliability.

411412003 Page 12

B. Recirc. Pump Seal 70029861

3. Poor design of the interface between CRD System Pressure and Seal Purge Flow requirements. lrhe existing design causes the purge line relief valve, FO25AIB, to rapidly open and close (chatter) when its setpoint is exceeded.

Basis:

The Seal Purge flow requirement is 1.5 to 2.5 gpm at approximately 1010 psig.

The CRD supply pressure is approximately 1400 PSIG. To achieve the required flow and reduction of the CRD supply pressure, a flow orifice and a flow throttling valve are used. The F025 relief valve passes approximately 10 gpm when the valve lifts. Therefore, when a downstream perturbation occurs restricting flow (i.e.

shutting the CIV) the purge line pressure increases, actuating the F025 relief valve, passing 10 gpm to the floor drain system. Since the makeup flow is restricted to 1.5 to 2.5 gpm the pressure in the line will rapidly decrease (blowdown) and reseat. The pressure will then again rapidly increase to the lift setpoint and the process repeats. This is what has been termed as "Chatter'. The valve vendor has stated that "chattering' as described above will cause degradation of the relief valve disc in a few minutes. Typically, disc degradation could be as minor as a seating surface anomaly, which would create a leak path.

The leak path could be minimal initially but would ultimately increase with time.

Unchecked seat leakage has the potential to erode the seating surface material.

This condition has been seen in F025 A and B relief valve nine and eight times respectively (going back to 1988). During the investigation, the seating surfaces of the removed F025A(B) relief valve were inspected. Both valve discs were severely eroded, confirming a long-term leakage condition. See attachment 7 for As-found testing and inspection results.

Organizational and Program Issues: Latent design issue from original design.

Corrective Action:

Type/Status:

Effectiveness:

Corrective Action:

Type/Status Effectiveness:

Corrective Action:

Type/Status:

Effectiveness:

Temporary Modifications(T-mods) No.03-014 and 03-015 installed gag on F025A and B relief valves during planned outage in March 2003 to ensure purge flow is not diverted.

Remedial/Complete None. T-mod to be removed during RF1 1.

Remove T-mods 03/014 and 03/015. Implement DCPs 80059696 and 8009697 to install an isolation valve upstream of F025A(B). Cut relief valve discharge piping and establish operator observations of relief valve discharge piping to monitor valve leakage.

CRCA 0120 Due Date: 5/15/2003 E-REP05 Fix the condition/Open Monitoring of the discharge piping for F025A(B) during routine operator rounds See VERF 0170 Process a modification request and consider the following options eliminate the F025A(B) chatter Eliminate the F025A(B) relief valve Increase the set point and adjust the blowdown Maintain existing relief valve and adjust the blowdown only Remove the T-mod and use under enhanced procedure controls (during fill and vent of seals CRCA 0130 Due Date: 6/30/2003 E-REP05 Prevent Recurrence/Open Monitoring of the discharge piping for F025A(B) during routine operator rounds See VERF 0170 4/14/2003 Page 13

B. Recirc. Pump Seal 70029861 Corrective Action:

Type/Status:

Effectiveness:

Implement modification to eliminate F025A(B) approved in operation 0130.

CRCA 0140 Due Date: 12/01/2004 E-REP05 Prevent Recurrence/Open Monitoring of the discharge piping for F025A(B) during routine operator rounds See VERF 0170 4/14/2003 Page 14

B. Recirc. Pump Seal 70029861

4.

Degradation of Flow Control Valves and flow restricting orifice causes particulate debris in purge flow.

Basis:

Inappropriate throttle valve model results in valve seating surface degradation.

The potential exists for the introduction of additional particulates, resulting from internal erosion, in the purge flow beyond control rod drive (CRD) filters. The currently installed throttle valve is a Kerotest full port, Y-Pattem globe with a standard plug design typically used for on-off service. Valves specified for throttling service typically are provided with special plug, designed for the application. The valves are currently operated near the full closed position. Root cause team members G. Delp and J. Thompson reported an audible noise near the valve during purge flow. The audible noise is typical of cavitation/erosion of the seating surfaces. Also, this condition could potentially be accelerated by the suspected degradation of the downstream flow orifice.

Corrective Action:

Type/Status:

Effectiveness:

Corrective Action:

Type/Status:

Effectiveness:

Corrective Action:

Type/Status:

Effectiveness:

Process a modification request to replace manual full port globe valve with reliable throttle valve. (V804N805). This modification should include installation of a pressure regulator, if relief valve is not removed (see CRCA 06). This modification should include the installation of the dual canister filter and upgrades to the CRD inlet filter described in the next section.

CRCA 0150 Due Date: 6/30/2003 E-REP05 Prevent Recurrence/Open None. This modification will result in improved seal life and will not be evident for several cycles, following the implementation of this corrective action.

Inspect and repair flow orifice. (NOTF 20137242 and 20137243.)

CRCA 0160 Due Date: 05/30/2003 E-REP05 Fix the Condition/Open Based on results of inspection, determine if preventive maintenance is required VERF 0180 Due Date: 08/31/2003 E-REP05 Replace 804(805) throttle valves in kind during RF1 1.

CRCA 0160 Due Date: 05/30/2003 E-REP05 Fix the Condition/Open None - See corrective action above to replace the throttle valves with new design.

4/14/2003 Page 15

B. Recirc. Pump Seal 70029861

5. The present Control Rod Drive (CRD) inlet filter size permits particulates to enter the seal purge tlow trom CRD flow that are large enough to reduce seal life due to accelerated wear to seal faces.

Basis:

The CRD System utilizes two centrifugal pumps (one in standby) to supply water to the CRD system and to the Recirc Seal purge lines. Suction is taken from the Condensate System through a duplex suction filter rated at 50 microns, into the running pump, and through a duplex discharge filter rated at 20 microns nominal, 50 micron absolute. The seal purge lines are supplied after the discharge filter through individual throttle valves. SOER 83-4 recommended that CRD suction filters be upgraded to 5 micron to improve the quality of water supplied to the seal purge. A questionnaire was sent to peer CRD engineers regarding CRD filtration. The results are tabulated in the OEF attachment. The majority of plants responding had better filter ratings than Hope Creek. River Bend and LaSalle plants had added separate seal purge line filters rated at 1 micron.

This investigation identifies poor CRD water quality as a potential contributing cause of the seal degradation. The data compiled indicates that some seal degradation could be caused by CRD System particulate and that improvement is warranted to maximize seal life.

Long-term improvement in CRD system reliability can also be expected with improved filtration. Small particles can be lodged in directional control valves and ball check valves causing leak by and rod motion problems. Additionally, early plant work history indicates that the original flow control valves used to control seal purge were failing possibly due to particle contamination. These needle valves flow control valves were replaced with the full port globe valves currently installed.

An improvement in CRD system filtration should be evaluated and implemented if cost beneficial. The Salem RCP seal injection design should be used in the design change evaluation. The filter sizing should be coordinated to support the 1 micron purge line filter recommended by the vendor.

Organization and Programmatic Issues: SOER 83-4 recommended that CRD suction filters be upgraded to 5 micron to improve the quality of water supplied to the seal purge. In 1996, a root cause team reviewed recirc. pump seal degraded performance but did not review the SOER 83-4. Reasons that the recommendations were not implemented previously during earlier reviews of the seal performance can't be determined.

Corrective Action:

Type/Status:

Effectiveness:

Corrective Action:

Type/Status:

Modify system to install one micron inline dual canister filter in purge line upstream of throttle valve. See CRCA 0150.

Prevent Recurrence/Open The determination this corrective action is effective will be increased seal life. This will not be evident for several cycles following the implementation of the corrective actions. Reducing particulate size in seal purge flow is a recognized strategy for improving reliability, therefore tracking of effectiveness is recommended.

Reduce rating of CRD filter media. This change should be coordinated with filter size called for in the purge line dual canister filter to step down particle size such that the purge line filter can achieve one micron absolute.

See CRCA 0150 Prevent Recurrence/Open 4/14/2003 Page 16

B. Recirc. Pump Seal 70029861 Effectiveness:

The determination that this corrective action is effective will be increased seal life. This will not be evident for several cycles following the implementation of the corrective actions. Reducing particulate size in seal purge flow is a recognized strategy for improving reliability, therefore tracking of effectiveness is not recommended.

4/14/2003 Page 17

B. Recirc. Pump Seal 70029861 Contributing Cause(s)

6.

Lack of flow indication downstream of F025 relief valve prevented operators from having an accurate knowledge of actual seal purge flow.

Basis:

Flow indication is currently located upstream of the F025 relief valve. This arrangement provided no indication that flow was diverted from the pump seals through the relief valve. Early indication of the insufficient purge flow would have provided the opportunity to take corrective action before the seal faces were damaged.

Corrective Action:

Type/Status:

Effectiveness:

If relief valve is removed, current location of flow indication is adequate (see CRCA 0130). OR If relief valve remains in service, Operations should perform permanent round Inspection of F025A(B)relief valve discharge piping. (See CRCA 0120)

NA None required.

7.

Organization did not display high sensitivity to purge loss of purge flow.

Basis:

It was perceived that it was acceptable to operate the pump with insufficient purge flow. This may have originated in the GE SIL 511 that recommended reduced seal purge flow. The SIL also describes two GEJBWR3 plants that operated the pump safety without purge for over 20 years. This SIL indicated that those plants had cleaner than average water. The vendor indicates that while operation without purge flow is within the original design configuration of the pump and the mechanical seal, the presently install seal faces are designed for optimum performance with seal flow. (See Attachment 11).

Organization did not respond to indications of insufficient purge flow identified in June 2002, just two weeks before a mid-cycle outage. Opportunity to take corrective action was missed Corrective Action:

Type/Status:

Effectiveness:

Revise HC.OP-SO.BB-0002 to provide additional guidance about how long we should operate the pumps without purge flow (See CRCA 0110).

Prevent Recurrence/Open None Required 4/14/2003 Page 18

B. Recirc. Pump Seal 70029861 Other Issues

1. Leak-off Line Issues Each reactor recirculation pump has two leak-off lines, one staging flow line and one uncontrolled leak off line.

The staging flow is controlled leak off taken from between the 1"t and 2nd stage seals. The flow is directed to the Drywell Equipment Drain Sump. Staging flow under normal operating conditions is 0.75 gpm. There is a flowmeter FSH N007A(B) in the line with high and low flow alarms at 1.4 gpm and 0.36 gpm, respectively.

The uncontrolled leak off line acts as a drain for any water that makes it's way across the 2nd stage seal faces. This uncontrolled leak off is directed to the Drywell Equipment Drain Sump.

Under normal operating conditions this flow should be 0 gpm. There is flowmeter FSH N002A(B) in the line with a high flow alarm at 0.1 gpm. If the leakage does not got down the leak off line, it bubbles out the top of the seal canister and goes to the Drywell Floor Drain, which has a lower limit for unidentified leakage than the Drywell Equipment Drain.

During Cycle 11, before the planned outage of March 2003, the 'B' Recirculation pump showed one alarm. This alarm indicated high uncontrolled leak off, and took place in May 2002. The alarm was reset shortly afterwards. Neither the staging flow or uncontrolled leakage flow alarms tripped during the months and weeks leading up to the planned outage.

Inspection of the seal faces upon removal showed them to be severely degraded, a condition that should have caused the uncontrolled leak off line to alarm on high flow. Upon inspection during the March 2003 outage, 2-3 gpm was observed leaking from the overflow port above the uncontrolled leakage line. The high flow alarm (0.1 gpm) was not in alarm.

The flow devices in the leak off line are Rotameter flow switches. Engineering walkdown during the March 2003 planned outage verified that the elevation of the rotameters for both

'A' and 'B' pump is lower than the elevation of the mechanical seal. Engineering should evaluate the reason for failure of flow alarms. Iterns to consider include: uncontrolled leak off lines are clogged, the flow is not able to unseat the disk of the Rotameter or slope is incorrect.

If the leakage is not directed to the Drywell Equipment Drain, it is goes to the Drywell Floor drain and is considered unidentified leakage. Unidentified leakage has a lower technical specification allowable limit. Drywell Equipment Drain has a higher technical specification allowable limit. The ability to identify and quantify the Recirc pump seal leakage could prevent or defer the need to take the plant off line.

NOTF 20139444 was initiated to perform a functional test during RF1 I on the flow alarms.

NOTF 20139446 was initiated to perform a test during RF11 on Recirc pump seal uncontrolled seal leak off line to determine why leakage does not flow through this line.

NOTF 20139711 was initiated to perform an evaluation to replace the existing design flow switch and install a flow meter.

2.

Multiple Souces of Unidentifed Leakage:

In February 2003, Operations implemented the procedure for Drywell Leakage Source Identification HC.OP-GP.ZZ-0005. No additional leakage sources were identified beyond the suspected 'B' Reactor Recirc Pump mechanical seal leak. Drywell walkdown after shutdown revealed two previously unidentified leaks: approximately 1.0 gpm from vent valve Hi GB GB-V244 associated with the chilled water system, and approximately 0.5 to 1.0 gpm from RACS valve ED-V034. Notification 20134744 was written to address the reasons these leaks were not identified during the implementation of HC.OP-GP.ZZ-0005.

4/14/2003 Page 19

B. Recirc. Pump Seal 70029861 ATTACHMENTS

1. Background, Event Chronology and Methodology

2.

References

3.

Operating Experience Listing

4.

Technical Issues Fact Sheet

5. Technical Issues Problem Analysis

6.

Technical Issues Cause Evaluation

7. As-found Testing of F025A&B Purge Line Relief Valves

8. Fault Tree Analysis

9. Change Analysis

10. Flowserve Inspection Report dated 3/1212003; B Reactor Recirc Pump and N-7500 Mechanical Seal

11. Flowserve letter dated 3/1212003, Purge Flow to A Reactor Recirculation Pump

12. B Recirculation Pump Failure Mode Matrix 4/14/2003 Page 20

B. Recirc. Pump Seal 70029861 Attachment I Background, Event Chronology and Methodology Background and Event Chronology Each recirculation pump has two full-pressure redundant mechanical seals to prevent pump shaft leakage. Either seal is capable of withstanding full reactor pressure. The seal cartridge assembly is an integral pump part and is replaced without removing the pump motor. Clean purge water is supplied to the seals from the Control Rod Drive Hydraulic System. A staging flow of 0.75 gpm is intentionally routed from the 2nd stage seal cavity to the Drywell Equipment Drain Sump. This line has a high flow alarm at 1.4 gpm and a low flow alarm at 0.36 gpm. A leakoff line is supplied to drain any undesired leakage from beyond the 2nd stage seal. This line has a high flow alarm at 0.1 gpm.

Failures of the seals can be determined with the pump in service by monitoring seal cavity pressures, the staging flow alarm, and the leakoff flow alarm as follows:

Failure of the number I seal: Both seal cavities will approach the same pressure, and the staging flow switch will trip due to high flow.

Failure of the number 2 seal: Pressure in the 2nd stage seal cavity will decrease and the undesired leakoff line flow switch will trip on high flow.

Failure of both seals: High flow alarm on both flow switches. For catastrophic seal failure, the leakage would approach about 50-60 gpm as limited by the breakdown bushing.

The N-6000 mechanical seal canisters installed in A and B Recirculation Pumps were changed to the model N-7500 in 1996 and 1997 respectively. Since that time the B pump has required replacement three times. The A pump seal has been was replace in 1999 and that seal is still in service with acceptable cavity pressure and temperature readings. The B Pump mechanical seals fail before expected service life. The B Pump seal was replaced in each of the last two refueling outages, RF09 and RF10, and most recently in March 2003 after less than an entire operating cycle in service.

Coming out of RF10 in November 2001 the seal was functioning properly with the 2nd stage seal cavity at 500 psi. Following the shutdown for the Safety Relief Valve leak in December 2001 an immediate decrease in 2nd stage seal pressure was identified. This pressure continued to degrade until it stabilized at 380 psig in November 2002. Beginning in December 2001 the 2nd stage pressure began to increase until i reached approx. 420 psi on March 7, 2003.

Between February 13, 2003 and March 7 2003, both seal cavities temperatures steadily raised 40 degrees F, indicating that both seal stages were had degraded. Drywell Floor Drain Flow rose from 1.2 gpm on February 13, 2003 to 4.1 gpm just before planned shutdown on March 7, 2003.

Inspection of the seal faces after removal confirmed that both seals had sustained significant damage. It is noteworthy that although the cavity pressures did indicate seal degradation, the staging flow did not alarm on high or low flow (FSH-007B) and the leakoff flow did not alarm on low flow (FSH-007A).

The A Recirculation Pump seal was last replaced in 1999, has 4.5 years of service and is expect to last until its scheduled replacement in RF12 in fall 2004. The expected service life for the Recirculation pump mechanical seals is six years.

Methodology The Root Cause Team used the Technical Issues Process to begin the fact finding, ground the 4/14/2003 Page 21

B. Recirc. Pump Seal 70029861 team and begin understand potential causes of the failed seal. The team used Fault tree analysis to identify the causes and change analysis to understand the differences between the A and B Recirculating Pump seal performances.

.1 4/14/2003 Page 22

B. Recirc. Pump Seal 70029861 References REFERENCES Root Cause Team Gene Nagy - Team Leader John Morrison - Root Cause Qualified Denise Boyle John Thompson Gene Delp Craig Johnson Glenn Gardner Persons contacted Mark Bergman, Reactor Recirculation System Engineer Jim Hinkle, HC Operations Jim Kepley, HC Operations John Jackamonis, Mechanical Maintenance Technician Joe Flanagan, Reliability Programs Vibration Engineer Frank Todd, Thermal Performance Engineer Tom Roberts, Reliability Programs Engineering Supervisor Richard Vaughn, Operations Procedure Writer Harold Koehler, PPL Susquehanna Marty Santic, Excelon Quad Cities Art Olsen, Sr. Seal Service Engineer Flowserve Corporation Gerry Lenzen, Section Head, Primary Pump, Flowserve Corporation Dave Zagras, Sr Seal Designer, Flowserve Corporation Allan Rollo, Crosby Valve Representative Documents SH.SE-DG.ZZ-0003, Technical Issues Process HC.OP-IS.BF-0102 Control Rod Drive System Valves - COLD SHUTDOWN - INSERVICE TEST HCOP-SO.BB-0002(Q), Reactor Recirculation System Operation. Revisions 27, 28, 29, 35, 38 and current revision 43 HC.OP-GP.ZZ-0005, Drywell Leakage Source Identification HC.OP-AB.CONT-0006, Drywell Leakage GE Licensing Topical Report, NEDO-24083, Recirculation Pump Shaft Seal Leakage Analysis P&ID M-43-1 Recirc System 4/14/2003 Page 23

B. Recirc. Pump Seal 70029861 P&ID M-46-1 CRD System DITS 3.32 CRD System DITS 3.30 Recirc System Calculation BF-7 Seal Purge Lines pressure drop Vendor Manual PNI-B31-CO01-0124 Recirc Pump Vendor Manual 322556 Reciro Pump Seal Cartridge PN1-C11-DO1 1-0100 Restricting Orifice PN1-C11-1020-0009 CRD Process Flow Diagram HC.OP-SO.BB-0002(Q) Recirc System Operation PN1-C1 1-D003-0106 CRD Discharge Filter Assy.

PN1-C1 1-D003-0111 CRD Discharge Filter Rating PN1 -C11 -F025-0140 Relief valve P-0501 Line Index PN1-B31-CO01-0137 Recirc Pump stress report PM122-002 CRD Pump Suction Filter Data Sheet GE SIL 511, Reduced Seal Purge Flow for Bryon Jackson Recirc Pumps, dated 04/30/1990 GE SIL 203, Extending Recirculation Pump Seal Life, dated 10/29/1976 SE SIL 459, Byron Jackson Recirculation Pump Shaft & Cover Cracking, 12/15/1987.

4/14/2003 Page 24

B Recirc Pump Seal Reliability 70029861 Equipment History and OEF Listing Equipment history was searched in SAP and DCRMS for MMIS records. Operating Experience was collected from INPO See-In searches and e-mail inquiries to BWROG CRD Committee members. The results are summarized below by category.

PSV-025A(B) Work History Document Summary WO 880429080 PSV-F025A leaking by; valve replaced WO 910503133 PSV-F025A leaking by; valve replaced.

WO 920113165 PSV-F025A leaking by; valve replaced WO 920312196 PSV-F025A removed and tested for repeatable setpoint at 1300 psi; found repeatable within 20 psig; reset to 1250 psig WO 950404142 PSV-F025A leaking by; valve replaced WO 950426248 PSV-F025A set too high 1330#; reset to 1275#; installed WO 950712209 PSV-F025A leaking by; set press found 11 80#; seal purge secured incorrectly.

WO 951130308 PSV-F025A lifted due to seal purge laced in service with HV-3800A closed; valve replaced WO 960617050 PSV-F025A leaking by 0.5 gpm; valve replaced; pressure readings CR 960617050 obtained (995 psig at V771); Engineering action plan started; CR concluded that pressure control valve was the problem; no follow up.

WO 990516126 PSV-F025A leaking by; valve replaced; CR concluded PCV should CR990516126 be replaced; CRCA closed with no action.

CR 70001487 PSV-F025A leaking by; replaced by WO 60003459 ; CRCA 09/27/1999 80005141 initiated to replace V804/805, closed to NUTS 80007363 which was closed with no action.

WO 890112083 PSV-F025B leaking by; valve replaced WO 890314105 PSV-F025B leaking by; valve replaced WO 920308117 PSV-F025B lifted due to seal purge laced in service with HV-3800B closed; valve replaced WO 951130308 PSV-FO25B lifted due to seal purge laced in service with HV-3800B closed; valve replaced WO 960411189 PSV-F025B leaking by; install pressure gage at V773 - read 1 000#;

valve replaced; Engineering Action Plan started.

CM990520075 PSV-F025B leaking by; valve replaced 5/4/00 60008520 CM990520075/

PSV-F025A lifting; valve replaced 5/4/00; CR attributed failure to 60003459 unspecified CRD system pressure perturbations; no follow up.

CR990520075 CR 960330084 Level 1 on B Seal. Procedure changes made to SO.BB-0002 for seal vent/purge and operation below 300 psig.

B Recirc Pump Seal Reliability 70029861 Flow Control Valves V804/805 (A/B purge line throttle valve) Work History Document Summary WO 871107039 FCV-DO12A not controlling flow; repaired valve WO 880902129 FCV-DO12A not controlling flow; repaired valve WO 891111105 FCV-DO12A not controlling flow due to internal blockage; repaired valve 4EC-3062 DCP to replace FCV-DO12A with V804 throttle valve 9/90 901024104 60004042 V804 degraded, not controlling flow; 10/12/2001 replaced valve 70029958 V804 installed backwards WO 871107035 FCV-DO12B not controlling flow; repaired valve WO 880409093 FCV-DO12B not controlling flow; repaired valve 4HM-0326 DCP to replace FCV-DO12B with V805 throttle valve 9/88 60033512 V805 leaks by seat (Open) 11/16/2002 Recirc Pump Seals Work History B Recirc Pmp Seal 60027904 Replace B Seal 3/11/03 - Shutdown one month prior to RF1 1 60021483 Replace Seal 10/30/01 (RF10) 60004748 Replace Seal 5/11/00 (RF09) 990318011 Replace B Pump Mechanical Seal 11/07/96 960826064 Rpl B Seal due to abnormal operating parameters - close out to 990318011 970524003 Replace B. Recirc Pump Mechanical Seal Dec 95-Jan 96 (RF06)

A Recirc Pmp Seal 60009576 Replace Seal CM990415247 Replace Seal - #2 seal pressure trending down CR990410137 CR00960413149 Reduction in A pump seal cavity pressure WO 961125352 IA-P-201 increas in #2 Leakage - Replace 4EO-3635 11/20/97 OEF Document Summary

B Recirc Pump Seal Reliability 70029861 B Recirc Pump Seal Reliability 70029861 E-Mail on FCVs All plants responding use a Flow Control Valve; Hope Creek is outlier using throttle valve. Susquehanna using Circle Seal RV99-291.

Pilgrim has bad performance with Kates 701 valves, will replace with Circle Seal P6-781. Columbia using ITT Conoflow valves and work well. NMP2 has good performance from Circle Seal P6-781.

E-Mail on PCVs Several plants responded to inquiries. Almost all have had similar relief valve problems. Susquehanna believes relief valve is prone to leak at operating pressures 20% below setpoint, looking to increase setpoint; River Bend believes relief valve is prone to leak at operating pressures; looking to eliminate relief valves; instrumented purge line and saw no perturbations in 3 months; moved flow orifice after FCV and moved relief upstream of Fl. NMP2 has had better performance of relief valves after changing valve-in/out sequences; isolation valves can leak so need to vent also to prevent overpressure.

INPO O&MR Discusses seal degradation; Recommends: 1. having detailed 432 procedures for fill and vent of Recirc pumps, including removal of gases from high points in seals. 2. Seal purge configuration should allow for proper venting; establish monitoring program for seal temperatures, pressures and drywell leakage. 3. prove training on seal leakage detection and actions.

INPO SOER 83-Discusses seal failures at several plants. Noted steps taken by 4

some plants: Maintenance practices to ensure proper installation; replacing 25 micron CRD Suction Filters with 5 micron; initiate seal water injection in coordination with opening pump isolation valves to prevent crud intrusion and over-pressurization of the pump and piping; prevent interruptions to cooling water and purge by system operations; replace bleed-off switches with transmitters or high/low signals going to separate annunciators. Recommendations: Thirteen recommendations were provided regarding maintenance practices, design improvements parts handling and training.

Recommendations warranting further review are: 1. Replace CRD suction filters with 5 micron to improve water quality to seal purge.

INPO OE8087 Clinton - seal failure exceeded T/S DW leakage limits. Plant was not shut down in timely manner after seal leakage became excessive. Seal purge was also cut off prematurely causing heat damage.

INPO OE4676 Grand Gulf - plant shutdown due to seal leakage.

INPO OE6068 Browns Ferry - seal failed. Plant changed procedures to start seal purge before opening pump isolation valves.

INPO SER 12-LaSalle and Dresden Seal Failures - seals failed due to inadequate 83 cooling and heat stress applied by sudden cooling; procedures were changed to permit adjustment of seal purge flow as required if high temperatures are detected; seal temperatures may exceed desired limits if the pump is stopped without adequate purge flow.

GE SIL 203 Recommends seal cavity venting to avoid trapping air at low operating pressures; seal purge should be in service at all times except when the pumps are isolated; seal temperatures should not be allowed to exceed 200 F; pumps should not be operated for extended periods below 300 psig; avoid pump starts at low pressure.

B Recirc Pump Seal Reliability 70029861 GE SIL 256 Provides recommendations for maintenance of the seals.

GE SIL 258 Provides recommendations for maintenance of the seals.

GE SIL 511 Recommends reduction in seal flow from 4 gpm to 2 gpm range and plugging the cover drain hole to minimize thermal impacts on the pump shaft. Records indicate Hope Creek implemented recommendations.

GE SIL 517 SIL addresses single loop operation. Recommends maintaining seal purge flow in idle loop to prevent seal overheating damage.

FLOWSERVE This bulletin describes seal degradation due to air entrapment and TECHNICAL recommends that a rigorous venting procedure be developed the BULLETIN 002-vendor assistance.80-026

B Recirc Pump Seal Reliability 70029861 CRD Filter Survey Discharge Filter Size Suction Filter Seal Purge PLANT (microns)

Size (microns)

Filters Comments Hope Creek 20 nominal; 50 absolute 50 nominal None Original Design Peachbottom 20 25 None NMP 50 absolute 25 absolute None Upgraded - see some 5 nominal, 25 1 micron improvement is seal River Bend 2 nominal, 15 absolute absolute installed in 2001 life Cooper 20 nominal; 50 absolute 50 nominal

?

25 nominal, 50 Fitzpatrick 20 nominal; 50 absolute absolute None Upgraded Suction Columbia 5 nom, 25 abs 10 None filters KKL 5

5 None 10 micron, 98% 1 micron dual Added seal purge LaSalle 20 nominal; 50 absolute efficient cannister filters last outage Monticello 20 nominal; 50 absolute 5

none BNP 1&2 25 absolute 10 none 6-8 year seal life

B Recirc Pump Seal Reliability 70029861 Technical Issue Fact Sheet Title of Issue: Hope Creek 'B' Recirculating Pump Seal Reliability Issue Number: NUCR 70029861 Responsible Engineer: Gene Nagy Problem Statement:

1) 'B' Recirc Pump seal fails prior to expected life.

Goal(s):

Determine cause of most recent seal failure on B Recirc Pump which resulted in maintenance outage to replace seal in March 2003.

Determine if cause of historical poor performance of B Recirc Pump seal.

Design Information and

References:

See Attachment 2 Licensing Basis Information and

References:

See Attachment 2 Facts and SourceslReference Documents:

The following facts were obtained from a post event debrief with operations, maintenance and engineering:

1. 3/7 Hope Creek shutdown a/c:

Seal Leakage High Temp Seal Cavity Low Interstage Pressure

2. Floor Drain leakage increasing for - 1.5 months

3. Relief Valves require 80% reduction to reseat Lift set point = 1250 psig, reseat at 1000 psig

4. NOP CRD operating pressure 1455 psig

5. Relief Valve set pressure 1250 +/- 25 psig

+/- 50 psig (ICD card)

6. B seal replaced RF10, and RF09

7. Indications (rubs) of bowed shaft "B" Recirc Pump, also wear pattern on rotating seal face larger diameter than nose of stationary seal.

8. 2nd Stage Leakage - no flow to equipment drain via leak off line, bubbles out top of seal canister and goes to the floor drain

9.

Existing 804 / 805 valves = Kerotest Y Pattern globe valve

10. Seal Contamination

a. RF10(b-Seal) =10-20MR

b. (seal not failed) 1 or 2K after wipe down

c.

Mini outage

- 350 - 600 MR (During this cycle there is fuel failure)

d. 3/17/03

- 500 MRAD Contamination

e. Old Style

-100 to 125 MR

11. (6000 Seals)

(not establishing purge flow)

12. After procedure change to establish purge flow rad levels much lower

13. RACS to Thermal Barrier was found leaking (ED034 (Plug Valve))

14. Set screws backed out on plug when maintenance went into dry well to remove 'B' Seal corrected during power assension M:kShared\\B Recirc Pump Root CauseXATT 4 - B Seal Form 1 FACT SHEET.doc 4115103

B Recirc Pump Seal Reliability 70029861 Technical Issue Fact Sheet

15. TREND data indicates that changes in CRD temperature caused proportional changes in seal cavity temperatures. There appears to have been some purge how to seal.

16. A Recirc Pump #2 Seal Leak off line was re-slopped by maintenance. 'B' Recirc #2 Leak off slope not known.

17. #2 Seal Pressure decreased during first part of cycle. In October 2002 it started to increase.

18. During Reactor Power Changes and Associated

19. RCS pressure changes, 'B' Recirc Seal before 11/02 both Seal Stages see same pressure drop. (A-Pump the

1. 2 Seal pressure drop is %2 that of H 1 Heal)

20. After 11/02 B - Cavity #2 does not change pressure in response to RCS pressure changes.

21. 6/02 B-Relief noted as leaking (Steve Myers) 20102989

22. B-Recirc Shaft has BOW

23. B-Recirc Pump Auxiliary Impeller showed signs of rub.

24. When changed from 6000 to 7500 style seal, both showed sign of leakage at the end of the first cycle. Both changed out (1999 ) (refueling RF08 Outage)

25. RF09 alignment of B-Recirc Pump to MTT and replaced seal.

26. RF10 Seal replaced (B pump)

27. Between RF09 and RF10 balance shot made to B Pump.

28. RF10 Vib Probs moved

29. RF10 coupling Machined

30. Relief Valves

a. 5/04/00 - B replaced, A replaced

b. 6/15/02 - B reported leaking, Replaced 3/7/03

c.

4128/02 - A lifted due to isolation error

d. 3/11/03 -A leaking and replaced

31. CRD pump suction and disc Disch = 50 r Abs, filters = Suction = 50 r Microns

32. On 7500 Seals, 2nd stage only failed

33. B Pump BF-V804 installed backward (10/12/01 - installed)

34. Throttle Valves excessive noise suggesting (cavitation -+ may be source of particulate)

35. Seal Purge Flow changed from 4 gpm to 2 gpm in 1992 (GE SIL 511 response). No DCP or evaluation of system / components performed.

Assumptions:

Operating ExperiencelHistory and Sources:

See Attachment 3 Cause(s): See Attachment 6 - Cause Evaluation Sheet - Seal Fault Tree Analysis - Change Analysis 3-19-03 M:\\Shared\\B Recirc Pump Root Cause\\ATT 4 - B Seat Form 1 FACT SHEET.doc 4115/03 SH.SE-DG.ZZ-0003(Z)

PROBLEM ANALYSIS SHEET WhatIs Dstintive DoesDistinction Dimension Performance Deviation Similar Component What Is Distinctive

.Suggest A

About...

I Change?

IDENTITY IS:

COULD BE but IS NOT:

B pump shaft is bowed Yes (bowed shaft)

WHAT is the component (system, etc.)

8 Recirculation Pump Seal A. Recirc Pump Seal when compared with A pump shaft.

with the malfunction?

when compared with Yes (foreign material)

WHAT is the malfunction?

IS:

COULD BE but IS NOT:

1. 2nd staoe #2 fails before 1" Degraded 1" and 2ni stage seal faces Shaft o-rings

2. First time failure of 1 S stage sea

3. Visible silt/grit and high dose rate?

LOCATION IS:

COULD BE but IS NOT at:

Purge line routing for A Pump is shorter Maybe WHERE is the malfunction observed Hope Creek B Recirc. Pump A Recirc. Pump when compared with purge line for B (geographically)?

Pump?

WHERE on the component is the IS:

COULD BE but IS NOT from:

1" Quadrant damage and 1" and 2"d malfunction observed?

1 and snd stage seal faces Normal Wear stace failure, and accelerated ware on Yes (foreign material)

One quadrant more that the rest Loss of lubricating film at seal face balance of seal surface when compared Fixed surface was worst.

with previous seal inspections?

1i' and 2"d stage damage was similar, magnitude of damage slightly different.

TIMING IS first observed:

COULD BE but IS NOT observed:

First cycle for 7500 and short cycle No WHEN was the malfunction first In 7500 style seal in 1999, both A and NA when compared with 6000 series.?

observed?

B seals WHEN has it been observed since?

IS observed:

COULD BE but IS NOT observed:

Bowed shaft on Bwhen compared with No 2000 - B For A in 2000 and 2001 A pump shaft 2001 - B (changed for insurance, For B in 2001 seals faces SAT but other parts wom)

WHEN in the operating cycle of the IS first observed:

COULD BE but IS NOT first 2 0d stage pressure decreased after component /system is the malfunction B. Recirc latest failure - 4 to 6 weeks observed:

downpower when compared with no Yes (foreign material) first observed?

after refueling, following down power downoower or dynamic chances?

(12/01).

MAGNITUDE IS:

COULD BE but IS NOT:

Seal face scoring when compared with Yes WHAT is the EXTENT of the Unidentifed leakage > 3.5 gpm Unidentified leakage from other other seal failures?

malfunction?

Cavity seal temp increase sources Stage 2 pressure decreases Seal face degradation Bowed shaft on Bwhen compared with HOW MANY components are affected?

IS:

COULD BE but IS NOT:

A pump shaft?

Yes (bowed shaft)

B Reclrc Pump A Recirc Pump IS:

HOW MUCH of any one component is 1' and 24 stage seal faces COULD BE but IS NOT:

Both stages failed and more seal face Yes (foreign material) affected?

Shaft o-rings cutting when compared with previous failures?

PERSONNEL IS:

COULD BE but IS NOT:

Venting procedure when compared Yes (relief valve)

WHO is involved with the malfunction?

Operations (fills and vents system)

Engineering with relief valve setpoint?

Maintenance (replaces the seals)

Chemistry Rad Pro vl:\\Shared\\B Recirc Pump Root Cause\\ATT 5 - PROBLEM ANALYSIS SHEET.doc 4/15/03

B Recirc Pump Seal Reliability 70029861 CAUSE EVALUATION SHEET Existing data that supports this as the Data required to confirm this cause POSSIBLE CAUSE cause.

Conditions necessary to Existing data that tends to disprove this as D

collect required data Degrade Seal Faces the cause.

Datajrequired to disprove this cause Wear on lock bolts, lock bolt bushings, stationary Rotating face contact pattern (Maint 3(24)

None Bowed pump shaft face holder and shaft sleeve.

Third Party review of data Refueling Outage. This Wear pattern on rotating face is wider by 1/64 (15 would required mils) than stationary face removing the seal Fretting on pump shaft and sleeve package.

Indication of rub on auxiliary impeller on casing Damage isolated to single quadrant caused wear on stationary face springs and holder from stationary face following the rotating face which Is not perpendicular indicates bow of above seal package Quad cities OE Visible foreign material on seal surfaces Vibration data is inconclusive Difference between A and B seal performance Data on length and disassemble NA Shaft or yoke length on B pump Difference between A and B disassembly Vendor states this would not cause accelerated seal wear.

Nuclear Common Page I of 9 4115/2003

B Recirc Pump Seal Reliability 70029861 CAUSE EVALUATION SHEET Existing data that supports this as the Data re.uired to confirm this cause POSSIBLE CAUSE cause.

Conditions necessary to Existing data that tends to disprove this as collect required data Degrade Seal Faces the cause.

Data required to disprove this cause Cuts on Seal Surfaces None NA Foreign Material for Reactor High dose rate on seal from foreign material Coolant System due to Visible material on seal surface Insufficient Purge flow.

Spikes in seal cavity temperature when 8 pump I shut-off None None NA Insufficient purge flow due to Relief valve is damaged by a few minutes of Leak Test quarantined F025A/B @ 1000 psig on J. Barton (relief valve tech) and relief valve open or damaged chatter. F025 sized to at least 10 gpm, sees only test stand. (C. Johnson 3/26) quarantined valves.

during fill and vent 2-4 gpm during fill and vent.

Open and inspect quarantined valves. (C. Johnson Steps in Operations procedure for fill and vent 3/26) 805 valve used as Isolation valve In past 805 valve Is degraded None None NA Nuclear Common Page 2 of 9 411512003

B Recirc Pump Seal Reliability 70029861 CAUSE EVALUATION SHEET Existing data that supports this as the Data required to confirm this cause POSSIBLE CAUSE cause.

Conditions necessary to Existing data that tends to disprove this as Data required to disprove this cause collect required data Degrade Seal Faces the cause.

.Blowdown or reseat pressure matches system Leak Test quarantined F025A/B @ 1000 psig on J. Barton (relief valve tech) and Insufficient purge flow due to operating pressure test stand. (C. Johnson 3/26) quarantined valves relief valve setpoint too low Setpoint matches design pressure for Recirc pump NA NA inlet piping.

GE SIL No. 511 - Recommended reducing seal NA NA purge flow from 3-5 gpm.

Insufficient purge flow because design flow in purge line is too low None NA NA Nulea Co mo Pag 3 f41 20 Nuclear Common Page 3 of 9 411512003

B Recirc Pump Seal Reliability 70029861 CAUSE EVALUATION SHEET Existing data that supports this as the Data required to confirm this cause POSSIBLE CAUSE cause.

J Conditions necessary to Existing data that tends to disprove this as Data required to disprove this cause collect required data Degrade Seal Faces the cause.

Insufficient purge flow due to bad None None NA flow indicator Flow indicator is a simple design, not subject to Check calibration history None degradation due to flow Visual - plug was moving None None NA Insufficient purge flow due to obstruction in purge line.

Maintenance tested purge line flow while seal was None None out during mini-outage 3103.

Degraded V804 and V805 valves Inspect installed valves (G. Gardner and C.

Next outage RF1 1 Johnson to inspect valves. Add note to work Foreign material from CRD flow V805 Full closed = 2.5 gpm orders)

V804 installed backwards Particulates from CRD = 50 microns absolute A pump seal is not affected None NA Seal package is hot (high dose)

Nuclear Common Page4 of 9 411 512003

B Recirc Pump Seal Reliability 70029861 CAUSE EVALUATION SHEET Existing data that supports this as the Data reguired to confirm this cause POSSIBLE CAUSE cause.

{

Conditions necessary to Existing data that tends to disprove this as collect required data Degrade Seal Faces the cause.

Datajreuired to disprove this cause SOER More OE and Vendor recommendations NA Foreign material in CRD flow due Filter size = 50 microns absolute, 20 microns OE from CRD owners group to CDR filter media rating.

nominal A recirc pump seal not affected.

Foreign material in CRDlpurge Manual globe valve use d as flow control valve Inspect installed valves (G. Gardner and C.

Next outage RF1 1 flow due to design of V804 and Orgnldsg a edetp lwcnrlJohnson to inspect valves. Add note to work V805. - Use as flow control Original design was needle type flow control orders) degrades valves and causes Operates at <5% off seat particles to go to seal unfiltered.

V804 is installed backwards V804 failure history None None NA Nuclear Common Page5 of 9 4/15/2003

B Recirc Pump Seal Reliability 70029861 CAUSE EVALUATION SHEET Existing data that supports this as the Data required to confirm this cause POSSIBLE CAUSE cause.

Conditions necessary to Existing data that tends to disprove this as Data required to disprove this cause collect required data Degrade Seal Faces the cause.

Degraded flow orifice increases pressure on V804 Inspect flow orifice Initiate NOTF for RF1 I NA Foreign material in CRD/purge Validate PM on Flow orifice. Contact PM group (G.

flow due to lack of PM on Flow Gardner) orifice None None NA None None NA Foreign material in CRD/purge flow due to less than adequate PM on V804 and V805 Fails before PM (18 months)

Validate work order history and PM on V804 and NA V805 valves Decline in performance following downpower None None Seal hangs up when RCS GE SIL No. 203, recommends not operating below pressure drops causing gap 300 psig between seal faces None None NA Nuclear Common Page 6 of 9 4/15/2003

B Recirc Pump Seal Reliability 70029861 CAUSE EVALUATION SHEET Existing data that supports this as the Data required to confirm this cause POSSIBLE CAUSE cause.

Conditions necessary to Existing data that tends to disprove this as collect required data Degrade Seal Faces the cause.

Data reguired to disprove this cause None None NA B Recirc pump seal was rebuilt incorrectly in 2001 Experienced, qualified technicians Test package from 2001 NA Long history of B seal problems and not on A seal, Copy of vendor report same technicians work on both Pre-installation test in (2001)

Vendor observation on disassembly None None NA Faulty seal face material Long history of problems on B seal and not on A None NA seal.

Seal installed incorrectly in 2001 Long history of B seal problems and not on A seal, None None same technicians work on both Trained and Qualified technicians Vendor report - no assembly or installation deficiencies.

None None NA Nuclear Common Page 7 of 9 411512003

B Recirc Pump Seal Reliability 70029861 CAUSE EVALUATION SHEET Existing data that supports this as the Data required to confirm this cause POSSIBLE CAUSE cause.

Conditions necessary to Existing data that tends to disprove this as Data required to disprove this cause collect required data Degrade Seal Faces the cause.

No indication that flow was diverted out the None NA No flow indication downstream of damaged relief valve F025 relief valve None None NA None None NA Seal cavity temperature increases on downpower or shutdown Vendor stated operating 10 hours1.157407e-4 days <br />0.00278 hours <br />1.653439e-5 weeks <br />3.805e-6 months <br /> at 200 degrees several times over cycle would not couse seal degradation.

This is a symptom of less than adequate purge flow.

-1*. None NA Nuclear Common Page 8 of 9 411512003

B Recirc Pump Seal Reliability 70029861 CAUSE EVALUATION SHEET Existing data that supports this as the Data required to confirm this cause POSSIBLE CAUSE cause.

Conditions necessary to Existing data that tends to disprove this as Data required to disprove this cause collect required data Degrade Seal Faces the cause.

Data reuiedtodipovtiscas GE SI No. 511 -

w EBR lnshv Organization did not have high enough sensitivity to impact of insufficient purge flow GE SIL No. 511 - Two GE BWR/3 plants have operated pumps for over 20 years without purge.

CR70025441 Vendor stated our plant needs purge flow due to condition of RCS water.

None None NA Nuclear Common Page 9 of 9 4/1512003

B Recirc Pump Seal Reliability 70029861 AS-FOUND TESTING of F025A & B Performed By:

Jay Barton Observed By:

Craig Johnson, Gene Delp and Glenn Gardner Date Performed:

3/26/03 A. Planned Procedure:

1) Pressurize valve inlet to 1000 psig - Collect leakage for 1 minute

2) Increase pressure to 90% of set or 1125 psig - collect leakage for 1 minute

3) Increase pressure to determine setpoint - acceptance criteria is +/-3%

B. Actual Testing:

The F025B was installed on the test stand. The test stand accumulator isolation valve was opened to the valve inlet. At this time, the accumulator water level was approximately % full with atmospheric pressure above the water. The F025B began leaking immediately with approx 1 ft of head on its inlet. The leakage was collected using a 1000 ml beaker. 450 ml (or 0.118 gal) of leakage was collected for the 1st minute.

When the F025A valve was later installed on the test stand, only minor leakage was detected when 1 ft of head was applied to the inlet. As the test pressure was increased, the valve began leaking excessively, although less than the F025B.

Due to the volume of leakage of both valves, neither the leakage collection at 1000 psig or the determination of set pressure could be performed. Also, due to test stand limitations for discharge water collection, it was decided that the leakage quantification would be performed at 200 psig for 1 minute for both valves. The following are the results:

1) F025B = 12,890 ml/min or 3.4 gal/min

2) F025A = 9430 ml/min or 2.5 gal/min C. Post Testing Disassembly:

Following testing, both valves were fully disassembled & inspected.

1) F025B:

Set Pressure Adjusting Nut: Finger tight Nozzle: Seating area damage/washout (not repairable)

Disc: Gouges in seating surface (not repairable)

Debris: Black particles in seating area

2) F025A:

B Recirc Pump Seal Reliability 70029861 Set Pressure Adjusting Nut: Tight Nozzle: Moderate wear (repairable)

Disc: Gouges in seating surface (not repairable)

Debris: Black particles in seating area

B Recirc Pump Seal Reliability 70029861 Change Analysis Worksheet Problem Statement: B Recirculating Pump seals fail before expected life.

Date: March 28, 2003 Problem Situation Problem Free Differences Effects(s) of Differences Situation B Recirc Pump A Recirc Pump B Recirc Pump A Recirc Pump seal in B seal life - one Impact on budget, Seals replaced in service since 9171999, cycle or less generation and outage 5/11/00, 10/30101 A seal life -4.5 schedule.

and 3111103 years, planned replacement RF12 B Recirc Pump A Recirc Pump More opportunities When pump speed or taken OOS service shutdown after swap for seal to RCS pressure drops the most often with RCS to shutdown cooling.

momentarily hang-Quad-ring on the pressurized, up when pump stationary faces hangs up transition to speed drops momentarily before shutdown cooling on following the rotating A Recirc. Pump.

faces as it is lowered.

This increases the gap between the seal face and could allow larger particles to slip in between the faces.

B cavity seal A cavity seal Per vendor, this Lack of purge flow allows temperature temperature remains indicates seal RCS to accumulate in increases on steady on downpower purge flow is not seals, increase wear on downpower or or shutdown. July present when seal faces due to shutdown.

2002 downpower no pump is turned off.

particulate intrusion. B seal cavity temp pump operated most of increase.

cycle w/o purge. Purge in-service on B pump through July 2002.

As-found As-found F025B seating Wear F025B indicates test/inspection test/inspection area is not longer leakage, longer F025B seating area F025A had moderate repairable time w/o purge. Less damage and washed damage to seating wear on F025A indicates out - not repairable area - repairable shorter time wlo purge.

Excessive wear on A seal - No signs off Excessive wear on Reduced Seal life.

seal parts i.e.,

excessive wear parts seen on B lockbolts, lockbolt seal indications of bushings, stationary bowed shaft on B face holder pump B pump differs from A pump comes apart Shaft length and Unknown vendor instructions exactly like the vendor axial position are during disassembly instructions different and reassembly 4/11/03

U Kecirc Hump 6eal Reliability 70029861 Problem Situation Problem Free l Differences Effects(s) of Differences Situation I

B Recirc Pump A Recirc Pump B pump does not A pump has expected Shaft length and Per vendor, Art Olsen this have full pump lift lift axial position are would not irnlpact seal (clearance off the different wear bottom)

Fill and vent procedure do not protect F025B relief valve Installed seal in 9/99 using Rev 33 Rev 35 - 3/15/00 major rewrite of fill and vent to protect F025, uses throttle valve for isolation.

Throttle valve ineffective as isolation.

A pump seal in service 4.5 years to date B pump seal replace 5/11/00 and 10/30/01 since Rev 35 changes.3/10/03 seal used Rev 43, using V100(V101) Seal Purge Isolation valve for isolation Hand held on B purge supply header can feel vibration and pulsing surges during normal operation.

Hand held on A purge supply header, pipe feels still during normal operation 4/11/03

too00 4i r(oI ATTACHIMEN-10 F-lowserve Corporation FLOWSERVE 2300 Vemon Ave Vemon, CA 90058-0017 Tel. 320-267-4831 Fax 320-384-0678 12 March 2003 PSE&G Hope Creek Generating Station Lower Alloways Creek Hancocks Bridge, NJ 08038 Attn: Carl Caltabiano, Mechanical Maintenance Superintendent

Subject:

Inspection of 'B' Reactor Recirc Pump and N-7500 Mechanical Seal Flowserve has completed the inspection of the B recirculation pump and N-7500 mechanical seal at the Hope Creek station. The seal had been service for 17 months and experienced increasing leakage in both stages. The removed seal has been removed, disassembled and inspected. The pump seal cavity, auxiliary impeller and coupling components were inspected. A replacement seal was assembled by station personnel and has been installed in the pump.

The following were findings from the pump inspection:

Drag marks on the ID of the seal cavity in the pump cover, for approximately 80% of the circumference. These marks are at the elevation of the auxiliary impeller.

Drag marks on OD of auxiliary impeller, over entire circumference. The marks on one side were heavier.

Fretting on the lower shaft land above the 0-ring groove. There were marks around the entire circumference.

The following conditions were found in the seal inspection:

Heavy gouges on both stationary faces. On each, there were 12-15 parallel gouges across the face in a space of approximately 4 inches.

Heavy wear marks on both sets of lock bolts from. contact with the stationary face holder bushings. The wear is uniform and there is no evidence of cocking of the face.

Wear on the stationary face holders and bushings.

Rub marks on the shaft sleeve lower ID.

All elastomers are intact.

The seal had high radiation and contamination levels and reddish deposits throughout.

The contact between the auxiliary impeller and pump cover indicates high shaft runout.

The clearance between the rotating and stationary components at the hydrostatic bearing and thermal barrier (stuffing box) is much closer than at the auxiliary impeller.

The rub marks on the shaft sleeve and pump shaft are suspected to be caused by high runout of the pump shaft. The heavy wear on the lock bolts is evidence of unusually high loads on the stationary face, consistent with high shaft runout.

The following contributed to failure of the stationary faces:

1) Loss of seal purge flow

2)

High runout of the shaft. The increased runout could be a result of imbalance in the shaft-impeller assembly or a bow in the shaft.

Page 2 B Recirc Pump & Seal Inspection The radiological conditions in the seal indicate there was insufficient seal purge flow to the B recirc pump. The concentration of particulate at the seal faces increased the amount of material being deposited between the faces. The high shaft runout accelerated the damage to the stationary face, leading to the initiation of leakage.

These conditions also caused the rapid increase in leakage.

The seal inspection revealed no deficiencies in assembly or installation.

Observation of the removal and installation of the replacement seal and the coupling assembly showed no deficiencies and demonstrated the ability to install the seal and restore the pump to the design configuration.

Flowserve recommends the following:

1) Restoration of seal purge flow to the B pump, including measures to verify proper flow during operation.

2) Verify proper seal purge flow to the A pump.

3) Replacement of the B recirc pump internals with the replacement 4t generation assembly.

Section Head, Primary Pumps Gerry Lenzen Phone 320-284-0676 Fax 320-284-678 Cell 320-2674831 glenzen@flowserve.com

f,

ex7 X &

1 -VJ V gIV t I I w"

Flowserve.i Corporation FU)WSERVE 2300 Vemon Ave Vemon, CA 90058-0017 Tel. 320-267-4831 Fax 320-384-0678 12 March 2003 PSE&G Hope Creek Generating Station Lower Alloways Creek Hancocks Bridge, NJ 08038 Attn: Peter Koppel, Engineering

Subject:

Purge Flow to A Reactor Recirculation Pump Flowserve has been informed that the purge relief valve on the Hope Creek Station 'A' reactor recirculation pump is lifting, reducing the supply of purge flow to the A' recirc pump and mechanical seal.

If the supply of purge flow to the pump is reduced or secured, there is an increased chance of particulate being deposited between the seal faces. This can result in damage to the stationary face and the initiation of leakage. Any seal or pump transients would increase the chances of particulate being deposited and leakage being initiated.

This includes the normal starting procedure. Transients causing sudden pressure changes in the seal cavity or sudden changes to purge flow or controlled bleed off flow should be prevented. Operation without purge flow is within the original design configuration of the recirc pump and mechanical seal. The presently installed seal faces are designed for optimum performance with seal purge.

Operation of the 'A' recirc pump without purge flow would not be expected to cause rapid degradation of the mechanical seal. If leakage were initiated during routine operation, the rate of increase would be expected to be slow and somewhat predictable. The recent behavior of the 'B' seat would be the best estimate of leakage progression. Close monitoring of seal temperatures, staging pressures and leakage indications should be maintained during this period.

Gerry Lenzen Section Head, Primary Pumps Phone 320-284-0676 Fax 320-284-0678 Cell 320-2674831 glenzen~flowserve.com

B Recirc Pump Reliability 7002986 1 2 B Recirc Pump Failure Mode Matrix

.Failure Mode i

Worn Pump Pump Sha Symptom Imbalance Misalignment Bearing Bowed Shaft Thrusting High Vibrations X

X X

Impeller Rub Single B

B Spot Impeller Rub B

B B

Circumferential Stuffing Box Rub B

B B

Circumferential Stuffing Box Rub Single Spot Worn Lock Bolts B

B Worn Lock Bolt B

Bushings B

Worn Stationary Face B

E Holder Worn Seal Face Holder Springs B

B Stationary Face Cocking B

Seal Damage Radial B

Cuts in One Quadrant Rotating Face Contact Width Greater Than B

B Stationary Face Nose Eccentric Orbit of Rotating Seal Face X

Random Seal face X

Contact Pattern Low frequency Random X

Motion Seal Face Separation X

Seal Damage Radial X

X Cuts Across Seal Face Pump Shaft Sleeve Quad Ring Polished X

Pump Shaft Fretting B

B B

Pump Shaft Sleeve Fretting B

Legend:

X Typical - not apparent in Bravo Recirc Pump B

Typical - Bravo Recirc Pump Has This Symptom