View source for U-601474, Forwards Response to Confirmatory Action Ltr CAL-RIII-89-16 Re Investigations of Reactor Recirculation Pump Seal Failure & Complications Involving Drywell Chiller A.Util Quarantined Equipment That Malfunctioned & Evaluated History

{{Adams
| number = ML20246J473
| issue date = 06/30/1989
| title = Forwards Response to Confirmatory Action Ltr CAL-RIII-89-16 Re Investigations of Reactor Recirculation Pump Seal Failure & Complications Involving Drywell Chiller A.Util Quarantined Equipment That Malfunctioned & Evaluated History
| author name = Holtzscher D
| author affiliation = ILLINOIS POWER CO.
| addressee name = Davis A
| addressee affiliation = NRC OFFICE OF INSPECTION &amp; ENFORCEMENT (IE REGION III)
| docket = 05000461
| license number = 
| contact person = 
| document report number = CAL-RIII-89-16, U-601474, NUDOCS 8907170381
| document type = CORRESPONDENCE-LETTERS, INCOMING CORRESPONDENCE, UTILITY TO NRC
| page count = 9
}}

=Text=
{{#Wiki_filter:=-         .. ..
g.
                                                                                                                                 .Dc D '
U-601474:
[        -
L42- 89 (06 -30 )-LP
                                                                                                .1A.120
                /LLINDIS POWER 00MPANY
                                                                      ' CLINTON POWER STATION, P.O. box 678, CLINTON, ILLINOIS 61727 June 30, 1989 Docket No. 50-461' Mr. A..B. Davis >
                     ' Regional Administrator Region'III.
                    'U.S.~ Nuclear Regulatory Commission 799-Roosevelt Road Glen Ellyn, Illinois 60137

==Subject:==
Response to Confirmatory Action Letter LQAL-RIII-89-016) dated June 1. 1989

==Dear Mr. Davis:==

                          .. This letter is in response to Confirmatory Action Letter (CAL)
CAL-RIII-89-016 dated June 'l', 1989. In the CAL, the Nuclear Regulatory Commission (NRC) requested Illinois Power Company (IP) to submit a-formal report of IP's findings and conclusions of the investigations into the Reactor Recirculation (RR) pump seal failure and-the concurrent
                    -complications involving drywell chiller A and the motor driven reactor feedwater pump regulating valve. As requested in the' CAL, IP
                     . quarantined the equipment that malfunctioned, evaluated-the history of the equipment and notified the NRC.Special Inspection Team Leader or the appropriate team member prior to performing work on the equipment.
Attachment A to this letter provides the details.concerning IP's investigation of and conclusions regarding the RR pump seal failure.
Attachments B and C, respectively, provide.the details of.IP's investigation of and conclusions regarding the drywell chillers and the motor driven reactor feedwater pump regulating valve.
If there are any questions on this report, please contact me.
Sincerely yours, b/
D. L. Holtzscher Acting Manager -
Licensing and Safety JAB /krm Attachments (3) cc;      NRC Resident Inspector                                                                                ,i
                               !!RC Clinton Licensing Project Manager Illinois Department of Nuclear Safety
           '8907170381 890630
* y ~3 1989                          'l PDR    ADOCK 05000461 u_. _ _

    ~
j
            *                                                                                          ?
l       '.                               Attachment A                                          1 B Reactor Recirculation Pumn Seal Failure
            'Backcround During the first refueling outage (RF-1) at Clinton Power Station (CPS),
the seal assemblies for Reactor Recirculation (RR) pumps A and B were reworked and replaced. The installation of RR pump A' seal assembly was completed February 8, 1989, in accordance with Maintenance Work Request (MWR) B11970. The iv.stallation of the B seal assembly was completed March 24, 1989, in accordance with MWR D02563.
On May 25, 1989, the outer seal pressure of RR Pump B droppod from 490 pounds per square inch gauge (psig) to approximately sixty psig for a period of eight and one half hours while the pump was operating at slow speed. During this period, drywell equipment drain flow increased by one gallon per minute (gpm), and the outer seal and seal staging flow alarms were received in the Main Control Room. These alarms are' characteristic symptoms of upper seal failure. Preparations were underway to replace the seal assembly when, at 1930 hours, on May 25, 1989, outcr seal pressure returned to normal (490 psig) and all alarms cleared. Nuclear Station Engineering Department (NSED) personnel, the RR pump vendor, and an independent consultant discussed the changes in outer seal pressure, and determined that the RR pump upper seal appeared to have rescated. Based on this information, replacement of the seal assembly was determined to be                  {
unnecessary at the time, and plant operation was continued.                              !
On June 1,1989, the Reactor Recirculation (RR) Flow Control Valves were throttled down in preparation for shifting the RR pumps to fast speed. The RR B loop experienced greater than normal flow oscillations as shown on General Electric Transient Analysis and Recording System (CETARS).
(General Electric (GE) has reviewed this data and concluded that these oscillations were not a contributing factor in the seal failures.) Upon being transferred to fast speed, the B pump outer seal pressure immediately dropped frcm 490 psig to thirty-eight psig. The loss of seal pressure and the flow to the drywell floor drain in excess of sixty gpm were indicative of failure of both the upper and lower seals in the B pump. Based upon this indication, operating personnel isolated the RR B loop.
Subsequent inspection of the B pump revealed significant upper and lower seal damage. Particles of stellite, carbon and stainless steel, no larger than 1/4 inch in diameter, were found outside the pump body. A spectral analysis of the particles was performed by the Radiation Protection Engineering Group. The analysis confirmed that no radiological concern or potential for hot particles existed as a result of the upper and lower seal failures. Following removal of the seal assembly from the B pump, no such material was found below the seal cavity.
An examination of the B seal assembly internals showed that the upper seal was completely destroyed and that the material from the upper seal had exited through various upper portions of the seal assembly. The material from the upper seal constituted the major part of the material found around the B pump. All components in the upper seal showed severe galling, gouging and deformation.
(1)
_________-__a

       .                                Attachment A B Reactor Recirculation Pump Seal Failure The lower seal faces were also inspected. The lower rotating face was found to have broken into eight pieces and to display evidence of heat-checking but no discoloration. The lower stationary seal was unbroken and showed no discoloration. The parts remaining in the seal assembly of pump B were verified to have been properly installed.
The condition of the upper seal in pump B indicates that the event commenced with a failure of the upper rotating and stationary seals.
Failure of the upper seal resulted in pieces of the seal migrating upward through the assembly until they reached the seal leakage chamber. These materials damaged the capscrews which retain the floating throttle sleeve.
Three of the four capscrews broke or sheared, allowing relative movement between the throttle sleeve and seal sleeves. This movement resulted in an unseating of the lower rotating seal 0-ring. At this point the lower seal no longer functioned. Water at reactor vessel temperature and pressure flowed past the seal assembly at a maximum of sixty-four gpm. With the isolation of RR loop B this flow caased and only Control Rod Drive (CRD) injection flow at three to five gpm and relatively low temperature entered directly into the lower seal cavity. This quenching caused cracking of the lower rotating face.
There are four possible causes of failure of the upper rotating and stationary seals: pump operation at low pressure; foreign material; design or material defects; and installation error. These four causes are discussed below.
Pump Operation at Low Pressure GE Service Information Letter (SIL) 203, which was issued in 1976 and revised in 1980, recommends that RR pump operation and RR pump starts at low pressure (less than 300 psi) be minimized. Sulzer Bingham, the pump vendor, concurs in that recommendation. It is postulated that low pressure operation starves cooling water to the upper seal (the lower seal is not as subject to overheating due to cooling water starvation) allowing the seal to heat excessively. This overheating would cause the upper rotating seal to deteriorate.
The RR pump B at CPS was operated at low pressure for an estimated 1000 hours and RR pump A was operated for approximately 1200 hours at low pressure. The pump A seal assembly was inspected to determins the effect of the low pressure operation. No evidence of other than normal wear was found. Furthermore, no discoloration of either the lower or upper seal of RR pump A was noted. The vendor stated that discoloration of the lower seal would be present if low pressure operation failure had occurred. The effects of low pressure operation on the RR B pump seal assembly was indeterminate due to the severe transient and resultant damage to the seal assembly. Therefore, it is not likely that pump operation at low pressure is the cause of the failure.
(2)

y
       .~*           ;                                                             Attachment A l-                                         B Reactor Recirculation Pumo Seal Failure 1:
1 Foreien Matter 1^
Foreign matter, if left in the seal cavity during assembly,'could have damaged the upper seal. This is unlikely since MWR D02563 required that maintenance personnel maintain system cleanliness at class B in accordance with CPS Procedure 1019.02, System Cleanliness. Additionally, a Quality-Control inspector verified that class B cleanliness was maintained.
l                  Desien or Materia 1' Defects A design or material, defect could have resulted in the spontaneous breakup of the upper rotating seal.                           This is unlikely since the design and manufacturing process of the pump and seal assembly have been proven in numerous installations including the seals used at CPS during the first operational cycle.         IP did install new seal assemblies during RF-1 which are an improved design and have had limited use in the industry; however, there is no evidence to indicate that this design is defective.
Installation Error Mechanical stress due to thrust forces could have damaged the rotating seals if some error were made during seal adjustments. Failure to properly set clearances on the bench prior to installation would have resulted in improper field installation.                           Even with proper bench adjustment, some field installation error could have resulted in improper seal settings. However, a review of procedure checklists and work documents, and discussions with maintenance personnel, have not identified any seal adjustment error.

== Conclusion:==

The exact cause of the. pump upper rotating and stationary seal failures has not been determined.          IP has considered the four possible causes discussed above and believes that the most probable cause of failure is improper seal setting either on the bench or in the field. No evidence to specifically support the other possible causes exists (i.e., no wear or discoloration to support damage from operation at low pressure; no foreign material observed in the seal cavity to support damage from this cause; and no industry history of seal failures indicating design or material defects).                                     Since no evidence exists to either support or refute improper seal setting as the cause of seal failure, and due to the fact that this cause could have rea"Ited in undue mechanical stress and subsequent failure of the upper rota-i.x seal, it is considered the most probable cause of the seal failures Corrective A tions Taken and Results achieved
: 1)        IP, with the assistance of experienced personnel on site from the pump vendor (Sulzer Bingham), Niagara Mohawk, Commonwealth Edison, GE and an independent consultant, rebuilt the B pump seal assemblies.
No errors were identified in the rebuilding or installation of the seals assemblies.
(3)

Attachment A B Reactor Recirculation Pump Seal Failure
: 2)    Vibration monitoring of.the pump at 0 psig, 500 psig and at normal reactor vessel pressure was performed. No abnormal vibration was noted during the vibration monitoring.
Corrective Actions Taken to Prevent Recurrence IP is taking the following actions to minimize the potential for seal failure in the future and to provide for early detection of the onset of such a failure if it should occur:
: 1)    Mandatory training on the seal assembly is being provided to appropriate maintenance personnel. This training involves hands-on work on a seal assembly mockup.
: 2)    During future seal assembly work, a qualified / experienced seals consultant will be present to ensure the seal assemblies are installed correctly. Upon completion of the third refueling outage (RF-3), IP will re-evaluate the need to have a qualified / experienced seals consultant present during seal assembly work.
: 3)    A static pressure test rig will be acquired to allow bench testing of rebuilt seals under static loads.
: 4)    Operators have been provided criteria to determine whether a degraded seal condition exists. In addition, operators have been provided specific actions to take in response to a degraded seal. These actions include contacting engineering and the Plant Manager.
: 5)    The NRC Resident Inspector will be notified of any degradations to the seals that may occur between the present and February 1990.
: 6)    The installation of an RR pump monitoring vibration system during RF-3 will be evaluated.
l l
(
i (4)

F
  ~  '
Attachment B Drvwell Chillers D.asiaf9J1B51 Drywell Chiller A-tripped on low chiller condenser water pressure. IP believes that the trip occurred because the chiller was attempting to respond to the increased heat load caused by the Reactor Recirculation (RR) pump seal failure and subsequent associated steam leak. The drywell chiller system is only required to operate during normal plant operations and during a loss of offsite power, not during other abnormal conditions.
The design cooling load for one train of the drywell chiller system includes latent heat load based upon a five gallon per minute reactor coolant leak. An RR pump seal failure (greater than five gallons per minute reactor coolant leak) is beyond the design heat load for the drywell chiller, therefore, the chiller would be expected to trip.
Although the drywell chiller trip occurred per design, the Nuclear Station Engineering Department (NSED) developed a plan to investigate the drywell chiller trip to determine whether any deficiencies in hardware or design existed.      Included in the plan were steps to vent sensing lines, install instrumentation for data acquisition, and cycle the chillers from one train to the other.
As a result of the planned testing, the following anomalies were found and corrected as noted:
During the testing, the service water pressure indication on the A chiller was rapidly oscillating (vibrating) although the pressure in the service water pipe remained relatively constant. Variances of twenty to thirty p.tig on either side of the average Service Water (WS) pressure reading vare observed. The US pressure reading on the B chiller was steadier than the reading on the A chiller and although the indication for the B chiller oscillated, the oscillations were not as rapid and not as large in magnitude (five to ten psig) as the A chiller indication oscillations. IP installed pulsation dampers on both chillers to decrease pressure oscillations on the pressure indication instruments.
When one of the two operating WS pumps was secured to observe the effect of this transient on A and B chillers, the US pressure dropped from approximately 100 psig to about thirty psig on B chiller and to about fifty psig on A chiller. This is below the design value low pressure setpoint of seventy-five psig. IP determined that the design setpoint was unnecessarily high and that a lower setpoint would provide for adequate operation of the WS system. Therefore, IP reduced the US low pressure setpoint from the existing seventy-five psig to twenty-five psig.
During the testing, chiller A tripped four times on " Evaporator Low Water Temp" even though the actual chilled water temperature was well above the setpoint. This trip was determined to be caused by a bent temperature sensing device. IP replaced the existing bent temperature sensing device with a new sensing device.
(1)

Attachment B Drvwell Chillers The response of the controller which controls the WS flow to the chiller condenser appeared to be more rapid on the A chiller than on the B chiller. More extensive testing of the controller concluded that performance of the controllers is similar and acceptable. No further actions were taken.
Root Cause During the testing of the A drywell chiller, the low chiller condenser water pressure was due to rapid pressure oscillations sensed by the WS instrumentation. During these oscillations the low pressure setpoint was reached and the chiller tripped. As previously discussed, IP has installed pulsation dampers on both chillers and has_ reduced the low pressure setpoint.
The tripping of the A chiller on " Evaporator Low Water Temp" is believed to be due to the temperature element not being concentrically located in the chilled water tube and due to the temperature element being bent at several places and touching the chilled water tube.    (The other side of the tube has refrigerant.)
Because the element was touching the chilled water tube, it was sensing the refrigerant temperature at many places within the tube and therefore, sensing lower temperature than the average chilled water temperature. As previously discussed, a new temperature sensing device has been installed.
Conclusion 4
The drywell chiller trip occurred per design and did not impact the RR pump seal failure. The corrective actions implemented will provide for more reliable operation of the drywell chillers.
(2)

L      '
I Attachment C Feedwater Reculatinz Valve

===Background===
The Feedwater Regulating Valve (1FWOO4) is the discharge valve for the notor driven reactor feedwater pump. The valve, manufactured by Paul Monroe Corporation, is a flow regulating drag valve with a hydraulic operator and electronic control.
The valve has functioned erratically since initial plant operation at CPS.
l-     -The most serious problem experienced with the valve has been a " lock-up" l        condition in which the valve failed to respond to the feedwater demand l        signal. On two previous occasions where the valve " locked-up" IP required that the valve vendor assist with the valve repair and testing to assure that the valve operation was adequate.
: Valve 1FWOO4 failed on June 1,1989, during power reduction following a failure of the RR pump seal. Valve 1FW004 failed to close in response to both manual and automatic control signals. The failure of the valve to respond resulted in an increase in reactor water level. Operators were not able to control reactor water level and therefore inserted a manual scram signal.. The valve remained " locked up" in the full open position even though it was receiving a control signal for full close.
A troubleshooting plan was developed and. approved by management to investigate this problem. The troubleshooting consisted of checking the electric / electronic and hydraulic control functions of the valve, as well as the mechanical motion of the valve. During this troubleshooting effort, the following anomalies were found and corrected as noted:
The servo (solenoid) valve that controls hydraulic operation of IFWOO4 did not meet resistance specifications at both coils. The valve was replaced.
The position transducer was found with two " dead spots" and two bad crimps on the trans(2cer connections. The transducer was replaced.
The servo card enclosure cover pressed against the J1 connector wire when closed. Also, the J1 connector was found to interrupt the signal when " moved". A new J1 connector was made and installed that does not press against the cover or interrupt the signal.
A ground stud on the mounting of the servo-amplifier board had a non-conducting carbon steel washer under the nut. The washer was replaced with a good conducting washer.
The Signal Current Hysterisis (SCH) module was not installed on the servo card. This device prevents valve modulation near the close position. The vendor has indicated that the servo card adjustments provide sufficient control to prevent valve modulation at the seat, and therefore this component is not required.
(1)

1*       '
                                                   .                                  Attachment C L                     .-
Feedwater Rerulatine Valve Root.Cause:
                                        .The troubleshooting determined that the electronic control failed toi energize the solenoids in response to a control signal. The root cause of-this failure has been traced to these two factors:
: 1)     Vibration of the control' cabinet causing loose electrical connections
: at the solenoid pin connectors on the servo-amplifier board due to
                                                     .the cabinet being located on top of the_ valve.
: 2)     Failure of the rectifying circuit on the servo-amplifier board.
IP installed a new control cabinet on a nearby wall and installed the servo-amplifier board in the new control cabinet to reduce the susceptibility of the servo-amplifier board to vibration. Also, a new servo-amplifier board was installed and tested to IP's specification. The combination of installin6 the new control cabinet, and installing the new servo-amplifier board should provide for reliable operation of the i                                              feedwater. valve. IP has also installed electronic monitoring equipment'to l                                             monitor the valve during plant startup.
l                                             The actuator for 1FWOO4 valve, unique in its electronic / hydraulic control, has no other application at CPS. IP is adding four preventive maintenance (PM) activities to the PM's for the 1FW004 valve to assist in maintaining proper valve operation.' Additionally, as a long term action, IP is evaluating the feasibility of a plant modification to replace the IFWOO4 valve and/or its control system.
(2)
________ ___ ____ ______ ___ __                                                                                                      i}}