NRC-95-0033, Forwards Response to Re Violations Noted in Insp Rept 50-341/95-03.Corrective Actions:Set multi-discipline Task Force to Evaluate Repetitive Failures of Ite MCC Auxiliary Contacts & Barton Differential Pressure Switches
| ML20082M272 | |
| Person / Time | |
|---|---|
| Site: | Fermi  |
| Issue date: | 04/17/1995 |
| From: | Mckeon R DETROIT EDISON CO. |
| To: | NRC OFFICE OF INFORMATION RESOURCES MANAGEMENT (IRM) |
| References | |
| CON-NRC-95-0033, CON-NRC-95-33 NUDOCS 9504240166 | |
| Download: ML20082M272 (10) | |
Text
Robert McKoon -
Amstant Vcc President
}
and Manager, operations Edison *00NorthDuneHwy*na" 7"""*
w m=
j April 17,1995 i
i 1
i U. S. Nuclear Regulatory Commission Attention: Document Control Desk Washington D.C. 20555 j
i
References:
- 1) Fermi 2 -
t NRC Docket No. 50-341 NRC License No. NPF-43
- 2) NRC Inspection Report 50-341/95003, dated March 17,1995 l
Subject:
Response to Notice of Violation i
l Enclosed is Detroit Edison's response to the Notice of Violation contained in Reference 2. Detroit Edison met with the NRC on March 8,1995 to share i'
actions being taken to improve resolution of equipment problems. The initiatives discussed in. that meeting will help prevent additional problems such as those cited in the first violation.
i The following commitments are being made in this letter:
1.
Approximately 36 additional Banon Differential Pressure Switches that have had noted performance problems still require further evaluation. This will be complete by December,1995.
2.
The Motor Control Center auxiliary contact multi-discipline f
team's effons will continue in order to determine the root cause.
l Additional actions that required further evaluation include:
i Continued review of past and future unexplained failures. This e
includes use of the troubleshooting guide and Technical and I
Engineering Services analysis of removed / replaced auxilian contacts.
Future review of Deviation Event Report 93-0312 to determine effectiveness of the main contactor cormctive actions.
1 i
Ot O,
- f' o t}
.. u u j
D CK,O 000 Q
U. S. Nuclear Regulatory Commission April 17,1995 NRC-95-0033 Page 2 of 3 Review of altematives developed by the team to modify or
=
replace the auxiliary contacts.
3.
Training will be provided to Instrumentation and Controls (I&C) repairmen by Barton on lessons learned from experience on calibrating and maintaining Barton switches.
4.
Training will be provided to maintenance personnel, including mechanical maintenance, regarding their responsibilities on the use of consumables.
5.
Procedure 35.306.008, " Motor Control Center Load Compartment,"
will be revised to delete the use of Cramolin. This action will be completed by July 31,1995.
6.
Incorporate requisite consumable control measures into site orientation and requalification training manual.
7.
Detroit Edison will continue to evaluate the past use of the other ten cleaners and consumables (other than Cramolin) noted in the violation.
If there are any questions related to this response, please contact Elizabeth A.
Hare, Compliance Engineer, at (313) 586-1427.
Sincerely, Enclosure cc:
T. G. Colburn J. B. Manin M. P. Phillips T.
Vegel Region Ill
Enclosure to NRC-95-0033
. Page 1 of 8 r
I.
Statement of Noncompliance 1 10 CFR Part 50, Appendix B, Criterion XVI, requires,in part, that measures shall be established to assure that conditions adverse to quality, such as failures, malfunctions, deficiencies, deviations, defective material and equipment, and nonconformances are promptly identified and corrected. In the cases of significant conditions adverse to quality, the measure shall ensure that the cause of the condition is determined and corrective action taken to preclude repetition.
a.
Contrary to the above, from about May 1989 to December 1994, the licensee failed to implement corrective action to preclude repetition e
of multiple safety related component failures, a significant condition adverse to quality, caused by repetitive auxiliary contact problems associated with ITE/telemechanique starter contactor failures.
b.
Contrary to the above, from March 1989 to December 1994, the licensee failed to determine the root cause and take effective corrective action for a significant condition adverse to quality.
Namely, deficiencies associated with Barton Differential Pressure Switches, Model Numbers 288A,289A and 580A, resulted in safety related equipment not being able to perform its intended function such as the High Pressure Coolant Injection and Residual Heat Removal System Minimum Flow Valves not operating automatically.
Reason for the Violation The common thread to these violation examples is poor communication and team work between the organizations involved in identifying the problems, evaluating the concern, prescribing corrective actions, and implementing the corrective actions. A lack of problem ownership led to inadequate fo, low-up to verify the effectiveness of corrective actions.
More specific, after the failure of ElINO21 Barton switch on December 10,1994, Deviation Event Report (DER) 94-0751 was written. On December 21,1994, DER 94-0787 was written due to failure of the P50N513B Barton switch. During the root cause evaluation of these two DERs, it was determined that the corrective actions put in place by the j
original Barton DER (DER 89-0283) were not implemented uniformly.
Specifically, if a Barton switch was found to be unsatisfactory and the two j
previous as found sets of data was also unsatisfactory, the Maintenance j
i
)
Enclosure to NRC-95-0033 Page 2 of 8 Instmment Control (I&C) supervisor was to perform an evaluation. This was not being consistently performed, therefore instruments that were having successive "as-found" unsatisfactory data were not always evaluated and trended for poor performance. Failure to perform this review resulted in the failure to identify the repetitive nature of the problem. In addition, the engineering personnel involved in developing corrective actions did not actively follow-up to the extent necessary to determine the effectiveness of the corrective actions.
Only the Residual Heat Removal (RHR) Bartons had a Potential Design Change (PDC) identified, although failures occurred on other systems.
The system engineers, who evaluated the PDCs, were not fully aware of the extent of the Barton failures because they focus more on system performance than on a specific type of component. Therefore, the system engim:er dispositioning the PDC did not recognize the repetitive Barton failires.
Repeat occurrences of ITE Motor Control Center (MCC) main contactor (contactor) problems were noted on various DERs since 1993. DER 93-0312. " Investigation of Failures Related to the ITE Contactors," was assigned as the lead DER to identify the root cause of the contactor problems. Plant Engineering coordinated efforts with Maintenance, Technical and Engineering Services (TES), and Technical Engineering Group in identifying two failure mechanisms for the main contactor. A sticky residue from Cramolin spray delayed or caused main contactors not to operate. Also, if the main contactor cover plate locking ketches are not fully engaged, contactor pickup can be prevented or delayed.
Cleaning to remove any Cramolin and verification of full ketch engagement was completed for all safety related main contactors during the recent refuel outage. Detroit Edison believes that these actions effectively solved the main contactor problem.
Problems with MCC auxiliary contacts had also been reponed in DERs associated with DER 93-0312. This DER was remaining open pending further review of potential failure mechanisms. Both of the two failure mechanisms identified for the main contactor were thought to be possible causes for problems with the auxiliary contacts. The delayed / sticky movement of the contactor could affect the forces on and resultant closure of the auxiliary contacts. Thus Cramolin indirectly may have affected the auxiliary contact function. Due to Cramolin, the contactor may not have returned to the full rest (deenergized) state and thus impeded the auxiliary contact. Note that the ketch being disengaged would have caused the same non-rest reset to occur.
Enclosure to NRC-95-0033 Page 3 of 8 Current activities, as a result of keeping the DER investigation open, indicate that the most probable root causes for the main contactor problems have been addressed but the failure mechanisms for the auxiliary contacts have not been fully identified. The issue and investigation have been complex with a problem involving the auxiliary contacts still remaining after the main problems have been solved.
Corrective Steos Taken and the Results Achieved Multi-discipline task forces were set up to evaluate the repetitive failures of the ITE MCC auxiliary contacts and the Barton Differential Pressure Switches. Specifically with the Barton Differential Pressure Switches, evaluation criteria have been prepared to review the root cause of those Bartons in the failure group. This evaluation includes the component history, switch function, whether it is subjected to pressure transients (potential over-range condition), observations from field walkdown and tubing design. To date, eight Bartons have been reviewed with respect to this evaluation criteria. The initial engineering recommendation is to modify seven of the eight. The modifications include replacement with a different model switch, setpoint adjustment and/or tubing reorientation, based on the individual switch application against the evaluation criteria.
The eighth Barton switch will not be changed, pending review of future surveillance data.
Approximately 36 additional Bartons that have had noted performance problems are still being reviewed. These reviews have been prioritized such that QA-1 Bartons will be completed first and all reviews will be complete by December 1995, at which time additional modifications may be required on an individual switch basis. All necessary design changes will be processed and prioritized through the Project Evaluation Review Committee (PERC).
An Engineering Functional Analysis (EFA) was prepared andjustified continued operation with respect to the auxiliary contacts. Considering the low rate of failures experienced, it is concluded that the problems experienced are within random failure rate and as a result have no significant impact on overall plant safety. The NRC has reviewed this EFA and found Detroit Edison's justification for continued operation acceptable. The failures will be further evaluated to determine causes and climinate or reduce the degraded condition.
Enclosure to NRC-95-0033 Page 4 of 8 A troubleshooting guideline for evaluation of the failure of a motor operated valve to stroke has been reissued to Operations and Maintenance personnel to ensure their sensitivity to finding a cause. This guideline is being presented to Operators as part of their requalification training in March and April 1995.
The multi-discipline team reviewing the auxiliary contact problems has performed a detailed review of the work requests and related information for the seventeen (17) unexplained failures of auxiliary contacts since 1989. The review looked at circuit function, schematics, DERs, work requests, contact involved, contact failure state, MCC location, AC or DC service, Q level, TES analysis, NPRDS, and corrective action taken. All of this was done to identify any trend or common cause.
In paiticular, for the six (6) unexplained failures for 1994, this review has clarified prior information. Two of the unexplained failures were removed from the list since there was no failure or the problem was not related to an auxiliary contact. Of the four remaining in 1994, two occurred after cleaning and disassembly of the main contactor and the auxiliary contact. These failures were considered to be related to prior maintenance activity. One contact was found to be cocked in its mounting and another had debris inside the auxiliary contact that had been previously disassembled. The final two have not had their review completed at this time.
This information is a result of the more complete review of work packages and TES auxiliary contact analysis of replaced parts. Both of these are a result of the prior actions from DER 93-0312.
One other recent action was the initiation of work requests to replace seven (7) auxiliary contacts. These had unexplained failures since 1989 but had not been replaced. They will be replaced and sent to TES for additional analysis.
The MCC auxiliary contact multi-discipline team's efforts will continue in order to determine the root cause. Additional actions that require further evaluation include:
Continued review of past and future unexplained failures. This includes use of the troubleshooting guide and TES analysis of replaced auxiliary contacts.
- Future review of DER 93-0312 to determine effectiveness of the main contactor corrective actions.
Enclosure to NRC-95-0033 Page 5 of 8 Review of alternatives developed by the team to modify or replace the auxiliary contact or the MCC in its entirety since it is obsolete and replacement parts are difficult to obtain.
Corrective Steps That Will Be Taken to Avoid Further Violations The first fundamental problem was recognition of the scope of the problem. Maintenance recognizes its role in identifying repetitive problems. The Preventive Maintenance group has recently written a number of DERs for components with repetitive unsatisfactory as-found data. Information on monitoring Barton switch performance data is being communicated to the I&C supervisors. The position of component engineer is being added to that organization. It is Detroit Edison's expectation that component engineers within the maintenance department will be in a better position to recognize recurrent component problems.
This will also promote ownership with respect to component problems and their resolution.
The second fundamental problem was identification and resolution of existing issues. The Technical Issues Report will be utilized more in the future to document and track component level problems. In the past, this report was geared more towards a specific system performance or problem. It has been proven a useful tool with respect to resolving those issues. Issues like the auxiliary contacts and the Barton Differential Pressure Switches, that are repetitive problems encompassing numerous systems, are being evaluated and added to the Technical Issues Report.
The third fundamental problem was prioritization of engineering work load. Detroit Edison has recently initiated the PERC. The charter of this committee is to prioritize major issues and activities. This type of committee has been found to be successful at other utilities in controlling and properly prioritizing top issues. Another area where prioritization is being redefined is for DER significance. A board ofline managers convenes each morning to assign the significance level and management involvement in DERs. This significance will determine how far the problem resolution needs to be taken. For equipment related DERs, Safety Engineering is performing a search of the last two years for past corrective maintenance and DERs. This information is provided to the DER evaluator for consideration of a recurring equipment problem.
Additionally, training will be provided to I&C repairmen by Barton on lessons learned from experience on calibrating and maintaining Barton switches.
i Enclosure to NRC-95-0033 Page 6 of 8 J
Date When Full Compliance will be Achieved As discussed above, the reviews of the additional 36 Bartons will be complete by December,1995. The schedule for completing these reviews takes into account Detroit Edison's desire to perform the reviews to the maximum extent practical with Plant Engineering personnel. Avoiding the use of contract engineers where possible will ensure that the knowledge and experience gained by engineers during the review process will be retained. All necessary design changes will be processed and i
prioritized through the PERC. In the interim, Detroit Edison will continue to monitor individual Barton performance during system surveillance testing and the calibration process. The corrective actions that are being taken will help to ensure continued compliance with 10 CFR 50 Appendix B, Criterion XVI.
II.
Statement of Noncompliance 2 10 CFR Part 50, Appendix B, Criterion XV, requires, in part, that measures shall be established to control materials, parts, or components which do not conform to requirements in order to prevent their inadvertent use. This measure shall include, as appropriate, procedures for identification, documentation, segregation, and disposition.
f Contrary to the above, from about May 1989 to December 1994 adequate measures were not established to ensure that only approved and authorized lubricants were used to clean / lubricate safety related electrical components. Consequently, licensee maintenance technicians used non approved cleaners such as Cramolin and up to ten other cleaners to clean / lubricate safety related electrical components.
Reason for the Violation Administrative controls for the procurement, storage, identification and issue of cleaning agents / lubricants such as Cramolin were adequate.
Cramolin was approved per the Spare Parts Reference System (SPRS),
however, this approval was for Balance of Plant (BOP) applications only.
The manufacturer's label on the can also stated that it could be used for contact cleaning. Furthermore, the Detroit Edison material identification label affixed to the can coded Cramolin as NQ, indicating that it is not to be used on safety related components without specific approval. The violation was the result of a weakness in the control of the material after it was issued for use.
i Enclosure to NRC-95-0033 Page 7 of 8 The program to ensure compliance with Criterion XV after the point of issue, places the responsibility on the user, i.e. craft or maintenance personnel, for ensuring that the cleaner / lubricant is approved for the specific application in question. This is similar to the expectation that a qualified, trained maintenance worker will use the correct tools on a job.
The specific reason for the violation in this case is that these consumable material control program requirements were not adequately communicated to maintenance workers.
Corrective Stens Taken and the Results Achieved With regard to Cramolin several actions were taken to preclude future inappropriate use. Procedure 35.306.008,"Mctor Control Center Load Compartment," Revisior 29 dated Februsy 22,1994, had a caution statement added that identified a potential failure of the main contactors caused by the presence of residual cleaners such as Cramolin. This was done to ensure Cramolin was no longer used on the main contactors.
Cramolin was never an authorized cleaner for auxiliary contacts in safety related applications. For the past six months, auxiliary contacts that have failed have been examined to determine if any conditions existed which could have contributed to their failure to operate. No Cramolin was noted on auxiliary contacts. Cramolin has only been found on a few of the main contactors that were cleaned during the recent refueling outage.
Corrective actions taken also included removing Cramolin from Detroit Edison stock at Fermi 2 and adding it to the Restricted Engineering Component List (RECL) so that it will not be purchased. All Requisition On Stores (ROS) issued for the Cramolin stock number have been evaluated with respect to their application for safety-related concerns.
A review of the consumable material program at Fermi 2 determined that the control of consumables up to the point of issuance to the craft was acceptable. The primary area of concern identified was the expectation regarding the correct use of consumables by the craft once they were issued. It was concluded that these expectations are not unreasonable and are within the bounds of what is considered to be normal craft capability provided that they are clearly communicated. To address this concern, a series of discussions were held with Electrical Maintenance personnel to stress the use of approved material on safety related equipment.
l Enclosure to NRC-95-0033 Page 8 of 8 The review did not identify any specific instances of inappropriate use of the other ten cleaners noted in the violation. This review is continuing.
The corrective steps taken and to be taken to address the issue of craft awareness of their responsibilities should be effective in controlling the inappropriate use of these cleaners and other consumables as well as Cramolin.
In summary, several specific actions were taken to preclude the inappropriate use of Cramolin on site. To address the broader concern, discussions were held with Electrical Maintenance personnel to communicate expectations regarding the use of an unapproved material on safety related equipment. The primary objective of these discussions was to ensure that personnel were aware of their responsibilities in this area.
Corrective Steps That Will Be Taken to Avoid Further Violations An action plan has been developed to implement the necessary corrective actions. The following actions are to be taken:
1.
Training will be provided to maintenance personnel, including mechanical maintenance, regarding their responsibilities on the use of consumables.
2.
Procedure 35.306.008, " Motor Control Center Load Compartm:nt,"
will be revised to delete the use of Cramolin.
3.
Incorporate requisite consumable control measures into alte orientation and requalification training manual.
4.
Detroit Edison will continue to evaluate the past use of the other ten cleaners and consumables (other than Cramolin) noted in the violation.
Date When Full Compliance will be Achieved All actions will be complete by July 31,1995.
-