ML20249B931
| ML20249B931 | |
| Person / Time | |
|---|---|
| Site: | San Onofre  |
| Issue date: | 06/22/1998 |
| From: | Nunn D SOUTHERN CALIFORNIA EDISON CO. |
| To: | NRC OFFICE OF INFORMATION RESOURCES MANAGEMENT (IRM) |
| References | |
| 50-361-79-10, 50-361-80-09, 50-361-80-9, 50-361-98-05, 50-361-98-5, NUDOCS 9806250061 | |
| Download: ML20249B931 (44) | |
Text
i.
IIg sourHERN CAufoRNIA l
E EDISON
"*ia ' ~" "
i Vice President An EDISON INTERNATIONAL
- Company June 22,1998 l
U.S. Nuclear Regulatory Commission l
Document Control Desk '
Washington, D. C. 20555 Gentlemen:
Subject:
Docket Numbers 50-361 and 50-362 Response to Inspection Report 98-05 Regarding Linestarters l
San Onofre Nuclear Generating Station, Units 2 and 3 i
Reference:
Letter from Thomas P. Gwynn (USNRC) to Harold B. Ray (SCE) dated May 21, L
1998 i
The referenced letter transmitted the results of an inspection conducted between March 23 and April 17,1998. The inspection was associated with the inoperability of the Unit 2 turbine-driven auxiliary feedwater pump and thejamming of a linestarter mechanical interlock. The referenced l
letter outlined additional information needed by the Nuclear Regulatory Commission (NRC). The j
requested information was:
i 1)
Supporting information for the SCE position that "this [the failure of the mechanical
. interlock] was not a matter that was avoidable by reasonable quality assurance measures or l-management controls."
2)
"Whether there were prior opportunities to detect and correct the condition that rendered the linestarter inoperable."
l l-3)
"Whether other equipment could be affected by this condition."
l The enclosure to this letter provides SCE's responses to the requested information. In summary, the failure was discovered during the planned replacement of the component. As a result of the failure, SCE proactively accelerated the planned replacement of the remaining components and conducted an extensive analyses to determine the root cause of the failure. The 1998 mechanical
' interlock failure (HV9305) was the first and only failure due to grit. The failed component had i
been installed more than 20 years ago and has been determined to have a design uniquely vulnerable to the condition which led to this failure. The HV9305 mechanical interlock is conservatively estimated to have had more than 100 actuations prior to being found failed. There l
San Clemente. CA 92674-0I2S i e V v 5
[ h-
\\s P. O. Box 128 4
,n,
949-368-1480 9906250061 99(Mn22 r
\\
Fax 949-368-1490 -
PDR ADOCK OS000361 1
i -
6 PDRr 4
I -,
Document Control Desk June 22,1998 are 172 reversing linestarters with this type of mechanical interlock and it is estimated that there have been more than 17,000 actuations of this type ofinterlock with only the one grit-related
' failure.
The mechanical interlock is unique compared to other plant equipment in that it is not sealed, it is not visible / accessible during routine inspection, the interlock contains relatively soft parts, and the operating forces are relatively small. The interlock is located on the backside of the reversing linestarter. The linestarter is housed within a motor control center cubicle which has a gasketed door. The interlock is therefore exposed to the environment only if the cubicle door is open. The grit has a fine " dust" appearance to the naked eye.
The Failure Analysis Report (FAR) identified the root cause of the failure as being grit that resulted in the subsequent galling of a soft part. The FAR concluded that the source of the grit was from guniting activities prior to 1993 and most likely, prior to startup. Based on the results ofreexamining the FAR conclusions, SCE has concluded that the grit deposition occurred during plant construction. The grit has the same elemental, visual, and size characteristics as unmixed Portland cement. The deposition pattern of the grit was indicative of a slow even deposition.
Constmetion activities included having large quantities of unmixed Portland cement on site, as well as having the Motor Control Center (MCC) cubicle doors open for wiring and terminations.
SCE has concluded that there was no breakdown of the quality assurance / management controls during construction, or during subsequent plant operation. The quality assurance / management
-l L
controls during construction, including those related to cleanliness, were consistent with NRC requirements and industry standards at the time. Periodic inspections by Bechtel and Edison personnel were performed to assure that the requirements were being met. NRC Inspection Reports from construction were reviewed and no non-conformances relative to the MCCs were identified. The quality assurance / management controls during operation are based on the same NRC requirements and standards as during construction. SCE has complied with the i
requirements applicable to the components, from construction and installation practices, to inspections during preventive maintenance and the required Technical Specification surveillance.-
SCE has concluded that there were no overlooked opportunities to detect and correct the condition that rendered the linestarter inoperable. This is based on a review of a variety of potential sources, both internal and external to SCE. Root c,ause reports, NRC Information Notices, and Generic Letters, as well as specific discussions with the MCC vendor, are examples of the sources reviewed.
SCE has concluded that there is no other equipment affected by this condition. This is based on an assessment of other equipment that could be susceptible to the same failure mechanism as the u
0 t
r a j
Document Control Desk June 22,1998 mechanical interlock. As described above, the interlock is unique compared to other plant equipment. SCE has found no evidence that the grit has caused a failure in plant equipment other than the one mechanicalinterlock.
The referenced letter provided Southern California Edison (SCE) the opportunity to request a pre-decisional enforcement conference or respond to the apparent violation. SCE requests a pre-decisional enforcement conference.
SCE believes that the enclosed additional information will be beneficial to the NRC in determining that the provisions of NRC Enforcement Policy section VI.A are appropriate and an apparent violation should not be assessed. If there is any additional information you require, or ifyou have any questions, please contact me.
Sincerely, i
N i
e--=-
)
Dwight E. h nn Vice President Engineering and Technical Support l
Enclosure:
Linestarter Request for Additional Information Response cc:
E. W. Merschoff, Regional Administrator, NRC Region IV J. Lieberman, Director, Office of Enforcement l
J. W. CliPord, NRR Project Manager L
K. E. Perkins, Director, Walnut Field Creek Office, NRC Region IV.
l J. A. Sloan, NRC Senior Resident Inspector, Units 2 and 3 i
l I
I L__--_--___-------_---_----_-----_--__------.-
ENCLOSURE l
LINESTARTER REQUEST FOR ADDITIONAL INFORMATION RESPONSE l
I REOUESTED INFORMATION l
The cover letter to inspection Report 98-05 (Reference 1) requested the following:
l
" please provide additionalinformation to support your position that this was not a matter that was avoidable by reasonable quality assurance measures or management controls. Additionally.
l we request that you provide your perspective on whether there were prior opportunities to detect and correct the condition that rendered the linestaner inoperable, and whether other equipment could be afrected by this condition."
BACKGROUND Previous Correspondence Licensee Event Report (LER) 1998-003 (References 2 and 3), described the discovery on February 5,1998, of a Unit 2 jammed reversing linestarter mechanical interlock associated with valve 2HV9305. The discovery was made during a planned replacement of the interlock. Based on the condition of the interlock when it was removed, SCE assumed that the valve would not have opened upon demand.
The Emergency Core Cooling System (ECCS) valve associated with thejammed mechanical interlock is the inboard Train A containment emergency sump isolation valve. This valve is normally closed and receives a recirculation actuation signal to open. The valve had been stroked
(
successfully on January 6,1998, as a pan of the valve's Insersice Testing requirements. Unit 2 shut down on Januanj 24,1998, for a planned midcycle outage. For approximately 27 hours3.125e-4 days <br />0.0075 hours <br />4.464286e-5 weeks <br />1.02735e-5 months <br /> j
during the time period ofJanuary 6,1998 to January 24,1998, the redundant train of recirculation was not available because of planned heat exchanger maintenance.
The planned replacement of the mechanical interlocks on all safety-related reversing linestarters was a part of the corrective actions associated with an unrelated 1995 f ailure. The 1995 failure was attributed to manual cycling performed as a part of preventive maintenance practices. The l
replacements were scheduled to be completed by the end of Cycle 10 for each unit. As a result of the 1998 failure, SCE conservatively accelerated the planned replacements. The replacement of l.
the remaining mechanical interlocks on both units was completed in March of 1998 (last three Unit 3 new interlocks installed).
l l
Equipment Information The safety-related Motor Control Centers (MCCs) were purchased from the Square D Company.
The failed reversing linestaner mechanical interlock was located in an MCC 2BE cubicle. MCC 1
ENCLOSURE LINESTARTER REQUEST FOR ADDITIONAL INFORMATION RESPONSE 2BE is located on 50' of the Control Building. The 50' elevation of the Control Building includes four separate switchgear rooms which house the safety-related Units 2 and 3 Train A and B MCCs. Each reversing linestarter has an electrical and mechanical interlock. The function of the interlocks is to prevent both contactors from being energized simultaneously, thus rendering the component non-functional. The mechanical interlock is mounted on the backside of the linestarters. In the installed condition, the mechanical interlock and its moving parts are not entirely visible or accessible. Each MCC cubicle has a gasketed door (see Figure 1). When the cubicle door is open, the mechanical interlock is exposed to the environment via openings on the front, top and sides (see Figure 2). Typically, an MCC cubicle comes with the linestarter assembly (including the mechanical interlock) installed. Only the plant specific wiring and terminations have to be completed afler installation of the MCC.
Submitted Failure Analysis Report Reference 4 provided detailed information with respect to the potential increased core damage risk associated with thejammed interlock, as well as the Failure Analysis Report (FAR) of the jammed mechanical interlock. The FAR concluded that thejammed interlock was caused by the presence ofgrit between the interlock metal guide post and the Delrin (a relatively soft plastic) sliding cam that resulted in the subsequent galling of the sliding cam. The grit was very fine (10-100 microm) ed tan in color. (For comparison, Portland cement panicles are in the same size range.) To the nak J eye, the grit is not distinguishable from common dust. However, if specifically looking for it, the grit does have a specific coloration and physical appearance. The grit also tends to stick where it lands.
The FAR concluded that the grit was consistent in color, visual appearance and elemental makeup to tan colored gunite. Gunite is used as a means to prevent soil erosion. The gunite is a mixture containing Portland cement. The tan color is attributed to pigment that is added so that when applied, the appearance is like dirt rather than like cement.
In preparing the FAR, significant effort was expended to try and determine the exact source of the grit. Samples were taken from replaced interlocks, spare MCC cubicles from both units, stmetural features in the switchgear rooms, and the inside fiberglass lining of the HVAC duct that supplies one of the switchgear rooms. The grit was not found on the structural features of the rooms, the lining of the HVAC duct, or equipment that had been replaced or disassembled and cleaned. However, equipment areas that were not normally accessible and had not been subject to disassembly, did show signs of the grit. Based on this information, the following conclusions were drawn: once the grit was deposited the grii did not migrate to other places, the grit deposition was not an on-going phenomenon, and the grit was not introduced through the HVAC system. In addition, one of the interlocks examined had been replaced in iW3. This particular interlock showed no traces of the grit when inspected in 1998. Based on this evidence, the FAR concluded the introduction of the grit occurred prior to 1993 and most likely prior to plant startup.
2
O l
l l
ENCLOSURE LINESTARTER REQUEST FOR ADDITIONAL INFORMATION RESPONSE Regarding the effects of the grit, the FAR concluded that the grit affected the interlock, rather than the other MCC components, because the interlocks have unique features which make them susceptible to grit induced problems. The mechanical interlocks have an unenclosed structure that is not visible / accessible without disassembly, soft parts that could gall, and moving parts that are subject to relatively low operating forces.
Site Construction San Onofre Nuclear Generating Stations (SONGS)is located on a relatively small area ofland on the beach, with Interstate Freeway 5 in close proximity. The concrete batch plant was located at the south end of the plant on a blufr. Steep excavation had to be performed for site construction, and over the years, activities were performed to prevent slope erosion.
An area on the east side of the freeway (commonly referred to as the Mesa) was utilized for warehousing of equipment prior to installation. The circulating water intake and outfall piping was constructed on the Mesa, and then moved in sections to the plant side. This necessitated concrete work to be performed on the' Mesa as well.
It was typical for the equipment to go from the warehouse to the fmal installed location, within a few days. Control Building equipment was installed in various stages of the construction process.
Some equipment was installed prior to the internal plaster walls being put in place. In addition, a consvuction opening existed in the west wall of the Control Building at elevation 56' to facilitate constn.ction rdvities within the building.
Once switchgear was installed, the field wiring 'vork began. As the wiring was pulled, the wires were coiled in the cabinet. Once the wires were pulled, the termination crews would terminate i
the wires. The switchgear was not tested and turned over as a single component. Startup testing and turnover was performed as individual pieces of equipment or groups of equipment were j
ready. Because the Control Building was common, the majority of Unit 3 equipment was installed in the same time frame as the Unit 2 equipment.
Plant Operation Date The low power license for SONGS Unit 2, was issued on February 16,1982. The full power license was issued on September 7,1982.
REEXAMINATION OF FAR CONCLUSIONS Reference 1 indicates there are " questions concerning the quality assurance controls that were required and the actions that were in place to preserve and protect the equipment during the time 3
t ENCLOSURE LINESTARTER REQUEST FOR ADDITIONAL INFORMATION RESPONSE when the grit was introduced into the linestaners." As discussed in the Background section, the FAR concluded that grit was introduced prior to 1993 and most likely prior to plant startup.
Because of a lack of dermitive time frame relative to when the grit was introduced, SCE reexamined the FAR conclusions, in order to better identify the source and define the timing of the introduction of the grit.
Efforts were made to better identify activities that might have generated the grit, as well as the timeframe of the activities. An extensive review of construction-related documents was performed to define when specific activities occurred for MCC 2BE (e.g., receipt, installation, turnover) Material samples were taken to validate the information relative to "guniting" being the source of the grit. Dust samples from equipment around the station were taken to determine
'if the grit deposition was localized, unique to particular components, or unique to particular time periods. Attachment I provides details with respect to the efrorts undertaken, and the conclusions reached.
l The additional etTorts resulted in the following conclusions:
- 1) The grit material deposition is not unique to the switchgear rooms, the 50' elevation of the Control Building or the MCC manufacturer.
l.
- 2) The grit deposition was not associated with a single event.
l
- 3) The grit source is a Portland cement-based activity.
' 4) The deposition took place when the cubicle door (s) were open for a prolonged period of time.
SCE has concluded that the deposition did take place during construction activities. The potential sources during construction, which are broad in nature (batch plant, grouting, and "guniting"), would not be expected to have specific precautions relative to protection of equipment. Therefore, the quality assurance measures and management control associated with equipment protection during construction are the focus in answering the first request for additional information.
REOUEST1 Provide additional infomiation to support your position that this was not a matter that was avoidable by reasonable quality assurance measures or management controls
. RESPONSE TO REOUEST 1 SCE has researched the quality assurance measures and management controls that were in place during the construction of SONGS Units 2 and 3, as related to the introduction of the grit into the linestarter interlock. Attachment 2 provides the details of the research and the conclusions L
reached.
l 4
L
l.
ENCLOSURE LINESTARTER REQUEST FOR ADDITIONAL INFORMATION RESPONSE Title 10 of the Code of Federal Regulations, Part 50, Appendix B, (Reference 5) Criterion 11,
" Quality Assurance Program," and Criterion XIll, " Handling, Storage, and Shipping," provide the regulatory basis for the quality assurance controls required to preserve and protect the equipment during construction (as well as today). During construction, the day to day quality assurance j
efTorts were under thejurisdiction and quality program of Bechtel Power Corporation. ANSI l
N45.2.2-1972, (Reference 6) was used to implement the 10 CFR 50, Appendix B, Criterion XIll requirements.
l The ANSI N45.2.2 standard includes four component classifications, dependent on the protective measures required, to prevent damage, deterioration, or contamination. For example, fuel is classed as a Level A componera, while structural steelis classed as a Level D compnent. The MCCs were classed as Level B components in accordance with ANSI N45.2.2. ANSI N45.2.2 does indicate manufacturer recommendations shall be considered. The MCC manufacturer did not include any requirements above those specified in ANSI N45.2.2.
During construction, afler equipment is installed, ANSI N45.2.3 (Reference 7) applies. This standard delineates five zones relative to cleanliness requirements. The switchgear rooms were designated as Zone IV. ANSI N45.2.3 for Zone IV requires regulating the use of tobacco and eating thr material and equipment protection or for health and fire hazards, in addition to maintaining good construction site housekeeping practices. No other specific requirements are stated. The classification of the switchgear rooms as Zone IV was reviewed and deemed appropriate. Zone Ill requires specific personnel and material access control in addition to the Zone IV requirements, while Zone 5 only requires good construct;on site housekeeping practices.
The procedures which implemented both ANSI Standards did have provisions for periodic l
inspection of the areas. Evidence that inspections occurred were found.
During construction, the NRC conducted inspections and issued Inspection Reports. Several of the reports include mention of specific tours of areas to determine if the storage, handling and protection requirements were being met. In one such report (Inspection Report 79-10, dated April 20,1979), specific reference to inspection of the storage, handling and protection of installed safety-related electrical components was mentioned. The inspection found that the storage and protection of safety-related electrical switchgear met the requirements. In Inspection Report 80-09, dated July 10,1980, storage conditions in the permanent warehouse were fbund to meet or exceed the requirements of the standards.~ This particular inspection report did mention non-conformances, unrelated to MCCs.
None of the standards or procedures reviewed specifically delineate requirements for keeping dust out of MCCs. The requirements are general in nature. Around a construction site, in areas such as the Control Building, activities such as cable pulling, equipment installation, tratlic, etc., would tend to generate dust. Therefore, the presence of a minor accumulation of grit, which is visually similar to dust, would not be cause to question equipment capabilities.
5 l
l
ENCLOSURE LINESTARTER REQUEST FOR ADDITIONAL INFORMATION RESPONSE Personnel familiar with other nuclear construction sites were contacted to determine if any l
additional precautions were taken at other sites that might have precluded deposition of the grit on the interlock. No additional precautions were identified. The SONGS procedures were deemed typical.
Based on the information found in the applicable ANSI standards, the implementing procedures, the documentation that provides evidence ofprocedure implementation, and the NRC Inspection Reports, SCE concludes that there were no programmatic failures of the quality assurance or management controls during plant construction relative to the introduction of the grit into the interlock. Furthermore, SCE believes that implementation of the requirements in the ANSI Standards would not have prevented the deposition of the grit on the interlock, nor would it have provided for the detection and removal of the grit.
While SCE has concluded that the grit was deposited during construction, the quality assurance measu:es and management controls during operation reference the same standards and are similar to those during construction. The response to Request 2 provides an assessment of the prior opportunities to detect and correct the condition during operation. No overlooked opportunities were found. Therefore, SCE has concluded that there has been no breakdown in quality assurance and management controls during operation relative to the MCCs.
REOUEST 2 Provide your perspective on whether there were prior opportunities to detect and correct the condition that rendered the linestarter inoperable.
RESPONSE TO REOUEST 2 The Response to Request I details the quality assurance and programmatic controls that were in place, and SCE's conclusion that they were consistent with industry practice and vendor requirements. Thus, SCE believes that during the pre-operational phase, there were no prior opportunities to detect and correct the condition that rendeced the linestaner inoperable.
The Reference 1 Inspection Repon provides the inspector's view that during operation there were no missed opportunities to detect and correct the condition that rendered the linestarter inoperable. The report states, in part: "..the inspector did not consider that a visual inspection of the breaker cubicle would have reasonably been expected to result in identification of the grit as an adverse condition.
Similar testing had been performed on the other valves with line starters, providing an even broader basis that there was not a problem with the dust or grit in the cubicles.
The inspector considered that the level of expen attention given to the 1995 failure evaluation would have resulted in an assessment of the grit ifit was reasonably identifiable as being unusual or contributing to the 1995 failures."
6
ENCLOSURE LINESTARTER REQUEST FOR ADDITIONAL INFORMATION RESPONSE SCE evaluated several potential post-operation opportunities to determine if an opportunity to detect and correct the condition that rendered the interlock inoperable was overlooked. The opportunities examined, and SCE's conclusions relative to those opportunities, are summarized in. This included reexamining the 1995 root cause evaluation, reviewing other SONGS failure analyses, reviewing past SONGS LERS, contacting the linestarter manufacturer, examining testing results associated with valves, examining NRC documents, and examining industry LER abstracts to determine if an opportunity to detect and correct the condition was overlooked. Based on the evaluation, SCE concludes that no missed opportunities exist.
The 1995 linestarter failure was attributed to a past preventive maintenance practice that resulted in significant side loading of the sliding cam. The post-maintenance test following the preventive maintenance would result in a blown fuse. The photographic evidence as well as the actual physical linestarters from the 1995 failure were reexamined, and the analyst determined that the failure was in no way associated with grit contamination.
Cycling of the valves (and the associated cycling of the interlocks) may occur due to testing, maintenance activities, or plant operational considerations It is conservatively estimated that the HV9305 mechanical interlock had more than 100 actuations prior to being found failed. A relative comparison of the estimated number of actuations that HV9305 experienced (prior to the interlock failure) to the estimated number of actuations that other valves powered from MCC 2BE experienced was made. The comparison showed that there were interlocks with more actuations that had not '. ailed, despite being exposed to the same grit. Furthermore, one particular interlock was examined and showed more grit (and had galling) than that seen in HV9305, and that particular interlock had not failed prior to its replacement. There are 172 Square D safety-related reversing linestarters with a mechanical interlock in use. HV9305 is not typically stroked, e>. cept for testing or maintenance activities. Conservatively, if each interlock had the same estimated number of actuations as the HV9305 interlock, there would have been over 17,000 actuations with only the one grit-related failure.
l Based on the review of potential opportunities, SCE concludes that there were no overlooked opportunities to detect and correct the condition that rendered the linestarter inoperable.
REOUEST 3 Could other equipment be affected by this condition?
RESPONSE TO REOUEST 3 SCE performed a review of electrical and mechanical equipment to determine if potential grit contamination could affect operability. This review included an assessment of typical SONGS electrical and mechanical equipment. This assessment is provided in Attachment 4. For each type of equipment, the assessment includes a generic discussion as to how equipment design 7
ENCLOSURE LINESTARTER REQUEST FOR ADDITIONAL INFORMATION RESPONSE i
features preclude grit-related failures. In addition, the assessment provides a discussion of the SONGS programs (e.g., Root Cause, TS Surveillance, Preventive Maintenance, etc.) that would likely identify grit-related equipment issues.
The grit poses no threat to static components because no frictional surfaces are involved in component operation. The grit is nonconductive and is chemically inert unless there is moisture i
present. The MCCs are inside a building and not exposed to moisture. The only possible i
negative effect of a thin layer on static components, would be a slight decrease in heat transfer from the surface of the component in question. This would be no worse than the effect from a thin layer of any other type of dust deposit which may develop between preventive maintenance activities. Tenninations would not be affected because the grit is nonconductive and the termination surfaces are in direct contact with one another. Since it has been concluded that the grit deposition has stopped, if the grit were to have been an issue relative to circuit continuity, the many " functional" tests of the terminations since operation would have detected it. Based on the above discussion, static components are not considered susceptible.
In summary, SCE's assessment concluded that the grit would only be an issue for equipment having a combination of exposed moving parts, small operating forces and soft friction surfaces.
Based on the review of typical plant electrical and mechanical equipment, only the type of interlock found in the reversing linestarter satisfies these prerequisites.
The failure of the linestarter mechanical interlock is unique in that MCC cubicle doors were open to the atmospheie during construction (for wiring and tenninations), which allowed the grit to enter in a suflicient quantity to be deposited on the moving parts of the mechanism. Because the interlock is not normally visible /acccssible without disassembly, opportunity to observe the grit was not available. Even if the interlock was examined, the dust like appearance of the grit would not lead to questioning the ability of the equipment to function properly. The interlock is a I
relatively low force device with a relatively soft part compared to many plant components. All of the similar interlocks utilized in SONGS Units 2 and 3 MCCs have been replaced, and therefore, are no longer subject to grit contamination. The surveillance testing, preventive maintenance programs, and other monitoring programs at SONGS have resulted in numerous (many J
thousands) actuations and inspections of components since plant construction. SCE has found no l
evidence that grit has caused a failure of any plant equipment other than the one mechanical interlock.
l 8
l w-____--___
E ENCLOSURE
' LINESTARTER REQUEST FOR ADDITIONAL INFORMATION RESPONSE REFERENCES
Report 50-361/98-05; 50-362/98-05," dated May 21,1998
- 2. Letter, R W. Krieger (SCE) to Document Control Desk (USNRC), " Licensee Event Report No.1998-003," dated March 9,1998
Nos.1998-001-01 and 1998-003-01," dated April 17,1998 i --
l.'
Supplemental Information," dated April 7,1998 i
5.
ANSI N45.2.2 - 1972," Packaging, Shipping, Receiving, Storage and Handling ofitems for Nuclear Power Plants"
- 7. ANSI N45.2.3 - 1973," Housekeeping During the Construction Phase of Nuclear Power Plants" 8.
NRC Regulatory Guide 1.38," Quality Assurance Requirements for Packaging, Shipping, j.
Receiving, Storage and Handling ofitems for Water-Cooled Nuclear Power Plants," dated March 1973 l;
9.
NRC. Regulatory Guide 1.39, " Housekeeping Requirements for Water-Cooled Nuclear Power j.
. Plants," dated September 1977 i:
ATTACHMENTS _
4 L
- 1. Reexamination of the FAR Conclusions
- 2. Quality Assurarice Measures and Management Controls
- 3. Linestarter Prior Opportunities to Detect and Correct
- 4. Impact of Grit on Other Equipment 1:
9 t -.
e
(
ENCLOSURE Y;9%
I.
f a
.N gI ^ L' ':igf? - -
l
(
i, sQ s'
.r w t
- b.,, > r ll [".
j l
$1U
\\
l
(.a Q f.sG e r
f
- 3 I
j
'A... c!.
.i 3
7
' 4,,'.
jp ~< l%N
.., ;c; g. l ;nf-
....[,
j.)
1.' jm? I j'h-
{
,...w,
- r..
- p g
g y;. <
' l *#. '. ' _.,,
,,q' V {_f If}D'5' l
4
' s Ml; 6v...@, g.
m(
l'
'..{'t]
l$ '
^siU
- l : % e.ph.,.
l
+ :
..) 7
,;j) f'??'p?.Qt$y., '
I l
p-(i,5 Q s,q -
2..
- "' i y
i
- - ((^.., }.k l
l 1
' $; 6,';!f.);
I
?f5h?
l Q: (\\f
.R-
.w
. gr
- g, s
2.l?' :.-
.I.' M:'S
<. n: n
.-f ',
t.w?
n b
,g; g l
'<n
. M; 7p.; ;.d.
y_n I
l Figure 1 Motor Control Center (MCC) CUBICLE 1
l l
10
ENCLOSURE n m.
7,, -
q_
,egy; i rlC
.I ti f
v l
3, :, : :
- 93. s c
2 4... a
.. tf
- 3. - -
us.:e Aa.e
&Jt4 J44'
>: -6 2AL@dSSW e
Figure 2a MCC CUBICLE WITH LINESTARTER Mechanical lnterlock
~
. az a
.N ' -diff "p' r;:
l L>
s 2a u=.
aux au.auw,=x;a..=.uu.a.a:
1 Figure 2b REVERSING LINESTARTER 11 i
l
'4 l'
ATTACHMENT 1 REEXAMINATION OF THE FAR CONCLUSIONS Because the Failure Analysis Report (FAR), included in Reference 4, did not identify a specific source of the grit nor a definitive time frame relative to when the grit.was introduced, SCE reexamined the FAR conclusions, in order to better identify the source and define the timing.
l Efforts were made to better identify activities that might have generated the grit, as well as the l-
' time frame of the activities. An extensive review of construction-related documents was l
performed to define when specific activities associated with MCC 2BE (e.g., receipt, installation,
' turnover) occurred. Material samples were taken to validate the information relative to "guniting" being the source of the grit. Dust samples from around the station were taken to determine if the grit deposition was localized, unique to particular components, or unique to particular time periods.
L Figure I represents the obtained information relative to when specific guniting activities took place. The guniting activities were at various locations surrounding the plant. Individual guniting activities are relatively short in duration. Recollections of additional guniting activities,
- early in construction, could not be confirmed.
It was determined that "gunite" is applied utilizing one of two processes, dry mix or wet mix.
The terminology gunite and shotcrete is often interchanged. Dry mix refers to the process where
- the material (containing dry Portland cement and sometimes pigment), water and air are mixed at the nozzle mixing chamber. Wet mix refers to the process where the material and water is premixed. The material is then mixed with the air at the nozzle mixing chamber. The documentation indicates that the process used to apply gunite at SONGS, is in fact, the wet mix shotcrete (rather than the guniting referred to in Reference 4 FAR).
MCC 2BE was originally received at the warehouse in December of 1976. Installation of MCC 2BE was complete and signed-off in October of 1977. Wiring and terminations were L
accomplished between mid-1977 and mid-1981.
i Figure 2 and the accompanying Table 1, depicts the additional potential source and " dust" h
samples from equipment, as well as the sample analysis results.
The FAR identified that the tan color of the grit was a distinguishing characteristic. Based on the limited quantity of tan guniting activities during construction, and the close elemental match of the grit to concrete, unmixed Portland cement was examined. Unmixed Portland cement matches the grit in elemental, visual and size characteristics with the exception Jsmall amounts of Titanium.
Examination of small quantities of Portland cement under various lighting conditions, showed that Portland cement can also appear to have a tan coloring.
I
ATTACHMENT 1 REEXAMINATION OF THE FAR CONCLUSIONS l-Based on the information obtained and the analysis performed, several conclusions were made.
- 1) The grit deposition is not unique to the switchgear rooms, the 50' elevation of the Control
]
Building or the MCC manufacturer. This is based on finding material that matches the l
interlock grit in several other plant locations, including equipment provided by manufacturers L
other than the MCC manufacturer. The definition of match is based en elemental, visual, and '
)
size characteristics.
I L
- 2) The grit deposition was not associated with a single event. This is based on the uniform i-deposition found in the multiple interlocks previously exarained (which would be indicative of a relatively long, slow deposition) and finding material that matches the interlock grit in multiple plant locations.
- 3) The grit source is a Podland cement-based activity. The only Ponland cement-based source that the FAR examined was gunite from a hillside. This gunite matched the coloring of the grit found in the interlock (tan). However, depending on the lighting condition, Portland cement can have a tan coloration, as described above. The grit does include a small amount of titanium which is not present in Portland cement. However, the titanium could be from L
multiple sources including specific pigmentation activities or slight cormption of the samples due to other processes such as painting.
- 4) The grit deposition took place during Unit 2 and 3 construction. This is based on:
Grit in multiple plant locations.
a The opponunity to have several MCC cubicle doors open for prolonged periods of time
- (which would be required to get the deposition patterns seen) for wiring and terminations.
The interlock that was replaced in 1993 showed ne signs of grit.
'e Construction Portland cement-based activities did occur over a long period of time.
. Samples of cured Portland cement, even after being cmshed as fine as possible are still composed of panicles that are typically an order of magnitude larger than particles found in unmixed Portland cement. This would lead to a conclusion that the source of grit was from an unmixed i
source. Funhermore, the size differential in crushed finished product verus unmixed product, would dismiss activities such as jack-hammering poured Portland cement, chiseling, etc., as the source of the interlock grit.
Table 2 summarizes the activities which were evaluated as potential sources of the grit material.
L 2
l,l!I!lj lIlli
- ,i >y 9.n.
- i i;
,4
=?
- =
P A
M
+
YT
+
V I
I T
+
CA 1
G
+
T 1 N N
I
~
E ET
+
MR N I
I U U I
CG
+
AI G T F T
+
A E
+
R 2
CT O
H
+
(
S g
fv
+
N O
+
L T
a I
A
+
g-T I
mmmm.-
w S
=
=
h l1 G
N u_
a 5
A N
9 a
C D.
T i
t._
19 9
u 4
- N i
R
_ =-
s_
b i
r A.
A
=
c WS a
0 1
t 9
e 13 3
9 E
r h
E t
E8 o
,x MT m
T9 E
2 N
d i
n r.i T
c I
r E1 ac W e
R a
R 1
m mT C.
o ee ei s 9e se
.C G
D s
2 N
m5 t
8 E
a1I 1 e9P
- =
A E 9 E
VU n LF ea Nv G
nL C
.m E
d E
u2AU aL T
9 ts L
m9WS f
v 4D3co D
L a1 C
u
=
c s4EQ8 o R. MM
~,
1 E9 v
U 0SI -
n1ID1
- o. W Y L:
= w_
C - e N
9 s r.
N uUUM 7 7 sT S1 t
99,s A
a r N Ns86.
M e
cCe R
g
.RTo 0 os 0
a F
g Dh0 0
N 0.o
.u M. = LE At
? n 0
0 Es Uu AS
+
Lat R RE
.cM RRar u
N U
00 o ER N
s S1 1
O uTEC. C c c. NP C11 A =
EE y
nE1 R m =
GR i
&N At1 E
- c. W A O G o= o r cNe s
N m
rPFCCcc A
MS S
{-
,A q
RW^
g s
g-
_ r c,
- y a
k
-c
- .k g
+
%%g.N%
~
iN a
n mrO o
u 2
cS m
y.
r_
emI wD e
b wsE N
.3tF Q'
?,j
.l
.I!
11
.3 T
~[7 s
5-e s
v u
r3 r
iu c
l 0
q
]~
r e
I
=
f N 2nS T
n
('
i q:
q E E iEL MRsP i
1 l
Uc 7
CGaM l
.i A
AIT F vS
,g o
l l
L!
T s
z_
A t
l w
i I
r 8-e T
P 5
n I
N m
2 2
O o
[
e
,p:0 T
n T
A a
=
o T
i r
S s
e B
s U
u k.
s s
l I
l-
[
O
[
1
=
I 1
5 Y
/
l 4e l
41 23 1
f_.
E N
I l-0 S
c TA N
sL O
N 6
aP
_n T
n TI o
t M c
S o
A n
p A i
T uS r
a S
A D
A T
G a
B s
T A
s U
e sE N
O u
C T
R e
s u
s aR A
I s
L g U a O R
R
[
[
r S E
O E
Y r L E
A F
uA e
E eT E
W I
G E
_ A b.a"Iu18 G
i N E
E T
_Y E
rE N
R R
F' 1
A uT L
oO A
O E
G t P L
E-t mD N
1 A
6 uA U
C
]
C D
3 U
n7 g
W,1 N
s A
t N
o fP K
A
(
mm E
S E
oo 9O cc R
j 2_
a A T
=
~
lO_
O N
nO N
l OO S
[tlr I
N_.%
lf O
H 2
C 2
hT N
A l[
A S
'.b u
( :E o
-(
~
a D
i g^,
R t
s
,.3 W
g
,f
=
=
l;;
y* = j-L.
=_
4 eh p
t
[.
6 1
v lmS 3
.19_L
/
. V N
l.
7 g
O
/
m
/*
I.D E
U
~
g
~
s*
Tli)l ll l
,~
SAN ON0FRE NUCLEA 1
I 4
a se osco ruermy se catco mrmy ny ;
yppy.p5g
..w=a c==
N
\\
gr*
~ --
g i
"J x
7 u,
wh
@lE
~
g b
K.
ol e,8 o F esi,8 a
5 9
N 7
LEG FACIUTIES MAINTENANCE BLUFF STABiUZA PROJECT NUCLEAR CONSTRUCTION 1992 - 1 FACIUTIES MAINTENANCE U-1 SEA WALL RE CONTRACTOR UNKNOWN SAN DIEGO BUFF 1 CONTRACTOR UNKNOWN 1979-1980 CONTRACTOR UNKNOWN 1975 - OVERLAYED CONTRACTOR UNKNOWN SWITCHYARD SLOPE 1 (1) GEPERAL TIME SPAN PROV p.
REPRESENT MANY N)lVIDU
.y SOUTHERN CAUFORNIA E DISON
1 GENERATING STAil0N
- S3ec ATk@o#
t 00 gvaBS$"td a
NO 6tF#
9 40
+
f
~
cac
{C I i
~
4 f
j g,,,,
[
suomia I
.suosvoia tg
~
s 01 N^T*
l 0 000 0 "f
0
- - =
- a. -
/.%
. - -r s
g a
3 + ~ f + ( Tg iM U
l GULF DF $ANTA CATALINA PACIFIC DCEAN i
l l
l 4D @
l 1993-1994 SHOTCRETE m
5 JR SHOTCRETE 1985 & 1995 m 1 1984 - 1986 i
i ATTACHMENT 1 93 ssorenETt r-,
i snoToenETE 9-2s-79 ggg1
_3.
TNE SPAN CAN SHOTCRETE & GUNITING ACTIVITY MAP
= ACTMTES.
y ew J
es m SAN ON0FRE NUCL l
l SAN DIE (
x x
x x
x 9
y 4
ATCHISON TOPEKA ANQ l
i
~
n.- s f7" d
O ps?
7 aws f;'
-j l
iso e
"-*7*a wT y a
,m g 4
A v7
.j =L M y ~,j gg
'~
,c l
[
Ub '5}o 7
y[
,_l?,
4-
" l 20
(
' ^- R v
una ii WL 91.6
.k i
tear
"" g ar il 12 21 l
4
\\
\\
22 'O
==c' 0
/
g g -
2k
~~
x u
k-...,a g
- -2 a
x I
O
^=m=4t n.
- t a souwt,m cAurmn.:
I I AmmmAt t E DISO N
~
at hl N b k -- '
L g'pggiun" 3EERATING STATION CABD
%h1$$E5 EWAY
- ~
- ~
3:77 2 A FE RAILROAD 14/15 h
SUBSTATION
\\
-x 3'
SUBSTATION O
9 SUBSTATION C
tu 16
=
j
^
gD l7 SUBSTATION
~
y g"
J e,
s b1,3 -o 0C.
N p.68 q., "')
-g1
' m =-",0 0) pgg
[
O 3
TAM; BLDG 23l
- Tunawc nAToa stro r-4;oloio f
c R ( p]
o
,g an i
w 9frOfoEsooG1-02_.
iEND ATTACHMENT 1
=-
FIGURE 2 SNT TAPE SAMPLES POTENTIAL SOURCE AND EQUIPMENT
?TIAL SOURCE SAMPLES SAMPLES d
m
~
- ~ ~
s le k
o o
c T,
i e
ic a
ly n
n tr t
a p
r t,
p mT t
i s
a n
o u,
T.
T p
it t
r i i o
o m
iml e
f ei s
i s
g T.
e T, i dt t
a a
g r
l a
t g
r n
n u
T n
t cy r
a im m
lat i
r i
e it s
o a s s g a
i l
la r oi i
Y R
gnT
. c
,e d,
a a
k, s d
l it r
s i k tt.
Ta i t c
t t
t a e t
n t
R t
Li r l i a i n
n n
e c r
r r gt g
a o
o pa i
r aii im ga r r A
y ge gg s
r s
s c
c r
ep ga e
et M
la ee
,t c yl ed d
l h
h it, t
t t
n t
ct n
ana c n n
i al c c lt t t s
s r a a
s ig ig zr s
a a
o r
a r aaa a c
- y t s s
M A
U h
e lt,
- l l'a it, l
l
/i t
wd r r t s v
y t t t c as, t t l g
we t
t t lt s
l r
r r
l r a si o a
a.
e e ir i S
e a igl r
s s
aa i r r
s -
s v v is i
v d g gm h s s dt 6
T a
1 23 01 0
1 i -
31 3 )l s
C 5656 66
?
8 u4 u 05 L
01 01 00 X33 56 5 tn6 2l 3l 25 e
DH DH XX LLL Ac Ac NN U
l 1 1 i1 s
s n
n a
BB BB BB BBB BB B pB Bo Bo BB S
B 22 33 23 222 22 3 (3
- c 2 c 22 E
R r
S a
I eg S
h Y
c t
L iw A
m s
N o
e o
S h
A E
r t
r L
n a
P i
D t
e N
n g
M d
A m
c A
te e
h 1
a S
c t
i e
m w
E o
N l
l b
o s
P t
aO C
2 A
no TI U
T
/ T e
n r
1 P T
a i
t I d
N s
nR e
E C
ta M
C eC mS co P
M hE l
I c
C U
esm s
aD C
Q e
t E
h o t E M
Tr o L
AL A
P N
M m
O m m
o I
s s
s s
s T
e e
e e
A f
I l
l l
1 1
le r
S k
D ib 1 6 ic ic&
l c
c c
6 6
1 1
i4 5
N p
l A C H l
b 0 b2 6
6 b
ib 5 c
D 3
T e
o u
l uX u3 uA uN l
1 1
CL mB
- B C
C C
i r
E m
e B B B
B r 2 r3 B
B C2 3
C3 C
e e
C2 C2 t
2 a
n C & &
C & C n&
n&
C &
C &
t t
M i
S d*
M M
M1, e
e M
M 6
3 3
P e5 5
5 6
3 C
C 0 8
0 0
m3 e0 0
c0 eX3 e2 e2 I
r r
r d5 d5 r
r U
mE aD D aX aLL a1 a1 aA aN Q
aB pB B
pB pBB oB oB pB pB J 2 S2 3
S2 S22 L2 L3 S2 S2 E
D a
a n
e e
N o
A A
r r
i t
g A
p l
o o
d n
n i
g B
g g
i i
E r
g c
C s a l
l n
l d
ta a
d d
e d
t e e l
r r
R Dr B B
ib e
e B
B t
t g
U nA l
d o
u o
o n
n l
l l
r o
r r
r e
e O
o r
l t
B T
P P
t t
t S
it n
l C
3 C
C 4
6 n
n n
a o
o
'0 o
o 5
3 e
c C
L o
s A
L
'0 ie 2
5 0
5 2
2 5
D 5
U 8
8 U
U ITN n
E o
ei T
l tpa O
mg A n
1 2
3 4
5 6
7 P
ai S s e
D
si Tt i, T, l
t h a gs r
t t
t ni
/ a eh s,
i dl r
d e
g s
ws l
, i s c e
n n
,l t
s s
r r ipi c
Z Z
t t u
ii e
t a
Y R
ff it h r
i, r
e gg a
t i
r c p T
T oo gt Ti e
e T
i R
s nt i
gg oii T
T r
c c
i A
y nn nrd t
s r
r s
a a
o t
p t
t t
n i i t
t t
e i i
L l
r r
u a
a a l
,g g
g r
t, t,
t n
oo al r t
s aa e
e t
M A
cc n
n n
sl c
c r
e e
e t t d,
t, a
a d
m m
m i
U hh t
nii r
r igg a r r t
t e
e e
s i
s gg l l d,
c c
c t
t r
ir n
d d
d T
i a
n n
n 8 4 045 d
d s
a a
a L
23 255 t
t t
t l
l l
U AA NNN BB BBB a
lt l
o o
o t
l r
r r
a a
S 33 33 3 S
S S
P P
P ER 3
3 3
S I
d d
d n
n n
S a
a a
Y 2
2 2
L s
s s
i it t
t A
i n
n n
S N
U U
U E
A g
g g
L n
n n
P D
i i
i M
t r
r r
N n
u u
u e
d d
d A
S A
m i
l l
l a
a a
N m
n n
n E
e o
o o
C lb o
it t
it aO C
R i
a a
a U
rI r
r r
fT e
e e
O p
p p
i P o
o o
S d
d d
L t InR n
n n
A lo l
l n
n n
r r
r o
o o
o o
eC ai ai ai IT o
o o
t t
t mS dc d c d c c
c c
hE le u l
r le u N e u n
n n
r r
E i
i i
l l
l t
t t
c a s as a s T
n n
n aD t
t t
s n s n s n t
no no no O a
a a
I c
I c I
c P
T T
T E
tAL L
P A
M N
s s
O A
e e
I S
c l
T l
c r
i4 i&
I o
3 b
D f
T e
buA u4 C5 D
d l
N p
C B N
A e
E m
C3 C B uo s n a
i M
S C & C 3, m
te z r
et M
M 0
3 1
a P
8 05 2
6 1
ri I
e2 c25 2
8 4
cl k
a t
U aA ANN ob o
o r
r 1
1 1
t i c
r 1
pB pBB 2
2 h a l
Q S3 S33 4
8 4
S t
B M
s E
D a
a r
f ah e
t a
r g
n e
c o
N o
r n
c a
dd oet s ld l
l a
r i s A
A g
c i
Bo eAwe d
o A
tp h
g r
g c
at n
n c
h y
S dd c
imed s
i E
r it t
k f a r
c i
i tc a
o o
B Wc le t
e o
a a
rl g
Aa o C
i w
w s a a
a Ar aoe r
r e e r
r s
S e
er l n2 Ck R
Dr e
e V
n dh nt d o#
Oas t
t c s l
U nA n
n 0
V e
nc nd oi at r e t
i l
l O
o e
e 8
k s
h iw aCma l
o l
c el ah c P
P 4
4 E
e i
c t
s r o wt a b
wr)
S a
'5
'3 1
1 i
r k eAat ki g m
m f
c 4
6 c w L
o it t
e cnCkd A
L 3
3 no no it f
n a
o ic up e l
o s o
c a l
f lo2 n
l pr wOa o U
U UR UR U Bua BO(
r r B oa#
ITN n
E o
ei T
l tpa O
mg 8
9 0
I 2
3 4
5 n
1 1
1 1
1 1
P ai S seD
a n
n i
s C
S Z
Z t
l i
h h
e e
T u
s e
ig ig c
c a
a e
c h
h r
r t
t Y
R a
i r
i RA is n,
T T
Ti T
i s
t o
o o
o y
Z.
n n
e e
n n
M lan t
t t,
F F
t t
n n
n h
h n
n M
A e
e e
U m
m m
ig ig e
e h
h m
m e
e e
e e
S c
c c
h h
h h,
c c
T d
d d
c c
c c
d d
r n
n t
t t
t L
a a
a n
n R
=
a -
a a
a a
a m
m m
m lt t
l l
l l
U t
t t
r r
r r
r o
o o
o o
o o
o o
S P
P P
N N
N N
P P
E S
I SYLAN A
D t
N ne A
n 1
i e N m
l o
baO C
TI
/T 1 P t I r
r nR o
o o
o o
o lo lo r
r r
r r
r r
eC l
lo lo lo lo lo o
c c
o o
l o
o mS c
c c
c c
c c
n n
i i
i i
i i
i i
hE n
n n
n n
n n
i c
y y
aD n
n n
n n
n n
a a
t a
a a
a a
a a
r r
E T
T T
T T
T T
G G
tAL P
M A
S ro e
c c
T e
f l
ip ip p
l N
p de m
e e
e e
m m
E m
s n a
a a
l l
a uo p
lp p
lp S
e eti S
S S
m m
m m
M e
e e
e z e
S S
S S
e te t
t a
a a
a a
t t
P r
ri e
r r
r I
c cl t
t i c
t s
t.
t c
c n
n n
n n
U o
ob n
i i
ir ia o
o h
h a o
a a
a Q
S S s C
P P
P P
C C
t E
D k
n o
n t
r t p m
inc n
N o
c a o t
m e
s i
e nt u
od s
E ip w
id im#
A t
2 t t p
e u
s nt ae lo f
r c
l c
lo s
f e a lP w c
o ee c
o r
t ot md gt a
c a
C s a F
s s o
eg a e e
el l
r e el en e r,e io iss ner d n e e h
ew idt c e tsh n
nl R
Dr a
it i i
a z
a lpt uWu u
eo i
t t b
ib w U
nA ig ia) s.
h b ab e
t u
s s r
r b
s r
r l
t g H
r e u
ut O
o prDd ed eo T gT '
'0 n
'2 s
r i
w wh t
s y
f t
n i
t l
t t
a ueUe wb e nf ' nf e
,a
,e S
a
- n f
'0 w
'0 w r
i o 7
r mkeiMh f c c3 u 7 loe 6
b a
c e
mF L
o r a 1
1 5i 4
2d 2
3 3
lum o#l f
n b
t t
8 r
A L
BMD(eHo t
I od 1
l r
I e
u e
n Bw Clof UCis UB Ut Uh Uu UA o
IT N
n E
o ei T
l t
pa 6
7 8
9 0
1 2
3 4
O mn 1
1 1
1 2
2 2
2 2
g P
ai S s eD 7!!
l'
i' Il lljjl1 lllII J
~
n Z
wa i
r e
T, d
c a
o r
n t,
Z le S
t i
n iT Z
c a
e b
h s
t o
e a
ig i
i l
T T
r t
u n
c h,
t o
e a,
a n
s i
e e
T r
a i
c c
t Y
R C
C T
a a
o i
e r
r R
n s
h T
h d
e t
t, is ig c
A y
h n
ig ix a
n n
n, o
r h,
t, Z.
Z, Z.
o M
la n
n t
t t
t t
t t
n n
n mn n
n n
n M
A e
e e
e e
e e
i o
U m
m m
ei m
m m
m e
e e
r s S
c c
c u u e
e e
e c
c c
c plc d
d d
d d
d d
T n
n n
t n
n n
n n
s o L
a a
a oc a
a a
a l
l l
l l
l l
U n
d t
my t
t t
t r
r r
r r
o o
o n
o o
o o
l S
P P
P Aa P
P P
P ER S
I y
S t
Y itn L
au A
q N
no A
gn D
t i
N n
d e
ne A
m p
1 e N m
e r
d lb o
o aO C
o o
l r
TI c
lo
/ T n
c 1 P n
i e
i t I r
r r
nR o
o o
g r
r r
ng 8
l l
l n
o o
l Ti o
an eC o
o o
a l
l c
c c
r o
o o
t mS o
c c
c rh n
n n
i i
i t
n n
n og hE h
i i
i i
c y
y y
yl aD a
a a
ig n
n n
ad r
r r
r a
a a
r t
n E
G G
G B
T T
T Ga tAL PM c
t A
e c
r y
c S
t r
r l
l le D
e e
o o
o t
T e
f f
n l
p p
p f
hs d
d e
f N
p m
m m
u E
m a
a a
B r )e e
e m
ot s n s n e
a S
S S
f e uo uo C
o M
r i
i S
e e
e g
dc et et t
t a
t a d
P e
e e
ie e n e z e z n
t t
es r
o ri ri I
c c
c Dlpu c cl cl k
a r
r l
t t i t i c
t U
n n
o nmeyd ob ob o
r o
o h
aad n h a h a l
o Q
C C
S SS( a Ss S s B
P t
t E
D l
n n
n n
n o U
o r
N o
mr mit n
a e2 h
i A
t uia n
t ua e
e s
r#
u p
l i
r e
or e
lom wg o
i e
a f
c on o n i
E r
Ced c n s
e cw l i o z
t c
C n
ia eo e at g
t e t
s a el p nf b wn a
s s o l
t e e nr v e ed aa R
Dr ieg t
il e a r
aiS b n n bi f sl o
et wd wt n
s U
nA t
h ae hl k
e a
f p
ri r
i r
r s t o c e A
n O
o umn u
luyt s
tr t r
e o t, e b n l
s u rh oi t o t,
N np 2 e a
oio o
i n
B l f t
ne b
S a
'0 D i
'0 a hum i
nnh.
f is r
d y
em c
it r
ea f imr g f
e Sb n
3t d 3
L o
o o t
c, nd t
s fud ad ut Wd pc a
ae A
L 2 an o ne 2
BAWlB Bo Ae mD u
n o
UKc UK Nat MM l
l r
I o e T
cr N
Dua n
E o
eq ei l t rS T
a pe A
u n n
5 6
7 8
9 0
1 O
mg 2
2 2
2 2
3 3
N S
go P
ai E iN S se T
D O
N 12 e
,1l l
1' l
l
1
(
i
! /I'able 2 l
Potential Grit Source Activities Evaluation l
Pre or Post Disposition Potential Source Operation Discussion "Goniting" Activities Pre Possible See Figure 1. Process used was shotcrete which would be in a slurry form. However, the mixture is water lean and could produce pockets of dry Portland cement.
Not consistent with uniform deposition. However, elemental match including the Titanium.
Consistent with MCC doors being open for wiring and terminations.
"GunitiL ~ Activities Post Dismissed See Figure 1. Process used was shotcrete, which would be in a slurry form. However, the mixtur;:is water lean and could produce pockets of dry Portland cement.
Not consistent with MCC cubicles not generally open all at once, or for long periods of time during operation.
Block Work Post Dismissed Not consistent with MCC cubicle doors not opened all at once, or for long periods of time during operation.
Plant Side Batch Pre Possible Time frame fits -long term operation Plant General Concrete Activities Local" grouting" Pre / Post Dismissed Not consistent with the uniform deposition. Not Activities consistent with grit found in multiple plant locations.
Intake & Outfall Pre Possible The intake and outfall piping was fabricated at the Piping Activities at Mesa during construction. The fabrication included the Mesa batching of concrete as well as application of mortar for the final 1 1/2" of outer circumference.
Not consistent with,rit found in multiple plant Plaster Wall Repair Post Dismissed t
Work on 50' Control locations.
Building Not consistent with MCC cubicle doors not typically open all at once, or for long periods of time during operation.
Grouting of the Post Dismissed Not consistent with grit found in multiple plant Transformer Sole location?
Plates in the Switchgear Rooms Not consistent with MCC cubicle doors not typically open all at once, or for long periods of time during Installation of MCCs operation.
BRA and BRB 9
ATTACHMENT 2 QUALITY ASSURANCE MEASURES AND MANAGEMENT CONTROLS As discussed in the Enclosure, SCE has concluded that the depositior. of grit took place during construction activities. Therefore, the quality assurance measures and management controls assessment is focused on construction requirements.
REGULATORY. INDUSTRY STANDARDS AND PROCEDURAL REQUIREMENTS Title 10 of the Code of Federal Regulations, Part 50, Appendix B (Reference 5) establishes the criteria fer a quality assurance program. The following criteria apply to the preservation and protection of equipment as it relates to the interlock grit:
Criterion II - Quality Assurance Program
"..The quality assurance program shall provide control over activities affecting the quality of structures, systems, and components, to the extent consistent to their importance to safety. Activities affecting quality shall be accomplished under suitably controlled conditions. Contiolled conditions include the use of appropriate equipment; suitable environmental conditions for accomplishing the activity, such as cleanness; and assurance that all prerequisites for the given activity have been satisfied. The program shall take into account the need fcr special controls, processes, test equipment, tool and skills to attain the required quality, and the need for verification of quality by inspection and test..."
Criterion XIII - Handling, Storage, and Shipping
" Measures shall be established to control the handling, storage, shipping, cleaning, and preservation of material and equipment in accordance with work and inspection instructions to prevent damage or deterioration.
ANSI N45.2.2-1972 (Reference 6) establishes the requirements relative to the protection and control necessary to assure the requisite quality of those important parts of the plant are preserved from the time itemr. are fabricated until they are incorporated in the plant. The following summarizes the pertinent portions of the ANSI standard:
The levels of protective measures are categorized according to Levels (Levels A through D), based on the protective measures needed to prevent damage, deterioration, or contamination, based on the phpical characteristics of the item.
Manufacturer's documented standard or minimum requirements shall be considered.
Motor Control Centers (MCCs) are classified as Level B components.
1
ATTACHMENT 2 QUALITY ASSURANCE MEASURES AND MANAGEMENT CONTROLS Items classified as Level B are those that are sensitive to environmental conditions and require measures for protection from the effects of temperature extremes, humidity and vapors, g forces, physical damage, and airborne contamination and should not require the special protection required for Level A items.
During packaging, Level B items require a high degree of protection and the package shall be designed to avoid the deleterious effects of shock, vibration, physical damage, water vapor, salt spray, condensation, and weather during shipping, handling, and storage. This package shall be equivalent to that for Level A items, except that certain extremes (e.g.,
shock, vibration, temperature, and humidity) need not apply during package design. An inspection is performed prior to packaging to assure dirt, oil residue, metal chips or other forms of contamination have been removed, by approved cleaning methods.
During sl.ipping, Level A, B, and C items shall be covered for protection from environmental conditions for open carriers. When level A, B, and C items cannot be adequately protected from weather or environment on open carriers, closed carriers shall -
be used.
Upon receipt the following is required:
=
. Visual inspection or examination prior to unloading to detect unusual conditions r
such as environmental damage (water or oil marks, damp conditions, dirty areas, or salt film).
Item inspection shall verify that the specified packaging and shipping requirements have been maintained and shall be performed in an area equivalent to the level of
- storage requirement for the item.
Item inspection criteria include cleanness
- as well as inspection to assure that internal and external areas are within the specification requirements for dirt, soil, mill scale, weld spatter, oil, grease or stains.
For Level B items in storage, the following is required:
Storage within a fire resistant, tear resistant, weather tight, and well ventilated building or equivalent enclosure.
(1)
Section 2.7.1 of the ANSI standard," Level A," defines atmospheric contaminants ", atmospheric contaminants (e.g., rain, snow, dust, dirt, salt spray, fumes).
(2)
If inspections for cleanness were performed prior to scaling and shipping, and inspection upon receipt indicates that there has been no penetration of the sea'ed boundary, inspection for internal cleanness is optional.
l 2
e l...
ATTACHMENT 2 QUALITY ASSURANCE MEASURES AND MANAGEMENT CONTROLS i
Precautions against vandalism.
An area situated and constructed so that the item will not be subject to flooding; i
the floor shall be paved or equal, well drained.
Placement on pallets or shoring to permit air circulation.
An area with uniform heating and temperature control or its equivalent to prevent condensation and corrosion.
' Minimum temperature of 40*F and maximum temperature of 140 F or less, if so stipulated by a manufacturer.
Cleanliness and good housekeeping practices at all times while the item is in the storage area.
Cleaning of the storage area as required to avoid the accumulation of trash, discarded packaging materials, and other detrimental soil.
J While ANSI N45.2.2, in classifying items mentions dust, only the specific requirements for packaging and receipt of Level B items mention dust or dirt. ANSI N45.2.2, for storage of Level A components, require "...a ventilation system with filters to provide an atmosphere free of dust and harmful vapors..." nis requirement is specifically called out as additional for Level A items (i.e., the specific ventilax,n system is not required for Level B items).
Bechtel Procedure WPP/QCI-008, " Material Receiving, Pre-Installation Storage and Handling,"
established the control for receiving, storing, and handling permanent plant equipment at SONGS, Units 2 and 3. This procedure covered equipment from receipt at the Mesa or plant, until installation. Procedure WPP/QCl-008 referenced ANSI N45.2.2 and Regulatory Guide 1.38, Revision 0 (Reference 8). The procedure included provisions for periodic inspection by qualified personnel. The inspection frequency for areas housing Level B items was a minimum of once eveiy two weeks. The procedure also referenced WPP/QCI-610," Pre-Operational Maintenance of Electrical Equipment."
i Bechtel Procedure WPP/QCI-008, Appendix II, " Storage of Electrical Equipment," included instructions specific to the storage of electrical equipment. This Appendix to the procedure implemented the ANSI N45.2.2., Level B, storage requirements for electrical equipment.
(
Procedure WPP/QCI-610, " Pre-Operational Maintenance Of Electrical Equipment," established
- the minimum requirements for pre-operational maintenance of electrical equipment. " Pre-1 Operational" was defined as that period of time between receipt of the equipment and its release 3
L-_______________-__-__
(.
ATTACHMENT 2 QUALITY ASSURANCE MEASURES AND MANAGEMENT CONTROLS to Start Up. This procedure included requirements to megger the bus sections, and visually inspect for damage, the control circuitry, relay, meters, etc.
'After the items had been installed, ANSI N45.2.3-1973 (Reference 7) applied. This standard defines housekeeping requirements for the control of work activities, conditions, and environments that could affect the quality ofimportant parts of a nuclear power plant during the l-construction phase. ANSI N45.2.3 implements the cleanness requirements for housekeeping on a-Zone basis (Zones I though V). The 50' elevation of the Control Building, where the MCC is installed, was designated a Zone IV area. The requirements for Zone IV areas are to regulate the l-use of tobacco and eating for material and equipment protection or for health and fire hazards, in addition to good construction site housekeeping practices. The classification of the Control
{
Building elevation 50' as a Level IV Zone has been reviewed and deemed appropriate. Zone Ill
{
requires specific personnel and material access control in addition to the Zone IV requirements, I
while Zone 5 only requires good construction site housekeeping practices.
I Bechtel Procedure WPP/QCI-009, " Housekeeping," implemented the requirements of ANSI N45.2.3., and referenced Regulatory Guide 1.39 (Reference 9). The procedure required, for
- Zone IV areas, that an undocumented inspection be performed at least once a week and a documented inspection be performed at least once each month by qualified personnel.
REQUIREMENT IMPI FMENTATION RRI ATIVE TO MCC 2BE L
MCC 2BE arrived December 31,1976, and was receipt inspected. The receipt inspection included verification that accessible internal and external areas were free from dirt and other contaminants. No exceptions were listed on the receipt inspection.
During the receiving process and after acceptance by the Quality Control organization, MCC 2BE
. was stored in the Mesa warehouse, in accordance the established procedures. The MCC vendor
}
manual did not impose any additional requirements outside of those specified in ANSI N45.2.2.
L SCE has located documentation which provides evidence that periodic monitoring of Level B
(
' areas was performed every two weeks, per the procedures described above.
{
l In mid-1977, MCC 2BE was transferred from the warehr use and installed in its permanent j
location, Room 308A of the 50' level of the Control Building. Installation and inspection was I
performed per procedure. Installation was signed offin October of 1977. No remarks were made on the inspection data report relative to dust. Turnover to the Station of MCC 2BE was completed in July of1981.
The storage and housekeeping requirements were implemented and consistently monitored by Bechtel and Edison personnel. Additionally, Edison performed audits of the Bechtel programs and procedures. During the preparation of this response, documentation was located that
' indicated the " quality department" was monitoring the housekeeping and storage 4
i
ATTACHMENT 2
- QUALITY ASSURANCE MEASURES AND MANAGEMENT CONTROLS requirements / programs, and initiating corrective actions as necessary, during SONGS Units 2 and l
3 construction.
L In addition to Bechtel and Edison personnel, the NRC conducted inspections and issued Inspection Repons during constmetion. Several of the Inspection Reports mention specific reviews / tours to determine if the storage, handling and protection requirements were being met.
In one such report (Inspection Report 79-10, dated April 20,1979), specific reference to inspection of the storage, handling and protection ofinstalled safety-related electrical components L
was mentioned. The inspector found that the storage and protection of safety-related electrical switchgear met the requirements. In Inspection Repon 80-09, dated July 10,1980, storage conditions in the permanent warehouse were found to meet or exceed the requirements of the standards. This particular Inspection Report did mention non-conformances unrelated to MCCs.
CONCLUSION Based on the information found in the applicable ANSI standards, the implementing procedures, the documentation that provides evidence of procedure implementation, and the NRC Inspection Repons, SCE concludes that there were no programmatic failures of the quality assurance or management controls during plant construction, relative to the introduction of the grit into the interlock. Furthermore, SCE believes that the ANSI Standard requirements, as imph.mented at SONGS, would not have prevented the deposition of the grit on the interlock, nor would ti.ey have provided for the detection and removal of the grit.
l The quality assurance measures and management controls relative to the protection of the MCCs during operation are similar to those during construction. The Topical Quality Assurance Manual includes references to ANSI N45.2.2, ANSI N45.2.3, Regulatory Guide 1.38, and Regulatory l
Guide 1,39 (Reference 9). The implementing procedures include the appropriate requirements.
[
' As described above, the requirements would not have prevented the deposition of the grit.
l details the assessment of operational prior opportunities to detect and correct the condition that rendered the lin starter inoperable. The results of the assessment described in
- Attachment 3 (no overlooked opportunities) provide further confidence that there has been no breakdown in quality assurance and management controls, relative to the protection of the MCCs.
5 Lt-------- ---- --------------- - ----- -----------
l' ATTACHMENT 3 LINESTARTER PRIOR OPPORTUNITIES TO DETECT AND CORRECT l.
1 Inspection Report 98-05 (Reference 1) included the inspector's evaluation of prior opportunities to detect or prevent the condition. The conclusions were: 1) regarding visual inspection - "... the inspector did not consider that a visual inspection of the breaker cubicle would have reasonably been expected to result in identification of the grit as an adverse condition," 2) regarding surveillance testing - "These activities demonstrated that the line starter functioned properly, and would not suggest a problem with grit. Similar testing had been performed on the other valves with line starters, providing an even broader basis that there was not a problem with the dust or grit in the cubicles," 3) regarding visual inspection preventive maintenance - "The inspector considered this guidance [ Procedure SO123-I-9.12] adequate," and 4) regarding the 1995 root cause evaluation - "The inspector considered that the level of expen attention given to the 1995 failure evaluation would have resulted in an assessment of the grit ifit was reasonably identifiable as being unusual or contributing to the 1995 failures."
i SCE reviewed several sources ofinformation to determine if any potential opponunity to detect and correct the condition that rendered the linestarter inoperable was overlooked. Pre-operational opportunities would be addressed via the pre-operational quality assurance program which is discussed in the Enclosure in the response to Request 1, and in Attachment 1.' Post-l
- operation opponunities are addressed in this Attachment. Below is a discussion of the potential opponunities reviewed.
1 1995 FAILURE In early 1995, during the Unit 2, Cycle 7 refueling outage, Square D mechanical interlocks jammed during preventive maintenance. SCE concluded the mechanical interlocksjammed due to excessive wear of the sliding cams caused by manual cycling performed as part of the preventive maintenance program. Corrective actions included, among other things, the planned change out of all the Square-D mechanical interlocks prior to returning to service from the Cycle 10 refueling outages for both Units 2 and 3 (scheduled for 1999).
The 1995 root cause investigation did not detect /repon any presence of grit despite extensive laboratory work and analysis. Because of the fine dust appearance of the grit to the naked eye, it is not reasonable to expect that the grit would be identified by a prudent individual. The grit cannot be differentiated from normal dust accumulation without optical microscopy, or Scanning Electron Microscope (SEM) and Energy Dispersive X-Ray Spectroscopy (EDS). - During the preparation of this response, the 1995 root cause evidence was reexamined. Grit was found on L
the particular linestarter, however, it was determined that the cause of the 1995 failures was not attributed to grit.
OTHER FAILURE ANALYSIS AND ROOT CAUSE REPORTS At SONGS, the detailed analyses of component failures are performed by the Independent Safety L
l 1
l
l ATTACHMENT 3 LINESTARTER PRIOR OPPORTUNITIES TO DETECT AND CORRECT Engineering Group (ISEG). These analyses are documented in Root Cause Evaluations (RCEs),
' Failure Analysis Reports (FARs) and discussions in Action Requests (ARs). In preparation of this response, ISEG reviewed the photographic evidence and spectrographic analysis which are generally present in RCEs and FARs for evidence ofgrit to determine if a potential opportunity to detect and correct the issue was overlooked. Documentation in ARs was not sufficient for this type of review. However, the ARs were reviewed to determine if there was any evidence of the presence of grit. No evidence was identified.
l The review consisted of 106 RCEs and 105 FARs which were issued since 1990 (which is when the " root cause" group came into existence). There was no case where grit (defined as the proper elemental composition and similar physical appearance to gunite) was definitely identified. In addition, there was no evidence of soft parts being susceptible to galling. When reviewing the available photographic and spectrographic evidence, several RCEs were examined in more detail.
A brief description of these RCEs, and the rationale fbr concluding that the events and associated actions did not constitute a prior opportunity to detect and correct the failure mechanism associated with thejammed mechanical interlock is provided below. If the RCE had an associated Licencee Event Repon (LER), it is identified in parentheses.
RCE 90-017 Foreign Particle in Foxboro Transmitter (LER 90-014)
- In December of 1990, there was a failure of a containment spray pump discharge pressure indicator. The indicator failure was attributed to a transmitter which failed during subgroup relay testing associated with the containment spray system. The transmitter failure was caused by the jamming of the feedback coil by a small, black, ferromagnetic panicle, found in the vicinity of the 3
gap between the feedback coil (moving part) and its housing (stationary part). Since the black '
particle did not resemble any of the transmitter material, as determined by laboratory analyses, it was concluded that the particle was most likely introduced into the transmitter when the top cover was removed during calibration.
The root cause indicated that the transmitter failure was due to thejamming of the coil by a particle in the small clearances of a part, versus the presence of grit that resulted in the subsequent galling of a soft part. The particle found in the transmitter was ferromagnetic versus the non-
. ferromagnetic, mineral nature of the grit. Furthermore, the root cause attributed the introduction of the foreign material to most likely be from calibration activities. Therefore, this event did not represent an opportunity to detect and correct the condition that rendered the linestarter l
l l
RCE 91-007 Time Delay Relay in MG Cabinet (LER 91-003) j In March of 1991, a reactor trip occurred due to a failure of the relay which provides protection L
. against over or under generator excitation. The failure of the three month old relay was l
l
' determined to be foreign material on the low excitation current contacts. The foreign material was red in color and ferromagnetic. The foreign material was a resistive substance that effectively j
2 g
ATTACHMENT 3 l
LINESTARTER PRIOR OPPORTUNITIES TO DETECT AND CORRECT caused the circuit to be open, The RCE cencluded that the substance was most likely deposited during manufacturing.
The root cause indicated that the transmitter failure was due to foreign material, versus the presence ofgrit that resulted in the subsequent galling of a soft part. The foreign material on the contact was ferromagnetic in nature, versus the non-ferromagnetic, mineral nature of the grit.
Therefore, this event did not represent an opportunity to detect and correct the condition that
- rendered the linestarter inoperable.
RCE 96-004 Switchyard Insulator' Arc-Over In June of 1996, a flashover to ground occurred as crews were perfonning hot washing of insulators on the phase A bus disconnect rack to'one of the offsite circuits. The root cause report indicated that the most likely cause of the flashover was salt buildup on the insulators, and the existence of salt trapped in residual silicon grease at the rain shield cavity between the top and bottom insulators.
The surface contamination on the insulator did not have a similar elemental spectra to the grit associated with the interlock failure. The predominant elements can be related to specific L
materials found on the insulator. Furthermore, the insulators do not have any moving parts, unlike the mechanical interlock. Therefore, this event did not represent an opportunity to detect and correct the condition that rendered the I nestarter inoperable.
RCE 96-001 ' Auxiliary Feedwater Pump Steam Supply Trip / Throttle Valve Stuck Due to Binding of Stem Nut and Yoke In October of 1995, the Auxiliary Feedwater Pump steam supply trip / throttle valve, 2HV4716, l
would not electrically move to the trip latch reset position after being manually tripped. The root l
cause evaluation' indicated that the failure was caused by binding of the sliding stem nut in the l
valve yoke. The clearances between the sliding nut and the yoke had become fouled with grease l
and dirt. The contamination formed a wedge between the nut and the yoke, thereby requiring j
excessive force to move the sliding nut into the latch position, l
(-
The contamination associated with the stem nut and yoke binding was attributed to grease and i
dirt. The dirt did not have an elemental spectra similar to the grit. The primary elements found were copper, calcium, carbon, and oxygen. Also, this failure mechanism did not include the galling of soft parts as with the interlock. Therefore, this event did not represent an opportunity to detect and correct the condition that rendered the linestarter inoperable.
4 r
3
ATTACHMENT 3 LINESTARTER PRIOR OPPORTUNITIES TO DETECT AND CORRECT
' RCE %-002 Failed Fire Dampers (LER 95-011).
During the May to June time frame of1995, twenty five fire dampers were found not to function as required. The root cause evaluation indicates that the failed dampers were either corroded or dirty.' The root cause evaluation attributed the cause to programmatic and organizational deficiencies associated with the preventive maintenance program for the dampers.
i The debris on the fire dampers was general corrosion consisting of mst and dirt verus tan grit, l
Therefore, this event did not represent an opportunity to detect and correct the condition that rendered the linestarter inoperable.
L l
OTHER SCE LFRS A review was conducted of the Unit 1,2 and 3 LER database. The review utilized search criteria L
such as Motor Control Center (MCC), MCC, dirt, grit, contamination, foreign, and interlock.
l With the exception of the LERs referenced above, the search and subsequent review did not indicate any events or corrective actions which would lead to disassembly of components to look for grit contamination or galled soft parts.
i NRC/ INDUSTRY INFORMATION l
NRC Information Notice, Bulletin, and Circular, as well as Generic Letter databases were u
l screened and reviewed in a manner similar to that described for the SCE LERs. In addition, an L
industry LER abstract database was searched. No indiction of mechanical interlock susceptibility l
. to this type of failure mechanism was identified.
VENDOR INFORMATION i
Square D was contacted after the 1995 failure,just after the 1998 failure, and again,~as a pad of i
this evaluation. The manufacturer indicated that no other failures of the type shown for the mechanical interlock have been identified. Furthermore, when queried with respect to installation, maintenance, or housekeeping recommendations (that might have led SCE to be concerned about dust in the cubicles causing a malfunction), the vendor responded there were no recommendations beyond the requirements of ANSI N45.2.2. The vendor also indicated that l
. there was no requirement for removing the linestarter (making the interlock visible) during inspection or cleaning.
4 MCC PERIODIC MAINTENANCE b
Periodic maintenance of the MCCs is governed by procedure S0123-I-9.12, " Motor Control Center Cleaning, Inspection, and Megger Testing." The requirements in the current procedure
' existed from early operation, though the requirements were located in various procedures. The MCC cleaning is conducted every other refueling outage and was last performed on MCC 2BE on 4
_w
ATTACHMENT 3 LINESTARTER PRIOR OPPORTUNITIES TO DETECT AND CORRECT March 15,1995. Cleaning of the MCC includes: vacuuming the interior with a nonmetallic nozzle to remove any loose accumulations of dirt, dust, and debris; dislodging accumulation with a stiff
. nonmetallic bristle brush; using a clean cloth to clean electrical components and insulation; and cleaning the terminal blocks with a nonmetallic bristle brushc The procedure also includes a section delineating linestarter inspection. The inspection includes: verification that no excessive archirig, pitting or curling is present on the contacts; no excessive or uneven contact wear is present and the pad thickness meets criterion; the contact springs are in the proper position; and moveable contacts travel freely with no binding. The periodic maintenance delineated in the SCE procedures is consistent with that recommended by the manufacturer.
In 1993, as a part of correctivo Sctions for the failure of an auxiliary contact, the linestarter auxiliary contacts were replaced. This involved, in some cases, removing the linestarter (and associated mechanical interlock) to access the auxiliary contacts. No remarks or comments were made about excessive dirt being present.
Because the actual interlock mechanism is not in plain view, it is reasonable that the MCC periodic maintenance would not detect the contamination which led to the failure of the mechanicalinterlock.
VALVE ACTUATIONS Any time the valve is cycled, the mechanical interlock is cycled. Therefore, valve actuations can be considered opportunities to detect and correct the condition that rendered the mechanical interlock inoperable. In addition, ' if the actuation of the valve involves entering the MCC cubicle,
' this too could be considered an opportunity. The valves may be actuated for various reasons including testing, maintenance, and normal plant operation. Below is a brief discussion of the testing which would cause valve' actuation.
Generic Letter (GL) 89-10 Testing All motor operated valves (MOVs) that are in the scope of:he GL 89-10 Program, have been examined, setup and tested. Furthermore, a preventive maintenance program has been established. Initial setup and subsequent preventive maintenance activities require the valves to be cycled several times. Data acquisition is taken at the valve and/or breaker cubicle. The numerous cycles the valves have endured has_not provided any evidence that the mechanical interlock would fail. Note, thejammed interlock was found during planned replacement, and did not represent a demand failure.
l 1
Inservice Testing (IST)ffechnical Specification (TS) Valve Stroke Testing ASME Section XI requires quarterly valve stroking unless specific exceptions are noted in the IST program. HV9305 is stroked quarterly. Furthermore, the TSs for Emergency
- Core Cooling Systems, Containment Spray, and Containment Isolation Valves, require
)
5
4-lJ ATTACHMENT 3 LINESTARTER PRIOR OPPORTUNITIES TO DETECT AND CORRECT L
L that each automatic valve actuates to the correct position on a safety signal at least once i
. per 24 months. Note, that the original TS frequency was 18 months. However, in l
Operating License Amendments 73 and 61 (for Units 2 and 3, respectively), issued May l
31,1989, the frequency was changed to correspond to the 24 month fuel cycle. In addition, there are valve timing tests required by the IST Program and the TS. The valve stroking and timing tests do not involve entry into the MCC cubicles.
i.
1 The aggregate of periodic valve tests conducted in the past have not included failures i-attributable to the mechanical interlocks, and thus, would lead to confidence in the interlock reliability.
TS Instrumentation Testing L
The Inrt wentation section of the TSs includes Engineered Safety Feature response time l
testing arv' subgroup relay testing requirements on a refueling and 6 month frequency, l
respec'.r ely. Typically, the tests are grouped such that there is no duplication between the ~
TS valve testing described above, and the instrumentation tests. The response time tests l
and the subgroup relay tests do not require entry into the MCCs.
l-Because the interlock failure was attributed, in part, to the galling of a soR part, one would reasonably deduce that a number of actuations would be required to provide sufficient galling to jam the interlock. SCE made a proportional comparison of the estimated HV9305 actuations to the estimated number oiactuations of other valves in MCC 2BE. HV9305 is not typically
. stroked, except for testing or maintenance activities, whereas, for example, the Component Cooling Water (CCW) valves are stroked approximately eveiy two weeks, when the CCW trains are ahernated. The comparison showed at least six valves, whose linestaiters are located in MCC 1
i 2BE, had at least 20% more actuations prior to replacement, than HV9305 had prior to failure.
In fact, one of the interlocks was estimated to have had at least four times as many actuations.
The interlock associated with this valve was examined during the FAR development and was noted to have grit. There are 172 Square D safety-related reversing linestarters with a mechanical interlock in use. Conservatively, if each interlock had the same estimated number of actuations as the HV 9305 interlock, there would have been over 17,000 actuations with only the one grit-related failure.
i l'
CONCLUSION L
This review of potential opportunities to detect and correct the condition that rendered the r.
(;
linestarter inoperable, did not reveal any overlooked opportunities.
j:
6
L, ATTACHMENT 4 L
IMPACTS OF GRIT ON OTHER EQUIPMENT l
This attachment discusses why the grit associated with thejammed mechanical interlock does not impact other plant electrical and mechanical equipment, and the programs in place which might detect its presence. The assessment is based on a survey of typical plant components, and the engineering judgement of System Engineers familiar with the equipment. In general, a fine layer
'of grit would only be an issue for equipment having a combination of exposed moving parts, small operating forces (relative to the small spring forces in the mechanical interlock), and soft friction surfaces.
The grit poses no threat to static components because no frictional surfaces are involved in component operation. The grit is nonconductive and is chemically inert unless there is moisture present. The MCCs are inside a building and not exposed to moisture. The only possible negative effect of a thin layer on static components, would be a slight decrease in heat transfer from the sudace of the component in question. This would be no worse than the effect from a thin layer of any other type of dust deposit which may develop between preventive maintenance activities. Terminations would not be affected because the grit is nonconductive and the termination surfaces are in direct contact with one another. Since it has been concluded that the grit deposition has stopped, if the grit were to have been an issue relative to circuit continuity, the f
many " functional" tests of the terminations since operation would have detected it. Based on the E
above discussion, static components are not considered susceptible, and are excluded from the.
- discussion below.
The Square D reversing linestarter mechanical interlocks have been replaced and are no longer
. susceptible to the failure mechanism previously observed (because grit deposition has stopped).
L The mechanical interlocks on other reversing linestarters are sealed and thus would not be subject
- to the same failure mechanism. Table I summarizes the evaluation of the remaining plant L
equipment.
EQUIPMENT Motors j
Many motors utilized in the plant are of the totally enclosed design which would prevent the entrance of external contamir. ants during normal operation. Motors which do not have totally l
enclosed housings, are designed to draw ambient air through the internals to provide cooling.
Coarse air filters or screens are normally provided to decrease the exposure to external contaminants, but it is expected that the internals of the motor will gradually become coated with a layer of dust over time, and the motors are designed to' operate with this condition. The grit is j
not conductive, so it will not create additional current paths within the motor. In addition, the operating forces involved in motors are extremely Isrge and all bearing sudaces are metal against metal such that a thin layer of grit could not cause binding of the, moving parts. For these reasons, 1
I U-
ATTACHMENT 4 IMPACTS OF GRIT ON OTHER EQUIPMENT
' the presence of a fine layer of grit in the environment of a motor does not pose an operational concern.
Molded Case Circuit Breakers Molded case circuit breakers are sealed from outside contamination. In the unlikely event that contamination penetrated the breaker housing, the mechanism itself has large operating forces that would make it unlikely to bind. The grit is not conductive, so it will not create additional current paths within the breaker. The circuit breaker contacts are normally spring loaded and designed with a certain amount of overtravel after initial contact of their moving and stationary surfaces.
~ This overtravel provides a wiping action that tends to scrape away light surface contaminants and oxidation. For this reason, the grit would not be expected to affect operation even if small amounts had penetrated the housing over time.
6.9 KV,4KV and 480 VAC Air Circuit Breakers Air circuit breakers have large operating clearances (compared to the mechanical interlock) and large operating forces. All critical friction surfaces have metal to metal bearings. These factors
. make it unlikely that the mechanism would bind from a thin layer of grit. The grit is not conductive, so it will not create additional current paths within the breaker. The circuit breaker contacts are normally spring loaded and designed with a certain amount of overtravel after initial contact of their moving and stationary surfaces. This overtravel provides a wiping action that tends to scrape away light surface contaminants and oxidation. For this reason, the grit would not be expected to affect operation.
Non-Reversing Linestarters and Contactors The non-reversing linestaners and contactors do not have mechanical interlocks and are sealed.
These components have large operating forces and large mechanical clearances. These factors make it unlikely that the mechanism would bind from a thin layer of grit. The grit is not conductive, so it will not create additional current paths within the component. The contacts of thesc :omponents are normally spring loaded and designed with a certain amount of overtravel
. aAer mitial contact of their moving and stationary surfaces. This overtravel provides a wiping action that tends to scrape away light surface contaminants and oxidation. For this reason, the grit would not be expected to affect operation even if small amounts had penetrated the housing
. over time.
Switches Stand-alone switches normally have an outer casing which shields internal moving components from contamination. - Switches with exposed moving parts are normally used only as 2
lL___-
-e t
ATTACHMENT 4 IMPACTS OF GRIT ON OTHER EQUIPMENT subcomponents within other components that have an outer casing as well. This makes it unlikely that the grit would find its way into the moving portions of switch assemblies.
~ Relays Protective relays, Agastat timing relays and motor driven rotary (MDR) relays are sealed units in which the mov' g mechanism is protected from external contamination. As such, there is no m
expected impact from any past exposure to an environment containing grit.
Auxiliary relays encompass a wide variety of designs by many different manufacturers. In general, the larger, higher power at-jE j rdays have large clearances (compared to the clearance in the interlock) and large operating forces such 6at they would be unaffected by a thin layer ofgrit. In addition, these relays are generally equipped with a protective dust cover. The smaller, lower power auxiliary relays, generally, have smaller operating forces and clearances, however, they are normally sealed against external contaminant intrusion. The relay contacts are normally spring loaded and designed with a certain amount of overtravel after initial contact of their moving and i
stationary surfaces. This overtravel provides a wiping action that tends to scrape away light -
surface contaminants and oxidation, For this reason, the grit would not be expected to affect relay operation even if small amounts had penetrated the relay housing over time.
Failed relays are submitted to the Independent Safety Engineering Group (ISEG) for evaluation in accordance with the Relay Degradation and Failure Monitoring Program. To date, no grit-related relay degradation has been identified through this program.
Pumps, Turbines, Fans, Dampers, Valves, and Actuators The presence of an abrasive material, such as the grit found in the jammed mechanical interlock, poses a wear concern for components with moving parts. Pumps, valves and motors are
. components with moving parts that are susceptible to wear. Based on the worst case levels ofgrit contamination found in the plant, the grit does not present an operational concern for these components for the follow'mg reasons:
Pumps and valves have sufficient' operational forces, and internal wear clearances to overcome any friction induced by small amounts of an abrasive, without affecting operation.
During SONGS construction, all the piping systems were open during installation and l
inspection activities. It is likely that the contamination was present at this time. Internal contamination of the pumps and valves in water systems is not a concern since the grit would be washed away during the system flush prior to startup;* The sliding wear surfaces of pumps, turbines, and ' valves (bearings, wear rings, stems, and seat surfaces) are primarily comprised of hard carbon and/or stainless steels, unlike the rdatively soft parts on the jammed mechanicalinterlock.
3
)
d
1
+
e ATTACHMENT 4 IMPACTS OF GRIT ON OTHER EQUIPMENT l
- Grit deposition on soft seated valves could possibly cause seat damage resulting in increased
)
leakage. However, testing and maintenance performed on these valves would have reasonably identified and corrected any grit induced problems. For example, the critical valves with resilient seals at SONGS are the main and mini purge containment isolation valves, and the Component j
- Cooling Water (CCW) A and B train isolation valves.
The main and mini purge containment isolation valves are seat leak t sted per the requirements of 10CFR 50 Appendix J, Type C testing. ' A satisfactory test indicates the seating surfaces are not damaged. An unsatisfactory test would require corrective maintenance, which at a minimum would require cleaning of the resilient seating surface.
The valves also have periodic maintenance performed to replace the valves elastomeric components. The main purge seat (also known as a t-ring)is replaced on a 4 refueling outage interval. The mini purge t-ring is replaced on an every other refueling outage interval.
All purge valves have had their t-rings replaced since initial plant startup.
The CCW train isolation valves are leak tested as part of the Technical Specification CCW leakage surveillance on a 24 month interval. Excessive leakage from the train isolation valves
. ould result in corrective maintenance and repair of the valve. The t-rings are replaced on w
an as required basis, but in the past were replaced on a 4 refueling cycle interval. As stated -
previously, valves in water systems would not be susceptible to grit induced problems, since the water flow would generally remove any particulate from the valve seats when the systems are placed in-service.
Based on the above, it is reasonable to expect that any leakage from soft seated valves would be identified and corrective actions would be taken.
Ventilation system component (fans / dampers) tolerances are orders of magnitude greater
.than the grit, and along with the large operating forces, would make them unlikely to bind.
1 The operating solenoid for air-operated valve actuators would be the most susceptible o
component, if contamination were to get into the instrument air system. The solenoids themselves are sealed components, and moving parts are not exposed. Should a solenoid i
require maintenance, it would be replaced rather than being rebuilt in place, eliminating the possibility for the accumulation ofgrit during maintenance.
l The instrument air system is equipped with a dryer / filter train that has four parallel filters to i
remove particles greater than 5 microns. The particulate found in the jammed mechanical interlock was on the order of 10-100 microns.
)
4 p
.S 1
ATTACHMENT 4
]
IMPACTS OF GRIT ON OTHER EQUIPMENT Based on the above, the potential presence of a thin layer ofgrit on these components would have
' been either removed over time, or does not pose an operational concern.
' Level. Flow and Pressure Transmitters The moving parts of these components are primarily shielded inside an outer case. Periodic
. maintenance is performed to check transmitter operation. During these periods, the outer case is '
]
opened offering the opportunity for debris to contaminate the transmitter internals. This is a l
relatively small period of time, and likely not of sufficient duration for contamination to be a l
factor, as compared to the opening of the MCC cabinet doors for wiring and termination during constriction.
PROGRAMS l
There are a number of SONGS programs that might have detected previous issues with grit contamination, ifit were affecting plant equipment. These programs include the Root Cause
- Program, the Technical Specification (TS) Surveillance Program, the Relay Degradation 1
. Monitoring and Reliability Program, the Mctor-Operated Valve (MOV) Testing Program, the Oil
' Monitoring Program, and the Preventive Maintenance Program. These programs are designed to -
trend equipment degradation, identify causes and implement corrective actions. Each of these
- programs is discussed below. Action Requests (ARs ) are the current plant-wide mechanism for documenting equipment issues. A formal method to document these issues has existed since operation. The process has evolved over time and different systems have been utilized. Failures u-
, of equipment, whether occurring during testing, surveillance, or other meaas, are documented in ARs.
' Root Cause Program j
Since 1990, numerous components have been provided to the ISEG for evaluation. As discussed -
in Attachment 3, ISEG reviewed several analyses and did not identify any other grit-related
. failures.
TS Surveillance Program l
\\
l
. Safety-related pumps and valves are typically tested quarterly in accordance with the TS surveillance and Inservice Test (IST) Programs. Certain safety-related fans and dampers are l
tested in accordance with the TS. By the very nature of these tests, the associated electrical support components' (such as motors, circuit breakers, lineststers, contactors, switches, and i
i 5
i
!L____----__-_
.c
- ATTACHMENT 4 IMPACTS OF GRIT ON OTHER EQUIPMENT relays) ability to function is established. The requirements of the IST Program include trending of valve stroke times and pump operating parameters. Grit-related issues would be' indicated by negative trends or equipment inoperability, which would result in a root cause evaluation.
The TS Surveillance Program includes the Engineered Safety Feature Actuation System (ESFAS) subgroup relay testing, which cycles the safety-related equipment from relay actuation to component actuation. Failures would require generation of an AR and potential root cause evaluation. Instruments found cut-of-tolerance during performance of a surveillance require initiation of an AR for an engineering evaluation. In addition to the mentioned TS and IST r veillances, there are other surveillance which result in testing ofindividual plant components and systems.
1 Relay Degradation Monitoring and Reliability Program This program requires Maintenance to submit failed relays to ISEG for failure classification.
. These classifications are in turn, used to trend the overall performance of the plant relays, and to provide early detection of any age-related or other common mode relay failure mechanisms.
MOV Testing Program The MOV. Testing Program, which includes the scope of GL 89-10, has resulted in examination, performance of preventive maintenance, and testing all MOVs within the program scope. The prevernive maintenance performed includes: general cleaning, replacement ofgrease in the actuator housing, and inspection of gearing, stem lubrication, switches, electrical
~
contactors/ connections and bearings. No previous MOV issues or failures due to the type ofgrit found in the jammed mechanical interlock have been reported.
Oil Monitoring Program Oil from pumps, motors, valves, and transformers are periodically sampled and analyzed in accordance'with the program. Unexpected or unacceptable msults would be identified, documented, and evaluated for corrective actions. If the oil contained particulate contamination, the source would be identified to prevent recurrence. Also, pump and motor oil systems are
' sealed. This, alongwith the non-migratory nature of the grit, makes it unlikely for grit to be a cause ofoil contamination.
I Preventive Maintenance Program Detailed periodic preventive maintenance on plant equipment, ranging from inspection or
-6 I
L
= _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ - - _ _ _ _ _ _ - _ - _ _
.6 e.
l l
ATTACHMENT 4 l
IMPACTS OF GRIT ON OTHER EQUIPMENT lubrication to complete overhaul, is performed in accordance with manufacturer recommendations
~and SONGS experience. The preventive maintenance provides the opportunity for Maintenance
. and/or Engineering to closely inspect the internals of the equipment. Abnormal findings are documented on an AR and evaluated.
SUMMARY
I The failure of the linestarter mechanical interlock is unique in that the cubicle was open to the atmosphere, which allowed the grit to enter in a sufficient quantity to be deposited on the moving l
. parts of the mechanism. Because the interlock is not normally visible / accessible without i
disassembly, opportunity to observe the grit was not available. The intericek is also a relatively -
low force device, with relatively soft parts, compared to many plant components. All of the l-Square D interlocks utilized in SONGS Unit 2 and Unit 3 MCCs have been replaced. The surveillance testing, Preventive Maintenance Program, and other monitoring programs at SONGS, have resulted in numerous actuations and inspections of components since plant
.i construction. SCE has found no evidence that the grit has caused a failure in plant equipment other than the one mechanical interlock. Based on the above, there is reasonable assurance that the grit condition on the linestarter mechanical interlock will not adversely impact other plant equipment.
l i
i I
i-1 1
i 7
r L
-_--_j
l.
i l
1 l
l ATTACIIMENT 4 I
IMPACTS OF GRIT ON OTHER EQUIPMENT l
Component Henson Not Affected Program (s) to Detect ELECTRICAL Designed for large air flows filters in Motors IST.
place to mitigate.
Preventive Maintenance Many MOV motors are sealed Oil Monitoring (Bearings)
TS Surveillance l
Grit non conductive Bearing surfaces are metal to metal Large operating forces Molded Case Moving parts sealed to Preventive Maintenance Circuit Breakers prevent contamination TS Surveillance Large operating forces Grit non conductive l
Contact wipe provides self cleaning action 6.9kV,4kV, and Clearances are large compared IST 480V Air Circuit to contaminant size Preventive Maintenance Breakers
. Large operating forces TS Surveillance Bearing surfaces are metal to metal Grit non conductive e
Contact wipe provides self cleaning action Non-Reversing No mechanicalinterlock and sealed IST Linestarters and Large operating forces Preventive Maintenance Contactors Clearances are large compared no TS Surveillance e
contaminant size Grit non conductive Contact wipe provides se!f a
cleaning action l
l 8
1 4
J
ATTACHMENT 4 IMPACTS OF GRIT ON OTHER EQUIPMENT
)
i Component Reason Not Affected Program (s) to Detect ELECTRICAL (continued)
Switches Moving parts enclosed to prevent IST external contamination Preventive Maintenance a
TS Surveillance Protective Relays Moving parts enclosed to prevent Preventive Maintenance external contamination TS Surveillance Relay Degradation e
Monitoring and Reliability Agastat Timing Mov'mg parts enclosed to prevent IST Relays external contamination Preventive Maintenance TS Surveillance Relay Degradation e
Menitoring and Reliability Moving parts enclosed to prevent MDR Rotary IST a
Relays external contamination Preventive Maintenance TS Surveillance Relay Degradation l
Monitoring and Reliability Auxiliary Relays Clearances are large compared to IST a
contaminant size Preventive Maintenance TS Surveillance Relay Degradation Moving parts enclosed to prevent Monitoring and Reliability external contamination Large operating forces
=
Contact wipe provides self cleaning action 4
i l
9
{
E--_---_--_--_-------.----
,, 'e :
+
ATTACHMENT 4 IMPACTS OF GRIT ON OTHER EQUIPMENT Component Reason Not Affected Program (s) to Detect MECHANICAL :
Pumps / Turbines Large operating forces IST a
Wear surfaces are metal (bearings, Preventive Maintenance
=
e wear rings, etc.)
Oil Monitoring (bearings)
Clearances are large compared to e.
TS Surveillance
=
l contaminant size Valves Large operating forces IST
= ' Wear surfaces are metal (stems, seat, Preventive Maintenance rings, etc.)
MOV Testing Clearances are large compared to TS Surveillance
=
=
contaminant size Large operating forces Valves IST a
Periodic replacement (Soft Seat)
Prevertive Maintenance MOV Testing a.
TS Surveillance Valve Actuators-Large operating forces IST a
Motor Operated Wear surfaces are metal (gears, Preventive Maintenance
=
e i
bearing, stem / nut, etc.)
MOV Testing TS Surveillance Fans / Dampers Large operating forces TS Surveillance e
Clearances are large compared to Preventive Maintenance
=
e contaminant size IST d
Valve Actuators Wear surfaces are metal (stem, a
e Air Operated /
bushings, springs, etc.).
Preventive Maintenance Solenoids Large operating forces TS Surveillance Clearances are large compared
=
to contaminant size Solenoids are sealed Transmitters Components sealed inside casing Preventive Maintenance a'
=
c (Level, Pressure, p
Flow) i-I 10 i
I
__