ML20236R825

Insp Rept 50-440/98-11 on 980518-22.Violations Noted.Major Areas Inspected:Adequacy of Licensee Programs,Procedures, Training,Equipment & Supporting Documentation for Maint of medium-voltage & low-voltage Power Circuit Breakers

ML20236R825
Person / Time
Site:	Perry
Issue date:	07/16/1998
From:	NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III)
To:
Shared Package
ML20236R805	List: ML20236R803 ML20236R811 ML20236R825
References
50-440-98-11, NUDOCS 9807220299
Download: ML20236R825 (22)

See also: IR 05000440/1998011

Text

_

,

.

-.

,

.

U.S. NUCLEAR REGULATORY COMMISSION

REGION lil

Docket No:

50-440

License No:

NPF-58

Report No:

50-440/98011(DRS)

Licensee:

Centerior Service Company

Facility:

Perry Nuclear Power Plant

Location:

P. O. Box 97, A200

Perry,OH 44081

,

Dates:

May 18 - 22,1998

inspectors:

Z. Falevits, Reactor Engineer, Team Leader, Rlll

S. Alexander, Reactor Engineer, NRR

A. Pal, Electrical Engineer, NRR

T. Tella, Reactor Engineer, Rill

[

Approved by:

Ronald N. Gardner, Chief,

Engineering Specialists Branch 2

Division of Reactor Safety

.

l-

l

l

!

o

9907220299 980716

PDR

ADOCK 05000440

,

0

PDR

{

l.

.

.

-

_

-

- _ _ _ _ _ _ _ _ - _ _ - - _ _ _ - _ _ _ _ _ - _ _ _ -

1

.

"

.

.

EXECUTIVE SUMMARY

!

Per:y Nuclear Power Plant

. NRC Inspection Report 50-440/98'111(DRS).

A team insoection was conducted using the guidance of Temporary instruction (TI) 2515/137,

!

. Revision 1, " Inspection of Medium-Voltage and Low-Voltage Power Circuit Breakers." issued

I

March 1998. The purpose of this inspection was to verify the adequacy of licensae programs,

l

procedures, training, equipment and supporting documentation for the mainte'1ance of

l

medium-voltage and low-voltage power circuit breakers. The inspection team was comprised of

I

Region lil and Headquarters personnel. The following observations were made.

7

Calculatic.a PRDC-0007, " Voltage Drop of DC Control Circuits," dated May 18,1998,

lacked rigor in terms of design inputs and conclusions. In addition, design control

l

measures were ineffective in that design reviews failed to identify these deficiencies. A

safety assessment, performed during the inspection, was acceptable to support an

i

interim operability determination. The licensee indicated that a long term resolution

would be developed to conservatively demonstrate on a continuing basis that the DC

system could perform its intended function. A violation was cited. (Section E2.1)

The licensee initiated aggressive corrective action to address NRC concems identified in

e

1994 relative to inadequate breaker maintenance. Subsequently, in 1995/1996 all AC

!

power circuit breakers were refurbished. However, most safety related DC bruakers

had not yet been refurbished despite their age and the observed hardened grease in

!

most of the ABB low voltage breakers. The licensee committed to accelerate the

i

refurbishment plan for the DC breakers. (Section M2.1)

The preventive maintenance procedures for the low-voltage power circuit breakers

i

e

reviewed were generally of a type appropriate to the circumstances and included

!

appropriate qualitative and quantitative acceptance criteria. However, various errors

!

were identified in test reports reviewed for solid state trip devices. Similar errors were

-l

.

identified by the licensee as a result of the team's request to review additional test

reports. The instantaneous trip function of circuit breaker EF1804 was tested for

instantaneous pickup at 2270 amps instead of the upper limit of 2160 amps. A violation

l

was cited. (Section M3.1)

The licensee was using an unevaluated cleaner on the 5 kV and 15 kV medium power

o

,

'

circuit breakers. The maintenance procedures specified an ammonia-based glass

cleaner to clean the breaker's electrical components. The licensee used "Windex Glass

Cleaner" which was not an evaluated cleaner. (Section M3.2)

The team concluded that lack of training on the Gels contributed to the inadequate

e

implementation of the test procedures and considered this a weakness. (Section M5.1)

l

l

2

l

._

- - _ - _ - _ - _ _ _

_ - - _ _ _ _ _ _ _ _ _ _ - _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - - _ - _ _ .

-

.

..

,

.

There was very good evaluation and disposition of NRC, vendor, INPO, and other

e

industry experience information. The operating experience reviews (OER) program

' procedures were generally comprehensive and workable and the OER database was

complete with respect to circuit breaker information identified beforehand by the team.

The individual OER evaluations were generally thorough and the resolution of issues

expressed in NRC generic communications, operating experience reports, Part 21

reports and vendor letters was very good. This area was considered a strength.

(Sections M6.1)

Treatment of medium and low-voltage power circuit breakers under the Maintenance

e

Rule was satisfactory. (Section M6.3)

No specific audits or self assessments were performed in the areas of breaker

e-

maintenance or operations. This area was considered a weakness. (Section M7.1)

The implementation of the vendor manual control and vendor interface program in the

e

past was not effective. The program was suffering from inadequate involvement by

equipment-knowledgeable staff, insufficiently aggressive or tenacious follow up with

uncooperative vendors, and out-of-date vendor contact information. The latest version

of the vendor manual control and vendor interface program shou'd improve the state of

switchgear vendor manuals. This area was considered a weakness. (Sections M8.1)

No documented breaker failures were recorded that could be directly attributed to

e

breaker problems after the licensee refurbished all AC breakers in 1995/1996. The

team noted that since breakers had been swapped between switchgear cubicles and

breaker serial numbers were not being consistently recorded on work orders, the

repetitive breaker failure data could not be easily obtained. The team also noted that

the licensee had not yet established a program for trending breaker repeat failures and

routine maintenance deficiencies. (Sections M8.2)

I

,

3

l

w

_

. _ _ _ _ _ - _ _ _ _ _ _ _ _ _

1

\\

-

.

I

.

II. Maintenance

M2

Maintenance and Materiel Condition of Facilities and Equipment

M2.1 Low Voltage Breaker Maintenance

a.

Insoection Scooe

The team reviewed the licensee's maintenance and refurbishment program for low

voltage breakers.

l

b.

Observations and Findings

,

The team noted that a violation (Perry report 50-440/94006) had been issued by NRC in

1994 for inadequate and untimely corrective actions to address the grease hardening

concems in ABB breakers. Grease hardening in ABB breakers was a known problem

since at least 1989, when a 10CFR Part 21 Notice was issued. The team also noted

that most of the breakers refurbished by the licensee during 1995 indicated severe

hardening of grease and several other problems. All the safety related AC low voltage

breakers (except the spares) had been refurbished and appeared to be in good working

condition. However, there were twelve safety related DC breakers in Unit 1 and eight

safety related DC breakers in Unit 2 yet to be refurbished.

The team noted that the DC ABB breakers were approximately 20 years old and the

vendor recommended complete refurbishment of these breakers, at intervals to not

exceed a maximum of 10 years. Further, four DC breakers refurbished recently also

exhibited hardened grease. In view of these facts, the team was concerned about the

condition of the remaining DC breakers in service.

In response to this concem, the licensee informed the team that the refurbishment

schedule for the DC breakers would be accelerated and that the safety related DC

breakers would be refurbished by September 30,1999. The previous completion date

was April 17,2001 for safety related ABB K-Line DC breakers.

c.

Conclusion

The team determined that corrective actions, taken to address the concems noted by

the NRC with the AC breakers in 1994, were effective. However, the team was

concerned that the safety related DC breakers had not yet been refurbished in view of

their age and the observed hardened grease in most of the ABB low voltage breakers.

4

i

L

_ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ - _ _ _ _ - _ _ _ _ - _ _ _ - _ - _ _ - _ _

_ - _

_____

_

__ _

__

-_

.

-

.

.

-M3

Maintenance Procedures and Documentation

-

M3.1 Review of Maintenance Procedures for Low-voltage Power Circuit Breakers

l

a.

Insoection Scone

l'

The team reviewed the preventive maintenance procedures for the low-voltage power

circuit breakers. The procedures were compared to the maintenance section of the

manufacturer's instruction manuals and to the recommendations in Electric Power

Research Institute (EPRI) Nuclear Maintenance Applications Center (NMAC) Publication

l

NP-7410, Volume I, " Low-voltage Circuit Breaker Maintenance," Part 1," ABB K-Line.

The procedures were reviewed in light of the team's knowledge of good industry practice

- for breaker maintenance, as well as specific guidance in Tl 2515/137. These

documents and concepts provided a framework for assessing the quality of the

- maintenance procedures. The ultimate acceptance criterion applied by the team was

that the procedures were of a type appropriate to the circumstances and included

appropriate qualitative and quantitative acceptance criteria.

The team reviewed the following low-voltage circuit breakers procedures:

L

General Electric Instruction (GEI)-0009, ABB Low-Voltage Power Circuit Breaker

.

Types K-600 & K-600s Through L-3000 & K-3000s Mainter'ance, Revision 4.

gel-0124, Types SS-3, SS-4, and SS-5 Solid State Trip Devices Testing and

.

Calibration Using Multi-Amp CB-8160 Test set, Revision 4.

gel-0102, Maintenance and Calibration of Gould/ITE Solid State Trip Cue'ces

.

,

l

Type (SS-3, SS-4, SS-5), Revision 2.

gel-0115, Maintenance and Calibration of DC Over current Trip Device Type

.

l

OD-3 and OD-4, Revision 2.

'

gel-0012, Cleaning of Electrical Equipment, Revision 2.

b.

Observations and Findings

The team observed that the maintenance procedures were clear, detailed, incorporated

sign-offs on individual steps by craft persons and overall by approver and supervisor.

Data sheets provided a good record of the results of all measurements made and

l

breaker condition at time of maintenance.

The preventive maintenance procedures reviewed were generally of a type appropriate

to the circumstances and included appropriate qualitative and quantitative acceptance

criteria. Breaker maintenance was done at a five year frequency. However, some

breakers depending on operating experience, might be maintained at three years

,

i

frequency.

(1)

The team reviewed procedure gel-0009, Revision 4, in detail, and made the

following observations:

L

5

L

f

L

- - _ - __

_ _ - __ - - ______.

.

-

.

.

The procedure included the maintenance and testing recommendations

.

.

from the manufacturer's instruction book and the NMAC K-line

Maintenance Guide. In addition, the procedure had been revised to

' include industry experience.

The breaker closing and tripping operation at 90 Vdc were included in the

.

procedure. However, the vendor maaual recommended a minimum

voltage for tripping of 70 Vdc. Also, the verification at 90 Vdc was not

performed on all breakers. The team was informed that the closing and

tripping operations at the minimum and maximum operating volhnes

were performed at the service center during refurbishment. (Operating

breakers below the minimum control voltage that might be expected -

under the worst-case design basis condition and operation for tim

- particular breaker provided some assurance of opening or closing

operability at those voltages.) Opening and closing breakers electrically

at the minimum and maximum rated voltage for the device of concem

(i.e., trip coils, closing solenoids, closing spring release coils, and closing

spring charging motors) provided qualitative indication that there was

e

some margin to degraded mechanical condition of the breaker for-

diagnostic screening purposes.

(2)

The team reviewed procedures gel-0102 and 0124 for testing and calibration of -

solid state trip devices on K-line breakers and identifed that the limits for the

solid state trip device long and short time delays were not in agreement with the

vendor manual recommendations. The time delays are tabulated below:

Lona Time Delav (seconds) Vendor Manual Data

Perry Procedure Data

Min.

8 - 12

8 - 12

int.

20 - 32

19 - 32

Max.

60 - 98

57 - 98

Short Time Delav (seconds)

Min.

0.08 - 0.17-

0.08 - 0.17

Int.

0.20 - 0.32

0.20 - 0.32

Max.

0.35 - 0.50

0.35 - 0.50

CR 98-1148 was written to address this cormem.

(3)

Additionally, the team reviewed a sample of actual testing using these

procedures a'x! identified that the instantaneous trip function of circuit breaker

EF1804 used for emergency closed cooling pump motor 1P42C001 A was

inadequate. The team reviewed a data sheet from work order R93-683, dated

. November 14,1994, and identified that the electrician who determined the test

current values, based on data provided by engineering, made several errors in

computing the test current values which were used during performance of the

6

.

.

)

'

test. First, the electrician applied the long-time trip pickup multiplica, tion factor

!

(1.1) in addition to the instantaneous trip multiplication factor (12) to the trip unit's

nominal full-load current value of 150 amperes, yielding a nominal instantaneous

trip pickup value of 1980 amperes (150x12x1.1) instead of the correct value of

1800. amperes (150x12). Second, the electrician applied a previously used field

test tolerance of 15 percent to his incorrectly computed instantaneous pick up

value of 1980 amperes instead of the *20 percent tolerance prescribed in the

revision of the procedure in effect at the time. This yielded a minimum

instantaneous trip test current value of 1683 amperes (1980x0.85) instead of the

correct value of 1440 amperes (1800x0.8) and an upper test limit of

2277 amperes (1980x1.15)instead of the correct value of 2160 amperes

(1800x1.2).

. The electrician tested the trip device for no-pickup at 1685 amps (lower limit) and

instantaneous pickup at 2270 amps (upper limit). Because the trip function was

tested at a value higher than the maximum allowable trip current (and tripped at

that level, but not below it), the test did not provide assurance that the breaker

would trip at current levels within the allowable band. This potentially impacted

circuit coordination and circuit protection. The fact that the breaker did not trip

when tested at a current level within 15 percent of the lower limit was actually

conservative (although not in accordance with the current procedure), indicating

that this breaker was not likely to tilp prematurely. The team determined that the

errors made in the test report were not identified by the reviewer / approver that

signed off the test report. Other completed tests with similar errors were

identified as a result of the team's request to review additional completed tests.

The team considered the as left instantanooto trip test current of circuit breaker

EF1B04 to be inadequate.

Criterion XI, " Test Control," of 10 CFR Part 50, Appendix B, requires, in part, that

components be tested in accordance with written test procedures that .

incorporate the requirements and acceptance limits of applicable design

documents. The failure to ensure that testing of circuit breaker EF1804 was in

accordance with design requirements is considered an example of a violation of

Criterion XI (50-440/98011-01(DRS)).

Condition report (CR) 98-1140 was written to address this concem and an

operability evaluation was performed. The operability evaluation determined that

the circuit pivw. tion and coordination would not be affected and the tested value

would not degrade the safety function of systems, structures or components.

The team agreed with the conclusion of the operability evaluation.

(4).

In response to the team's concem about the use of 20 percent tolerance for

instantaneous pickup inriend of 15 percent as indicated in the vendor manual,

the licensee provided an ABB letter dated September 22,1992. The team noted

that the ABB letter indicated a 10 percent tolerance according to ANSI

C37.17-1979Property "ANSI code" (as page type) with input value "ANSI</br></br>C37.17-1979" contains invalid characters or is incomplete and therefore can cause unexpected results during a query or annotation process. for long time pickup and longer waiting timr= between tests. These

L.

were not included in the plant procedures gel 0102 and 0124. CR 98-1166 was

7

L

____________

_

-

-_

._

l

t

.

-

.

.

-

t

. written to address this finding. This item was considered unresolved pending

!

licensee completed corrective action following clarification from the vendor of use

' of the 15 percent tolerance for long time trip during single phase primary injection

testing and the vendor recommended cooling periods between short time and

instantaneous tests. (50-440/98011-02(DRS)).

c.

Conclusions

The preventive maintenance procedures reviewed were generally of a type appropriate

l

to the circumstances and included appropriate qualitative and quantitative acmptance

criteria. However, the team identified various errors in test reports for solid state trip

devices. Similar errors were identified by the licensee as a result of the team's request

to review additional test reports. As a result of the noted test errors, the team identified

that the instantaneous trip function of circuit breaker EF1B04 was tested for -

instantaneous pickup at 2270 amps instead of the upper limit of 2160 amps. The

breaker was subsequently evaluated and the evaluation determined that circuit

protection and coordination would not be affected.

M3.2 Review of I nhriu. tion and Cleaners used Durina Mahtenance of Power Circuit

Breakers

a.

Inspection Scope

The team reviewed preventive maintenance procedures used to perform maintenance

activities on the power circuit breakers. The team focused on lubrication requirements -

.

delineated in the procedure and compared them to vendor recommended and approved

lubricants and cleaners.

b.

Observations and Findings

Generic Electric Instruction, gel-0135, "ABB Power Breakers 5 KV Types 5HK250 and

5HK350 Maintenance," Revision 0, dated March 25,1994, section 4.2, item 9 specified

that a~ monia-based glass cleaner, or equivalent SC 1152404 Chemical Permit 284 was

m

an acceptable cleaner for 5 KV breakers. Section 5.4.4b stated, " Wipe the outside of

the arc chute with a clean cloth and a mild ammonia-based glass cleaner," Similarly,

gel-0136, "ABB Power Breakers 15 KV Types 5HK1000 Maintenance," Revision 0,

dated March 25,' 1994, section 4.2, item 10 also specified that ammonia-based glass

cleaner, or equivalent was an acceptable cleaner for 15 KV breakers. The team noted

that Chemical Permit 284 was issued on June 6,1996, for use of "Windex Glass

Cleaner" which contained butoxyethanol and isopropyl alcohol; however, the permit did

not allow use of this cleaner specifically on breaker components. The team determined

that the use of "Windex Glass Cleaner" was not approved by the breaker manufacturer.

ABB manual MS3.2.1.9-1D used, in part, for maintenance and surveillance of

medium-voltage breakers type 5HK250 and 350 stated in section 1.a that the

recommended lubricants for breaker mechanisms was Anderol 757 and for electrical

'

contacts, NO-OX-lD. The manual further stated that, "use of other lubricants risks

8

_

Ic

.

'

-

.

.

incompatibility with original materials or unproven performance." In addition, ITE Gould

instructions, IB-8.2.7-2, issue H "5 KV power circuit breakers type 5HK75,5HK150,

5HK250 and 5HK350, " stated in section 5.12 " Lubrication" that, "if grease should

become contaminated or parts are replaced, lubrication should be done with NO-OX-ID

or Anerol as applicable. Use of other greases have not been proved by test and are not

recommended."

l

The licensee could not provide the team with an engineering evaluation that concluded

l

that use of the "Windex Glass Cleaner" to clean the breaker arc chutes was acceptable.

!

The licensee informed the team that the use of Windex would be discontinued, the

procedures would be revised to match the vendor manual and an acceptable product

would be used. The licensee, however, did not issue a condition report, to determine

the effect this cleaner had on the breakers it was used on, until questioned by the NRC.

The team was concemed since use of unapproved sprays in the nuchar industry on

safety related breakers has resulted in breaker operational problems.

The licensee was evaluating the acceptability of the use of "Windex Glass Cleaner" to

clean safety related electrical breaker components. This issue is considered unresolved

pending licensee evaluation results. (50-440/98011-03(DRS)).

,

c.

Conclusions

The team concluded that the licensee was using an unevaluated cleaner on the 5 KV

and 15 KV medium power circuit breakers. The maintenance pror*d:r,c5 specified are

ammonia-based glass cleaner to clean the breaker's electrical components. The

licensee used "Windex Glass Cleaner," which was not evaluated for this application and

was not an approved cleaner by the vendor.

' M3.3 . Review of Medium Voltaos Breaker Testino

a.

Inspachon Scope

The team reviewed licensee's maintenance procedures for medium and low voltage

breakers to verify adequacy of testing.

b.

Observations and Findings

The team noted that procedure gel-135, Revision 0, "ABB Power Circuit Breakers 5 kV

Types 5HK 250 and 5HK 350 Maintenance," Section 5.6.2, specified bumishing the

main contact surfaces with a fine Scotch-Brite or bumishing stone, prior to measuring

~ the contact resistances to compare with values (60 micro-ohms) specified in Table 1

(Section 5.8.1) of the procedure. The team was concemed that the as-found contact

resistances were not measured, prior to cleaning and bumishing, and prior to severci

. closing operations and thus the as-found data was lost and could not be trended.

9

-

_ _ _ _ _ _ _ _ - - _ _ - _ - - _ _ _ _ _ _

-

-_-

_ -

_ _ _ _ _ _ - -

.. . _ _ . . .. _.

..

..

..

.

.

.

.

The team noted that ABB vendor manual for HK type Breakers (186.2.1.7D) specified a

maximum contact resistance as 40 micro-ohms for optimum performance. The vendor

manual specified that the main and arcing contacts be dressed with a fine file if the

contact resistance (and the milli-volt drop) exceeded 150 percent of the specified value.

The vendor manual also stated that the contact pressures be adjusted, if these values

were still over 150 percent of specified values, after the contacts were dressed.

However, the procedure specified only one contact resistance of 150 percent (60

micro-ohms) as the acceptance criteria instead of the twa step process specified in the

vendor manual. The licensee's maintenance procedure also specified several breaker

closing operations prior to meastfg 2 wotact resistances.

The licensee's methodology which did not take as found data and include provisions for

assuring that prerequisites (i.e. measuring contact resistances prior to cleaning and

bumishing) wete met was a concem since this data could be evaluated and trended by

engineering to determine if changes were needed to the maintenance frequency.

c.

Conclusions

The team was concerned that the as-fou'nd contact resistances were not measured,

prior to cleaning and bumishing, and prior to several closing operations and thus the

as-found data was lost and could not be trended. Procedure, gel-135, Revision 0, did

not include provisions for measuring the contact resistances (and milli-volt drops) of

breakers prior to cleaning and bumishing these contact surfaces.

M5

Maintenance Staff Training and Qualifications

M5.1 Training in Breaker Maintenance

a.

Inspection Scope

The team reviewed the licensee's program, procedures and lesson plans for training

maintenance personnel in electrical breaker maintenance.

b.

Observations and Findinas

The team selected three electricians to review their training in detail. The electrical

training coordinator and the selected electricians were interviewed. The training was

acceptable overall. However, the team noted that training was given to the electricians

on administrative procedures and on new Gels, but no training was given on the

subsequent revisions to the Gels. The team noted that gel-0124, " Types SS-3, SS-4

and SS-5 Solid State Devices Testing," Revision 4, changed the acceptance criteria for

instantaneous pickup from 15 percent variation toi20 percent. The team noted that

'

i

eight procedure instruction changes were issued to this procedure since it was revised

'

in June 1998. The team noted that during the testing of solid state devices using

gel-0124, the electricians made several errors (as mentioned in Section M3.1.b(3)). It

10

L _ _ __

_ - _ _ _ - _ _ _ - _ _ _ _ _ - _ _ - __

-

---

. _ -

-

.-_-

.

.

,

appeared that some engineering functions were delegated to the electricians. The team

- determined that lack of adequate training on these procedure revisions could have

contributed to the errors made by the electricians in implementing these procedures.

The team also noted that Section 5.1.5 of Perry's quality assurance (QA) plan stated, in

part, that training was not required for revisions to instructions. This might have

contributed to inadequate training.

c.

Conclusion

The team concluded that lack of training on the revisions to Gels contributed to the

inadequate implementation of the test procedures, and considered this a weakness.

M6

Maintenance Organization and Administration

M6.1 Evaluation of Ocaratina Exoerience Reoort (OER) Prociem and ImO smentation

a.

Inspection Scope-

The team reviewed procedures for obtaining, tracking, reviewing, disseminating, and

dispositioning industry operating experience information, including information from the

NRC, The Institute for Nuclear Power Operations (INPO), other licensees and vendors.

The team reviewed a printout of the Perry OER computer database entries pertaining to

circuit breakers in general, comparing the plant information to the team predetermined

list. Finally, the team reviewed the files for selected OER items including NRC

information notices (IHs) pertaining to medium-voltage circuit breakers in general and

Perry's ABB HK and GE Power-Vac type breakers in particular.

b_

Observations and Findinos

Review of NRC generic communications indicated that there were ten ins that were

either generically or specifically applicable to Perry ABB medium-voltage Type HK circuit

breakers. In most cases (only one exception identified), the review by OER personnel

for applicability of the information to Perry was thorough. With only two exceptions

among the selected items reviewed, the dispositions of the information were complete,

appropriate and timely. For example, IN 83-84, dealing with problems with puffer

pistons in ABB HK breakers was appropriately determined to be applicable to Perry

equipment. Although the team did not verify that all affected breakers were immediately

inspected for the reported problem, the licensee had incorporated instructions for

inspecting for the problem on a routins basis into HK breaker preventive maintenance

procedures.

The exception to appropriate determination of applicability involved the licensee's review

of IN 80-13 and GE riuclear anergy (NE) service information letter (SIL) 155 regarding

,,

L

GE SBM type switches. The information notice, SIL 155 and several service advice

letters (SALs) from the GE product department that manufactured the switches,

reported problems with age induced cracking of the contact cam followers that were

11

n

_

i

<

.

.

made of GE's polycarbonate material called Lexan@, used in SBM switches that were

made between 1972 and 1976. The review focused on switches received through GE

NE in the NSSS scope of supply and the licensee determined that none of those -

switches were affected. However, the OER and technical reviewers failed to recognize

that the switches had been widely used by switchgear original equipment manufacturers

including ABB. During the review of the ABB HK breaker vendor manuals, the team

found SBM control switch information and confirmed with the licensee and through the

walkdowns that the local breaker control switches on the ABB HK breaker cubicles were

GE SBM types. In response to this finding, the licensee committed to reevaluate the

applicability of the IN, SIL and SALs pertaining to SBM switches, determine whether any

of the switches installed or in the warehouse were among the affected date codes or

showed signs of degradation and to take appropriate corrective action.

IN 84-46 reported problems with breaker position verification and was determined to be

generically applicable to Perry. System operating instruction sol-R22 (Revision 4), for

medium-voltage switchgear and FTI-F0036 (Revision 2), the post maintenance test

manual indicated that unless the equipment being supplied power by the breaker

needed to be operated for some other reason (e.g... surveillance, operations, etc.) the

breaker would typically not be cycled (closed and opened) after being retumed to the

connected position. If maintenance had been performed on a breaker removed from its

>

cubicle or even from the connected position to the test or disconnect positions within the

cubicle, the breaker was functionally tested (closed and opened) in the test position.

- However, industry experience (including experience with HK breakers) has shown that

there are various things that can occur when moving a breaker back into the connect

position (i.e., with primary disconnects connected to the bus) that can lead to loss of the

ability for the breaker to be closed once reconnected, opened once closed, or the

closing spring recharged. Should this occur, a safety-related load might be rendered

inoperable, but this condition remain undetected for an extended period.

The team discussed the practice of operating the breaker as soon as possible after

reconnection to the bus (which runs the load equipment if the bus is energized),

consistent with or as allowed by plant conditions and operations, with the Perry electrical

maintenance engineer. The engineer agreed that as infrequently as breakers were

actually disconnected relative to the frequency of routine operation of most load

equipment, or bus switching operations, that the poct-reconnection retest should not

unduly tax the equipment. He agreed, therefore, to consider adopting this good

engineering practice which the team pointed out was generally subscribed to by

members af the circuit breaker users groups as well as EPRI-NMAC.

IN 91-78 reported problems with circuit breaker status (closed or open) indication in

which control power to the breaker closing or spring release coils and trip coils was

fused separately from power to the status indicating lights. In this situation, if a control

power fuse blew the indicating lights that were on could remain on and not alert control

room operators that control power for remote electric or automatic operation of the

- breakers was no longer available. According to the file, this IN had been correctly

screened as potentially applicable to Perry and referred to engineering for review. After

I

several iterations of review, the licensee ultimately determined that in all cases, breaker

,

12

- - _ - _ - - _ _ _ - _ _ - - -

.

.

control power and indicating light power had common fuses. However, during the

team's review, it appeared that one of the licensee engineer's questions in an internal

l

memorandum had not been adequatdy addressed. In the course of examining the

i

affected control and indication schematic drawings, the team discovered a feature of the

l

plant design that would require further review. The diagrams revealed that although trip

and closing coils and indicating lights did have common fuses, the red indicating lights

(breaker closed), at least in the control room, were wired in series with the trip coils such

that the control room red light (not necessarily the one on the local panel) not only

indicated that the breaker was closed and tripping power was available, but also that

there was circuit continuity through the trip coil. However, the drawings indicated that

the green indicating lights (breaker open) were not wired in series with the breaker

closing spring release coils. These lights when illuminated would then indicate that the

breaker was open and that closing power was available (common fuse), but would not

confirm continuity through the spring release coil branch circuit. The licensee could not

offer an explanation of why the closing and opening circuits were or should be wired

differently.

The team found the OER program to be very good; however, the following weaknesses

were noted:

IN 96-44 and 96-44, Supp.1 were not reviewed adequately for generic impact at

.

Perry. Procedure PAP - 1607, Rev.1 required generic implication svaluation.

OER review request was reissued to review for generic implication.

OMR-387 regarding missing retaining rings was not reviewed adequately.

-

Documentation indicated only Unit 1 breakers were reviewed for the missing

<

'

retaining rings. In response to the teams concem, OER review request was

reissued to include Unit 2 breakers.

Resolution of NRC generic communications and Part 21 reports on the subject of circuit

breakers was good. As a result of the OER Program, some procedures had bean

modified to incorporate the lessons leamed. For example:

As a result of the review of IN 88-75, operating procedures gel-0135 and gel-

.

0136 were modified to place the breaker control switch in OFF/STOP position in

order to reset the anti-pump seal-in as applicable and allow the breaker to close.

As a result of the review of IN 93-91, the appropriate maintenance procedure

j

.

was modified to add visual verification to ensure correct positioning of the

auxiliary switch contacts with the breaker racked in and closed.

i

c.

Conclusions.

j

The team concluded that, in general, there had been very good evaluation and

3

disposition of NRC, vendor, INPO, and other industry experience information. The OER

{

program procedures were generally comprehensive and workable, the OER database

was complete with respect to circuit breaker information identified beforehand by the

13

1

_

-

.

.

team (and, in fact, included some information of which the team was previously

unaware). With only one exception identified, the individual OER evaluations were

typically thorough and looked for the message in the information rather than dismissing

j

some pertinent information on the basis of Perry not having the exact type of equipment

used as an example in the information. The items selected for detailed review of their

ultimate disposition that had been determined, in the OER screening process, to be

applicable to Perry medium-voltage circuit breakers were typically appropriately and

promptly incorporated in plant procedures or other documents as applicable. This area

was considered a strength.

M6.2 - Reviewer and Anorover Raouirement for Comolated Maintenance Activities

a.

Insoection Scope

- The team reviewed selected completed maintenance work orders related to breaker

i

maintenance and testing activities.

b.

Observations and Findings

The team noted that some of the work packages were reviewed and approved by the

i

same individual. The team determined that reviewer and approver responsibilities for

completed electrical and mechanical maintenance work activities were not clearly

delineated in administrative procedures but were management ' expectations." In

contrast, the inspectors noted that instrument and control (l&C) maintenance procedure

mal-0504 clearly documented in Sections 6.9.2 and 6.9.3 the review and approval sign

off requirements. The l&C procedure required that the review be performed by the l&C

supervisor and that the approval be performed by an independent maintenance support

engineer. The licensee informed the team that the electrical and mechanical reviewer

and approver requirements will be reexamined and action will be taken to address this

concem. In addition, the review of completed work orders also revealed inconsistent

filling of identifying information on all pages. Some required signatures and breaker

serial numbers were also missing.

c.

Conclusions

The team concluded that reviewer and approver responsibilities for completed electrical

and mechanical maintenance work activities were not clearly delineated in

administrative procedures. The licensee informed the team that the electrical and

mechanical reviewer and approver requirements will be reviewed and the concem

addressed.-

M6.3 Treatment of Circuit Breakers Under 10 CFR 50.65. The Maintenance Rule

a.

Inspection Scope

The team reviewed the licensee's treatment of medium and low-voltage power circuit

breakers under 10 CFR 50.65, the Maintenance Rule (MR). The team reviewed

14

._

__ _.

._ - __-_______--_ -__-__-___ - ---- - -

--

.

-

.

scoping, classification of breakers according to system or as separate class of

components, treatment under 50.65(a)(1), or (a)(2), maintenance preventable

functional failures (MPFFs), and use of industry information as required by (a)(3).

b.

Observations and Findings

The team found that there was adequate justification for exclusion of those breakers not

included within the scope of 10 CFR 50.65 activities. For in-scope breakers, the scoping

bases considered were comprehensive. In general, at Perry, breakers that supplied

busses, supplied transformers for lower-voltage systems, and tied busses together were

treated as part of the electrical distribution system or subsystem of which they were a

part. Breakers that supplied power directly to components of other systems (e.g.,

mechanical fluid systems) or to load centers or motor control centers for those systems

were generally treated as part of the system to which they supplied power. The licensee

did not treat breakers as separate components for (a)(1) performance issues, however,

the licensee did elect to include " monitoring" of all safety-related 13.8 kV,4.16 kV and

480 Vac breakers under the electrical component group, to enhance the MR (a)(2)

performance criteria which were established for train or system functions.

The licensee hcd included the failere in 1996 (an unexpected trip on apparent overload

documented in PlF 96-2958) of breaker EF1D09 (fuel handling building HVAC supply

fan B), in its MR monitoring database. Although the failure was attributed to a

manufacturing defect (overcurrent trip unit phase sensor current transformer lead

reversal), which was also the subject of NRC IN 98-18, and hence, was not treated as

an MPFF per se, the licensee's MR procedure, PAP-1125, Revision 2, called for basing

performance criteria on all functional failures. The follow up corrective action described

in PlF 96-2958 with the vendor to prevent recurrence was satisfactory.

c.

Conclusions

Treatment of medium and low-voltage power circuit breakers under the MR was

satisfactory. In addition, the practice of including safety-related AC breakers in the

electrical component group to enhance performance criteria and the licensee's proactive

position in which functional failures attributable to other activities such as operations and

engineering, in addition to maintenance, formed the basis for performance criteria were

considered strengths.

M7.1 Self-Assessments. Audits and Surveillance

a.

Insoection Scoce

The team reviewed the licensee's self-assessments, audits and surveillance in the

areas of low and medium power electrical breakers.

I

15

_ _ _ _

L

,

-

.

,

'b.

Observations and Findinas

,

The team noted that the licensee did not perform specific self-assessments, quality

'

audits, and surveillance of Perry's breaker maintenance program. One QA audit

conducted during January 1996 in the area of maintenance did not evaluate breakers

but included reference to field observations to evaluate the preparation of breakers for

planned replacement during RF05.

c.

Conclusion

The team determined that no specific audits or self-assessments were performed in the

areas of breaker maintenance or operations. This was considered a weakness.

M8

Miscellaneous Maintenance issues

M8.1 Vendor Manuals and Vendor Interface

,

a.

Inspection Scope:

l

. The team reviewed procedures goveming vendor manuals and other vendor

information, including periodic recontact to ensure that all pertinent information has been

received. The team reviewed the binders of approved vendor technical manuals and

other technical documents compiled by the licensee under its system to determine if

they were complete and up to date relative to medium- and low-voltage safety-related

switchgear.

b.

Findinas and Observations:

I

The program as currently described by the latest revision to the procedures should

improve the state of vendor manuals for Perry switchgear and better ensure receipt and

!

appropriate disposition of updates to manuals and other technical information.

However, the program implementation has not been fully effective, particularly in the

area of periodic contact of Non-NSSS vendors of key safety-related equipment, in

l.

particular, switchgear. There was evidence that vendors had not been very cooperative

i'

in this effort in the past, but this fact was recognized by the licensee and documented in

several CRs, for example in 1992 and again in 1995, the result of which was the present

vendor manualimprovement program.

However, the efforts thus far under this initiative to contact vendors and receive timely

replies has not been effective. The writing of form letters once a year or in intervals of

several months with no response was clearly not a sufficiently aggressive approach. It

appeared that the licensee did not always write to the most appropriate vendor locations

or personnel. The effort suffered from a lack of adequate involvement by equipment-

knowledgeable personnel and did not comprise tactics that have proven more effective.

For example, the licensee did not establish the several vendor contacts (at different

locations) who were currently cognizant of the equipment of interest and familiar with the

associated technical documentation and conduct detailed comprehensive reconciliations

16

i

o

.

~

.

.

!

of all applicable equipment and information. Written responses that were received from

l

vendors indicating that the requested information was not available at that location, did

l

ot receive adequate follow up.

l

Several switchgear manual binders reviewed did not contain the most recent applicable

revisions; although, safety-related circuit breaker manuals had recently been more

completely updated than those for non-safety-related switchgear. For example,

File 162, the binder for safety-related ABB 4160-Vac Type 5HK breakers and associated

equipment contained an outdated renewal parts bulletin. An old version (different

number series, published by one of the previous manufacturing company names, of the

instruction book for the switchgear itself (i.e., cubicles, busses, etc.)) was in the file

whereas the new version (published by ABB) has been out since about 1991.

Revisions 3 and 4, adding manuals for a GE and ITE protective relay, respectively, were

entered incorrectly in that non-safety-related switchboard designators were listed as

applicable instead of the safety-related ones (i.e., % R22 S001 through S005 instead of

% R22 3007,' S006 and S009). In File 160 for safety-related low-voltage ABB K-line

breakers, and K3000 and K4000 breakers, only old versions of the switchgear

^ instruction books were in the file, in File 424 for the GE Power Vac medium-voltage

breakers, one change was not properly entered. The several binders for non-safety-

related breakers had similar, but more numerous, deficiencies.

c.

Conclusions:

The latest version of the vendor manual control and vendor interface program should

improve the state of switchgear vendor manuals. Implementation in the past was not

always effect!ve, suffering from inadequate involvement by equipment-knowledgeable

staff, insufficiently aggressive follow up with uncooperative vendors, and out-of-date

vendor contact information This area was considered a weakness, but improving.

M8.2 Review of Breaker Failun History and Trendina Prooram

a.

Insoection Sr2DR:

The team attempted to review data of failures of low and medium-voltage circuit

breakers over the last three years.

b.

Findinas and Observations:

The team requested breaker failure history data; however, the licensee could not easily

retrieve the information and the team could not effectively assess this area. The team

noted that safety related breakers were swapped between switchgear cubicles and

' breaker serial numbers were not consistently being recorded on work orders, thus the

repetitive breaker failure data could not be easily obtained.' Based on interviews with

engineers and maintenance staff and review of completed maintenance procedures, the

team concluded that no documented breaker failures were recorded that could be

directly attributed to breaker problems after the licensee refurbished all AC breakers in

1995/1996,

17

-_

_ _ _ _ - - _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ .

___

- -_ ._

i

b

x

.

. .

The team noted that the high pressure core spray pump (HPCS) 1E22C001 breaker

failed to close in January 1998 during testing. PIF 98-0125, dated January 26,1998,

)

- documented that HPCS pump 1E22C001 failed to start on demand during a surveillance

when breaker EH1304 failed to close. Troubleshooter g did not identify the cause of this -

failure. The breaker closed satisfactorily after it was racked back into the cubicle. The

I

as found condition of the breaker was not preserved after the pump failed to start and

the breaker was racked out before the engineers could inspect the breaker and record

the as found condition in order to determine the root cause. The team also noted that

the licensee had not yet established a program for trending breaker repeat failures and

routine rnaintenance deficiencies.

c.

Conclusons

' The team concluded that no documented breaker failures were noted that could be

j

directly attributed to breaker problems after the licensee refurbished all AC breakers in

<

1995/1996. The team determined that, since breakers had been swapped between

switchgaar cubicles and breaker serial numbers were not being consistently recorded on

{

work orders, the repetitive breaker failure data could not be easily obtained. The team

also noted that the licensee had not yet established a program for trending breaker

repeat failures and routine maintenance deficiencies.

I

lit. Engineering

E2

Engineering Support of Facilities and Equipment -

E2.1

Breaker Control Power issue

a.

Inspection Scope

The team performed an inspection to determine whether breaker operation was assured

at minimum operating voltage as specified in the vendor's manual or minimum

calculated voltage, whichever is the lowest as stated in TI-2515/137.

b.

Observation and Findinas

Calculation PRDC 0007, " Voltage Drop of DC Control Circuits," Rev. 4, dated May 18,

1998,'was reviewed to verify that electrically operated breakers (both medium and

low-voltage) were operable at the calculated minimum available voltage at the closing

and tripping coils. The design basis of the calculation was loss of offsite power with loss

of coolant accident. Since all emergency core cooling systems load breakers would

change state within the first minute, the battery terminal voltage at first minute was used

in this calculation. However, for a station blackout (SBO) recovery, battery discharge

- voltage (voltage at the end of four hours which is 105 Vdc) was used for circuits '

associated with preferred supply breakers and attemate preferred supply breakers. The

calculation indicated available voltage at the closing, tripping, and spring charging

circuits for 4 kV and 480 V power circuit breakers was above 90 Vdc. Testing at 90 Vdc

was included in the latest revision of both medium and low-voltage power circuit breaker

s

18

,

.

.

maintenance procedures. However, very few breakers were tested at 90 Vdc at the

plant. The team was told that all AC breakers were tested at the minimum operating

voltage during refurbishment at the ABB service center. The voltages used were: (1)

for 4 kV and 15 kV HK type breakers 90 Vdc for breaker close coil and charging motor

and 70 Vdc for tripping coil, and (2) for 480V K-line breakers-100 Vdc for closing coil

and 70 Vdc for tripping coil and 85 Vdc for charging motor. All DC breakers were

manually operated.

in reviewing the calculation, the team noted the following concems:

Inverter DC current input was not calculated properly. The inverter was a

.

constant kVA device and current was dependent on the available voltage at its

terminals.

The battery terminal voltage calculation did not consider the effect of aging factor

.

(1.25), temperature correction factor (1.04), and design margins (1.1).

Non-conservative control circuit lengths were used.

.

The calculation did not address the emergency diesel generator (EDG) output

.

breaker closing coil voltage during SBO recovery (i.e., at the end of SBO

duration of 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />).

The calculation did not address 13.8 kV breakers trip coil voltage.

.

The preliminary recalculation of two circuits (emergency service water pump

1P45-C0001B and EDG output breaker EH-1102) indicated inat the available voltage at

closing and trip coils will be below 90 Vdc. As a result, CR 98-1167 was initiated.

Criterion ill, " Design Control," of 10 CFR Part 50, Appendix B, requires that the design

basis be correctly translated into specifications, procedures, instructions, or drawings.

The failure to: (1) properly calculate inverter DC current input, (2) consider the effect of

aging factor, temperature correction factor and design margins in the calculation, (3) use -

correct control circuit lengths, (4) address the EDG output breaker voltage during SBO

recovery and (5) consider the 13.8 kV breakers trip coil voltage is considered a violation

of Criterion lli(50-440/98011-04(DRS)).

,

Based on the present condition of the divisional batteries, as demonstrated by required

- service and capacity tests, an initial assessment of affected circuit breakers indicated

that no immediate operability concems existed. The minimum calculated DC control

voltage at the circuit breakers was estimated to be 93 Vdc under design basis

,

requirements. The licensee committed to perform a detailed calculation to fully

substantiate this initial assessment.

i

{

l

19

I

_ _ _ - _ -

_ - _ _ _ _ - _ - _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ - _ - _ _ - _ _ _ _ _ - _ _ - _ - - _ _ _ _ _ _ _ _ _ _ _

..

.

.

c.

Conclusions

The team identified that calculation PRDC-0007," Voltage Drop of DC Control Circuits,"

Revision 4, dated May 18,1998, lacked rigor in terms of design inputs and conclusions.

In addition, design control measures to provide verification of adequacy of design were

ineffective in that design reviews failed to identify these deficiencies. Based on review

of selected calculations, the team was concemed about the adequacy of design basis

calculations in general. A safety assessment, performed during the inspection in

response to the team's concems about the voltage calculation, was acceptable to

support an interim operability determination. The licensee indicated that a long term

resolution would be developed to conservatively demonstrate on a continuing basis that

the DC system could perform its intended function. A violation was cited.

V. Manaaernent Meetinos

X1

Exit Meeting Summary

On May 22,1998, the inspection results of the team inspection were presented to the licensee

at an exit meeting. The i.ansee did not identify any material provided during the inspaction

report period as proprietary.

l

20

..

..

,

.

PARTIAL LIST OF PERSONS CONTACTED

Licensee

N. Bonner, Director, Maintenance /First Energy

R. Collings, Manager, QA, First Energy /QAS

L. McGuire, Supervisor, PNMD/MSPU

S. Moffitt, Manager, PNED/First Energy

L. Myers, Site Vice President

S. Sanford, Compliance Engineer, PNSD/ PAS /CU

R. Schrauder, Director, PNED, First Energy

D. Watkins, Maintenance Engineer, PNMD/CEI

M. Zeal, PES /NSSS, PNED/CEI

.

NBCL

J. Clark,' Resident inspector

' C. Lipa, Senior Resident inspector

INSPECTION PROCEDURE USED

Tl 2515/137 Inspection of Medium-Voltage and Low-Voltage Power Circuit Breakers

IP 40500

Effectiveness of Licensee Controls in Identifying, Resolving and Preventing

Problems

ITEMS OPENED, CLOSED,~ AND DISCUSSED

Opened

50-440/98011-01

VIO

Inadequate calculated test parameter values and as left

instantaneous trip test current of circuit breaker EF1B04.

50-440/98011-02

URI - (1) Limits for the solid state trip device long and short time delays

were not in agreement with the vendor's manual

recommendations, (2) use of 20 percent tolerance for

instantaneous pickup instead of 15 percent as indicated in the .

. Vendor Manual, and (3) 10 percent tolerance for long time pickup

and longer waiting time between tests not in procedures.

50-440/08011-03

URI

Use of an unapproved cleaning agent such as Windex Glass

Cleaner to clean safety related electrical breaker components.

50-440/98011-04.

VIO ' Failure to: (1) properly calculate inverter de current input (2)

consider the effect of aging factor, temperature correction factor

and design margins in the calculation, (3) use correct control

circuit lengths and additional calculational deficiencies.

21

_ _ - _ _ . .

. - - - _ _ . _ _ _ _ - _ - _ - - _ _ _ _ _ - _ _ - - - _ - _ - _ _ _ _ - - _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ - _ _ _ - - _ _ - _ _ _ _ _ - _ _

.

,

s.-

,

,

LIST OF ACRONYMS USED

AC

Altemate Current

CR

Condition Report

DC

Direct Current

EDG

Emergency Diesel Generators

-EPRI

Electric Power Research Institute

gel

Generic Electric Instructions

GENE

General Electric Nuclear Energy

HPCS

High Pressure Core Spray

l&C

Instrumentation and Control

IN

information Notice

-INPO

Institute of Nuclear Power Operation

kV-

Kilo Volts

MPFF

Maintenance Preventable Functional Failures

,

MR

Maintenance Rule

l

NMAC

Nuclear maintenance Applications Center

NRC

Nuc' ear Regulatory Commission

!

OER

Operating Experience Reports

QA

Quality Assurance

SAL

Service Advice Letter

SBO-

Station Blackout

i

SIL

Service Information Letter

Tl

Temporary Instruction

j

URI

Unresolved item

'

Vdc

Voltage in direct current

!

' VIO

Violation

i

l

l

l

i

i

l

l

i

l

22

i

-

_ _ .

.. Y