IR 05000341/1985006
| ML20135E259 | |
| Person / Time | |
|---|---|
| Site: | Fermi  |
| Issue date: | 09/09/1985 |
| From: | Guldemond W, Maura F, Tomlinson E NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III) |
| To: | |
| Shared Package | |
| ML20135E242 | List: |
| References | |
| 50-341-85-06, 50-341-85-6, NUDOCS 8509160342 | |
| Download: ML20135E259 (11) | |
Text
i
)
..
..
i U. S. NUCLEAR REGULATORY COMMISSION l
REGION III
Report No. 50-341/85006(DRS)
Docket No. 50-341 License.No. NPF-33 Licensee:
Detroit Edison Company 2000 Second Avenue Detroit, MI 48224 Facility Name:
Fermi Nuclear Power Plant, Unit 2 Inspection At:
Fermi 2 Site, Newport, MI Inspection Conducted: January 23-27, February 4-6 and August 13-16, 1985 Inspectors:
f!f!8f
'
Date A
i1 E. Tomlinson 9/f/ff Date
'
hWe
'L Approved By:
W. Guldemond, Chief
-
f!f!ff Operational Programs Section Date '
Inspection Summary Inspection from January 23 to August 16, 1985 (Report No. 50-341/85006(DRS))
Areas Inspected:
Special, announced inspection by Region III and NRR:PSB personnel of licensee's program to determine and correct cause of emergency diesel generatcr bearing failures, and review the containment integrated leak rate test report, dated February 19, 1985.
The inspection involved 136 inspector-hours onsite by two NRC inspectors including 36 hours4.166667e-4 days <br />0.01 hours <br />5.952381e-5 weeks <br />1.3698e-5 months <br /> onsite during off shifts.
An additional 45 inspector-hours of inspection effort were required at the regional office by one NRC inspector.
Results:
No violations or deviations were identified.
8509160342 850910 PDR ADOCK 05000341 G
pyg
.
-
__
!_
'
..
..
t
DETAILS 1.
Persons Contacted l
Detroit Edison Company L. Burkholder, Maintenance Foreman
+J. Conen, Licensing Engineer
!
-R. Dunlea, Startup Test Engineer R. Eberhart, Rad Chem Engineer
+J.-Fix, Plant Support Engineer
- J. Green, System Engineer
- W. Jens, Vice President Nuclear Operations R. Laubenstein, NS0
+J. Leman, Superintendent, Maintenance and Modifications
- R. Lenart, Plant Superintendent-W. Miller, Supervisor, Operational Assurance, Nuclear QA J. Nyquist, Assistant to Superintendent Fairbanks Morse
- C. Ankrum, Manager, Analytical Engineering E. Greene, Manager, Customer Services
- T. Skinner, Manager, Product Service T. Stall, Service Representative A. Wickman, Service Representative
'
!
Failure Analysis Associates
- L. Swanger, Managing Engineer Power and Energy International, Inc.
'
S. Chen, President
- Denotes persons attending the Bethesda, MD meeting of February 6,1985.
+ Denotes persons attendir.g the exit meeting of August 16, 1985.
The inspectors also contacted other licensee personnel including members of the Operations, Quality Assurance and Maintenance departments.
2.
Introduction On January 10, 1985, during the performance of a twenty-four hour surveillance test, Emergency Diesel Generator (EDG) No. 11 tripped on
low lubricating oil pressure approximately fourteen minutes into the test. The' engine had been operating at design load for approximately nine minutes.
Immediately after the trip the operator noted that the AP across the lube oil filter was 0 psid, and across the strainer was
$26 psid with the standby oil pump running.
Lube oil level was normal.
An initial inspection of the engine through the inspection covers and
-
-
. _ - - - - - -.
.
- -. -_ -
,
. -
.
.
- -...
. - _
.
..
of the lube oil filter and strainer revealed pieces of metal which were determined to be bearing material and pieces of oil scraper rings and piston skirt.
Inspection of the lube oil filters and strainers for the other three EDGs revealed bearing flakes in the No. 12 EDG components only.
Oil samples were taken for analysis from all four EDGs. The licensee formed a Task Force to determine the cause of the failure, and implement corrective actions.
The NRC inspection consisted of a review of the licensee's investigation, including a review of the operating and maintenance history of the engines, selected test data, results of the analysis of lube oil samples and filtered material, a visual inspection of the failed components and additional bearings and journals at the request of the inspectors, the lube oil piping configuration including the air-oil boost system modifications, discussions with representatives of the engine manufac-turer and other licensee consultants, the evolution of the emergency diesel generators operating procedures, licensee's correspondence with.the NRC on this subject (letters dated February 12, March 6, 14, and 15, 1985), and witnessing the second periodic inspection of the lube oil filters performed in August 1985 for EDGs #11 and 12.
3.
Inspection Results a.
Component Inspection The following damaged components were noted:
Diesel No. 11 - Upper crankshaft connecting rod bearing shells Nos. 1 through 7, of which Nos. 2, 3, and 4 were totally destroyed.
Upper crankshaft main bearing shells Nos.1 through 7 and 13.
Upper crankshaft journals Nos. 2, 3, and 4.
Upper pistons Nos. 2, 3, and 4.
Connecting rods Nos. 2, 3, and 4.
Diesel No. 12 - Upper crankshaft main bearing shells Nos. 1, 2, 3, 7, and 8.
One upper crankshaft connecting rod bearing.
Other components inspected and found undamaged were:
Diesel No. 11 - Upper crankshaft remaining connecting rod bearing shells, main bearing shells, and connecting rods; fuel injector crankshaft lobes; all lower crankshaft main bearing shells; pistons Nos. 1 and 5 through 12; all upper cylinder liners; Diesel No. 12 - Upper crankshaft; upper crankshaft remaining connecting rod bearing shells and main bearing shells; pistons; upper cylinder blocks; connecting rods; fuel injector camshaft lobes;
,
Diesel No. 13 and No. 14 - Upper crankshaft connecting rod bearing shells Nos. 1 through 6 and main bearing shells No. 1 through 6 were
'
inspected and found to be acceptable for reuse.
,
.
-_
. - -
_
. -_
..
..
,
b.
Analysis of Lube Oil
!
Samples of the lube oil from each engine were collected and analyzed.
The results were satisfactory, demonstrating no fuel or water in-leakage, low solubles, and viscosity within the operating range.
No detrimental contaminants were found.
,
c.
Procedures Controlling Engine Operation
!
~
The inspt:ctors reviewed the evolution of Procedure 23.307, Emergency Diesel Generator System, from its initial Revision 0 to the present (Revision 7).
It was noted that prior to Revision 3 the procedure required manual prelubrication of two to three minutes in length
!
prior to manually starting the engines from either the local panel or the control room.
On May 21, 1983, a Temporary Change Request was approved to reduce the prelubrication of Engines Nos. 11 and 12 to one minute. On January 24,.1984 a major revision to procedure 23.307 was issued as Revision 3.
Steps for starting the engines from either the local or the control room panels were combined and the
!
prelubrication requirement was deleted.
That action was in agreement i
with the vendor's recommendation following the installation of lube oil system modifications (see Paragraph 3.d).
Since the event, the procedure has been revised (Revision 7) to require a two
'
minute manual prelubrication prior to all engine manual starts and l
continuing the prelube until the engine has reached rated speed.
I d.
Lube Oil System The inspectors reviewed.the lube oil system configuration, including the modifications made'to reduce the voids in the piping supplying the upper crankshaft and providing a lube oil booster / accumulator
which upon receipt of an engine start signal would feed the upper-crankshaft bearings with oil as the engine started to rotate.
These modifications were. completed during November 1982 (Diesel No. 11),
(-
December 1982 (Diesel No. 12), and November 1983 (Diesels No. 13 and 14).
In reply to an inquiry, the engine manufacturer informed the
licensee by letter dated July 20, 1983, that prelube requirements
!
could be eliminated once the modifications were carried out provided the booster accumulators remained filled and the. rest of the keepwarm/
standby lube oil system remained operational while the engine was
'
shutdown.
The licensee discontinued the practice of prelubricating
the engines prior to each planned start per the vendor's recommenda-tion.
It should be noted that while four other licensees have
-
installed the modification, Fermi 2 has been the only site to discon-
tinue mar.ual prelubrication of its engines prior to planned start.
!
A review of the piping configuration, the size of the accumulator (~ one gal.), and the observed (empty) condition of the jumper lines from the booster supply header to each upper main bearing during the inspection of Diesels Nos. 13 and 14 led the inspectors to conclude that the air booster / accumulator modification was not capable of perfor:r.ing its intended function. At best, only the bearings closest
l l
m
,,
,.-
-
_m.,
,, ~,, -.
,- ~. - _
--.-.---m--
..
..
to the accumulator supply line were receiving oil during engine starts.
This was later confirmed by calculations performed by the licensee's consultant.
The following additional inspections were performed by the licensee
.
in an effort to help identify possible causes of the bearing failures.
The inspection results were negative as all components were found to be operating properly.
(1) lube oil headers, valves, and pipes
,
(2) lube oil pumps (3) relief valves (main lube oil pump, filters, and strainers).
Some metal flakes were found in Diesel No. 11 strainer relief / bypass valve, and Diesel No. 12 filter relief / bypass valve, which were determined to be bearing material (4) lube oil cooler bypass control valves (5) air booster / accumulator (functional test)
(6) lube oil pressure switches and gauges (7) lube oil temperature switches
!
!
(8) lube oil filter and strainer aP gauges
-
i l
(9) crankcase pressure switches A special test was conducted on Diesel No. 12 to determine the time-
.
pressure response of the lube oil system during the starting sequence.
j The test showed that-in a prelubricated engine it takes approximately 3.5 seconds before the oil pressure at the remote end of the upper header reaches 10 psi.
Based on an analysis performed by the licen-see's consultant (FaAA), it takes a 10 psi header pressure to support
!
an adequate oil film on the bearings.
Using data obtained during the
!
preoperational test orogram, the inspectors determined that the upper i
crankshaft will experience 8 to 10 revolutions before the oil pressure reaches 10 psi. Although suggested by the inspectors, no tests were performed on a non prelubricated engine to determine the performance characteristics of the lube oil system including the time required to obtain adequate lubrication.
c e.
Engine History and Data Reviews As discussed previously, the diesel damage was mainly restricted to the bearings of the upper crankshaft most remote from the lubricating
. oil distribution system supply.
The licensee considered several possible causes, including installation errors, misalignment, con-tamination of lube oil, lube oil pump failure or cavitation, obstruc-tion in lube oil system, engine overload, inadequate bearing material
'S
_
..
-
.-
.
.
-
... -
.
.
-
-..
._...
- aa-
->_1.
n-r.,a,..
r.-
-
-.m J
-
..
,
or design, and electrical discharge through bearing. All such causes were evaluated and eliminated due to lack of evidence or lack of consistency in the observed damage.
The licensee concluded that the failure could only be attributed to the lack of proper lubrication.
The inspectors agree with the licensee's conclusion based on all the physical evidence available.
The remaining question at this time was why Diesels Nos. 13 and 14 were not damaged.
In an effort to answer this question, the inspectors reviewed selected data obtained during
preoperational testing, and the operating and maintenance history of
all four diesel engines.
In an attempt to explain the observed differences in bearing condition between the four engines, the number of engine starts were first divided into pre-lubed and non prelubed starts, with the latter being of greater interest.
The results are shown in Table 1.
,
Table 1 Summary of EDG Starts Condition EDG No. 11 EDG No. 12 EDG No. 13 EDG No. 14 Total Starts 305 205 159 175 With Manual Prelube 132
25
'
Without Manual Prelube 173 152 134 145 No significant differences between the non prelubed starts of the four diesels were found to account for the gross difference in bearing condition. As a j
result, the non prelubed starts were further subdivided to account for the
!
elapsed time between engine start and its previous shutdown.
The results are
!
shown in Table 2.
Table 2 Summary of EOG Starts Without Manual Prelube
i Elapsed Time Between Start and Previous
'
Shutdown EDG No. 11 EDG No.12 EDG No.13 EDG No. 14
!
> 24 hr.
24
23
> 16 hr.
32
31
!
-> 12 hr.
37
33
> 8 hr.
39
' 32
'
> 4 hr.
43
38
> 3 hr.
45.
44 l
> 2 hr.
63-54
> 1 hr.
70
74
< 1 hr.
82
71 5 45 min. < 1 hr.
3
3 I 30 min. 7 45 min.
5'
4 s 20 min. < 30 min.
4'
4 5 15 min.-< 20 min.
5
4 5 10 min. <'15 min.
6
7
.
...
.
_
.
-
...
-- -.
-
.
. - -.
..
. -. -
- _
..
_ -
-
-
_.
.
.__
.
-
- _-
.
-
.
..
!
Elapsed Time Between Start and Previous Shutdown EDG No. 11 EDG No.12 EDG No.13 EDG No. 14
)
> 5 min. < 10 min.
21
17 i
7 5 min.
22
32
- 1 hr. unknown
<
duration
16
0 As can be seen, the differences between Diesels Nos. 11 and 12, and Nos. 13 and 14 are not significant.
In most cases, for starts with
> 1 hr elapsed time, Diesels Nos. 13 and 14 experience was very similar to Diesels No. 11 or 12 (80 to 106 percent of the starts-experienced by the damaged diesels).
The range of least similarity in " dry" starts exists from four hours to 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> following the preceeding shutdown.
Even in that range the undamaged diesels experienced from 69 to 89 percent of the " dry" starts suffered by
l the damaged diesels.
No significant difference exists for " dry" starts of less than 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> following the preceding shutdown.
There-
.
'
!
fore, there is no obvious answer why Diesels Nos. 13 and 14 suffered no' damage.
!
A review of the maintenance history and operating data for the four diesels failed to account for the differences in engine damage.
While Diesel No.11 suffered an overload condition on September 16, 1984, its duration (~ one minute) and magnitude (~ 137% of rated)
are not considered to have contributed to the problem.
!
f.
Corrective Action
'
The following components were replaced during engine reassembly:
(1) upper crankshaft of Diesel No. 11
'
(2) upper main bearings of Diesel No. 11 and 12
(3) upper connecting rod bearings of Diesels No. Il and 12
?
(4) three upper pistons (Nas. 2, 3, 'and 4) of Diesel No.11
'
L (5) three clinder liners (Nos. 2, 3, and 4) of Diesel No. 11 (6)
three connecting rods (for the 3 replaced pistons) of
Diesel No. 11
)
l (7) one lower rod bearing (No. 5) dented during inspection
,
of Diesel No.13
The operating procedures have been modified to require a minimum
[
of two minutes of prelubrication prior to planned starts and to
,
continue prelube pump operation until the diesel reaches full speed.
'
,
p
.
-, -,,
-
- - - - -
y-r
-
y
.,
,_-
e,
<,,s
,
,+n y
.-c,g m -
- _ _ _ ___ __ --_
-
..
By letter dated March 14, 1985 the licensee committed to the following:
(1) measure the gap between bearing and bearing cap after twenty starts without manual prelubrication or every 18 months which-ever occurs first.
The acceptance criteria will be 5,.002".
(2) perform an oil filter inspection and replacement quarterly.
The inspection will consist of two discs from each stack (there are seven stacks per filter).
Initiate a bearing gap check if more than an average of two flakes of bearing material per disc are found in Diesels Nos. 11 and 12, or one flake per disc for Diesels Nos. 13 and 14.
(3) obtain a monthly lube oil sample while the engine is operating and perform an analysis for metal concentrations.
(4) perform a spectrographic analysis of the material deposits on the lube oil filter discs replaced quarterly.
(5) establish a program to evaluate and trend the obtained data for use in prediction and detection of bearing failure.
Provide a program for the use of the data, and any preliminary results, six months after receipt of the Operating License.
The inspectors reviewed the licensee's commitments and informed the licensee and NRR that to inspect only two discs per stack is unacceptable in that it amounts to an inspection of approximately 0.25% of the total filtering surface area.
To require a finding of over two flakes of bearing material over such a small sample of filter material does not provide assurance that bearing failure will be detected at an early stage.
g.
Quarterly Oil Filter Inspection Results The inspector witnessed the second quarterly lube oil filter inspections for diesels #11 and 12, performed on August 14 - 15, 1985.
The inspections were performed in accordance with Preventive Maintenance Work Orders No. 652125 and 652126 for diesels No. 11 and 12, respectively, and with Maintenance Instruction No. MI-M136, Revision 2.
After the filter elements were removed, each filter housing deck plate was wiped and the rags inspected for metal particles. A few particles >1/16 inch were found in the filter for diesel #12.
Diesel No. 11 filter deck plate contained enough metal particles > 1/16 inch to trigger an inspection of the diesel main and connecting rod bearings.
The inspector witnessed the inspection of one of the filter elements for diesel #12.
No foreign particles could be found on the outside surface of the element, however, an inspection of approximately 75%
of the filter discs resulted in the finding of at least four particles of paint > 1/16 inch length.
This demonstrated the need to actually inspect a significant percentage of the filter discs.
-
o The inspector witnessed the licensee's inspection of diesel #11 main and connecting rod bearing on both the upper and lower crankshafts.
The inspection was conducted in accordance with procedure 34.000.14, Revision 2, Emergency Diesel Generator - Inspection.
The results of the bearing gap check were satisfactory.
The inspector visually inspected all upper crankshaft main and connecting rod bearings and the upper crankcase compartment.
No signs of bearing distress were noted. 'A slight bearing shell misalignment was noted on upper main bearing #11, however, the misalignment (< 1/16 inch) does not appear to be affecting the bearing performance and was noted for future reference only.
The licensee has concluded that the bearing material'found in both filters are remanents of the original damaged bearings.
No estimate could be made as to how long it is going to take to flush out all old bearing material into the oil filters.
h.
Results of the Filter Deposits and Lube Oil Samples As of August 16, 1985 the licensee had conducted six filter inspections.
The filters inspected and their service history is as follows:
Date Filter Number of Hours of EDG EDG No.
Installed Removed EDG Starts Operation
03/03/85 05/08/85
9.5
05/08/85 08/14/85
11.3
02/25/85 05/08/85 7*
21.1
05/08/85 08/14/85
12.5
02/08/85 06/07/85
17.9
02/09/85 07/25/85
25.9
- 0ne of the starts was automatic without manual prelubrication.
The inspector reviewed the available results of the filter deposit samples and the monthly oil samples analyzed for metal concentra-tion.
The filter deposits come from a very small fraction of the total filter area and are not representative of the filter average.
At the present, there is not enough history to tell whether trending is possible, but due to the nature of the samples, a correlation to incipient bearing failure does not appear feasible.
A review of the oil samples also showed there is not enough history yet to trend; however, two significant items were noted:
(1) The May 27 lube oil sample results of diesel #14 showed
!
extremely high values (ranging from 22 to-120 times the maximum acceptable values given in the test report) for copper, iron, lead and zinc.
In addition, a note in the report states that two slivers of metal (3/16" and 3/8" long) were found in the sample and were determined to be " free-machining " brass, i
l probably from a brass fitting.
The report recommends resampling at the earliest possible date.
,
!
,
9-i
- - - - -
.
-
-
--
_.. -
. -
...
- -
...
o
,
(2) For diesel #11 it was noted that the first oil sample analyzed had several metals which slightly exceeded the maximum acceptable values given in the test report.
Of interest is the fact that s
the second (May 17) and third (June 3) samples showed the concentration of these metals to decrease.
Dilution of these metals through addition of new oil has been ruled out, since any automatic additions of lube oil have been'small.
,
The record shows that the May 27 (diesel #14) sample was not
!
received by the Detroit Edison Company Engineering Research Laboratory until June 18, or 22 days after the sample was taken.
By the time it was analyzed and the results transmitted to the Fermi site, the next month's sample was due to be taken.
The inspector questioned what appears to be an excessive amount of time required to transport the sample and obtain the results.
The inspector also recommended that the trend program report to a
NRR, due September 20, 1985, include explanations on any abnormal lube oil sample test results, and that a sample of new lube oil be analyzed and the results included to establish a reference level.
i i.
Summary
,
(1) The bearing failure could only be attributed to the lack
-
of proper lubrication during engine starts since the air i
booster / accumulator modification was not capable of performing its intended function.
(2) No significant difference in the number of " dry" starts
or maintenance histories could be found between the four
"
diesels; therefore, there is no obvious answer as to why
Diesels #13 and #14 suffered no damage.
,
(3) The station procedures have been modified to require manual
t prelubrication of the diesels prior to and during the acceleration phase of every planned start.
This should prevent bearing failures due to lack of proper lubrication during engine start.
(4) In addition to-the procedure revision, the licensee has
committed to perform bearing gap measurements every 18 months or after twenty starts without manual prelubrication,
-whichever occurs first, and following the finding of a
'
specific number of bearing material flakes in the lube oil
,
filters. The filters are being replaced and inspected every quarter. Analyses of lube oil and filter deposit material are being performed in an effort to determine if a trend program can be established to predict and detect bearing failure.
!
(5) The filter inspection program and material deposit analysis l
as proposed by the licensee does not appear to be yielding
}
meaningful data.
._
,
--
_
,_ _ _._
_-
_. _ -
,_-
_,
.-
-
.._
.-
.*s
,s r
(6) The licensee will report to NRR (license condition).the results of the first six months of its corrective action program by September 20, 1985.
The report will contain the licensee's proposal to continue or modify the present program.
Region III plans to participate in the review l
of the licensee's proposal.
No violations or deviations were identified.
4.
Review of Reactor Containment Building Integrated Leak Rate Test Report The inspector reviewed the licensee's " Reactor Containment Building Integrated Leak Test Report", submitted to the NRC on February 19, 1985 and determined that the licensee has adequately performed, reviewed, and evaluated the preoperational Type A, B and C containment tests.
,
The final Type B and C test results were:
Type B - 244.1 scf/ day Type C - 746.5 scf/ day Total - 990.6 scf/ day or 0.14 La where the acceptance criteria is 5 0.6 La'or s 4281 sct/ day.'
No violations or deviations were identified.
5.
Exit Interview The inspector met with licensee representatives (denoted in Paragraph 1)
<
at the conclusion of the inspection on August 16, 1985 and summarized the scope and findings of the inspection.
The licensee representatives acknowledged this information.
The inspector also discussed the likely informational content of the inspection report with regards to documents reviewed by the inspector during the inspection. The licensee did not identify any such documents as proprietary.
1 f
,
t
{.
i
.'
i
[
P
'
'
,
, _.
.,, -, -
_
-s.
..--
c
-
,_..-
-,,,, -
_
_
~
,..