IR 05000440/2007008

IR 05000440-07-008, on 8/16/2007 - 8/23/2007, Perry Nuclear Power Plant, Supplemental Inspection; Inspection Procedure 95001, Inspection for One or Two White Inputs in a Strategic Performance Area.

ML072630594
Person / Time
Site:	Perry
Issue date:	09/20/2007
From:	Burgess B NRC/RGN-III/DRP/RPB6
To:	Allen B FirstEnergy Nuclear Operating Co
References
IR-07-008
Download: ML072630594 (16)

Text

September 20, 2007Mr. Barry S. AllenSite Vice President FirstEnergy Nuclear Operating Company Perry Nuclear Power Plant P. O. Box 97, 10 Center Road, A-PY-290 Perry, OH 44081-0097SUBJECT:PERRY NUCLEAR POWER PLANTNRC SUPPLEMENTAL INSPECTION REPORT 05000440/2007008

Dear Mr. Allen:

On August 23, 2007, the U.S. Nuclear Regulatory Commission (NRC) completed asupplemental inspection at your Perry Nuclear Power Plant. The enclosed report documents the inspection results, which were discussed on August 23 , 2007, with you and other members of your staff.The NRC performed this supplemental inspection to assess your evaluation of a WhiteMitigating Systems Performance Index (MSPI) in the Emergency Alternating Current Power System area of the Mitigating Systems cornerstone resulting from four Emergency Diesel Generator failures that occurred over a three year period. We conducted this inspection in accordance with Inspection Procedure 95001, "Inspection For One Or Two White Inputs In A Strategic Performance Area," and examined activities conducted under your license as they relate to safety and compliance with the Commission's rules and regulations and with the conditions of your license.The objectives of this inspection were to: (1) provide assurance that the root causes and thecontributing causes for the risk significant performance issues associated with the White MSPI are understood; (2) provide assurance that the extent of condition and extent of cause of the issues are identified; and (3) provide assurance that corrective actions are sufficient to address the root causes and contributing causes, and to prevent recurrence.Based on the results of this inspection, no findings of significance were identified and weconcluded that you understood the root causes and contributing causes of the issues, that you identified the extent of condition and extent of cause of the issues, and that your corrective actions were sufficient to address the causes and to prevent recurrence of the issues.

B. Allen-2-In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of thisletter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS), accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/Bruce L. Burgess, ChiefBranch 6 Division of Reactor ProjectsDocket No. 50-440License No. NPF-58Enclosure:Inspection Report 05000440/2007008 w/Attachment: Supplemental Informationcc w/encl:J. Hagan, President and Chief Nuclear Officer - FENOCJ. Lash, Senior Vice President of Operations and Chief Operating Officer - FENOC D. Pace, Senior Vice President, Fleet Engineering - FENOC J. Rinckel, Vice President, Fleet Oversight - FENOC R. Anderson, Vice President, Nuclear Support - FENOC Director, Fleet Regulatory Affairs - FENOC Manager, Fleet Licensing - FENOC Manager, Site Regulatory Compliance - FENOC D. Jenkins, Attorney, FirstEnergy Corp.

Public Utilities Commission of Ohio Ohio State Liaison Officer R. Owen, Ohio Department of Health

SUMMARY OF FINDINGS

IR 05000440/2007008; 8/16/2007 - 8/23/2007; Perry Nuclear Power Plant; SupplementalInspection; Inspection Procedure 95001, "Inspection For One Or Two White Inputs In A

Strategic Performance Area."The Davis-Besse Resident Inspector performed this supplemental inspection. No findingsof significance were identified. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 4, dated December 2006.

Cornerstone: Mitigating SystemsThe NRC performed this supplemental inspection to assess the licensee's evaluation of a

WhiteMitigating Systems Performance Index (MSPI) in the Emergency Alternating Current Power System area of the Mitigating Systems cornerstone resulting from four Emergency Diesel Generator start failures that occurred over a three year period. This supplemental inspection was performed in accordance with Inspection Procedure 95001, "Inspection For One Or Two White Inputs In A Strategic Performance Area." The inspectors concluded that: (1) the licensee understood the root causes and contributing causes of the risk significant performance issues that resulted in the White MSPI; (2) the licensee identified the extent of condition and the extent of cause of the issues; and (3) the licensee's corrective actions were sufficient to address the root causes and contributing causes, and to prevent recurrence.A.

NRC-Identified and Self-Revealing Findings

No findings of significance were identified.

B.Licensee-Identified Violations

None.

3

REPORT DETAILS

01INSPECTION SCOPEThe NRC performed this supplemental inspection to assess the licensee's evaluation of aWhite Mitigating Systems Performance Index (MSPI) in the Emergency Alternating Current (AC) Power System area of the Mitigating Systems cornerstone resulting from four Emergency Diesel Generator (EDG) failures that occurred over a three year period. The licensee determined that the Emergency AC Power System MSPI crossed the Green-to-White threshold during the first quarter of 2007.02EVALUATION OF INSPECTION REQUIREMENTS02.01Problem Identification a.Determine that the root cause evaluation identifies who (i.e., licensee, self-revealing, orNRC) and under what conditions the issue was identified.The inspectors determined that the root cause evaluation adequately identified who andunder what conditions the issue was identified.On April 21, 2007, the licensee generated Condition Report (CR) 07-19009 identifyingthat the NRC mitigating system performance index (MSPI) for Emergency AC Power had crossed the Green-to-White threshold during the fist quarter of 2007. The licensee determined that an MSPI failure to run condition occurred on January 31, 2007, when the Division 2 EDG developed a jacket cooling water system leak during a surveillance test run. The licensee determined that, due to the jacket water leak, the Division 2 EDG would not have fulfilled its mission time without intervention.The licensee's evaluation of the White Emergency AC Power System MSPI included areview of four EDG failures that occurred over a three year period that contributed to the index.b. Determine that the evaluation documents how long the issue existed, and prioropportunities for identification. The inspectors determined that the root cause evaluation adequately identified how longthe issue existed and prior opportunities for identification.The Emergency AC Power System MSPI was categorized as White for the first quarterof 2007 as a result of four EDG failures that occurred between August 2004 and January 2007.The licensee reviewed the EDG failures that occurred between August 2004 andJanuary 2007. The first failure, in August 2004, was associated with a jacket water leak that was determined to be the result of a degraded jacket water system flange. The second failure, in February 2005, was associated with the EDG governor. The governor was determined to have failed by a previously unknown failure mechanism and resulted 4in a 10 CFR Part 21 notification from the vendor. The third failure, in November 2005,was associated with a diesel generator output breaker malfunction that occurred during testing. The licensee determined that a breaker control device spring pin had loosened.

There were no previous identified occurrences of this issue at Perry or in the industry.

The fourth failure, in January 2007, was associated with a jacket water leak from the pipe coupling. The licensee determined that the coupling fasteners had loosened over time due to vibration. c.Determine that the evaluation documents the plant-specific risk consequences (asapplicable) and compliance concerns associated with the issue.The inspectors determined that the licensee's evaluation adequately documented theplant-specific risk consequences and compliance concerns associated with the White Emergency AC Power System MSPI that was reported for the first quarter of 2007.As documented in CR 07-19009, "Emergency AC NRC Performance Indicator is White,"the increase in the Emergency AC Power System MSPI that resulted from failure to run on January 31, 2007, was calculated to be 1.25 x 10

-6 core damage frequency (CDF). The threshold from Green-to-White was greater than 1.0x10

-6 CDF and therefore, thedifference in MSPI values (1.25 x 10

-7 CDF) quantitatively represented only a slightincrease in plant risk and was not considered to be significant. Therefore, although the Emergency AC Power System MSPI was characterized as White, the specific risk consequences of this increase were of low significance.02.02Root Cause and Extent of Conditiona. Determine that the problem was evaluated using a systematic method(s) to identify rootcause(s) and contributing cause(s).The inspectors determined that the licensee adequately evaluated the issues associatedwith the White Emergency AC Power System MSPI using systematic methods to identify root and contributing causes.Licensee personnel completed a root cause evaluation of the circumstancessurrounding the four EDG failures over the previous 12 quarters that resulted in the Emergency AC Power System MSPI crossing the Green-to-White threshold in the second quarter of 2007. The EDG failures were evaluated through a variety of systematic methods as documented in the following condition reports:*CR 07-19009, "Emergency AC NRC Performance Indicator is White"A root cause evaluation was conducted using a management oversight and risktree (MORT), an event and causal factors chart, a streaming analysis, and selected portions of performance improvement international organizational and programmatic deficiencies chart to investigate the January 31, 2007, failure of the Division 2 EDG to run. The investigation team determined that the cause of the excessive leakage from the jacket water pipe coupling was vibration induced loosening of the coupling bolts that allowed the seal to relax and leak. The investigation also revealed that the site had a basic understanding of the new 5MSPI reliability standards and had the awareness that another unreliability failurewould cause the emergency AC power indicator to go White. The licensee concluded that there was a less than adequate implementation of the change management process for the new MSPI program.*CR 05-07531, "Division 1 DG [Diesel Generator] Output Breaker Failed to CycleDuring PMT [Post-Maintenance Testing]" An apparent cause evaluation was conducted using problem solvingflowchart and event and causal factors charting methodologies to investigate a November 9, 2005, failure of the Division 1 EDG output breaker to close during a post maintenance test activity. Licensee personnel determined that an actuating arm spring retaining pin had been dislodged sometime between the removal of the breaker from breaker cubicle and the subsequent re-installation.*CR 05-00936, "Division 2 DG Failed to Quick Start for SVI-R43-T1348" A root cause evaluation was conducted using fault tree analysis and failure modeand effects analysis methodologies to investigate a February 7, 2005, failure of the Division 2 EDG to start within five minutes after completion of a 24-hour rated load run. The investigation of the failure determined that the de-energized position of the governor sub-assembly pilot valve plunger unexpectedly changed due to internal bias drift and thermal growth. This was determined to be a new failure mechanism. As a result, a 10 CFR Part 21 notification was generated by the vendor to communicate this new failure mode to the industry. *CR 04-04154, "Division 2 DG Inoperable Due to Jacket Water Leak"An apparent cause evaluation was conducted using failure mode analysis andcausal analysis methodologies to investigate an August 11, 2004, Division 2 EDG jacket water pipe coupling leak that rendered the EDG inoperable and unavailable. The jacket water coupling leakage was attributed to the installation of a damaged jacket water coupling. The coupling was removed during EDG maintenance. When the coupling was re-installed, the damaged condition was not identified prior to installation.The inspectors reviewed the four evaluations described above and concluded that thesystematic methodologies that were utilized were appropriate and that these evaluations adequately addressed the individual equipment and organizational issues thatcontributed to or influenced these events. The inspectors noted that the licensee had determined that no common themes between the events existed.b.Determine that the root cause evaluation was conducted to a level of detailcommensurate with the significance of the problem.The inspectors reviewed licensee documents and interviewed station personnel anddetermined that the root cause evaluation was conducted to an adequate level of detail commensurate with the significance of the issue.

6c. Determine that the root cause evaluation included a consideration of prior occurrencesof the problem and knowledge of prior operating experience.The inspectors determined that the root cause evaluation included adequateconsideration of prior occurrences of the problem and knowledge of prior operating experience.The inspectors determined that licensee personnel conducted a detailed review ofprevious EDG generator reliability issues that extended back to 2002. Licensee personnel reviewed condition reports to identify and categorize previously identified reliability issues. Through document reviews and interviews with licensee personnel, the inspectors determined that licensee personnel had reviewed the FENOC database for previous occurrences. Licensee personnel also reviewed industry operating experience to identify similar events. d.Determine that the root cause evaluation addresses the extent of condition and theextent of cause of the problem.The inspectors determined that the root cause evaluation adequately addressed theextent of condition and the extent of cause of the issue.The following specific extent of condition and extent of cause reviews were conductedfor each of the EDG failures:*CR 07-19009, "Emergency AC NRC Performance Indicator is White"Following the identification of the pipe coupling leak on the Division 2 EDG, thelicensee reviewed other plant applications of similar pipe couplings. The review included the Division 1 and 3 EDGs, and emergency service water system (ESW). The licensee determined that any corrective actions relative to the Division 2 EDG should be applied to the Division 1 EDG. The licensee concluded that Emergency Service Water (ESW) system couplings were not susceptible to the same vibration issues associated with EDG applications.*CR 05-07531, "Division 1 DG Output Breaker Failed to Cycle During PMT"Following the identification that an actuating arm spring retaining pin had beendislodged sometime between the removal and re-installation of the Division 1 EDG output breaker, licensee personnel conducted a comprehensive extent of condition review. This review included breakers installed in the plant, breakers stored in the warehouse and breakers undergoing refurbishment. Following this review, the licensee concluded that no additional similar problems existed.*CR 05-00936, "Division 2 DG Failed to Quick Start for SVI-R43-T1348" Following the identification that the de-energized position of the governorsub-assembly pilot valve plunger had unexpectedly changed due to internal bias drift and thermal growth, licensee personnel conducted an extent of condition and extent of cause review. The scope of this review included the high pressure 7core spray (HPCS) EDG, the reactor core isolation cooling (RCIC) systemand the feedwater turbine control system. These systems were selected because they potentially utilized electro-hydraulic controls that were similar to the Division 2 EDG controls. Licensee personnel determined that the HPCS EDG and feedwater turbine control system were not susceptible to the same failure mechanism. Licensee personnel determined that the RCIC system failed to meet surveillance testing acceptance criteria in May 2002 due to a turbine governor null voltage shift. Licensee corrective actions included a revision to RCIC surveillance testing procedures to record and adjust the null voltage as necessary.*CR 04-04154, "Division 2 DG Inoperable Due to Jacket Water Leak"The licensee's extent of condition and extent of cause review concluded thatthis issue was an isolated event. The licensee reviewed available operating experience related to jacket water leaks in the process of making their determination.02.03 Corrective Actionsa.Determine that appropriate corrective action(s) are specified for each root cause, or thatthere is an evaluation that no actions are necessary

.The inspectors determined that appropriate corrective actions were specified for eachroot cause.The following specific corrective actions were implemented for each of the EDG failures:

• CR 07-19009, "Emergency AC NRC Performance Indicator is White"Following the identification of the pipe coupling that had leaked on the Division 2EDG, the licensee performed a torque check of similar couplings for Division 1 and 2 EDGs. The licensee put in place work orders to replace gaskets in similar couplings during the next available outage times for the EDGs.*CR 05-07531, "Division 1 DG Output Breaker Failed to Cycle During PMT Following the identification that an actuating arm spring retaining pin had beendislodged sometime between the removal and re-installation of the breaker, licensee personnel completed an examination of similar breakers and verified that no additional problems existed. Licensee personnel also revised procedures associated with the removal, installation, and inspection of breakers to include a verification that the actuating arm spring retaining pin was in place. *CR 05-00936, "Division 2 DG Failed to Quick Start for SVI-R43-T1348" Following the identification that the de-energized position of the governorsub-assembly pilot valve plunger had unexpectedly changed due to internal bias drift and thermal growth, licensee personnel revised EDG preventive 8maintenance procedures to add null voltage bands, acceptable operationcriteria, and criteria for the notification of the EDG system engineer. In addition, licensee personnel revised EDG surveillance testing procedures to record the null voltage and to prescribe acceptance criteria. The licensee reviewed present and previously recorded readings to identify adverse trends. Licensee personnel generated Engineering Change Request 04-0049 to upgrade the EDG governors.*CR 04-04154, "Division 2 DG Inoperable Due to Jacket Water Leak"Following the determination that a damaged jacket water coupling was notidentified during the inspection of the coupling prior to installation, licensee personnel replaced the damaged coupling and trained maintenance personnel involved with EDG maintenance activities. This training included a review of the event as well as a review of proper handling, storage and visual inspection of components prior to removal and/or installation. In addition, the licensee subsequently replaced the Division 1 and Division 2 jacket water couplings with a more robust model.To address the root causes associated with the White Emergency AC Power SystemMSPI, the following corrective actions were planned and/or completed:

(1) performance of a gap analysis for the MSPI process to determine if knowledge deficiency exists for engineering and work management;

(2) performance of a gap analysis for the MSPI process as it relates to the required change in knowledge, culture, and mind set for operations section; and

(3) the licensee planned to reapply the change management process to the MSPI program in order to increase the effectiveness of the program. b.Determine that the corrective actions have been prioritized with consideration of the risksignificance and regulatory compliance.The inspectors determined that the corrective actions were adequately prioritized withconsideration of risk significance and regulatory compliance.Corrective actions associated with the EDG failures prior to 2007 had been completed,with the exception of Division 1 EDG governor replacement activities that are scheduled to be completed in the next refueling outage. Additionally, interim procedure steps were established to prevent the previous problem from recurring. Some of the corrective actions associated with the January 31, 2007, jacket water leak had been completed and the remaining corrective actions were scheduled to be completed in a reasonable time frame. The inspectors interviewed the corrective action owners and determined that the corrective actions had been properly prioritized.c.Determine that a schedule has been established for implementing and completing thecorrective actions

.The inspectors determined that an acceptable schedule was established for theimplementation and completion of corrective actions in accordance with the licensee's corrective action program. Some of the remaining corrective actions included Division 1 EDG governor modifications, corrective actions associated with the pipe coupling jacket 9water leak, the gap analyses, and MSPI change management implementation. Theinspectors determined that a reasonable schedule was established for the remaining corrective actions.d.Determine that quantitative or qualitative measures of success have been developed fordetermining the effectiveness of the corrective actions to prevent recurrence.The inspectors determined that adequate qualitative and quantitative measures ofsuccess were developed for determining the effectiveness of corrective actions to prevent recurrence.The licensee conducted effectiveness reviews for the identified EDG failures thatoccurred before 2007 to assure these performance issues have been adequately addressed to prevent recurrence. The inspectors reviewed the completed effectiveness reviews and concluded that the corrective actions were effective. The licensee had actions in place to perform effectiveness reviews once corrective actions were completed for the jacket water leak that occurred in January 2007.03MANAGEMENT MEETINGSExit Meeting SummaryThe inspectors presented the inspection results to Mr. Allen and other members oflicensee management and staff at the conclusion of the inspection on August 23, 2007.

The licensee acknowledged the information presented. The inspectors asked the licensee whether any materials examined during the inspection should be considered proprietary. No proprietary information was identified.

1

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee

J. Hagan, Chief Nuclear Officer, FENOC

D. Pace, Senior Vice President, Fleet Engineering and Services, FENOC

J. Lash, Chief Operating Officer, FENOC

J. Rinckel, Vice President, Oversight, FENOC

B. Allen, Vice President, Perry

K. Krueger, Plant General Manger, Perry

F. Cayia, Director, Performance Improvement, Perry

J. Shaw, Director, Engineering, Perry

K. Cimorelli, Director, Work & Outage Management, Perry

M. Wayland, Director, Maintenance, Perry

M. Loberling, Manger, System Engineering, Perry

D. Evans, Manager, Operations, Perry

J. Lausberg, Manager, Regulatory Compliance, Perry

T Hilston, Manager, Design Engineering, Perry

LIST OF ITEMS

OPENED, CLOSED AND DISCUSSEDOpened None.Opened and

Closed

None.

Discussed

None.

2

LIST OF DOCUMENTS REVIEWED

The following is a list of documents reviewed during the inspection.

Inclusion on this list doesnot imply that the NRC inspectors reviewed the documents in their entirety but rather that