IR 05000255/2012010: Difference between revisions
StriderTol (talk | contribs) (Created page by program invented by StriderTol) |
StriderTol (talk | contribs) (Created page by program invented by StriderTol) |
||
| Line 19: | Line 19: | ||
=Text= | =Text= | ||
{{#Wiki_filter: | {{#Wiki_filter:uly 31, 2012 | ||
== | ==SUBJECT:== | ||
PALISADES NUCLEAR PLANT - NRC SUPPLEMENTAL INSPECTION REPORT 05000255/2012010, AND ASSESSMENT FOLLOW-UP LETTER | |||
- NRC SUPPLEMENTAL INSPECTION REPORT | |||
==Dear Mr. Vitale:== | ==Dear Mr. Vitale:== | ||
On June 29, 2012, the U.S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection pursuant to Inspection Procedure 95001, | On June 29, 2012, the U.S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection pursuant to Inspection Procedure 95001, Supplemental Inspection for One or Two White Inputs in a Strategic Performance Area, at your Palisades Nuclear Plant. The enclosed inspection report documents the inspection results, which were discussed on June 29, 2012, with you and other members of your staff. | ||
at your Palisades Nuclear Plant. The enclosed inspection report documents the inspection results, which were discussed on June 29, 2012, with you and other members of your staff. | |||
In accordance with the NRC Reactor Oversight Process Action Matrix, this supplemental inspection was performed to follow-up on a White finding in the Mitigating | In accordance with the NRC Reactor Oversight Process Action Matrix, this supplemental inspection was performed to follow-up on a White finding in the Mitigating Systems Cornerstone with low to moderate safety significance which occurred in the fourth quarter of 2011. This issue involved the inadvertent actuation of the overspeed trip mechanism on the turbine-driven auxiliary feedwater pump turbine during surveillance testing and was previously documented and assessed in NRC Inspection Reports 05000255/2011013 and 05000255/2011017. The NRC was informed on May 2, 2012, of your staffs readiness for this inspection. | ||
, of your | |||
The objectives of this supplemental inspection were to provide assurance that: | The objectives of this supplemental inspection were to provide assurance that: (1) the root causes and the contributing causes for the risk-significant issues were understood; (2) the extent of condition and extent of cause of the issues were identified; and (3) corrective actions were or will be sufficient to address and preclude repetition of the root and contributing causes. | ||
The NRC determined that the staff at Palisades Nuclear Plant performed | The NRC determined that the staff at Palisades Nuclear Plant performed an acceptable evaluation of the White finding. The evaluation identified the primary root cause of the issue to be inadequate preparation and instruction related to maintenance on the turbine-driven auxiliary feedwater pump overspeed trip mechanism linkages during the turbine overhaul performed during refueling outage 1R21. Specifically, the licensee failed to perform several vendor-recommended maintenance activities such that the reliability of the turbine-driven auxiliary feedwater pump turbine overspeed trip device was not maintained. To correct this issue and prevent recurrence, the licensee revised the maintenance and post-maintenance testing procedures to include these maintenance activities and verify the reliability of the overspeed trip mechanism. No findings were identified during this inspection. | ||
A supplemental inspection for these issues is scheduled in September 2012. | The NRC has determined that inspection objectives stated above have been met. Therefore in accordance with IMC 0305, Operating Reactor Assessment Program, this performance issue shall not be considered in the Action Matrix after the end of the third quarter of 2012. However, due to the performance issues in the Initiating Events cornerstone resulting in White (service water coupling failure) and Yellow (loss of a direct current bus) findings in the fourth quarter of 2011, the closure of this finding will not change Palisades Nuclear Plants status in the Degraded Cornerstone Column of the ROP Action Matrix. The licensee remains in Column III (Degraded Cornerstone) of the Action Matrix until resolution of the issues associated with that cornerstone. A supplemental inspection for these issues is scheduled in September 2012. | ||
In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's Agencywide Document Access and Management System (ADAMS). | In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's Agencywide Document Access and Management System (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room). | ||
ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room). | |||
Sincerely, | Sincerely, | ||
/RA/ | /RA/ | ||
John B. Giessner, Chief Branch 4 Division of Reactor Projects Docket Nos. 50-255 License No. DPR-20 | John B. Giessner, Chief Branch 4 Division of Reactor Projects Docket Nos. 50-255 License No. DPR-20 | ||
===Enclosure:=== | ===Enclosure:=== | ||
Inspection Report | Inspection Report 05000255/2012010 w/ Attachment: Supplemental Information | ||
REGION III== | REGION III== | ||
Docket No.: | Docket No.: 50-255 License No.: DPR-20 Report No.: 05000255/2012010 Licensee: Entergy Nuclear Operations, Inc. | ||
50-255 License No.: | |||
DPR-20 Report No.: | |||
05000255/2012010 Licensee: Entergy Nuclear Operations, Inc. | |||
Facility: Palisades Nuclear Plant Location: Covert, MI Dates: June 25, 2012, through June 29, 2012 Inspectors: J. Draper, Reactor Engineer, Lead Inspector S. Shah, Reactor Engineer Approved by: J. Giessner, Chief Branch 4 Division of Reactor Projects Enclosure | |||
Dates: June 25 , 2012 , through June 29 , 2012 Inspectors: | |||
J. Draper , Reactor Engineer, Lead Inspector S. Shah, Reactor Engineer Approved by: J. Giessner | |||
, Chief Branch 4 Division of Reactor | |||
=SUMMARY OF FINDINGS= | =SUMMARY OF FINDINGS= | ||
Inspection Report (IR) | Inspection Report (IR) 05000255/2012010; 6/25/2012 - 06/29/2012; Palisades Nuclear Plant, | ||
Unit 1; Supplemental Inspection - Inspection Procedure (IP) 95001 Two regional inspectors performed this inspection. | |||
===Cornerstone: Mitigating Systems=== | ===Cornerstone: Mitigating Systems=== | ||
The NRC staff performed this supplemental inspection in accordance with IP 95001 | The NRC staff performed this supplemental inspection in accordance with IP 95001 Supplemental Inspection for One or Two White Inputs in a Strategic Performance Area, to assess the licensee=s evaluation associated with the failure and inoperability of the turbine driven auxiliary feedwater (TDAFW) pump in May 2011 due to the invalid actuation of the turbine mechanical overspeed trip during testing. The NRC staff previously characterized this issue as having low to moderate safety significance, or white, as documented in NRC IR 05000255/2011017. During this supplemental inspection, the inspectors determined that the licensee performed a comprehensive evaluation of the self-revealed TDAFW pump failure, which occurred during a routine Technical Specification (TS) surveillance requirement (SR) test. | ||
=s evaluation associated with the failure and inoperability of the turbine driven auxiliary feedwater (TDAFW) pump in May 2011 due to the invalid actuation of the turbine mechanical overspeed trip during testing. The NRC staff previously characterized this issue as having low to moderate safety significance, or white, as documented in NRC IR 05000255/2011017. During this supplemental inspection | |||
, the inspectors determined that the licensee performed a comprehensive evaluation of the self-revealed TDAFW pump failure , which occurred during a routine Technical Specification (TS) surveillance requirement (SR) test | |||
The licensee identified the root cause of the failure to be inadequate preparation for the TDAFW turbine overhaul such that the reliability of the overspeed trip function was not ensured. The licensees evaluation determined that this root cause was limited to this overhaul activity, and has taken corrective actions to revise the maintenance and testing procedures so as to prevent recurrence of the issue. | |||
@ | Given the licensee=s acceptable performance in addressing the inoperable TDAFW pump, the white finding associated with this issue will only be considered in assessing plant performance for a total of four quarters in accordance with the guidance in Inspection Manual Chapter 0305, | ||
AOperating Reactor Assessment Program.@ Inspectors will review the licensees implementation of corrective actions during a future inspection. | |||
Findings | Findings No findings of significance were identified. | ||
=REPORT DETAILS= | =REPORT DETAILS= | ||
| Line 94: | Line 70: | ||
{{a|4OA4}} | {{a|4OA4}} | ||
==4OA4 Supplemental Inspection== | ==4OA4 Supplemental Inspection== | ||
{{IP sample|IP=IP 95001}} | |||
===.01 Inspection Scope=== | ===.01 Inspection Scope=== | ||
The NRC staff performed this supplemental inspection in accordance with Inspection Procedure (IP) | The NRC staff performed this supplemental inspection in accordance with Inspection Procedure (IP) 95001 to assess the licensees evaluation of a white finding, which affected the Mitigating Systems Cornerstone in the reactor safety strategic performance area. The inspection objectives were to: | ||
provide assurance that the root and contributing causes of risk | * provide assurance that the root and contributing causes of risk-significant issues were understood; | ||
-significant issues were understood; provide assurance that the extent of condition and extent of cause of risk | * provide assurance that the extent of condition and extent of cause of risk-significant issues were identified; and | ||
-significant issues were identified; | * provide assurance that the licensees corrective actions for risk-significant issues were or will be sufficient to address the root and contributing causes and to preclude repetition. | ||
-significant issues were or will be sufficient to address the root and contributing causes and to preclude repetition. | |||
The licensee entered the Degraded Cornerstone Column of the | The licensee entered the Degraded Cornerstone Column of the NRCs Action Matrix in the fourth quarter of 2011 as a result of a yellow finding in the Initiating Events Cornerstone originating in 4Q2011. Palisades also has a white finding in the Initiating Events Cornerstone and a white finding in the Mitigating Systems Cornerstone, both originating 4Q2011. The White finding in the Mitigating Systems Cornerstone was associated with the inoperability of the turbine driven auxiliary feedwater (TDAFW) pump and is the subject of this inspection. On May 10, 2011, during routine Technical Specification (TS) surveillance requirement (SR) testing, the TDAFW pump, P-8B, tripped due to the unexpected actuation of the mechanical overspeed trip linkage. The finding was characterized as having white safety significance based on the results of a Phase 3 risk analysis performed by a region-based senior reactor analyst (SRA), as discussed in NRC IR 05000255/2011013 and finalized in NRC IR 05000255/2011017. | ||
The failure of TDAFW pump P-8B during the TS SR test was attributed to an inadequate maintenance procedure that resulted in improper maintenance, including greasing, of the turbine overspeed trip mechanism during the October 2010 refueling outage. Following the trip, the licensee took immediate corrective actions to remove grease from the overspeed trip mechanism knife-edge which contributed to the trip. This action and the subsequent post-maintenance testing (PMT) restored the TDAFW pump to operable status on May 12, 2011. | |||
-8B during the TS SR test was attributed to an inadequate maintenance procedure that resulted in improper maintenance, including greasing, of the turbine overspeed trip mechanism during the October 2010 refueling outage. Following the trip, the licensee took immediate corrective actions to remove grease from the overspeed trip mechanism knife | |||
-edge which contributed to the trip. This action and the subsequent post | |||
-maintenance testing (PMT) restored the TDAFW pump to operable status on May 12, 2011. | |||
The licensee staff informed the NRC staff on May 2, 2012, that they were ready for the supplemental inspection. In preparation for the inspection, the licensee performed a root cause evaluation (RCE), CR | The licensee staff informed the NRC staff on May 2, 2012, that they were ready for the supplemental inspection. In preparation for the inspection, the licensee performed a root cause evaluation (RCE), CR-PLP-2011-5723, Revision 1, to determine the root and contributing causes of the TDAFW pump failure and to identify any organizational or programmatic issues or weaknesses in safety culture that contributed to the white finding. The licensee then performed a self-assessment of the root cause evaluation to assess the RCE quality. | ||
-PLP-2011-5723, Revision 1, to determine the root and contributing causes of the TDAFW pump failure and to identify any organizational or programmatic issues or weaknesses in safety culture that contributed to the white finding. The licensee then performed a self | |||
-assessment of the root cause evaluation to assess the RCE quality. | |||
The inspectors reviewed the | The inspectors reviewed the licensees RCE and self-assessment in addition to other corrective action program documents, procedures, work orders, and vendor information. | ||
-assessment in addition to other corrective action program documents, procedures, work orders, and vendor information. | |||
In addition, the team viewed the turbine | In addition, the team viewed the turbine-driven auxiliary feedwater pump as well as interviewed approximately 20 plant employees from various areas and levels of the licensees organization. These inspection activities were performed to aid the inspectors in determining whether root and contributing causes and the contribution of safety culture components were understood and corrective actions taken or planned were appropriate to address the causes and preclude repetition. | ||
-driven auxiliary feedwater pump as well as interviewed approximately 20 plant employees from various areas and levels of the | |||
===.02 Evaluation of the Inspection Requirements=== | ===.02 Evaluation of the Inspection Requirements=== | ||
02.01 Problem Identification a. IP 95001 requires that the inspection staff determine that the | 02.01 Problem Identification a. IP 95001 requires that the inspection staff determine that the licensees evaluation of the issue documents who identified the issue (i.e., licensee-identified, self-revealing, or NRC-identified) and the conditions under which the issue was identified. | ||
-identified, self | |||
-revealing, or NRC-identified) and the conditions under which the issue was identified. | |||
The inspectors determined that the RCE adequately discussed the identification of the issue. During the testing of P | The inspectors determined that the RCE adequately discussed the identification of the issue. During the testing of P-8B on May 10, 2011, the auxiliary feedwater pump tripped on overspeed, at which time the test was stopped and the pump was declared inoperable. The inspectors verified that this information was documented in the licensees RCE. In the problem statement of the root cause report, the licensee identified that the failure of the auxiliary feedwater pump was self-revealing when it tripped unexpectedly during the performance of TS Surveillance Procedure, RO-97, Auxiliary Feedwater System Automatic Initiation Test Procedure, resulting in additional unavailability time. Condition Report (CR) CR-PLP-2011-02350 was generated to document the failure. The root cause evaluation was completed under condition report CR-PLP-2011-05723 as a corrective action for the parent CR-PLP-2011-02350. | ||
-8B on May 10, 2011, the auxiliary feedwater pump tripped on overspeed, at which time the test was stopped and the pump was declared inoperable. The inspectors verified that this information was documented in | |||
-revealing when it tripped unexpectedly during the performance of TS Surveillance Procedure, RO | |||
-97, | |||
-PLP-2011-02350 was generated to document the failure. The root cause evaluation was completed under condition report CR-PLP-2011-05723 as a corrective action for the parent CR | |||
-PLP-2011-02350 | |||
b. IP 95001 requires that the inspection staff determine that the licensees evaluation of the issue documents how long the issue existed and prior opportunities for identification. | |||
This test operated the pump on recirculation to Condensate Storage Tank. The licensee did not identify any issues related to the overspeed trip mechanism during these two surveillance tests. The | The licensees RCE documented that the auxiliary feedwater pump was likely inoperable since the 10-year maintenance was last performed during the refueling outage on October 29, 2010, which was when the knife-edge was greased. As part of the maintenance activity, the turbine over speed mechanism was tested using an electric motor per procedure WI-214382 Auxiliary feedwater turbine K-8 over speed trip test using turbine trip overspeed device (TTOD). The turbine did not experience an overspeed trip during this test. The pump also successfully completed surveillance procedure QO-21B - P-8B, IST Auxiliary Feedwater System on 15 February 2011 under Work Order 52293482. This test operated the pump on recirculation to Condensate Storage Tank. The licensee did not identify any issues related to the overspeed trip mechanism during these two surveillance tests. The licensees past operability recommendation performed following the May 10, 2011 trip determined P-8B was inoperable between October 29, 2010 and May 10, 2011. | ||
-8B was inoperable between October 29, 2010 and May 10, 2011 | |||
The | The licensees RCE found that the 10-year overhaul performed during 1R21 refueling outage did not adequately refurbish, replace, and test governor and over speed trip components on P-8B. The licensees evaluation concluded that they had the opportunity to obtain the updated vendor bulletin (that included recommendations for measurement of trip lever shoulder stud clearance, trip lever lateral movement checks, governor valve seat rib location verification and documentation of visual checks of trip pin condition and orientation) since August 2007, but did not. | ||
The inspectors determined that the licensees evaluation was adequate with respect to identifying how long the issue existed and prior opportunities for identification. | |||
c. IP 95001 requires that the inspection staff determine that the licensees evaluation documents the plant specific risk consequences, as applicable, and compliance concerns associated with the issue(s). | |||
The | The NRC determined the May 2011 pump failure was a White finding, as documented in Inspection Report (IR) 05000255/2011017. The inspectors found that the licensees root cause evaluation did not address the plant specific risk consequences resulting from failure of the auxiliary feedwater pump. Although the quantitative risk consequences were documented by the licensee in EA-PSA-SDP-P8B-11-05, Risk assessment of Turbine Driven Auxiliary Feedwater Pump Trip, Revision 1, the inspectors observed that the RCE itself did not clearly state the actual (or potential) safety consequences arising from the failure condition. The licensee captured this observation in condition report, CR-PLP-2012-04767. The inspectors determined that this omission did not impact the effectiveness of the RCE as the detailed risk assessment was previously provided by the site and analyzed by NRC risk experts. | ||
The licensee evaluated the compliance concerns and determined the inoperability of the TDAFW was a violation of TS and Licensee Event Report (LER) 2011 01 has been submitted. The inspectors determined that the | The RCE included a discussion of nuclear, radiological and environmental safety significance and stated that no actual consequences resulted from the inoperability of auxiliary feedwater system. The licensees risk assessment concluded that the change in core damage frequency was less than 1E-06/yr, or Green. The differences between the NRCs risk evaluation and the licensees risk evaluation are discussed in IR 05000255/2011013. Interviews with licensee personnel indicated the licensee understood the differences between their evaluation and the NRC's Probabilistic Risk Assessment (PRA) evaluation. | ||
The licensee evaluated the compliance concerns and determined the inoperability of the TDAFW was a violation of TS and Licensee Event Report (LER) 2011-004-01 has been submitted. The inspectors determined that the licensees evaluation in this area was adequate. | |||
====d. Findings==== | ====d. Findings==== | ||
| Line 170: | Line 116: | ||
The licensee used the following systematic methods to complete the root cause evaluation: | The licensee used the following systematic methods to complete the root cause evaluation: | ||
data gathering through interviews, document review, and discussions with the turbine vendor, Elliott; | * data gathering through interviews, document review, and discussions with the turbine vendor, Elliott; | ||
* contracted MPR Associates, Inc., to perform an overspeed trip mechanism linkage force calculation; | |||
* event timeline; | |||
* failure mode analysis; | |||
* failure fault tree; | |||
* hazard, barrier and target analysis. | |||
The inspectors determined that the licensee evaluated the issue using a systematic methodology to identify root and contributing causes. | The inspectors determined that the licensee evaluated the issue using a systematic methodology to identify root and contributing causes. | ||
b. IP 95001 requires that the inspection staff determine that the | b. IP 95001 requires that the inspection staff determine that the licensees RCE was conducted to a level of detail commensurate with the significance of the issue. | ||
The licensee used a root cause evaluation team that consisted of members of mechanical maintenance, operations, and engineering, as well as a consultant. This team performed a detailed evaluation that identified several maintenance deficiencies on the overspeed trip mechanism linkage that, when combined, made the pump susceptible to invalid trips at speeds below the overspeed trip setpoint. These maintenance deficiencies included failing to: rotate the knife edge, replace the latch plate, replace the reset lever springs, verify there is no foreign material in the governor, adjust the trip mechanism backseat, and verify the overspeed gap is correct. These deficiencies that were the root cause of the failure were identified through the failure mode analysis performed by the RCE team. | The licensee used a root cause evaluation team that consisted of members of mechanical maintenance, operations, and engineering, as well as a consultant. This team performed a detailed evaluation that identified several maintenance deficiencies on the overspeed trip mechanism linkage that, when combined, made the pump susceptible to invalid trips at speeds below the overspeed trip setpoint. These maintenance deficiencies included failing to: rotate the knife edge, replace the latch plate, replace the reset lever springs, verify there is no foreign material in the governor, adjust the trip mechanism backseat, and verify the overspeed gap is correct. These deficiencies that were the root cause of the failure were identified through the failure mode analysis performed by the RCE team. | ||
The RCE | The RCE teams failure mode analysis evaluated each potential failure mode (several of which were discussed in Elliott Technical Service Letter ETSL-T-2012, Spurious Trips on YR Turbines, issued on August 1, 2007) to determine whether evidence supported or refuted it as a possible cause. The licensee determined that while each individual failure mode would not have caused, by itself, the failure of the pump on May 10, 2011, the combination of several of these failure modes put the turbine in a condition where it was susceptible to inadvertent actuation of the overspeed trip mechanism. | ||
-T-2012, | |||
In addition to the root cause, the RCE team identified several more fundamental problems that contributed to the failure and the root cause. Some of these contributing causes were deficiencies associated with the control of critical maintenance, the control of supplemental workers, and the implementation of new vendor information. Based on the comprehensive evaluation performed for this root cause evaluation, the inspectors concluded that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem. | In addition to the root cause, the RCE team identified several more fundamental problems that contributed to the failure and the root cause. Some of these contributing causes were deficiencies associated with the control of critical maintenance, the control of supplemental workers, and the implementation of new vendor information. Based on the comprehensive evaluation performed for this root cause evaluation, the inspectors concluded that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem. | ||
c. IP 95001 requires that the inspection staff determine that the | c. IP 95001 requires that the inspection staff determine that the licensees RCE included a consideration of prior occurrences of the issue and knowledge of operating experience (OE). | ||
-driven equipment, both internally and from the industry. However, the RCE did not include an evaluation of previous events associated with the contributing causes, specifically, examples of issues with the control of critical maintenance and supplemental workers. This OE evaluation, though, was performed in the | |||
-assessment of the RCE, so the inspectors found this evaluation to be acceptable. | The licensees RCE included an evaluation of internal and external OE. This OE evaluation included previous similar events involving overspeed protection issues associated with steam-driven equipment, both internally and from the industry. | ||
However, the RCE did not include an evaluation of previous events associated with the contributing causes, specifically, examples of issues with the control of critical maintenance and supplemental workers. This OE evaluation, though, was performed in the licensees self-assessment of the RCE, so the inspectors found this evaluation to be acceptable. | |||
One of the issues identified in the licensees RCE is the failure to incorporate vendor recommendations into the maintenance procedure for the TDAFW turbine overhaul. | |||
This was identified as one of the contributing causes to the TDAFW failure. In response to this deficiency, the licensee initiated a corrective action to ensure that these recommendations were evaluated and incorporated into the maintenance procedure. | |||
The licensee also has a corrective action to review preventative maintenance activities and equipment overhaul activities to ensure that appropriate vendor recommendations and information have been reviewed and incorporated into the work instructions and procedures. | |||
Based on the licensees evaluation and conclusions, both in the RCE and in the RCE self-assessment, the inspectors determined that the licensees RCE included a consideration of prior occurrences of the problem and knowledge of prior OE. | |||
d. IP 95001 requires that the inspection staff determine that the licensees RCE addresses the extent of condition and extent of cause of the issues. | |||
The | The licensees evaluation considered the extent of condition associated with the degraded turbine overspeed trip device as well as the work performed by the maintenance contractors who lubricated the incorrect linkages. With respect to the turbine trip, the licensee determined that no other auxiliary feedwater system equipment was adversely affected based on subsequent surveillance test performances. With respect to the degraded overspeed trip device, the licensee determined that no other steam-driven equipment in the plant was susceptible to this same type of failure. Finally, with respect to the work performed by the maintenance contractors, the licensee determined that the maintenance on the TDAFW pump was the only work performed by this maintenance crew during the refueling outage. | ||
The inspectors concluded that the | The licensees evaluation also considered the extent of cause associated with the planning and preparation for maintenance on critical equipment by supplemental workers. The licensee reviewed other outage work, equipment overhaul work, and preventative maintenance activities to look for weaknesses in the control of critical tasks and contract workers. The licensee also looked for the incorporation of vendor information, which was a contributing cause. The licensee identified that the lack of a final procedure relative to supplemental workers was a weakness, and has finalized procedure EN-MA-101-01, Critical Maintenance Identification and Oversight. No significant deficiencies were found. The inspectors concluded that the licensees RCE addressed the extent of condition and the extent of cause of the issue. | ||
e. IP 95001 requires that the inspection staff determine that the | e. IP 95001 requires that the inspection staff determine that the licensees root cause, extent of condition, and extent of cause evaluations appropriately considered the safety culture components as described in IMC 0310, Components Within the Cross-Cutting Areas. | ||
-Cutting Areas | |||
The licensee also identified a weakness in the cross-cutting area of Problem Identification and Resolution in the component of Operating Experience, and a weakness in the cross | The licensee identified several weaknesses in the cross-cutting area (cross-cutting areas represent attributes of safety culture) of Human Performance, including in the cross-cutting components of Resources and Work Control. The licensee also identified a weakness in the cross-cutting area of Problem Identification and Resolution in the component of Operating Experience, and a weakness in the cross-cutting component of Accountability, which is not associated with a cross-cutting area. Specifically, the licensee identified that work authorizing documents did not include details appropriate for outage work on high critical equipment (Resources); the Maintenance organization did not recognize or prepare for outage work on high critical equipment as performed by supplemental workers (Work Control); available vendor and industry operating experience was not fully reflected into work authorizing documents or referenced maintenance procedures (Operating Experience), and managers and supervisors did not reinforce adherence to procedures among those doing the work on the TDAFW pump turbine overhaul (Accountability). These weaknesses correlate to the cross-cutting aspects H.2(c), H.3(a), P.2(a), and O.1(b), respectively, as described in Inspection Manual Chapter (IMC) 0310 dated October 28, 2011. | ||
-cutting component of Accountability, which is not associated with a cross | |||
-cutting area. Specifically, the licensee identified that work authorizing documents did not include details appropriate for outage work on high critical equipment (Resources); the Maintenance organization did not recognize or prepare for outage work on high critical equipment as performed by supplemental workers (Work Control); available vendor and industry operating experience was not fully reflected into work authorizing documents or referenced maintenance procedures (Operating Experience), and managers and supervisors did not reinforce adherence to procedures among those doing the work on the TDAFW pump turbine overhaul (Accountability). These weaknesses correlate to the cross | |||
-cutting aspects H.2(c), H.3(a), P.2(a), and O.1(b), respectively, as described in Inspection Manual Chapter (IMC)0310 dated October 28, 2011. | |||
The licensee addressed these weaknesses with a series of corrective actions including reviewing work activities to verify critical maintenance is identified and reviewing work orders and preventative maintenance activities to verify vendor information has been incorporated. The inspectors determined that the | The licensee addressed these weaknesses with a series of corrective actions including reviewing work activities to verify critical maintenance is identified and reviewing work orders and preventative maintenance activities to verify vendor information has been incorporated. The inspectors determined that the licensees RCE included a proper consideration of whether weaknesses in any safety culture components were root causes or significant contributing causes of the issue. | ||
====f. Findings==== | ====f. Findings==== | ||
| Line 220: | Line 166: | ||
: (2) an evaluation that states no actions are necessary is adequate. | : (2) an evaluation that states no actions are necessary is adequate. | ||
The licensee implemented immediate corrective actions to restore the auxiliary feedwater | The licensee implemented immediate corrective actions to restore the auxiliary feedwater pumps operability. They cleaned the grease from the knife edge, checked the trip linkage and cleaned the steam traps. In addition, to address the root causes, they revised FWS-M-6, Auxiliary Feedwater Turbine Maintenance, to incorporate recommendations from the Elliott Technical Service Letter ETSL-2012, industry experience and the results from the root cause evaluation. This included identification of critical steps and checks of linkage and trip pin clearances with provisions for recording of as-found data for review by plant personnel. Changes implemented by the licensee also included requirements for periodic replacement of trip mechanism springs, latch parts, trip pin and plunger. Post-maintenance testing instructions in the maintenance procedure were changed from suggestions to requirements. The corrective actions for the root and contributing causes listed in RCE CR-PLP-2011-05723 appear to be appropriate. | ||
To address the contributing causes such as poor vendor information control, the licensee established a program to ensure that the system engineer validates any vendor recommendations, vendor bulletins or operating experience prior to performance of preventative maintenance on equipment that requires overhaul or refurbishment. The licensee also implemented programs to ensure that the requirements of critical maintenance were met in accordance with procedure EN-MA-101-01, Critical Maintenance Identification and Oversight, and work package evaluations adequately addressed supplemental worker performance during outages. Although substantial progress was achieved in the area of supplemental worker performance, the inspectors identified instances where additional licensee oversight was required. | |||
During refueling outage 1R22, the licensees quality assurance staff identified that prior to the pressurizer manway installation, the licensee failed to brief supplemental workers in accordance with plant procedure EN-HU-102, Human Performance Tools. The briefing was performed using a vendor pre-job brief checklist and did not contain all the information required by procedure EN-HU-102. The brief was led by the vendor personnel with no Entergy supervision. The licensee captured this issue in CR-PLP-2012-03560 and CR-PLP-2012-03561. Also, on May 2, 2012, the licensee identified that steam generator primary manway ultrasonic testing readings were outside the acceptable range. The licensee completed an apparent cause evaluation (CR-PLP-2012-03497) and concluded that a supplemental worker had programmed the incorrect stud material configuration which resulted in the ultrasonic readings being outside the required range. The procedure used by the worker did not identify the stud material and the worker assumed that the material used at Palisades was similar to the ones used at other facilities. Tensioning activities for the steam generator were stopped by management and they determined a more appropriate course of action to ensure that the elongation measurements for the steam generator man ways were acceptable going forward. Also, during the same refueling outage, on April 29, 2012, during reactor head reassembly, damage was caused to the incore instrument flange #6 tree assembly and bull nose. The root cause evaluation (CR-PLP-2012-03411) performed by the licensee concluded that the supplemental workers did not positively verify proper alignment nor perform the procedure step for guiding the bullet nose into the nozzle and that procedure RFL-R-16, Reactor Vessel Closure Head Installation, did not adequately provide proper instructions or acceptance criteria. Work was stopped and the reactor head was safety placed on the head stand. Although there were no consequences of safety significance in these three instances, the inspectors stressed the importance of ensuring proper oversight of supplemental workers to prevent recurrence of such events. | |||
The inspectors determined they had reasonable assurance that the licensees proposed corrective actions were appropriate and addressed each root and contributing cause so this area may be closed. The inspectors note this area should be a continuing focus at the site. | |||
b. IP 95001 requires that the inspection staff determine that the licensee prioritized corrective actions with consideration of risk significance and regulatory compliance. | |||
The | The licensees immediate corrective actions restored the auxiliary feedwater pump to operable status. After restoring the pump, the licensee tested P-8B to ensure that the pump would perform its intended functions when required. The resident inspectors witnessed this testing and observed that the auxiliary feedwater pump successfully passed the surveillance test. | ||
-8B to ensure that the pump would perform its intended functions when required. The resident inspectors witnessed this testing and observed that the auxiliary feedwater pump successfully passed the surveillance test. | |||
The | The licensees corrective actions to address the root and contributing causes were prioritized in accordance with procedure, EN-LI-102, Corrective Action Process, and Root Cause Evaluation Quality Checklist/Score Sheets. The corrective actions were scored based on licensing, regulatory performance and nuclear safety factors. The licensees plan to verify vendor information was implemented based on the risk significance of the equipment. The licensee implemented programs to ensure that critical steps were identified in accordance with plant procedure EN-MA-101-01, Critical Maintenance Identification and Oversight, prior to performance of critical maintenance activities. The inspectors reviewed the licensees plans for accomplishing the above tasks and noted that the risk significance of the equipment was being appropriately considered. Based on the guidance provided in the licensees procedures and the prioritization of the corrective actions in accordance with these procedures, the inspectors determined that the corrective actions were prioritized with consideration of the risk significance and regulatory performance. | ||
-LI-102, | |||
-MA-101-01, | |||
c. IP 95001 requires that the inspection staff determine that the licensee established a schedule for implementing and completing the corrective actions. | c. IP 95001 requires that the inspection staff determine that the licensee established a schedule for implementing and completing the corrective actions. | ||
The licensee established due dates for the corrective actions in accordance with procedure EN | The licensee established due dates for the corrective actions in accordance with procedure EN-LI-102, Corrective Action Process, requirements for timeliness. | ||
-LI-102, | |||
-PLP-2011-05723; however, many of the due dates for the action items were contained throughout the | Some of the due dates were captured in CR-PLP-2011-05723; however, many of the due dates for the action items were contained throughout the licensees corrective actions program in various condition reports. The inspectors determined that the licensee adequately established a schedule for implementing and completing the corrective actions. The corrective actions for this RCE were completed in June 2012. | ||
All CAPRs have been completed and effectiveness reviews for those CAPRs will be completed by November 2013. The inspectors considered the timeline for completion of corrective actions to be appropriate. | |||
d. IP 95001 requires that the inspection staff determine that the licensee developed quantitative and/or qualitative measures of success for determining the effectiveness of the corrective actions to preclude repetition. | d. IP 95001 requires that the inspection staff determine that the licensee developed quantitative and/or qualitative measures of success for determining the effectiveness of the corrective actions to preclude repetition. | ||
As documented in root cause evaluation, CR-PLP-2011-05723, the licensee established measures for determining the effectiveness of the corrective actions. These measures included: | As documented in root cause evaluation, CR-PLP-2011-05723, the licensee established measures for determining the effectiveness of the corrective actions. These measures included: | ||
-PLP-2012-0362 will ensure future control of vendor documents, including changes, for plant equipment and have bounded the vendor document control deficiencies identified; No K-8 related plant unexpected Limited Condition of Operation (LCO) due to maintenance activities; and Revised auxiliary feedwater pump procedure updated to reflect results of the RCE for all aspects of turbine overhaul work. | * Enhanced monitoring of maintenance and corrective action records related to K-8 equipment to ensure that any additional failures are given appropriate management attention; | ||
* Verify corrective actions resulting from evaluation of CR-PLP-2012-0362 will ensure future control of vendor documents, including changes, for plant equipment and have bounded the vendor document control deficiencies identified; | |||
* No K-8 related plant unexpected Limited Condition of Operation (LCO) due to maintenance activities; and | |||
* Revised auxiliary feedwater pump procedure updated to reflect results of the RCE for all aspects of turbine overhaul work. | |||
The licensee staff entered these corrective action items into their corrective action program to ensure that these effectiveness reviews and enhanced monitoring were performed. The inspectors determined that quantitative and qualitative measures of success had been developed for determining the effectiveness of the corrective actions to preclude repetition. | The licensee staff entered these corrective action items into their corrective action program to ensure that these effectiveness reviews and enhanced monitoring were performed. The inspectors determined that quantitative and qualitative measures of success had been developed for determining the effectiveness of the corrective actions to preclude repetition. | ||
e. IP 95001 requires that the inspection staff determine that the | e. IP 95001 requires that the inspection staff determine that the licensees planned or taken corrective actions adequately address a Notice of Violation (NOV) that was the basis for the supplemental inspection, if applicable. | ||
The NRC issued an NOV to the licensee on January 3, 2012. The Notice of Violation associated with the White finding that was the subject of this IP 95001 inspection identified one violation of NRC requirements. In particular, a violation of TS 3.7.5 occurred from October 29, 2010, to May 10, 2011, due to the safety related steam driven auxiliary feedwater pump P | The NRC issued an NOV to the licensee on January 3, 2012. The Notice of Violation associated with the White finding that was the subject of this IP 95001 inspection identified one violation of NRC requirements. In particular, a violation of TS 3.7.5 occurred from October 29, 2010, to May 10, 2011, due to the safety related steam driven auxiliary feedwater pump P-8B being inoperable as a result of maintenance not being prescribed by documented instructions or procedures. The NRC has concluded that information regarding the reasons for the violation, the corrective actions taken and planned to be taken to correct the violation and prevent recurrence, and the date when full compliance was achieved, is already adequately addressed on the docket in NRC IR No. 05000255/2011013 and in the licensees response to the NRC dated November 28, 2011. During this inspection, the inspectors confirmed that the licensees RCE and planned and taken corrective actions addressed the NOV. The licensee restored full compliance on May 11, 2011, by cleaning the grease from the knife edge, checking the trip linkage and performing a surveillance run of the pump. | ||
-8B being inoperable as a result of maintenance not being prescribed by documented instructions or procedures. The NRC has concluded that information regarding the reasons for the violation, the corrective actions taken and planned to be taken to correct the violation and prevent recurrence, and the date when full compliance was achieved, is already adequately addressed on the docket in NRC IR No. 05000255/2011013 and in the | |||
====f. Findings==== | ====f. Findings==== | ||
| Line 269: | Line 207: | ||
{{a|4OA6}} | {{a|4OA6}} | ||
==4OA6 Exit Meeting== | ==4OA6 Exit Meeting== | ||
Regulatory Performance Meeting On June 29, 2012, the NRC met with the licensee to discuss its performance in accordance with IMC 0305, Section 10.02.b.4. During this meeting, the NRC and licensee discussed the issues related to this White finding that resulted in Palisades Nuclear Plant being placed in a Column of the | On June 29, 2012, the inspectors presented the inspection results to Mr. Anthony Vitale, Vice-President of Operations, Palisades Nuclear Plant, and other members of his staff. | ||
The inspectors asked the licensee if any of the material examined during the inspection should be considered proprietary. The licensee did not identify any proprietary information. | |||
Regulatory Performance Meeting On June 29, 2012, the NRC met with the licensee to discuss its performance in accordance with IMC 0305, Section 10.02.b.4. During this meeting, the NRC and licensee discussed the issues related to this White finding that resulted in Palisades Nuclear Plant being placed in a Column of the NRCs ROP Action matrix with additional oversight. This discussion included the causes, corrective actions, extent of condition, extent of cause, and other planned licensee actions. | |||
ATTACHMENT: | ATTACHMENT: | ||
| Line 277: | Line 218: | ||
=SUPPLEMENTAL INFORMATION= | =SUPPLEMENTAL INFORMATION= | ||
==KEY POINTS OF CONTACT== | |||
Licensee | |||
: [[contact::A. Vitale]], Vice President, Operations | : [[contact::A. Vitale]], Vice President, Operations | ||
: [[contact::M. Sicard]], Special Projects | : [[contact::M. Sicard]], Special Projects | ||
: [[contact::A. Bono]], Engineering Inspection Support | : [[contact::A. Bono]], Engineering Inspection Support | ||
: [[contact::C. Arnone]], NSA Director | : [[contact::C. Arnone]], NSA Director | ||
Nuclear Regulatory Commission | |||
: [[contact::J. Jandovitz]], Acting Chief, Division of Reactor Projects, Branch 4 | : [[contact::J. Jandovitz]], Acting Chief, Division of Reactor Projects, Branch 4 | ||
: [[contact::J. Giessner]], Chief, Division of Reactor Projects, Branch 4 | : [[contact::J. Giessner]], Chief, Division of Reactor Projects, Branch 4 | ||
: [[contact::T. Taylor]], Senior Resident Inspector, Palisades | : [[contact::T. Taylor]], Senior Resident Inspector, Palisades | ||
: [[contact::A. Scarbeary]], Resident Inspector, Palisades | : [[contact::A. Scarbeary]], Resident Inspector, Palisades | ||
: [[contact::J. Ellegood]], Senior Resident Inspector, Donald C. Cook | : [[contact::J. Ellegood]], Senior Resident Inspector, Donald C. Cook | ||
==LIST OF ITEMS== | |||
LIST OF | |||
===OPENED, CLOSED AND DISCUSSED=== | |||
===Opened=== | |||
None. | |||
===Closed=== | |||
: 05000255/2011013-01 VIO Improper Lubrication of Turbine Driven Auxiliary Feedwater Pump Linkages Attachment | |||
==LIST OF DOCUMENTS REVIEWED== | |||
}} | }} | ||
Revision as of 00:26, 12 November 2019
| ML12213A225 | |
| Person / Time | |
|---|---|
| Site: | Palisades |
| Issue date: | 07/31/2012 |
| From: | Jack Giessner Reactor Projects Region 3 Branch 4 |
| To: | Vitale A Entergy Nuclear Operations |
| References | |
| EA-11-227 IR-12-010 | |
| Download: ML12213A225 (21) | |
Text
uly 31, 2012
SUBJECT:
PALISADES NUCLEAR PLANT - NRC SUPPLEMENTAL INSPECTION REPORT 05000255/2012010, AND ASSESSMENT FOLLOW-UP LETTER
Dear Mr. Vitale:
On June 29, 2012, the U.S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection pursuant to Inspection Procedure 95001, Supplemental Inspection for One or Two White Inputs in a Strategic Performance Area, at your Palisades Nuclear Plant. The enclosed inspection report documents the inspection results, which were discussed on June 29, 2012, with you and other members of your staff.
In accordance with the NRC Reactor Oversight Process Action Matrix, this supplemental inspection was performed to follow-up on a White finding in the Mitigating Systems Cornerstone with low to moderate safety significance which occurred in the fourth quarter of 2011. This issue involved the inadvertent actuation of the overspeed trip mechanism on the turbine-driven auxiliary feedwater pump turbine during surveillance testing and was previously documented and assessed in NRC Inspection Reports 05000255/2011013 and 05000255/2011017. The NRC was informed on May 2, 2012, of your staffs readiness for this inspection.
The objectives of this supplemental inspection were to provide assurance that: (1) the root causes and the contributing causes for the risk-significant issues were understood; (2) the extent of condition and extent of cause of the issues were identified; and (3) corrective actions were or will be sufficient to address and preclude repetition of the root and contributing causes.
The NRC determined that the staff at Palisades Nuclear Plant performed an acceptable evaluation of the White finding. The evaluation identified the primary root cause of the issue to be inadequate preparation and instruction related to maintenance on the turbine-driven auxiliary feedwater pump overspeed trip mechanism linkages during the turbine overhaul performed during refueling outage 1R21. Specifically, the licensee failed to perform several vendor-recommended maintenance activities such that the reliability of the turbine-driven auxiliary feedwater pump turbine overspeed trip device was not maintained. To correct this issue and prevent recurrence, the licensee revised the maintenance and post-maintenance testing procedures to include these maintenance activities and verify the reliability of the overspeed trip mechanism. No findings were identified during this inspection.
The NRC has determined that inspection objectives stated above have been met. Therefore in accordance with IMC 0305, Operating Reactor Assessment Program, this performance issue shall not be considered in the Action Matrix after the end of the third quarter of 2012. However, due to the performance issues in the Initiating Events cornerstone resulting in White (service water coupling failure) and Yellow (loss of a direct current bus) findings in the fourth quarter of 2011, the closure of this finding will not change Palisades Nuclear Plants status in the Degraded Cornerstone Column of the ROP Action Matrix. The licensee remains in Column III (Degraded Cornerstone) of the Action Matrix until resolution of the issues associated with that cornerstone. A supplemental inspection for these issues is scheduled in September 2012.
In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's Agencywide Document Access and Management System (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
John B. Giessner, Chief Branch 4 Division of Reactor Projects Docket Nos. 50-255 License No. DPR-20
Enclosure:
Inspection Report 05000255/2012010 w/ Attachment: Supplemental Information
REGION III==
Docket No.: 50-255 License No.: DPR-20 Report No.: 05000255/2012010 Licensee: Entergy Nuclear Operations, Inc.
Facility: Palisades Nuclear Plant Location: Covert, MI Dates: June 25, 2012, through June 29, 2012 Inspectors: J. Draper, Reactor Engineer, Lead Inspector S. Shah, Reactor Engineer Approved by: J. Giessner, Chief Branch 4 Division of Reactor Projects Enclosure
SUMMARY OF FINDINGS
Inspection Report (IR) 05000255/2012010; 6/25/2012 - 06/29/2012; Palisades Nuclear Plant,
Unit 1; Supplemental Inspection - Inspection Procedure (IP) 95001 Two regional inspectors performed this inspection.
Cornerstone: Mitigating Systems
The NRC staff performed this supplemental inspection in accordance with IP 95001 Supplemental Inspection for One or Two White Inputs in a Strategic Performance Area, to assess the licensee=s evaluation associated with the failure and inoperability of the turbine driven auxiliary feedwater (TDAFW) pump in May 2011 due to the invalid actuation of the turbine mechanical overspeed trip during testing. The NRC staff previously characterized this issue as having low to moderate safety significance, or white, as documented in NRC IR 05000255/2011017. During this supplemental inspection, the inspectors determined that the licensee performed a comprehensive evaluation of the self-revealed TDAFW pump failure, which occurred during a routine Technical Specification (TS) surveillance requirement (SR) test.
The licensee identified the root cause of the failure to be inadequate preparation for the TDAFW turbine overhaul such that the reliability of the overspeed trip function was not ensured. The licensees evaluation determined that this root cause was limited to this overhaul activity, and has taken corrective actions to revise the maintenance and testing procedures so as to prevent recurrence of the issue.
Given the licensee=s acceptable performance in addressing the inoperable TDAFW pump, the white finding associated with this issue will only be considered in assessing plant performance for a total of four quarters in accordance with the guidance in Inspection Manual Chapter 0305,
AOperating Reactor Assessment Program.@ Inspectors will review the licensees implementation of corrective actions during a future inspection.
Findings No findings of significance were identified.
REPORT DETAILS
OTHER ACTIVITIES
4OA4 Supplemental Inspection
.01 Inspection Scope
The NRC staff performed this supplemental inspection in accordance with Inspection Procedure (IP) 95001 to assess the licensees evaluation of a white finding, which affected the Mitigating Systems Cornerstone in the reactor safety strategic performance area. The inspection objectives were to:
- provide assurance that the root and contributing causes of risk-significant issues were understood;
- provide assurance that the extent of condition and extent of cause of risk-significant issues were identified; and
- provide assurance that the licensees corrective actions for risk-significant issues were or will be sufficient to address the root and contributing causes and to preclude repetition.
The licensee entered the Degraded Cornerstone Column of the NRCs Action Matrix in the fourth quarter of 2011 as a result of a yellow finding in the Initiating Events Cornerstone originating in 4Q2011. Palisades also has a white finding in the Initiating Events Cornerstone and a white finding in the Mitigating Systems Cornerstone, both originating 4Q2011. The White finding in the Mitigating Systems Cornerstone was associated with the inoperability of the turbine driven auxiliary feedwater (TDAFW) pump and is the subject of this inspection. On May 10, 2011, during routine Technical Specification (TS) surveillance requirement (SR) testing, the TDAFW pump, P-8B, tripped due to the unexpected actuation of the mechanical overspeed trip linkage. The finding was characterized as having white safety significance based on the results of a Phase 3 risk analysis performed by a region-based senior reactor analyst (SRA), as discussed in NRC IR 05000255/2011013 and finalized in NRC IR 05000255/2011017.
The failure of TDAFW pump P-8B during the TS SR test was attributed to an inadequate maintenance procedure that resulted in improper maintenance, including greasing, of the turbine overspeed trip mechanism during the October 2010 refueling outage. Following the trip, the licensee took immediate corrective actions to remove grease from the overspeed trip mechanism knife-edge which contributed to the trip. This action and the subsequent post-maintenance testing (PMT) restored the TDAFW pump to operable status on May 12, 2011.
The licensee staff informed the NRC staff on May 2, 2012, that they were ready for the supplemental inspection. In preparation for the inspection, the licensee performed a root cause evaluation (RCE), CR-PLP-2011-5723, Revision 1, to determine the root and contributing causes of the TDAFW pump failure and to identify any organizational or programmatic issues or weaknesses in safety culture that contributed to the white finding. The licensee then performed a self-assessment of the root cause evaluation to assess the RCE quality.
The inspectors reviewed the licensees RCE and self-assessment in addition to other corrective action program documents, procedures, work orders, and vendor information.
In addition, the team viewed the turbine-driven auxiliary feedwater pump as well as interviewed approximately 20 plant employees from various areas and levels of the licensees organization. These inspection activities were performed to aid the inspectors in determining whether root and contributing causes and the contribution of safety culture components were understood and corrective actions taken or planned were appropriate to address the causes and preclude repetition.
.02 Evaluation of the Inspection Requirements
02.01 Problem Identification a. IP 95001 requires that the inspection staff determine that the licensees evaluation of the issue documents who identified the issue (i.e., licensee-identified, self-revealing, or NRC-identified) and the conditions under which the issue was identified.
The inspectors determined that the RCE adequately discussed the identification of the issue. During the testing of P-8B on May 10, 2011, the auxiliary feedwater pump tripped on overspeed, at which time the test was stopped and the pump was declared inoperable. The inspectors verified that this information was documented in the licensees RCE. In the problem statement of the root cause report, the licensee identified that the failure of the auxiliary feedwater pump was self-revealing when it tripped unexpectedly during the performance of TS Surveillance Procedure, RO-97, Auxiliary Feedwater System Automatic Initiation Test Procedure, resulting in additional unavailability time. Condition Report (CR) CR-PLP-2011-02350 was generated to document the failure. The root cause evaluation was completed under condition report CR-PLP-2011-05723 as a corrective action for the parent CR-PLP-2011-02350.
b. IP 95001 requires that the inspection staff determine that the licensees evaluation of the issue documents how long the issue existed and prior opportunities for identification.
The licensees RCE documented that the auxiliary feedwater pump was likely inoperable since the 10-year maintenance was last performed during the refueling outage on October 29, 2010, which was when the knife-edge was greased. As part of the maintenance activity, the turbine over speed mechanism was tested using an electric motor per procedure WI-214382 Auxiliary feedwater turbine K-8 over speed trip test using turbine trip overspeed device (TTOD). The turbine did not experience an overspeed trip during this test. The pump also successfully completed surveillance procedure QO-21B - P-8B, IST Auxiliary Feedwater System on 15 February 2011 under Work Order 52293482. This test operated the pump on recirculation to Condensate Storage Tank. The licensee did not identify any issues related to the overspeed trip mechanism during these two surveillance tests. The licensees past operability recommendation performed following the May 10, 2011 trip determined P-8B was inoperable between October 29, 2010 and May 10, 2011.
The licensees RCE found that the 10-year overhaul performed during 1R21 refueling outage did not adequately refurbish, replace, and test governor and over speed trip components on P-8B. The licensees evaluation concluded that they had the opportunity to obtain the updated vendor bulletin (that included recommendations for measurement of trip lever shoulder stud clearance, trip lever lateral movement checks, governor valve seat rib location verification and documentation of visual checks of trip pin condition and orientation) since August 2007, but did not.
The inspectors determined that the licensees evaluation was adequate with respect to identifying how long the issue existed and prior opportunities for identification.
c. IP 95001 requires that the inspection staff determine that the licensees evaluation documents the plant specific risk consequences, as applicable, and compliance concerns associated with the issue(s).
The NRC determined the May 2011 pump failure was a White finding, as documented in Inspection Report (IR) 05000255/2011017. The inspectors found that the licensees root cause evaluation did not address the plant specific risk consequences resulting from failure of the auxiliary feedwater pump. Although the quantitative risk consequences were documented by the licensee in EA-PSA-SDP-P8B-11-05, Risk assessment of Turbine Driven Auxiliary Feedwater Pump Trip, Revision 1, the inspectors observed that the RCE itself did not clearly state the actual (or potential) safety consequences arising from the failure condition. The licensee captured this observation in condition report, CR-PLP-2012-04767. The inspectors determined that this omission did not impact the effectiveness of the RCE as the detailed risk assessment was previously provided by the site and analyzed by NRC risk experts.
The RCE included a discussion of nuclear, radiological and environmental safety significance and stated that no actual consequences resulted from the inoperability of auxiliary feedwater system. The licensees risk assessment concluded that the change in core damage frequency was less than 1E-06/yr, or Green. The differences between the NRCs risk evaluation and the licensees risk evaluation are discussed in IR 05000255/2011013. Interviews with licensee personnel indicated the licensee understood the differences between their evaluation and the NRC's Probabilistic Risk Assessment (PRA) evaluation.
The licensee evaluated the compliance concerns and determined the inoperability of the TDAFW was a violation of TS and Licensee Event Report (LER) 2011-004-01 has been submitted. The inspectors determined that the licensees evaluation in this area was adequate.
d. Findings
No findings of significance were identified.
02.02 Root Cause, Extent of Condition, and Extent of Cause Evaluation a. IP 95001 requires that the inspection staff determine that the licensee evaluated the issue using a systematic methodology to identify the root and contributing causes.
The licensee used the following systematic methods to complete the root cause evaluation:
- data gathering through interviews, document review, and discussions with the turbine vendor, Elliott;
- contracted MPR Associates, Inc., to perform an overspeed trip mechanism linkage force calculation;
- event timeline;
- failure mode analysis;
- failure fault tree;
- hazard, barrier and target analysis.
The inspectors determined that the licensee evaluated the issue using a systematic methodology to identify root and contributing causes.
b. IP 95001 requires that the inspection staff determine that the licensees RCE was conducted to a level of detail commensurate with the significance of the issue.
The licensee used a root cause evaluation team that consisted of members of mechanical maintenance, operations, and engineering, as well as a consultant. This team performed a detailed evaluation that identified several maintenance deficiencies on the overspeed trip mechanism linkage that, when combined, made the pump susceptible to invalid trips at speeds below the overspeed trip setpoint. These maintenance deficiencies included failing to: rotate the knife edge, replace the latch plate, replace the reset lever springs, verify there is no foreign material in the governor, adjust the trip mechanism backseat, and verify the overspeed gap is correct. These deficiencies that were the root cause of the failure were identified through the failure mode analysis performed by the RCE team.
The RCE teams failure mode analysis evaluated each potential failure mode (several of which were discussed in Elliott Technical Service Letter ETSL-T-2012, Spurious Trips on YR Turbines, issued on August 1, 2007) to determine whether evidence supported or refuted it as a possible cause. The licensee determined that while each individual failure mode would not have caused, by itself, the failure of the pump on May 10, 2011, the combination of several of these failure modes put the turbine in a condition where it was susceptible to inadvertent actuation of the overspeed trip mechanism.
In addition to the root cause, the RCE team identified several more fundamental problems that contributed to the failure and the root cause. Some of these contributing causes were deficiencies associated with the control of critical maintenance, the control of supplemental workers, and the implementation of new vendor information. Based on the comprehensive evaluation performed for this root cause evaluation, the inspectors concluded that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem.
c. IP 95001 requires that the inspection staff determine that the licensees RCE included a consideration of prior occurrences of the issue and knowledge of operating experience (OE).
The licensees RCE included an evaluation of internal and external OE. This OE evaluation included previous similar events involving overspeed protection issues associated with steam-driven equipment, both internally and from the industry.
However, the RCE did not include an evaluation of previous events associated with the contributing causes, specifically, examples of issues with the control of critical maintenance and supplemental workers. This OE evaluation, though, was performed in the licensees self-assessment of the RCE, so the inspectors found this evaluation to be acceptable.
One of the issues identified in the licensees RCE is the failure to incorporate vendor recommendations into the maintenance procedure for the TDAFW turbine overhaul.
This was identified as one of the contributing causes to the TDAFW failure. In response to this deficiency, the licensee initiated a corrective action to ensure that these recommendations were evaluated and incorporated into the maintenance procedure.
The licensee also has a corrective action to review preventative maintenance activities and equipment overhaul activities to ensure that appropriate vendor recommendations and information have been reviewed and incorporated into the work instructions and procedures.
Based on the licensees evaluation and conclusions, both in the RCE and in the RCE self-assessment, the inspectors determined that the licensees RCE included a consideration of prior occurrences of the problem and knowledge of prior OE.
d. IP 95001 requires that the inspection staff determine that the licensees RCE addresses the extent of condition and extent of cause of the issues.
The licensees evaluation considered the extent of condition associated with the degraded turbine overspeed trip device as well as the work performed by the maintenance contractors who lubricated the incorrect linkages. With respect to the turbine trip, the licensee determined that no other auxiliary feedwater system equipment was adversely affected based on subsequent surveillance test performances. With respect to the degraded overspeed trip device, the licensee determined that no other steam-driven equipment in the plant was susceptible to this same type of failure. Finally, with respect to the work performed by the maintenance contractors, the licensee determined that the maintenance on the TDAFW pump was the only work performed by this maintenance crew during the refueling outage.
The licensees evaluation also considered the extent of cause associated with the planning and preparation for maintenance on critical equipment by supplemental workers. The licensee reviewed other outage work, equipment overhaul work, and preventative maintenance activities to look for weaknesses in the control of critical tasks and contract workers. The licensee also looked for the incorporation of vendor information, which was a contributing cause. The licensee identified that the lack of a final procedure relative to supplemental workers was a weakness, and has finalized procedure EN-MA-101-01, Critical Maintenance Identification and Oversight. No significant deficiencies were found. The inspectors concluded that the licensees RCE addressed the extent of condition and the extent of cause of the issue.
e. IP 95001 requires that the inspection staff determine that the licensees root cause, extent of condition, and extent of cause evaluations appropriately considered the safety culture components as described in IMC 0310, Components Within the Cross-Cutting Areas.
The licensee identified several weaknesses in the cross-cutting area (cross-cutting areas represent attributes of safety culture) of Human Performance, including in the cross-cutting components of Resources and Work Control. The licensee also identified a weakness in the cross-cutting area of Problem Identification and Resolution in the component of Operating Experience, and a weakness in the cross-cutting component of Accountability, which is not associated with a cross-cutting area. Specifically, the licensee identified that work authorizing documents did not include details appropriate for outage work on high critical equipment (Resources); the Maintenance organization did not recognize or prepare for outage work on high critical equipment as performed by supplemental workers (Work Control); available vendor and industry operating experience was not fully reflected into work authorizing documents or referenced maintenance procedures (Operating Experience), and managers and supervisors did not reinforce adherence to procedures among those doing the work on the TDAFW pump turbine overhaul (Accountability). These weaknesses correlate to the cross-cutting aspects H.2(c), H.3(a), P.2(a), and O.1(b), respectively, as described in Inspection Manual Chapter (IMC) 0310 dated October 28, 2011.
The licensee addressed these weaknesses with a series of corrective actions including reviewing work activities to verify critical maintenance is identified and reviewing work orders and preventative maintenance activities to verify vendor information has been incorporated. The inspectors determined that the licensees RCE included a proper consideration of whether weaknesses in any safety culture components were root causes or significant contributing causes of the issue.
f. Findings
No findings of significance were identified.
02.03 Corrective Actions a. IP 95001 requires that the inspection staff determine that
- (1) the licensee specified appropriate corrective actions for each root and/or contributing cause, or
- (2) an evaluation that states no actions are necessary is adequate.
The licensee implemented immediate corrective actions to restore the auxiliary feedwater pumps operability. They cleaned the grease from the knife edge, checked the trip linkage and cleaned the steam traps. In addition, to address the root causes, they revised FWS-M-6, Auxiliary Feedwater Turbine Maintenance, to incorporate recommendations from the Elliott Technical Service Letter ETSL-2012, industry experience and the results from the root cause evaluation. This included identification of critical steps and checks of linkage and trip pin clearances with provisions for recording of as-found data for review by plant personnel. Changes implemented by the licensee also included requirements for periodic replacement of trip mechanism springs, latch parts, trip pin and plunger. Post-maintenance testing instructions in the maintenance procedure were changed from suggestions to requirements. The corrective actions for the root and contributing causes listed in RCE CR-PLP-2011-05723 appear to be appropriate.
To address the contributing causes such as poor vendor information control, the licensee established a program to ensure that the system engineer validates any vendor recommendations, vendor bulletins or operating experience prior to performance of preventative maintenance on equipment that requires overhaul or refurbishment. The licensee also implemented programs to ensure that the requirements of critical maintenance were met in accordance with procedure EN-MA-101-01, Critical Maintenance Identification and Oversight, and work package evaluations adequately addressed supplemental worker performance during outages. Although substantial progress was achieved in the area of supplemental worker performance, the inspectors identified instances where additional licensee oversight was required.
During refueling outage 1R22, the licensees quality assurance staff identified that prior to the pressurizer manway installation, the licensee failed to brief supplemental workers in accordance with plant procedure EN-HU-102, Human Performance Tools. The briefing was performed using a vendor pre-job brief checklist and did not contain all the information required by procedure EN-HU-102. The brief was led by the vendor personnel with no Entergy supervision. The licensee captured this issue in CR-PLP-2012-03560 and CR-PLP-2012-03561. Also, on May 2, 2012, the licensee identified that steam generator primary manway ultrasonic testing readings were outside the acceptable range. The licensee completed an apparent cause evaluation (CR-PLP-2012-03497) and concluded that a supplemental worker had programmed the incorrect stud material configuration which resulted in the ultrasonic readings being outside the required range. The procedure used by the worker did not identify the stud material and the worker assumed that the material used at Palisades was similar to the ones used at other facilities. Tensioning activities for the steam generator were stopped by management and they determined a more appropriate course of action to ensure that the elongation measurements for the steam generator man ways were acceptable going forward. Also, during the same refueling outage, on April 29, 2012, during reactor head reassembly, damage was caused to the incore instrument flange #6 tree assembly and bull nose. The root cause evaluation (CR-PLP-2012-03411) performed by the licensee concluded that the supplemental workers did not positively verify proper alignment nor perform the procedure step for guiding the bullet nose into the nozzle and that procedure RFL-R-16, Reactor Vessel Closure Head Installation, did not adequately provide proper instructions or acceptance criteria. Work was stopped and the reactor head was safety placed on the head stand. Although there were no consequences of safety significance in these three instances, the inspectors stressed the importance of ensuring proper oversight of supplemental workers to prevent recurrence of such events.
The inspectors determined they had reasonable assurance that the licensees proposed corrective actions were appropriate and addressed each root and contributing cause so this area may be closed. The inspectors note this area should be a continuing focus at the site.
b. IP 95001 requires that the inspection staff determine that the licensee prioritized corrective actions with consideration of risk significance and regulatory compliance.
The licensees immediate corrective actions restored the auxiliary feedwater pump to operable status. After restoring the pump, the licensee tested P-8B to ensure that the pump would perform its intended functions when required. The resident inspectors witnessed this testing and observed that the auxiliary feedwater pump successfully passed the surveillance test.
The licensees corrective actions to address the root and contributing causes were prioritized in accordance with procedure, EN-LI-102, Corrective Action Process, and Root Cause Evaluation Quality Checklist/Score Sheets. The corrective actions were scored based on licensing, regulatory performance and nuclear safety factors. The licensees plan to verify vendor information was implemented based on the risk significance of the equipment. The licensee implemented programs to ensure that critical steps were identified in accordance with plant procedure EN-MA-101-01, Critical Maintenance Identification and Oversight, prior to performance of critical maintenance activities. The inspectors reviewed the licensees plans for accomplishing the above tasks and noted that the risk significance of the equipment was being appropriately considered. Based on the guidance provided in the licensees procedures and the prioritization of the corrective actions in accordance with these procedures, the inspectors determined that the corrective actions were prioritized with consideration of the risk significance and regulatory performance.
c. IP 95001 requires that the inspection staff determine that the licensee established a schedule for implementing and completing the corrective actions.
The licensee established due dates for the corrective actions in accordance with procedure EN-LI-102, Corrective Action Process, requirements for timeliness.
Some of the due dates were captured in CR-PLP-2011-05723; however, many of the due dates for the action items were contained throughout the licensees corrective actions program in various condition reports. The inspectors determined that the licensee adequately established a schedule for implementing and completing the corrective actions. The corrective actions for this RCE were completed in June 2012.
All CAPRs have been completed and effectiveness reviews for those CAPRs will be completed by November 2013. The inspectors considered the timeline for completion of corrective actions to be appropriate.
d. IP 95001 requires that the inspection staff determine that the licensee developed quantitative and/or qualitative measures of success for determining the effectiveness of the corrective actions to preclude repetition.
As documented in root cause evaluation, CR-PLP-2011-05723, the licensee established measures for determining the effectiveness of the corrective actions. These measures included:
- Enhanced monitoring of maintenance and corrective action records related to K-8 equipment to ensure that any additional failures are given appropriate management attention;
- Verify corrective actions resulting from evaluation of CR-PLP-2012-0362 will ensure future control of vendor documents, including changes, for plant equipment and have bounded the vendor document control deficiencies identified;
- No K-8 related plant unexpected Limited Condition of Operation (LCO) due to maintenance activities; and
- Revised auxiliary feedwater pump procedure updated to reflect results of the RCE for all aspects of turbine overhaul work.
The licensee staff entered these corrective action items into their corrective action program to ensure that these effectiveness reviews and enhanced monitoring were performed. The inspectors determined that quantitative and qualitative measures of success had been developed for determining the effectiveness of the corrective actions to preclude repetition.
e. IP 95001 requires that the inspection staff determine that the licensees planned or taken corrective actions adequately address a Notice of Violation (NOV) that was the basis for the supplemental inspection, if applicable.
The NRC issued an NOV to the licensee on January 3, 2012. The Notice of Violation associated with the White finding that was the subject of this IP 95001 inspection identified one violation of NRC requirements. In particular, a violation of TS 3.7.5 occurred from October 29, 2010, to May 10, 2011, due to the safety related steam driven auxiliary feedwater pump P-8B being inoperable as a result of maintenance not being prescribed by documented instructions or procedures. The NRC has concluded that information regarding the reasons for the violation, the corrective actions taken and planned to be taken to correct the violation and prevent recurrence, and the date when full compliance was achieved, is already adequately addressed on the docket in NRC IR No. 05000255/2011013 and in the licensees response to the NRC dated November 28, 2011. During this inspection, the inspectors confirmed that the licensees RCE and planned and taken corrective actions addressed the NOV. The licensee restored full compliance on May 11, 2011, by cleaning the grease from the knife edge, checking the trip linkage and performing a surveillance run of the pump.
f. Findings
No findings of significance were identified.
4OA6 Exit Meeting
On June 29, 2012, the inspectors presented the inspection results to Mr. Anthony Vitale, Vice-President of Operations, Palisades Nuclear Plant, and other members of his staff.
The inspectors asked the licensee if any of the material examined during the inspection should be considered proprietary. The licensee did not identify any proprietary information.
Regulatory Performance Meeting On June 29, 2012, the NRC met with the licensee to discuss its performance in accordance with IMC 0305, Section 10.02.b.4. During this meeting, the NRC and licensee discussed the issues related to this White finding that resulted in Palisades Nuclear Plant being placed in a Column of the NRCs ROP Action matrix with additional oversight. This discussion included the causes, corrective actions, extent of condition, extent of cause, and other planned licensee actions.
ATTACHMENT:
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee
- A. Vitale, Vice President, Operations
- M. Sicard, Special Projects
- A. Bono, Engineering Inspection Support
Nuclear Regulatory Commission
- J. Jandovitz, Acting Chief, Division of Reactor Projects, Branch 4
- J. Giessner, Chief, Division of Reactor Projects, Branch 4
- T. Taylor, Senior Resident Inspector, Palisades
- A. Scarbeary, Resident Inspector, Palisades
- J. Ellegood, Senior Resident Inspector, Donald C. Cook
LIST OF ITEMS
OPENED, CLOSED AND DISCUSSED
Opened
None.
Closed
- 05000255/2011013-01 VIO Improper Lubrication of Turbine Driven Auxiliary Feedwater Pump Linkages Attachment