ML080600164: Difference between revisions

From kanterella
Jump to navigation Jump to search
(Created page by program invented by StriderTol)
 
(Created page by program invented by StriderTol)
Line 1: Line 1:
#REDIRECT [[IR 05000445/2007008]]
{{Adams
| number = ML080600164
| issue date = 02/29/2008
| title = IR 05000445-07-008, on 12/4/07-1/24/08, Comanche Peak, Final Significance Determination for a White Finding and Notice of Violation, Special Inspection
| author name = Collins E
| author affiliation = NRC/RGN-IV/ORA
| addressee name = Blevins M
| addressee affiliation = Luminant Generation Co, LLC
| docket = 05000445
| license number = NPF-087
| contact person =
| case reference number = EA-08-028
| document report number = IR-07-008
| document type = Inspection Report, Letter, Notice of Violation
| page count = 42
}}
See also: [[see also::IR 05000445/2007008]]
 
=Text=
{{#Wiki_filter:February 29, 2008EA-08-028Mike Blevins, Senior Vice President  and Chief Nuclear Officer
Luminant Generation Company, LLC
ATTN:  Regulatory Affairs
Comanche Peak Steam Electric Station
P.O. Box 1002
Glen Rose, TX 76043SUBJECT:FINAL SIGNIFICANCE DETERMINATION FOR A WHITE FINDING ANDNOTICE OF VIOLATION - COMANCHE PEAK STEAM ELECTRIC STATION -
NRC SPECIAL INSPECTION REPORT 05000445/2007008Dear Mr. Blevins:
On January 24, 2008, the U.S. Nuclear Regulatory Commission (NRC) completed its reviewsrelated to a Special Inspection at your Comanche Peak Steam Electric Station, Unit 1, facility. This Special Inspection Team was chartered to review the circumstances related to the failure
of Emergency Diesel Generator (EDG) 1-02 to start on November 21, 2007, and to evaluate the
actions taken in response to the problem.  The NRC's initial evaluation satisfied the criteria in
NRC Management Directive 8.3, "NRC Incident Investigation Program," for conducting a special
inspection.  The possibility that adverse generic implications were associated with the EDG
failure mechanism was the deterministic criterion met.  Additionally, the result of the NRC's
initial conditional risk assessment associated with this degraded condition indicated that a
special inspection was warranted.  The determination that the inspection would be conducted
was made by the NRC on November 30, 2007, and the inspection started on December 4,
2007.The inspection examined activities conducted under your license as they relate to safety andcompliance with the Commission's rules and regulations and with the conditions of your license.
The inspectors reviewed selected procedures and records, observed activities, and interviewed
personnel.The enclosed inspection report documents the inspection results, which were discussed onDecember 7, 2007, and again on January 10, 2008, with Mr. R. Flores and Mr. T. Hope,
respectively, and other members of your staff.  On January 24, 2008, an exit meeting was held
with Mr. F. Madden, Director, Regulatory Affairs, and other members of your staff to convey the 
Luminant Generating Company, LLC-2-final disposition of the inspection findings.  Following a discussion of the preliminary safetysignificance of this finding during the exit briefing, Mr. Madden indicated that Luminant Power
does not contest the characterization of the risk significance of this finding, and that you have
declined to further discuss this issue at a Regulatory Conference or provide a written response.
Accordingly, the NRC is issuing the final significance determination for the inspection finding as
discussed below.  On February 25, 2008, an additional exit meeting was held with Mr. T. Hope,
and other members of your staff to convey a revision to one of the inspection findings. This report documents one finding concerning a failure to satisfy Technical Specification (TS)Limiting Condition for Operation (LCO) 3.8.1 due to EDG 1-02 being in an inoperable conditionfollowing maintenance.  Following the discovery of this condition, the TS required actions were
satisfied however, the time period between the occurrence of the condition and the discovery of
the condition exceeded the TS allowed outage time for the EDG.  This finding has been
determined to be of low to moderate safety significance (White).  This finding does not
represent an immediate safety concern because of the corrective actions you have taken.
These actions included restoring EDG 1-02 to an operable status, ensuring that all other EDGs
were not in a similar degraded condition, and curtailing painting activities pending the
implementation of suitable measures to prevent the recurrence of a similar condition.You have 30 calendar days from the date of this letter to appeal the NRC's determination ofsignificance for the identified White finding.  Such appeals will be considered to have merit only
if they meet the criteria given in NRC Inspection Manual Chapter 0609, Attachment 2.  In
accordance with the NRC Enforcement Policy, the Notice of Violation is considered an
escalated enforcement action because it is associated with a White finding.You are required to respond to this letter and should follow the instructions specified in theenclosed Notice when preparing your response.In addition, we will use the NRC Action Matrix to determine the most appropriate NRC responseto this issue, and we will notify you by separate correspondence of that determination.The report also documents one NRC-identified finding of very low safety significance (Green). This finding was determined to involve a violation of NRC requirements.  However, because of
the very low safety significance and because it is entered into your corrective action program,
the NRC is treating the finding as a noncited violation (NCV) consistent with Section VI.A.1 of
the NRC Enforcement Policy.  If you contest this NCV, you should provide a response within
30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear
Regulatory Commission, ATTN.:  Document Control Desk, Washington, DC 20555-0001; with
copies to the Regional Administrator, U.S. Nuclear Regulatory Commission, Region IV, 611
Ryan Plaza Drive, Suite 400, Arlington, Texas 76011-4005; the Director, Office of Enforcement,
U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident
Inspector at the Comanche Peak Steam Electric Station.In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, itsenclosure, and your response (if any) will be available electronically for public inspection in the 
Luminant Generating Company, LLC-3-NRC Public Document Room or from the Publicly Available Records (PARS) component ofNRC's document system (ADAMS).  ADAMS is accessible from the NRC Web site at
http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).  To theextent possible, your response should not include any personal privacy, proprietary, or
safeguards information so that it can be made available to the public without redaction.Sincerely, /RA/Elmo E. CollinsRegional AdministratorDockets:50-445Licenses:NPF-87Enclosures:1.Notice of Violation
2.NRC Inspection Report 05000445/2007008
        w/AttachmentsAttachment 1:Supplemental InformationAttachment 2:Special Inspection Charter
Attachment 3:Significance Determination Evaluationcc w/enclosures:Fred W. Madden, Director
Regulatory Affairs
Luminant Generation Company LLC
P.O. Box 1002
Glen Rose, TX  76043Timothy P.  Matthews, Esq.Morgan Lewis
1111 Pennsylvania Avenue, NW
Washington, DC  20004Anthony Jones, Chief Boiler InspectorTexas Department of Licensing
  and Regulation
Boiler Program
P.O. Box 12157
Austin, TX  78711Somervell County JudgeP.O. Box 851
Glen Rose, TX  76043
Luminant Generating Company, LLC-4-Richard A. Ratliff, ChiefBureau of Radiation Control
Texas Department of Health
1100 West 49th Street
Austin, TX  78756-3189Environmental and Natural    Resources Policy Director
Office of the Governor
P.O. Box 12428
Austin, TX  78711-3189Brian AlmonPublic Utility Commission
William B. Travis Building
P.O. Box 13326
Austin, TX  78711-3326Susan M. JablonskiOffice of Permitting, Remediation
  and Registration
Texas Commission on
  Environmental Quality
MC-122
P.O. Box 13087
Austin, TX  78711-3087Environmental and Natural Resources Policy Director
Office of the Governor
P.O. Box 12428
Austin, TX 78711-3189Lisa R. Hammond, ChiefTechnological Hazards Branch
National Preparedness Division
FEMA Region VI
800 N. Loop 288
Denton, TX  76209
Luminant Generating Company, LLC-5-Electronic distribution:ROPreportsRIDSSECYMAILCENTERRIDSOCAMAILCENTERRIDSEDOMAILCENTERRIDSOEMAILCENTER
RIDSOGCMAILCENTERRIDSNRROD
RIDSNRRADIP RIDSOPAMAIL
RIDSOIMAILCENTERRIDSOIGMAILCENTER
RIDSOCFOMAILCENTERRIDSRGN1MAILCENTER
RIDSRGN2MAILCENTERRIDSRGN3MAILCENTER
RIDSNRRDIPMIIPBOEWEB ECollinsAHowellFuller - KSFC MaierVasquez - GMVD Furst, NSIR
Vegel - AXVChamberlain - DDCN Hilton, OE
Caniano -  RJC1Powers - DAPJune Cai, OE
ACampbell - ACCCJohnson - CEJ1John Wray, OE
DLoveless - DPLDAllen - DBAASanchez - AAS1
Herrera - MSH3SSanner - ESSStarkey, OE - DRS
Mary Ann Ashley, NRRPaulk -CJPM Burrell, OE
Dricks - VLDWMaier - WAMR Barnes, OE
DPowers - DAPDPelton - DLP1SUNSI Review Completed:      CEJ      ADAMS:    Yes    G  No        Initials:    CEJ     
  Publicly Available     
G  Non-Publicly Available     
G  Sensitive  Non-Sensitive R:\_REACTORS\_CPSES\2007\CP2007-08 CHY.wpdADAMS ML080600164RIV:RI:DRP/ESRI:DRP/E
SRAC:DRP/ASES/ACESCHYoung:vlh;mjsAASanchezDPLoveless
CEJohnsonGMVasquez    E-CEJ      E-CEJ  /RA//RA//RA/1/31/082/08/082/08/082/12/082/11/08D:DRSD:DRPC:ACESDRARARJCanianoDDChamberlainKSFullerATHowellEECollins /RA//RA//RA//RA//RA/2/11/082/21/082/14/082/22/082/28/08OFFICIAL RECORD COPYT=Telephone          E=E-mail        F=Fax
Enclosure 1NOTICE OF VIOLATIONLuminant Generation Company, LLC Docket No. 50-445Comanche Peak Steam Electric StationLicense No. NPF-87EA-08-028During an NRC inspection completed on January 24, 2008, a violation of NRC requirementswas identified.  In accordance with the NRC Enforcement Policy, the violation is listed below:Unit 1 Technical Specification (TS) 3.8.1, "AC Sources - Operating," requires that whilethe plant is in Modes 1, 2, 3, or 4, two diesel generators (DGs) capable of supplying the
onsite Class 1E power distribution subsystem(s) shall be operable.  For the condition of
one DG being inoperable, the required action is to restore the DG to an operable status
within 72 hours and within 6 days from the discovery of the failure to meet the Limiting
Condition for Operation (LCO), or be in Mode 3 within 6 hours and Mode 5 within 36
hours.Contrary to the above, from November 1, 2007, through November 21, 2007, while theplant was in Mode 1, one of the two DGs capable of supplying the onsite Class 1E
power distribution subsystem(s) was inoperable, and action was not taken to either
restore the DG to an operable status within 72 hours or be in Mode 3 within 6 hours and
Mode 5 within 36 hours.  Specifically, Emergency Diesel Generator (EDG) 1-02 was
made inoperable as a result of painting activities due to paint having been deposited and
remaining on at least one fuel rack in a location that prevented motion required to
support the operation of the EDG.  This condition caused EDG 1-02 to fail to start during
a surveillance test on November 21, 2007. This violation is associated with a White significance determination process finding.Pursuant to the provisions of 10 CFR 2.201, Luminant Generation Company, LLC is herebyrequired to submit a written statement or explanation to the U.S. Nuclear Regulatory
Commission, ATTN.:  Document Control Desk, Washington, DC 20555-0001 with a copy to the
Regional Administrator, Region IV, and a copy to the NRC Resident Inspector at the facility that
is the subject of this Notice, within 30 days of the date of the letter transmitting this Notice of
Violation (Notice).  This reply should be clearly marked as a "Reply to a Notice of Violation;EA-08-028," and should include for each violation:  (1) the reason for the violation, or, if
contested, the basis for disputing the violation or severity level; (2) the corrective steps that
have been taken and the results achieved; (3) the corrective steps that will be taken to avoid
further violations and (4) the date when full compliance will be achieved.  Your response may
reference or include previous docketed correspondence, if the correspondence adequately
addresses the required response.  If an adequate reply is not received within the time specified
in this Notice, an order or a Demand for Information may be issued as to why the license should 
Enclosure 1-2-not be modified, suspended, or revoked, or why such other action as may be proper should notbe taken.  Where good cause is shown, consideration will be given to extending the response
time.
 
If you contest this enforcement action, you should also provide a copy of your response, with
the basis for your denial, to the Director, Office of Enforcement, United States Nuclear
Regulatory Commission, Washington, DC 20555-0001. Because your response will be made available electronically for public inspection in the NRCPublic Document Room or from the NRC's document system (ADAMS), accessible from the
NRC Web site at http://www.nrc.gov/reading-rm/adams.html, to the extent possible, it shouldnot include any personal privacy, proprietary, or safeguards information so that it can be made
available to the public without redaction.  If personal privacy or proprietary information is
necessary to provide an acceptable response, then please provide a bracketed copy of your
response that identifies the information that should be protected and a redacted copy of your
response that deletes such information.  If you request withholding of such material, you mustspecifically identify the portions of your response that you seek to have withheld and provide in
detail the bases for your claim of withholding (e.g., explain why the disclosure of information will
create an unwarranted invasion of personal privacy or provide the information required by
10 CFR 2.390(b) to support a request for withholding confidential commercial or financial
information).  If safeguards information is necessary to provide an acceptable response, please
provide the level of protection described in 10 CFR 73.21.  Dated this 29
th day of February 2008.
U.S. NUCLEAR REGULATORY COMMISSIONREGION IVDockets:50-445Licenses:NPF-87
Report:05000445/2007008
Licensee:Luminant Generation Company, LLC
Facility:Comanche Peak Steam Electric Station, Unit 1
Location:FM-56, Glen Rose, Texas
Dates:December 4, 2007, through January 24, 2008
Team Leader:C. Young, P.E., Resident Inspector, Arkansas Nuclear One
Inspectors:A. Sanchez, Resident Inspector, Comanche Peak Steam Electric StationD. Loveless, Senior Reactor AnalystBranch Chief:C. Johnson, Chief, Project Branch ADivision of Reactor ProjectsApproved By:D. Chamberlain, Director Division of Reactor Projects
Enclosure 2-2-SUMMARY OF FINDINGSIR 05000445/2007008; 12/04/07 - 01/24/08; Comanche Peak Steam Electric Station (CPSES),Unit 1; Special Inspection in response to the failure of the Train B Emergency Diesel Generator
to start on demand on November 21, 2007.The report covered a 6-day period (December 4-7, 2007) of onsite inspection, with inofficereview through January 24, 2008, by a special inspection team consisting of two resident
inspectors and one senior reactor analyst.  Two findings were identified, including one Green
noncited violation, and one White violation.  The significance of most findings is indicated by its
color (Green, White, Yellow, or Red) using Inspection Manual Chapter 0609, "Significance
Determination Process."  Findings for which the significance determination process does not
apply may be Green or be assigned a severity level after NRC management review.  The
NRC's program for overseeing the safe operation of commercial nuclear power reactors is
described in NUREG-1649, "Reactor Oversight Process," Revision 4, dated July 2000.A.NRC-Identified and Self-Revealing FindingsCornerstone:  Mitigating Systems
*White.  A violation of Unit 1 Technical Specification 3.8.1, "AC Sources -Operating," was identified for the licensee's failure to satisfy Limiting Condition
for Operation 3.8.1 in that painting activities conducted on the Unit 1 Train B
EDG 1-02 resulted in paint being deposited and left in a location that caused the
EDG to become inoperable.  As a result, EDG 1-02 failed to start on demand
during the subsequent monthly surveillance test.  Following the discovery of the
condition, the required actions were satisfied; however, the time period between
the occurrence of the condition and the discovery of the condition exceeded the
allowed outage time.  This issue was entered into the licensee's corrective action
program as SMF-2007-03253.The finding was greater than minor because it was associated with the humanperformance attribute of the mitigating systems cornerstone, and it affected the
cornerstone objective to ensure the availability, reliability, and capability of
systems that respond to initiating events to prevent undesirable consequences.
The Phase 1 Worksheets in Manual Chapter 0609, "Significance Determination
Process," were used to conclude that a Phase 2 analysis was required because
the performance deficiency affected the emergency power supply system that is
a support system for both mitigating and containment barrier systems.  Based on
the results of the Phase 2 analysis, the finding was determined to have low to
moderate safety significance (White).  The senior reactor analyst determined
that a more detailed Phase 3 analysis was needed to fully assess the safety
significance.  Based on the results of the Phase 3 analysis, the finding was
determined to have low to moderate safety significance (White).  The Phase 1,
2, and 3 Significance Determination Process analyses associated with this
finding, including assumptions and limiting core damage sequences, is included
as Attachment 3 to this report.  The cause of this finding was determined to have
a crosscutting aspect in the area of human performance associated with work
Enclosure 2-3-practices in that the licensee failed to provide adequate supervisory andmanagement oversight of work activities, including contractors, such that nuclear
safety is supported [H.4(c)].  Specifically, the actions planned and taken to
assess and control the operational impact of the painting activities on the
functionality of the emergency diesel generator were not reflective of adequate
supervisory and management oversight of the activities (Section 2.1).*Green.  The inspectors identified a noncited violation of Unit 1 TechnicalSpecification 5.4.1.a, "Procedures," for an inadequate alarm response
procedure.  The inspectors determined that Procedure ALM-1302A, "Diesel
Generator 1-02 Panel," Revision 5, was inadequate in that it was ambiguous and
did not cause the responders to verify that the fuel racks were free as part of the
response actions to investigate the cause of the unit failing to start.
Consequently, the licensee failed to identify that the Unit 1 Train B Emergency
Diesel Generator 1-02 fuel racks were not free to move, which led to an
extended period of inoperability and a significant delay in diagnosing the cause
of the emergency diesel generator failure to start.  This issue was entered into
the licensee's corrective action program as SMF-2007-03426.The finding was determined to be more than minor because it was associatedwith the procedure quality attribute of the mitigating systems cornerstone, and it
affected the cornerstone objective to ensure the availability, reliability, and
capability of systems that respond to initiating events to prevent undesirable
consequences.  Using Manual Chapter 0609, "Significance Determination
Process," Phase 1 Worksheet, the finding was determined to have very low
safety significance (Green) because it was not a design or qualification
deficiency, did not represent a loss of safety function, did not represent an actual
loss of a single train for greater than its Technical Specification allowed outage
time, did not represent a loss of a non-Technical Specification Train of
equipment for greater than 24 hours, and did not screen as potentially risk
significant due to a seismic, flooding, or severe weather initiating event 
(Section 2.2).B.Licensee-Identified Violations
None.
Enclosure 2-4-REPORT DETAILS1.0SPECIAL INSPECTION SCOPEThe NRC conducted a special inspection at Comanche Peak Steam Electric Station tobetter understand the circumstances surrounding the failure of the Unit 1 Train B
Emergency Diesel Generator (EDG) 1-02 to start on demand during a monthly
surveillance test on November 21, 2007.  Following approximately 11 hours of
troubleshooting, EDG 1-02 was restored to an operable status.  In accordance with NRC
Management Directive 8.3, it was determined that the period of inoperability of the EDG,
both prior to and during the failure to start event, had sufficient risk significance to
warrant a special inspection.  The initial incremental conditional core damage probability
associated with the assumed period of EDG inoperability was estimated to be
1.76 x 10-5.  The possibility that adverse generic implications were associated with theEDG failure mechanism was the deterministic criterion met to warrant a special
inspection.
 
 
The team conducted the inspection in accordance with Inspection Procedure 93812,
"Special Inspection," and the inspection charter, which is included in this report as
Attachment 2.  The special inspection team reviewed procedures, corrective action
documents, operator logs, and maintenance records for the EDG system.  The team
interviewed various licensee personnel regarding the events that led up to and response
actions that followed the EDG failure, as well as design and operational characteristics
of the EDG and its support systems.  The team reviewed the licensee's root cause
analysis report, past failure records, extent of condition evaluation, immediate and long
term corrective actions, and industry operating experience.  A list of specific documents
reviewed is provided in Attachment 1.1.1Event SummaryOn November 21, 2007, at 10:20 a.m., EDG 1-02 failed to start on demand during amonthly slow start surveillance test.  Prior to this, the last successful surveillance test on
EDG 1-02 was on October 24, 2007.  The licensee's response to the failure to start is
described in Section 1.2 below.  Troubleshooting efforts were ultimately successful, and
EDG 1-02 was restored to an operable status at 3:08 a.m. on November 22, 2007.  The
failure was determined to be the result of fuel racks being stuck in the closed positions,
and not responding to a full open governor demand, thereby preventing sufficient fuel
from reaching the engine.  The failure mechanism is described in Section 1.4 below.
The cause of the fuel rack binding was ultimately determined to be a drop of paint on a
fuel rack which prevented the rack from being able to move through the fuel pump
housing.  The root and contributing causes of this failure are discussed on Section 1.3
below.Prior to the failed surveillance test on November 21, 2007, painting was conducted onand around EDG 1-02 and EDG 2-02 (Unit 2 Train B EDG).  The painting activities
associated with EDG 1-02 began on October 15, 2007, and continued through
November 8, 2007.
Enclosure 2-5-Included below is a timeline that includes significant elements pertaining to this event.Date/TimeEventOctober 15, 2007Painting begins on EDG 1-02 on top of engine and around
heads.October 15, 2007 -November 1, 2007Painting activities continue on a daily basis.October 24, 2007Successful monthly slow start surveillance test of EDG 1-02. No painting is done on this day.October 29, 2007 -November 1, 2007Painting occurs around the 6L fuel pump.November 1, 2007Painting in locations that could have reasonably resulted instray paint/drops on fuel rack(s) is completed.November 21, 2007      3:28 a.m.EDG 1-02 declared inoperable due to bar over in preparationfor monthly surveillance test.      4:09 a.m.Bar over completed.  Successful water roll via the air startsystem was completed.  EDG 1-02 declared operable.    10:17 a.m.EDG 1-02 declared inoperable for monthly surveillance test.
    10:20 a.m.EDG 1-02 failed to start on demand for monthly slow startsurveillance test.  Operations personnel believed the EDG did
not roll.  Troubleshooting commences.      4:49 p.m.Slow start attempt of EDG 1-02 resulted in EDG rolling up to90-100 rpm and failing to start.  Troubleshooting continues.      6:02 p.m.Fast start attempt of EDG 1-02 resulted in a failure to start. Fuel racks were observed not to move in response to a
governor demand.      7:39 p.m.Fuel racks were manually stroked.  Rack 6L was found to bestuck.  Rack 2L moved approximately half of its travel range,
then became bound.  Both racks were freed and stroked until
normal free range of motion was restored.      9:25 p.m.Walkdown inspections revealed residual paint on 6L, 4L, and4R fuel racks.  Residue of paint on 6L was wiped away.      9:32 p.m.Successful start and run of EDG 1-02.
November 22, 2007      3:08 a.m.EDG 1-02 was declared operable.
Enclosure 2-6-1.2Licensee Response to the Failure of the EDG to StartThe inspectors evaluated the licensee's implementation of procedures (abnormal, alarm,troubleshooting, and normal operations) and Technical Specifications, reviewed plant
management's control and decision making actions, and reviewed the troubleshooting
and investigating activities that occurred following the Unit 1 Train B EDG failure to start
during the monthly surveillance test on November 21, 2007.  The inspectors reviewed
corrective action documents, procedures, Technical Specifications, and operations logs.
The team performed system walkdowns and interviewed engineering, maintenance, and
operations personnel.The inspectors determined that, in general, the licensee responded to the event properlyand in accordance with plant procedures.  Nuclear equipment operators (NEO) quickly
identified that the EDG 1-02 failed to start and immediately responded to the local EDG
alarm panel.  The operations field support supervisor performed an inspection to look for
any obvious problems that could have caused the EDG to fail.  NEOs noted that it did
not sound like a normal start, and assumed that a possible issue associated with the
starting air system had something to do with the failure to start.  The licensee had
already declared the EDG inoperable prior to the attempted start in conjunction with the
surveillance test.  In response to the Unit Failure To Start alarm on the local EDG alarm panel, NEOsperformed the steps of the local alarm panel procedure, Alarm Procedure
Manual ALM-1302A, "Diesel Generator 1-02 Panel," Revision 5, which instructed the
operations personnel to investigate the cause of the failure.  This included checking for
proper operation and issues associated with the fuel racks, day tank, and the starting air
system.  One of the applicable steps was to "check fuel racks free."  This was
accomplished in accordance with the expectations of senior operations personnel by
visually verifying that there were no apparent conditions that would obstruct the motion
of the fuel racks.  No abnormalities were identified at this time.The operations staff reviewed drawings and diagrams, interviewed the NEOs, andconsulted with meter and relay representatives, system engineering department
personnel, and the mechanical services department to develop a troubleshooting plan.
Due in large part to the testimony of the NEOs that the EDG did not even roll in
response to the start attempt, the troubleshooting plan focused on the starting air
system as the suspected cause of the failed start.  The plan called for meter and relay
personnel to monitor various solenoids and relays during a subsequent slow start
attempt of the EDG.  This attempt resulted in the EDG rolling up to 90-100 rpm, and
again failing to start.  Indications now suggested that a fuel-related problem must exist,
and focus was shifted accordingly.  A third attempt was performed with the EDG in a
fast start configuration.  Again, the EDG failed to start.  Observers noted that the fuel
racks did not move from their closed positions in response to the mechanical governor's
attempt to drive the fuel racks to the full open position.  The licensee then attempted to
exercise the fuel racks and metering rods individually and discovered that two metering
rods (2L and 6L) were partially and fully bound, respectively.  Licensee personnel
physically exercised the metering rods until they were free to move, and removed
evidence of paint that was found to be on the 6L metering rod by the fuel pump housing
interface.  The licensee then performed a fourth attempt to start EDG 1-02, which was
Enclosure 2-7-successful.  The EDG was fully loaded, and operations personnel completed thesurveillance testing.  The EDG 1-02 was subsequently declared operable on the
morning of November 22, 2007 at 3:08 a.m.  The inspectors determined that the initial troubleshooting plan was too narrowly focusedon finding an EDG starting air problem (despite a successful water roll via the air start
system that occurred earlier that morning), as opposed to pursuing all likely causes of a
failed start.  If the focus of the response were broader, it is likely that the stuck metering
rod would have been discovered earlier, and the duration of EDG inoperability following
the failed start would have been reduced.Subsequently during the troubleshooting efforts, the joint engineering team developed aconfirm and refute matrix to process the results from troubleshooting.  Possible causes
that were analyzed over the course of troubleshooting included:*Starting air receiver discharge valves mispositioned*Manual stop button mispositioned
*Tachometers operational
*Malfunctioning of air start solenoid valves
*Mechanical governor bound
*Fuel supply to the engine
*Main control board handswitch
*Electronic governor not operating
*Fuel racks not functioning**Determined to be the cause of the failure
As described in Section 1.3 below, the Unit 2 Train B EDG was also in the process ofbeing painted.  Once the cause of EDG 1-02 inoperability was determined to be stuck
metering fuel rods, the operations staff inspected the Unit 2 Train B EDG and
determined that the same issue did not exist.  Operations also inspected the Units 1
and 2 Train A EDGs and determined that the stuck metering rods issue did not exist.
The Unit 1 Train B EDG was the only EDG affected.1.3Root Cause and Corrective Action Assessment.1Root Cause AnalysisThe inspectors reviewed and assessed the licensee's root cause analysis for technique,accuracy, thoroughness, and corrective actions proposed and taken.  The inspectors
reviewed the scope and processes used by licensee personnel to identify the root cause
for the failure of the Unit 1 Train B EDG to start during a monthly surveillance test.  The
inspectors compared information gained through inspection to the event information and
assumptions made in the root cause reports.  The inspectors interviewed licensee
personnel, reviewed logs, reviewed personal statements, and observed root cause team
meetings.  The inspectors evaluated the licensee's extent of condition review and
common cause evaluation.
Enclosure 2-8-The licensee captured the EDG 1-02 failure to start problem in the corrective actionprogram as SMF-2007-03253, and performed a root cause analysis in response to
determine the cause of the failure.  Evaluation techniques utilized by the licensee
included an Events and Causal Factors Chart and a Barrier Analysis.  The result of
these efforts identified the most probable root cause of the failure to be a drop of paint
that was deposited and adhered to the 6L fuel rack in a location that prevented the rack
(along with all other fuel racks) from moving in the open direction in response to the
governor demand associated with an EDG start signal.  This failure mechanism is
further discussed in Section 1.4 below.  Although there was no documented evidence of
the actual paint drop, there was paint residue observed which remained in the subject
location following the manual manipulation and freeing of the stuck fuel rack during
troubleshooting.  This residue was wiped off upon discovery.Additionally, the following four contributing causes to the failure were identified in thefinal root cause analysis:*Work practices of painters and other groups who performed daily inspectionsfailed to identify paint spatter and drops that should have been cleaned off
sensitive engine components.*The tools and techniques used by painters were not completely effective inpreventing paint spatter and drips.**Because the directions in alarm response procedure ALM-1302A were notspecific, the time period following the failure until the discovery of the cause of
the problem was extended.*The fuel control shaft break away force may have increased over time due towear and aging effects.  This may have added to the force required to overcome
the adhesion of the paint drop.*This issue was also identified early in the inspection process by the inspectors and isfurther discussed in Section 2.2 below.The root cause team assessed that the engineering confirm/refute evaluation performedduring troubleshooting, along with the subsequent investigative actions outlined below,
were effective in considering and ruling out all other potential causes of the failure:*Electrical and control circuitry problems were investigated and ruled out.  Due tothe initial reports that the field operator did not believe that the EDG even rolled
over, the root cause team investigated other possibilities that could have caused
the EDG not to have rolled, and still brought in the alarms that were received.
One viable possibility considered was a possible fault associated with the EDGStart/Stop hand switch in the control room.  The hand switch in question was a
piece of original equipment.  One of the corrective actions was to replace the
hand switch when the 6L fuel pump and metering rod was replaced after the
event.  The switch was bench tested, disassembled, and inspected, and it was
determined that the switch not only functioned properly without signs of
degradation, but it would not be physically possible to have the switch
Enclosure 2-9-manipulated to send a stop signal to the diesel while an operator takes theswitch to the start position.  The inspectors performed a visual inspection of the
switch internals and reviewed the testing methods and results.  The inspectors
concluded that the EDG start/stop control switch would not have caused the
EDG failure to start on November 21, 2007.*The starting air system was examined and proven to be functional.  Theinspectors confirmed this by performing system walkdowns.  A water roll check
was performed satisfactorily.*The fuel day tank was inspected to ensure proper alignment and fuel quality.
*Inspections of the joints that connect the fuel pump control shaft levers to thefuel racks were performed, and determined that none were exhibiting mechanical
binding.  The inspectors confirmed this by performing a system walkdown.*The 6L fuel pump was replaced and sent to the vendor for testing, disassembly,and inspection.  No abnormalities were identified, and internal binding of the
pump was determined not to be a cause of the event.*The capability of single paint drop to counter the force applied and prevent themotion of the fuel control shafts was assessed.  A spare fuel pump was
subjected to a series of field tests to determine the force required to overcome
the adhesion of a drop of paint in the location that had been identified.  The
results were consistent with the hypothesis that the force applied from the
mechanical governor could have been overcome by the presence of the paint
drop becoming wedged in the minimal clearance between the fuel rack and the
pump housing.  Another pull test was done to confirm that a fuel rack exposed to
various combinations of dirt and grit would not require appreciably more pull
tension to operate.Aspects of organizational and programmatic effectiveness were also evaluated by theroot cause team, and confirmed by the inspectors.  These included inadequate
supervisory and management involvement with the painting activities, work practices
employed during the job, and the less than comprehensive development of the
procedures and work packages associated with the activity.The extent of the condition that was determined to be the cause of the EDG 1-02 failurewas assessed by the root cause team.  All other EDGs were thoroughly inspected to
verify that the same condition did not exist, particularly with the Unit 2 Train B EDG 2-02,
which had been similarly painted in September and October.  All other EDGs were
verified to be free of the subject degraded condition.  Emergency Diesel Generator 2-02
successfully passed its monthly surveillance test on November 28, 2007.  The
inspectors reviewed the licensee's actions and concluded that the licensee's extent of
condition evaluation was adequate.The inspectors concluded, following interviews as well as a review of personalstatements made by the maintenance personnel, that the work practices of painters and
other work groups who performed daily paint clean-up inspections to identify paint
Enclosure 2-10-spatter and drops that needed to be cleaned off of sensitive engine components was avalid contributor to the event.  The inspectors also determined that neither
documentation nor feedback from the inspections to the painters or operations
management regarding the results of those inspections was performed.  The
communication of those results, to the right individuals, could have identified the need to
reinforce expectations, alter paint methods or barriers, or institute a stand down that
may have led to the prevention of the event.  At a minimum, communication between
organizations (maintenance, inspection, operations, and management) was not as
strong as it could have been for this work on highly risk significant, safety-related
equipment.Along with the discussion above, the inspections that were performed as part of thepostpainting activities were agreed upon between operations and maintenance.  Neither
the inspections nor any other applicable postmaintenance testing was specified by the
work order for performing the painting activities.  Also, there was no discussion
concerning foreign materials control exclusion (FME) controls.  FME has been a
significant issue with the licensee in the recent past, but no mention of this sensitivitywas made.  The inspections that were performed were not documented anywhere as
having been done nor were any of the findings stemming from the inspections. The inspectors found that the licensee assembled an effective root cause team.  Theroot cause team investigated every lead that was available to determine exactly why the
Unit 1 Train B EDG failed to start on November 21, 2007.  The inspectors determined
that the scope, methods, and rigor associated with the root cause analysis were
appropriate and consistent with the safety significance of the problem, and that the
evaluation was successful in determining and addressing the most probable root and
contributing causes of this issue..2Corrective Action AssessmentThe inspectors evaluated the scope, adequacy, and timeliness of the licensee'scorrective measures that were both planned and implemented in response to the cause
of the EDG 1-02 failure.  The inspectors concluded that the actions planned and taken
by the licensee were appropriate to address the degraded condition, to result in the
prevention of a future similar failure, and were consistent with the safety significance of
the event.  Corrective actions to be taken prior to resuming painting activities include:*Revise Procedure MSM-G0-0220 used for painting to require a shiftlymanipulation of the fuel racks in addition to a visual inspection of components to
be free of paint spatter/drops*Verify the information contained in the painting pre-job briefing book to ensure itcontains all sensitive areas on the EDG that should not be painted*Revise Procedure MSM-G0-0220 to include pictures and other informationcontained in the painters' prejob briefing book used during EDG painting*Revise Procedure MSM-G0-0220 to provide for "as you go" inspections andcleaning when painting is done around sensitive components
Enclosure 2-11-*Include this event in prejob briefings for future activities to heighten sensitivity tothe potential effects of paint spatter/drops in areas that can bind mechanical
components or block air pathways*Improve tools and techniques used by painters to minimize drops and spatter. Also research available FME barriers that could be used to shield sensitive areasAdditional planned corrective actions include:
*Develop a preventive maintenance activity to perform a fuel control shaft breakaway force test to monitor for potential degradation in the shaft linkage or
bearings*Revise alarm response Procedure ALM-1302A to remove ambiguity regardingchecking components for freedom of movement by providing specific instruction
to include a manual manipulation of the components1.4Scope of the Failure MechanismThe inspectors, through inspection and investigation, interviews of system engineers,reviews of EDG design documentation, and assessment of the licensee's root cause
analysis, developed a scope of the mechanism that was determined to be the root
cause of the EDG 1-02 failure.  The fuel pump control racks (fuel metering rods) were
prevented from moving from their normal standby (closed) positions in response to a
governor demand by the presence of a drop of paint that had adhered to the fuel rack in
a location where the rack enters the housing of the fuel pump (with very minimal
clearance) when moving in the open direction.  Since all fuel racks are mechanically
linked by the common fuel control shafts and cross shaft linkages, the motion of the
entire system in the open direction (back to the extensible link from the mechanical
governor) was inhibited by one fuel rack that was stuck in the standby (closed) position.
A torsion spring on the control shaft associated with each fuel pump control shaft lever
functions to allow continued motion of the system in the closed direction if one or more
individual fuel racks become bound.  However, the feature does not provide this function
for system motion in the open direction, as in the response to an EDG start signal.1.5Event PrecursorsThe root cause of the EDG failure to start was determined to be paint that was inadvertently dropped onto a fuel pump metering rod.  The inspectors reviewed
corrective action documents and interviewed system engineers in order to identify any
previous related issues that may have been precursors to the Unit 1 Train B EDG failure
to start.  The inspectors reviewed all available documented issues dating back to 1999
that fell into each of the following two categories:  (1) Previous similar or related EDG
failures, and (2) Previous issues involving equipment failures related to painting.  The
inspectors determined that there had been no previous EDG or painting related issues
that may have been precursors to this event.
Enclosure 2-12-1.6EDG Maintenance and TestingThe inspectors reviewed the licensee's EDG Maintenance and testing programs.  Theinspectors reviewed maintenance and testing records as well as the licensee's plans
and schedules related to preventive maintenance and testing of the EDGs.  The
inspectors also interviewed several system engineers to gain an understanding of the
licensee's approaches and programs involving EDG maintenance and testing.  The
inspectors determined that the licensee's EDG routine maintenance and testing
programs are adequate and that the licensee is following the program provisions.
However, the inspectors determined that these maintenance and testing practices for
painting activities were not adequate as discussed in Section 2.0.1.7Industry Operating Experience (OE)The inspectors reviewed the industry operating experience (OE) the licensee gainedthrough their normal review, as well as that which was referenced in the licensee's root
cause evaluation.  The inspectors conducted interviews of licensee personnel, reviews
of pertinent OE materials discovered independently as well as with the assistance of the
NRC's Operating Experience Section, and an evaluation of actions taken by the licensee
in response to relevant OE.  The specific documents reviewed during this review is listed
in Attachment 1 of this report.The inspectors determined that the licensee had appropriately reviewed andincorporated OE associated with the circumstances of the EDG failure, and that a failure
to incorporate applicable OE into station practices was not a contributing cause to the
EDG failure.  The inspectors reviewed several items of OE, inspection reports, and
licensee event reports (LERs).  The inspectors reviewed the licensee's responses to the
applicable cases.  The licensee did have all of the OE in their OE review system, with
the exception of LERs.  The licensee reviews industry OE that comes from INPO and
not specifically the LER database.  It appeared that the licensee had accounted for all
available OE at the time that could have reasonably been obtained and reviewed.All of the OE pertaining to notification events of inoperable diesels due to paintingdescribed gross painting errors that resulted in inoperable diesel generators (e.g.,
inappropriate/movable components being painted).  The licensee did take those events
into consideration when developing the work plan for painting of the EDGs in the
associated rooms.  The licensee held meetings well in advance of the scheduled
painting window, ensured that operations and maintenance personnel were
communicating, and developed a painter's handbook that presented precautions as well
as clear photographs of the areas and components not to paint.  The preparation was
adequate for the knowledge that the plant had on site at the time.  The sensitivity that
one paint drop in a specific, unintended location could render the EDG inoperable was
not considered by the licensee in their preparation and conduct of the EDG painting
activities, but this was not a subject of previous OE.One item that was not specifically incorporated into the procedures for painting the EDGwas a specific postmaintenance test to be performed to prove operability.  The
licensee's procedure described and recommended any of several postmaintenance
Enclosure 2-13-options, including visual inspections and equipment functionality tests.  This procedureand its weaknesses were discussed as part of the root cause evaluation in Section 1.3.The licensee sent two of its employees (a system engineer and a painting supervisor)on a benchmarking trip prior to cleaning up, painting, and relamping the EDG Rooms.
The licensee employees were aware of the potential to make the EDG inoperable by
painting activities, but did not get enough information to be as sensitive as necessary for
their painting activities.  After the failed EDG start, the licensee called the plants visited
during the benchmarking trip to ask more questions, and then discovered that one plant
had knowledge that very little paint or other foreign materials on the metering rods could
render the EDG inoperable.  The licensee could have possibly obtained this information
if their staff were to have asked more probing questions, given the work that was
planned at the site.  The inspectors concluded that the licensee was not fully effective in
addressing operating experience associated with painting impacts on emergency diesel
generator operability.1.8Potential Generic IssuesThe inspection team evaluated the circumstances surrounding the event and assessedthe root cause of the Unit 1Train B EDG failure to start.  The team interviewed
numerous licensee personnel and reviewed industry operating experience as well as
NRC generic communications with the goal of identifying any potential generic issues
that should be addressed as a result of the event.The inspection team concluded that, while painting activities occur at all plants, there areno specific generic concerns associated with this instance of procedural compliance.
The licensee has also issued an action in the corrective action program to issue an OE
report to INPO for future reference.2.0SPECIAL INSPECTION FINDINGS2.1Painting Activities Result in Inoperability of EDGIntroduction:  A White self-revealing violation of Unit 1 Technical Specification (TS)3.8.1, "AC Sources - Operating," was identified for the licensee's failure to satisfy TS
LCO 3.8.1 in that painting activities conducted on the Unit 1 Train B EDG 1-02 resulted
in paint being deposited and left in a location that caused the EDG to become
inoperable.  As a result, EDG 1-02 failed to start on demand during the subsequent
monthly surveillance test.  Following the discovery of the condition, the TS required
actions were satisfied; however, the time period between the occurrence of the condition
and the discovery of the condition exceeded the TS allowed outage time.Description:  On October 15, 2007, the licensee commenced painting activities thatoccurred on and around EDG 1-02.  A successful monthly slow start surveillance test
was performed on October 24, 2007.  Painting activities continued through November 1,
2007.  The inspectors reviewed Work Order (WO) 4-07-175968-00, which implemented
the painting activities on and around EDG 1-02 and specified that painting was to be
performed per the requirements of Procedure MSM-G0-0220, "General Plant Painting,"
Revision 2.  The inspectors noted that the WO did not contain requirements for
Enclosure 2-14-postmaintenance testing of the EDG, and that Procedure MSM-G0-0220, "General PlantPainting," Revision 2, contained the following steps:NOTE:  System engineer, operations, maintenance services or otherdepartments may provide useful guidance in determining appropriate protection
of equipment and post-painting functional testing.5.1.1.2  Painting conducted on equipment should be done in such a manner asto ensure paint does not bind components required to move.  Prejob briefings,visual verification of postpainting operation, equipment functional testing or other
similar activities are recommended practices that should be employed when
painting equipment.Through interviews, the inspectors determined that representatives from themaintenance services, system engineering, maintenance, and operations departments
discussed plans for verifying at the end of each day that the EDG remained operable.
The above requirement and guidance of the general plant painting procedure was not
referenced in this discussion.  It was decided that a senior operations department
personnel would perform a visual inspection at the end of each day to verify that
painting had not been done so as to affect the operability of the EDG.  This plan was
understood and executed, but was not documented, nor were any inspection results
documented.  Prejob briefs and postpainting inspections were focused on avoiding the
painting of components that were not supposed to be painted and were appropriate and
effective in that regard.  However, appropriate sensitivity to the potential functional
impact of stray drop(s) of paint in sensitive location(s) was not emphasized.On November 21, 2007, EDG 1-02 failed to start on demand during its next monthlysurveillance test.  Following approximately 11 hours of troubleshooting, EDG 1-02 was
successfully started.  This issue was entered into the licensee's corrective action
program as SMF-2007-003253-00.  The licensee performed a root cause analysis to
determine the cause of the failure.  The most likely cause of the failure was determined
to be a paint drop that had been deposited on the 6L fuel rack that caused the rack to
become stuck.  This prevented motion of all 16 fuel racks, thereby preventing the EDG
from receiving sufficient fuel to run.  Corrective actions planned and taken by the
licensee are discussed in Section 1.3 of this enclosure. Analysis:  The performance deficiency associated with this finding involved thelicensee's failure to ensure that the assumed operability of safety-related equipment was
not affected by the performance of scheduled maintenance activities.  Specifically,
painting was conducted on and around EDG 1-02 in such a manner that paint was
deposited and remained in a location that caused the EDG to become inoperable and
fail to start on demand during a subsequent surveillance test.  Postpainting verification
of equipment functionality was inadequate.  Consequently, the requirements of TS LCO
3.8.1.b and the associated required TS Actions B.4 and G.1 and 2 were not met.  The
finding was greater than minor because it was associated with the human performance
attribute of the mitigating systems cornerstone, and it affected the cornerstone objective
to ensure the availability, reliability, and capability of systems that respond to initiating
events to prevent undesirable consequences.  The Phase 1 Worksheets in Manual
Chapter 0609, "Significance Determination Process," were used to conclude that a
Enclosure 2-15-Phase 2 analysis was required because the performance deficiency affected theemergency power supply system that is a support system for both mitigating and
containment barrier systems.  Based on the results of the Phase 2 analysis, the finding
was determined to have low to moderate safety significance (White).  The senior reactor
analyst determined that a more detailed Phase 3 analysis was needed to fully assess
the safety significance.  Based on the results of the Phase 3 analysis, the finding was
determined to have low to moderate safety significance (White).  The Phase 1, 2, and 3
significance determination process analyses associated with this finding, including
assumptions and limiting core damage sequences, is included as Attachment 3 to this
report.  The cause of this finding was determined to have a crosscutting aspect in thearea of human performance associated with work practices in that the licensee failed to
provide adequate supervisory and management oversight of work activities, including
contractors, such that nuclear safety is supported [H.4(c)].  Specifically, the actions
planned and taken to assess and control the operational impact of the painting activities
on the functionality of the EDG were not reflective of adequate supervisory and
management oversight of the activities.Enforcement:  Unit 1 Technical Specification (TS) 3.8.1, "AC Sources - Operating,"requires that while the plant is in Modes 1, 2, 3, or 4, two diesel generators (DGs)
capable of supplying the onsite Class 1E power distribution subsystem(s) shall be
operable.  For the condition of one DG being inoperable, the required action is to restore
the DG to an operable status within 72 hours and within 6 days from the discovery of the
failure to meet the Limiting Condition for Operation (LCO), or be in Mode 3 within 6
hours and Mode 5 within 36 hours.  Contrary to the above, from November 1, 2007,
through November 21, 2007, while the plant was in Mode 1, one of the two DGs capable
of supplying the onsite Class 1E power distribution subsystem(s) was inoperable, and
action was not taken to either restore the DG to an operable status within 72 hours or be
in Mode 3 within 6 hours and Mode 5 within 36 hours.  Specifically, Emergency Diesel
Generator (EDG) 1-02 was made inoperable as a result of painting activities due to paint
having been deposited and remaining on at least one fuel rack in a location that
prevented motion required to support the operation of the EDG.  This condition caused
EDG 1-02 to fail to start during a surveillance test on November 21, 2007.  Following the
discovery of the condition on November 21, 2007, the licensee satisfied the TS required
actions by restoring the EDG to an operable status on November 22, 2007.  This
violation is the subject of the enclosed Notice of Violation:  VIO 05000445/2007008-01,
"Painting Activities Result in Inoperability of Emergency Diesel Generator."2.2Inadequate Alarm Response Procedure for EDG Failure to StartIntroduction:  The inspectors identified a Green noncited violation of Unit 1 TechnicalSpecification 5.4.1.a, "Procedures," for an inadequate alarm response procedure.  The
inspectors determined that Procedure ALM-1302A, "Diesel Generator 1-02 Panel,"
Revision 5, was inadequate in that it was ambiguous and did not cause the responders
to verify that the fuel racks were free as part of the response actions to investigate the
cause of the unit failing to start.  Consequently, the licensee failed to identify that the
Unit 1 Train B EDG 1-02 fuel racks were not free to move, which led to an extended
period of inoperability and a significant delay in diagnosing the cause of the EDG failure
to start.
Enclosure 2-16-Description:  On November 21, 2007, at 10:20 a.m., EDG 1-02 failed to start during aslow start monthly surveillance test.  Field operators responded to the EDG local alarm
panel.  Operators referenced the alarm response Procedure ALM-1302A, "Diesel
Generator 1-02 Panel," Revision 5, and reviewed the section for Alarm Window 6.6 "Unit
Failure To Start."  A limited number of system malfunctions that could have caused the
failure to start were indicated.  These included fuel rack or fuel oil day tank issues,
improper starting air alignment, failed timing chain, and a Governor malfunction.Operators implemented the "Operator Actions" section of the procedure, which includedactions to determine the cause of the unit failing to start.  The first action indicated was
to "Check fuel racks free and in max fuel position."  The field support supervisor (senior
reactor operator) believed that the appropriate action was to perform a visual inspection
of the fuel racks.  The fuel racks were not in the "max fuel" position.  The inspectors
later determined that, following the majority of postulated failed start scenarios, the fuel
racks would not be expected to remain in the "max fuel" position, even if they had
initially moved.  In accordance with the operators' training, the expectation for
performing this step was to visually inspect the racks.  However, the inspectors
determined that without observing them being in a position other than their normal
standby (closed) position, this visual check would not be sufficient to meet the intent of
the procedure step (i.e., to ensure that the racks were not stuck in the "no fuel" position,
which was a probable failure cause that was indicated earlier in the procedure).  The
operator completed this procedure step, as well as subsequent steps for starting air
alignment, EDG day tank alignment, and fuel quality with no abnormalities identified.
Field operator actions were completed at 11:05 a.m.  The licensee developed a troubleshooting plan and attempted two more starts of theEDG (both unsuccessful) before determining that the fuel racks and metering rods were
not responding to the Governor demand to open.  At 7:39 p.m. the licensee exercised
the fuel racks and discovered that two of the metering rods were stuck, with one fully
stuck in the closed position and one which became partially stuck following some motion
in the open direction.  Operations and maintenance performed followup inspections and
successfully started the EDG at 9:32 p.m.  The diesel was declared operable following
the surveillance run and post run inspections on November 22, 2007 at 3:08 a.m.The inspectors concluded that the field operators performed the actions of the alarmresponse Procedure ALM-1302A, in accordance with station procedures and training,
and operations management's expectations.  The inspectors further concluded that the
inadequacy of the alarm response procedure to give clear instruction and guidance to
ensure that the EDG fuel racks were verified to be free and not binding resulted in
missing an opportunity to identify the cause of the EDG failure to start in a timely
manner.  This missed diagnosis not only led to a narrowly focused troubleshooting effort
by the licensee, but also allowed the EDG to remain unnecessarily inoperable for
approximately an additional 8.5 hours.Analysis:  The performance deficiency associated with this finding involved thelicensee's failure to adequately establish clear procedure guidelines to implement alarm
response Procedure ALM-1302A.  This resulted in the licensee's failure to identify the
binding of the Unit 1 Train B EDG fuel racks and metering rods in a timely manner
following a failure to start.  The finding was determined to be more than minor because
Enclosure 2-17-it was associated with the procedure quality attribute of the mitigating systemscornerstone, and it affected the cornerstone objective to ensure the availability,
reliability, and capability of systems that respond to initiating events to prevent
undesirable consequences.  Using Manual Chapter 0609, "Significance Determination
Process," Phase 1 Worksheet, the finding was determined to have very low safety
significance (Green) because it was not a design or qualification deficiency, did not
represent a loss of safety function, did not represent an actual loss of a single train for
greater than its Technical Specification allowed outage time, did not represent a loss of
a non-Technical Specification train of equipment for greater than 24 hours, and did not
screen as potentially risk significant due to a seismic, flooding, or severe weather
initiating event.Enforcement:  Unit 1 Technical Specification 5.4.1.a requires that written procedures beestablished, implemented, and maintained covering the procedures listed in Regulatory
Guide 1.33, "Quality Assurance Program Requirements," Revision 2, Appendix A,
Section 5, for Abnormal, Off-Normal, or Alarm Conditions.  Contrary to the above, on
November 21, 2007, the licensee failed to adequately establish, implement, and
maintain a procedure for an alarm condition.  Specifically, alarm response
Procedure ALM-1302A, "Diesel Generator 1-02 Panel," Revision 5, was not adequately
established and maintained, which resulted in the licensee's failure to recognize that the
EDG 1-02 fuel racks and metering rods were bound and caused the failure of the EDG
to start on November 21, 2007.  Consequently, the EDG remained inoperable for
approximately 8.5 hours longer than necessary.  Because the finding was determined to
be of very low safety significance and has been entered in the licensee's correction
action program as SMF-2007-003426, this violation is being treated as an NCV
consistent with Section VI.A of the Enforcement Policy:  NCV 05000445/2007008-02,
"Inadequate Alarm Response Procedure for EDG Failure to Start."4OA6 Meetings, Including ExitOn December 7, 2007, and January 10, 2008, the results of this inspection werepresented to Mr. R. Flores, Site Vice President, and Mr. T. Hope, Regulatory
Performance Manager, respectively, and other licensee personnel who acknowledged
the findings.  Additionally on January 24, 2008, the final results of this inspection were
presented to Mr. F. Madden, Director, Regulatory Affairs, and other members of the
licensee staff who acknowledged the findings.  On February 25, 2008, an additional exit
meeting was conducted with Mr. T. Hope and other licensee personnel who
acknowledged the findings.  The inspectors confirmed that no proprietary material was
retained during the inspection.  ATTACHMENT 1:  Supplemental InformationATTACHMENT 2:  Special Inspection Charter
ATTACHMENT 3:  Significance Determination Evaluation
Attachment 1A1-1SUPPLEMENTAL INFORMATIONKEY POINTS OF CONTACTLicensee PersonnelJ. Bain, System EngineerJ. Bales, Maintenance Services
T. Bennette, Operations
M. Blevins, Senior Vice President and Chief Nuclear Officer
H. Davenport, System Engineer
D. Davis, Performance Improvement Director
R. Flores, Site Vice President
D. Goodwin, Manager, Shift Operations
T. Hope, Manager, Regulatory Performance
M. Kanavos, Plant Manager
D. Kross, Director, Operations
S. Lakdawala, Corrective Action Program Manager
F. Madden, Director, Regulatory Affairs
D. McGaughey, Manager, Shift Operations
G. Merka, Regulatory Affairs
J. Meyer, Technical Support Manager
W. Morrison, Maintenance Smart Team Manager
J. O'Quinn, Maintenance
W. Reppa, System Engineering Manager
D. Scott, Root Cause Analyst
S. Smith, Director, System Engineering
R. Sorrell, System Engineer
T. Terryah, System Engineering Manager
T. Tigner, Programs Supervisor
B. Wagner, PROMPT Team
W. Williams, Maintenance Services
M. Wisdom, System Engineering
NRCD. Allen, Senior Resident InspectorLIST OF ITEMS OPENED, CLOSED, AND DISCUSSEDOpened05000445/2007008-01VIOPainting Activities Result in Inoperability of EmergencyDiesel Generator (Section 2.1)
Attachment 1A1-2Opened and Closed05000445/2007008-02NCVInadequate Alarm Response Procedure for EDG Failure toStart (Section 2.2)
Closed None Discussed NoneLIST OF DOCUMENTS REVIEWEDProceduresNUMBERTITLEREVISIONALM-1302ADiesel Generator 1-02 Panel5
MSM-G0-0002General Plant Painting2
MSM-P0-3374Emergency Diesel Generator Monthly Run RelatedInspections
3OPT-215-1Offsite Transmission Network Operability Data Sheet14OWI-104-28Plant Equipment Operator Diesel Generator 1-02Operating Log
13OWI-104-26Control Room Diesel Generator 1-02 Operating Log11MSM-G0-0216Protective Coatings23
MSM-G0-0217Maintenance Protective Coatings-Concrete0
MSM-G0-0218Maintenance Protective Coatings-Steel0
CMP-CV-1009Application of Protective Coatings to Carbon SteelSurfaces in the Containment and Radiation Areas
Outside of Containment
0ODA-102Conduct of Operations24ODA-401Control of Annunciators, Instruments, and ProtectiveRelays 9ODA-407Guidelines on Use of Procedures12
Attachment 1A1-3OPT-214AEmergency Diesel Operability Test19SOP-609Diesel Generator System17
STA-426Industry Operating Experience Program1
STA-692Protective Coatings Program0
TSP-503Emergency Diesel Generator Reliability Program3Smart FormsSMF-2007-03253SMF-2007-03426SMF-2007-03302SMF-2007-02319WOs4-07-1765224-07-176543
5-05-501230-AA
5-07-502391-AK4-07-1759684-07-176544
4-07-176582
4-07-1754924-07-1765454-95-091357-00
4-94-078722-00Miscellaneous InformationEvaluation EVAL-2007-003253-02-00, "Root Cause Analysis""Post-Work Test Guide," Revision 12
LER 07-004-00, "Emergency Diesel Generator Failed Surveillance Test Due to Paint on FuelInjector Control Linkage"TUElectric Office Memorandum, CPSES-9125952, October 10, 1991
TUElectric Office Memorandum, CPSES-9108929, April 3, 1991
TUElectric Office Mamorandum, CPSES-91000861, January 11, 1991
Technical Evaluation TE# SE-90-1814
Cooper-Enterprise Clearinghouse R4/RV4 Preventative maintenance Program (PMP) forNuclear Standby Applications, Revision 0Operations Guideline 3, Attachment 4, "Operations Department Alarm Response Expectations,"August 2006
Attachment 1A1-4CPNPP Operations Logs, November 21-22, 2007Amercoat 220, Waterborne Acrylic Topcoat Product Datasheet, circa 1999
Information NoticesIN 93-76, "Inadequate Control of Paint and Cleaners for Safety-Related Equipment"IN 91-46, "Degradation of Emergency Diesel Generator Fuel Oil Delivery Systems"NRC Inspection DocumentsInspection Procedure 93812, "Special Inspection," 7/18/2007
"Special Inspection Charter to Evaluate the Comanche Peak Steam Electric Station DieselGenerator Failure to Start Event," November 30, 2007NRC Inspection ReportsML073060511 (RBS)ML072040388 (DC Cook IR 05000316/2007004)LIST OF ACRONYMSADAMSagency document and management system
CFRCode of Federal RegulationsCPSESComanche Peak Steam Electric Station
EDGemergency diesel generator
FMEforeign material exclusion
INPOInstitute of Nuclear Power Operations
LERlicensee event report
NRCNuclear Regulatory Commission
OEoperating experience
PARSpublicly available records system
NEOnuclear equipment operator
SDPsignificance determination process
SMFsmart form
WOwork order
A2-1Attachment 2November 30, 2007MEMORANDUM TO:Cale Young, Resident Inspector, ANOAlfred Sanchez, Resident Inspector, CPSESFROM: Arthur T. Howell III, Director, Division of Reactor Projects AVegel for/RA/
SUBJECT:SPECIAL INSPECTION CHARTER TO EVALUATE THE COMANCHEPEAK STEAM ELECTRIC STATION DIESEL GENERATOR FAILURE
TO START EVENTA Special Inspection Team is being chartered in response to the Comanche Peak SteamElectric Station emergency diesel generator (EDG) failure to start event on November 21, 2007.
You are hereby designated as the Special Inspection Team members.  Mr. Cale Young,
Resident Inspector, ANO, is designated as the team leader.  The assigned SRA to support the
team is David Loveless.A.BasisOn November 21, 2007, Comanche Peak Unit 1 diesel generator, DG-102, failed to startduring the monthly surveillance test.  After several failed attempts to start the diesel,
licensee engineers developed a trouble shooting plan to determine the cause of the
diesel failing to start.  During the trouble shooting efforts, licensee personnel identified
that two fuel rack linkage/metering rods (L2 and L6) on DG-102 appeared to be binding.
Additional inspections indicated that there were very small signs of paint on the metering
rods for the L2 and L6 fuel pumps, but not enough to prevent movement.  Painting
activities in all EDG rooms were suspended until further measures were taken to prevent
reoccurrence of this issue.  During the trouble shooting activities, each individual fuel
pump was manually operated by maintenance personnel and all but two moved freely.
Maintenance personnel were able to manually move, and subsequently free, the L2 and
L6 metering rods.  Operations personnel then performed the surveillance test
satisfactorily.  Maintenance personnel verified that the metering rods on the remaining
EDGs had free movement of all fuel rack linkage/metering rods.During further investigation into when painting had occurred inside the EDG room, it wasdiscovered that the painters continued to paint in the diesel room after the last
successful surveillance test.  This brings into question whether DG-102 would have
been able to perform its intended function if called upon from October 24 to
November 21, 2007.
A2-2Attachment 2This Special Inspection Team is chartered to review the circumstances related to thefailure of DG-102 to start, and to assess the effectiveness of the licensee's actions for
resolving these problems.B.ScopeThe team is expected to address the following:
1.Develop a chronology (time-line) that includes significant event elements.
2.Evaluate the licensee's response to the failure of the EDG to start.  Ensure thatplant personnel responded in accordance with plant procedures and Technical
Specifications.3.Assess the licensee's root cause determination for the EDG failure, the extent ofcondition review, the common cause evaluation and corrective measures.
Evaluate whether the timeliness of the corrective measures are consistent with
the safety significance of the problem.4.Develop a complete scope of the failure mechanism identified by the licensee'sroot cause determination.5.Identify previous EDG issues that may have been precursors to the November 1,2007, event.  Evaluate the licensee's corrective measures and extent of
condition reviews for those problems.6.Evaluate the licensee's EDG maintenance and testing programs.  Verify that theprograms are adequate and that the licensee is following the program provisions.7.Evaluate pertinent industry operating experience that represents potentialprecursors to the November 21, 2007, event, including the effectiveness of
licensee actions taken in response to the operating experience.8.Determine if there are any potential generic issues related to the EDG failure atComanche Peak Unit 1.  Promptly communicate any potential generic issues to
Region IV management.9.Collect data as necessary to support a risk analysis.  Work closely with theSenior Reactor Analyst during this inspection.
A2-3Attachment 2C.GuidanceInspection Procedure 93812, "Special Inspection," provides additional guidance to beused by the Special Inspection Team.  Your duties will be as described in Inspection
Procedure 93812.  The inspection should emphasize fact-finding in its review of the
circumstances surrounding the event.  It is not the responsibility of the team to examine
the regulatory process.  Safety concerns identified that are not directly related to the
event should be reported to the Region IV office for appropriate action.The Team will report to the site, conduct an entrance, and begin inspection no later thanDecember 4, 2007.  While on site, you will provide daily status briefings to Region IV
management, who will coordinate with the Office of Nuclear Reactor Regulation, to
ensure that all other parties are kept informed.  If information is discovered that shows a more significant risk was associated with this issue, immediately contact Region IVmanagement for discussion of appropriate actions.  A report documenting the results of
the inspection should be issued within 30 days of the completion of the inspection.This Charter may be modified should the team develop significant new information thatwarrants review.  Should you have any questions concerning this Charter, contact me at
(817) 860-8148.
A3-1Attachment 3ATTACHMENT 3SIGNIFICANCE DETERMINATION EVALUATIONComanche Peak Steam Electric StationEDG Inoperability Caused By Painting ActivitiesSignificance Determination Basis
1.Phase 1 Screening Logic, Results, and AssumptionsIn accordance with NRC Inspection Manual Chapter 0612, Appendix B, "IssueScreening," the team determined that this finding represented a licensee performance
deficiency.  The team then determined that the issue was more than minor because it
was associated with the equipment performance attribute and affected the mitigating
systems cornerstone objective to ensure the availability, reliability, or function of a
system or train in a mitigating system in that Emergency Diesel Generator DG-102
would not have started upon demand.The team evaluated this finding using the "SDP Phase 1 Screening Worksheet for theInitiating Events, Mitigating Systems, and Barriers Cornerstones," provided in Manual
Chapter 0609, Appendix A, "Determining the Significance of Reactor Inspection
Findings for At-Power Situations."  For this finding, a Phase 2 estimation was required
because the performance deficiency affected the emergency power supply system that
is a support system for both mitigating and containment barrier systems.
2.Phase 2 Risk EstimationIn accordance with Manual Chapter 0609, Appendix A, Attachment 1, "User Guidancefor Phase 2 and Phase 3 Significance Determination of Reactor Inspection Findings for
At-Power Situations," the senior reactor analyst evaluated the subject finding using the
Risk-Informed Inspection Notebook for Comanche Peak Steam Electric Station, Units 1
and 2, Revision 2.01a.  The following assumptions were made:a.The identified performance deficiency occurred some time between the last successful test on October 24, 2007, and the test failure that occurred on
November 21, 2007.b.In accordance with Manual Chapter 0609, Appendix A, Attachment 2, "SiteSpecific Risk-Informed Inspection Notebook Usage Rules," Rule 1.1, "Exposure
Time," the analyst determined the time frame over which the finding impacted
the risk of plant operations.  Because the exact time of failure was unknown, an
exposure time of t/2 from the last valid test was used.  This was 1/2 of the 28 days
between tests, or 14 days.  Therefore, for the phase 2 analysis, the exposure
time used to represent the time that the performance deficiency affected plant
risk was between 3 and 30 days.
A3-2Attachment 3c.Table 2 of the risk-informed notebook requires that when a performancedeficiency affects the diesel generators, the following initiating event scenarios
are applicable:  LOOP and LEAC.  Therefore, the analyst utilized these
worksheets from the risk-informed notebook.d.According to the risk-informed notebook, Table 1, for a 3-30 day exposure, the initiating event likelihood should be 3 for a loss of offsite power and 5 for a loss
of offsite power with loss of one vital 6.9kV bus.e.The analyst gave no operator action credit as discussed in ManualChapter 0609, Appendix A, Attachment 1, Table 4, "Remaining Mitigation
Capability Credit."  The requirements to have procedures in place and to have
trained the operators in recovery under similar conditions for such credit were not
met.The dominant sequences from the notebook were documented below:TABLE C.bFailure of Emergency Diesel Generator 102 to StartPhase 2 Sequences Initiating EventSequenceMitigating FunctionsResults  Loss of Offsite Power  2  LOOP-AFW-FB  8
  4  LOOP-EAC-REC5  6
  7  LOOP-EAC-TDAFW  6  Loss of Offsite Power with      Loss of One Vital 6.9 kV Bus  1  LEAC-PORV-HPR-MKRWST  8
  3  LEAC-PORV-HPI  7Using the counting rule worksheet, the result from this estimation indicated thatthe finding was of low to moderate safety significance (WHITE).  However, the
analyst determined that this estimate did not include a full coverage of the risk
related to the failure identified and that a better evaluation of the internal risk
would be necessary for fully assessing the risk related to external initiators.
3.Phase 3 AnalysisIn accordance with Manual Chapter 0609, Appendix A, the analyst performed a Phase 3analysis using the Standardized Plant Analysis Risk (SPAR) Model for Comanche Peak,
Revision 3.31, dated August 2006, to simulate the failed Diesel Generator 1-02.
Additionally, the analyst conducted an assessment of the risk contributions from external
initiators using insights and/or values provided by the licensee's probabilistic risk
assessment model, in the licensee's recent submittal for extension of completion times
for diesel generators (Reference 1), and simplified fire probabilistic risk assessment.Reference 1:  Letter dated November 15, 2007, Blevins to U.S. NRC, Subject:  "Comanche Peak Steam Electric Station (CPSES) Docket Nos. 50-445 and 50-446, Response to Request for Additional Information Related to Licence Amendment Request (LAR) 06-009
,Revision to Technical Specification (TS) 3.8.1, 'AC Sources - Operating; Extension of Completion Times for Diesel Generators."
A3-3Attachment 3
AssumptionsTo evaluate the change in risk caused by this performance deficiency, the analyst madethe following assumptions:A.The vital batteries at Comanche Peak will deplete after approximately 4 hours of fullpostaccident loads without an operating battery charger, assuming that operators do
not take actions to shed unnecessary loads from the vital dc buses.  This is the
value used in the licensee's probabilistic risk assessment.B.The Comanche Peak SPAR model, Revision 3.31, represents an appropriate tool forevaluation of the subject finding.C.The failure of Emergency Diesel Generator 1-02 was the result of binding of the fuelrack on at least one injection pump that was caused by painting activities on and
around the diesel.D.Emergency Diesel Generator 1-02 successfully started and loaded during asurveillance performed on October 24, 2007.  The diesel failed to start during a
surveillance on November 21, 2007, because the fuel rack on at least one injection
pump was bound to the extent that the entire fuel rack assembly was unable to leave
the "no fuel" position.E.Painting activities in and around the Emergency Diesel Generator 1-02 engineended on November 1, 2007.  Therefore, the conditions that caused the engine to
fail had to have been in place at that time for the root cause to be valid (See
Assumption C). F.The exposure time used for evaluating this finding should be determined inaccordance with Inspection Manual Chapter 0609, Appendix A, Attachment 2, "Site
Specific Risk-Informed Inspection Notebook Usage Rules."  Attachment 2 discusses
the approach to establishing the exposure time that should be used for the
significance determination process.  Step 1.1 states:"The exposure time used in determining the initiating event likelihood shouldcorrespond to the time period that the condition being assessed is reasonably
known to have existed.  If the inception of the condition is unknown, then an
exposure time of one half of the time period since the last successful
demonstration of the component or function (t/2) should be used."G.The appropriate exposure time (EXP), representing the time that Emergency DieselGenerator 1-02 was not functional, for use in this evaluation is 24 days.The exact time at which the residual paint that caused the binding of the fuelracks occurred is unknown.  However, it is reasonable to assume that the
condition existed after the completion of painting activities on November 1, 2007. 
A3-4Attachment 3Therefore, in accordance with Assumption F, Emergency Diesel Generator 1-02would not have started upon demand for the 20 days November 1 through
November 21, 2007.Additionally, the inception of the condition could have occurred any time betweenthe last successful run of the machine on October 24 and the completion of the
painting activities on November 1.  Therefore, in accordance with Assumption F,
Emergency Diesel Generator 1-02 would not have started upon demand for one
half of the period from October 24 through November 1, or for an additional 4-
day period.Based on these two arguments, the analyst determined that the appropriateexposure time was the sum of the 20 days that the machine was reasonably
assumed to have failed and one half the 8 day period that could have resulted in
the failure condition.H.Given the condition of the fuel rack and the interpretation by licensed operators ofannunciator response procedures, operators would not have been able to recover
Emergency Diesel Generator 1-02 prior to postulated core damage for sequences
less than 2 hours.  Licensed operators stated that the annunciator response
procedure would not have directed operators to manipulate the fuel racks by hand
nor does it require operators to request maintenance personnel perform such a task.I.The appropriate nonrecovery probability for sequences longer than 2 hours is 0.29. The analyst conducted a human reliability analysis using the SPAR-H method to
determine an appropriate nonrecovery probability.  To calculate this value, the
analyst used the following assumptions:a.The analyst assumed that nominal time was available for recovery diagnosis andaction.  The licensee recovered the diesel in 11 hours 11 minutes during
nonemergency conditions. Therefore, the analyst assumed that, if required,
recovery could have been reasonably performed within the 4 hour coping period
plus extended boil down times.b.The analyst assumed that emergency response personnel would be under highstress during the diagnosis and recovery.  This is based primarily on the belief
that recovery personnel would know that the consequences of the task would
represent a direct threat to plant safety.c.The complexity of this task was considered to be nominal.  There is someambiguity in the diagnosis.  However, there are only two fundamental paths in
diagnosis.  The probability that the wrong path would be initially investigated is
taken into account in the performance shaping factor for procedure quality.d.Procedures for the diagnosis were incomplete.  Solely following the proceduresavailable would not have led to recovery.  The basic items to consider were
available in the annunciator response procedure, although it is not clear that this
procedure would have been governing and/or utilized by the recovery personnel.
A3-5Attachment 3e.All other performance shaping factors were considered nominal for obviousreasons.J.Emergency Diesel Generator 1-01 would not have failed from the same cause asEmergency Diesel Generator 1-02 because painting activities had not been
conducted on that diesel.  Therefore, the analyst left the common cause failure
probability at it's nominal value.K.The nominal nonrecovery values used by the SPAR model are for the averagenonrecovery for either of two diesel generators.  Therefore, given that recovery of
Emergency Diesel Generator 1-02 would be handled separately, the analyst
adjusted the generic nonrecovery value to account for only Emergency Diesel
Generator 1-01 being the only machine available for random failure recovery.L.The nominal likelihood for a loss of offsite power was unaffected by the subjectfinding.M.Evaluating the risk contribution of this finding related to seismic events isappropriately conducted by utilizing the licensee's assessment found in Reference 1.
The conditional core damage frequency (CCDFSEISMIC) given by the licensee was 2.1 x 10-6/year.N.The licensee's fire risk model is an appropriate tool for evaluation of the subjectfinding.  The CCDF for fire (CCDFFIRE) provided by the licensee in Reference 1 was7.8 x 10-6/year.The analyst independently evaluated the risk change related to internal fires. These insights were then used to challenge and evaluate the results of the
licensee's model.  In all cases, the licensee's model covered the scenarios
posed by the analyst and included a larger scope of fires than was feasible for
the analyst to evaluate.O.Traditionally, the initiation of most high wind events, including those that cause aloss of offsite power, are included in the licensee's PRA and/or the SPAR model.
However, the licensee's assessment in their individual plant evaluation for externalevents did not include events that damage other pieces of equipment that may affect
risk.  As stated in Reference 1, the licensee estimated the CCDF for tornados
(CCDFWIND) given the failure of a diesel generator to be 2.3 x 10
-5/year.P.The best estimate of the risk contribution from the subject finding related to internalflooding is best evaluated using a ratio from the licensee's PRA as was discussed in
Reference 1.  In their evaluation, the licensee stated that the risk from internal
flooding derived from their internal events PRA was approximately 1 percent (P
FLOOD)of the total plant core damage frequency.Q.The ratio of sequences going to core damage in the first 2 hours to those goingthrough battery depletion is the same for internal and external initiators.  This
A3-6Attachment 3assumption permits the analyst to use the ratio from the internal events SPAR inapplying recovery to external initiators.R.The differences between the SPAR and the licensee's models were inconsequential. The analyst, in reviewing the differences between the models, determined that there
were several global differences including:  the lack of random failure recovery for
diesel generators in the licensee's model and the lack of convolution integrals in the
SPAR model.  However, the analyst determined that these differences were not of
consequence to this evaluation because the final results were within the same color
band. Internal Initiating EventsThe senior reactor analyst used the SPAR model for CPSES to estimate the change inrisk associated with internal initiators that was caused by the finding.  Average test and
maintenance of modeled equipment was assumed and a cutset truncation of 1.0E-12
was used.Consistent with guidance in the RASP Handbook, including NRC document,"Common-Cause Failure Analysis in Event Assessment (June 2007)," and Assumptions
3.C, 3.G, and 3.K, the SRA modeled the condition by adjusting the following basicevents in the SPAR model:    Basic Event
          Original Value      Conditional Value EPS-DGN-FS-1EG1      5.0 X 10
-3    TRUE EPS-XHE-XL-NR01H    7.72 X 10
-1    8.79 X 10
-1 EPS-XHE-XL-NR02H    6.48 X 10
-1    8.05 X 10
-1 EPS-XHE-XL-NR03H    5.56 X 10
-1    7.46 X 10
-1 EPS-XHE-XL-NR04H    4.84 X 10
-1    6.95 X 10
-1The SPAR baseline core damage frequency (CDF
BASE) was 1.80 x 10
-5/year.  Theevaluation case for the above change set resulted in a conditional core damage
frequency (CCDF
SPAR) of 3.78 x 10
-4/year.  The dominant core damage sequences weredocumented in the table below:
A3-7Attachment 3Initiating EventSequencePreponderant FailuresFrequencyLoss of OffsitePower20-03Failure of EDG 1-01 withBattery Depletion at 4 hours2.62 x 10-4/year20-06Failure of EDG 1-01 withBattery Depletion at 4 hours
combined with RCS Pump
Seal Failure .6.56 x 10-5/year20-45Failure of EDG 1-01 and theTurbine-Driven AuxiliaryFeedwater Pump with Core
Damage at 1 hour.2.37 x 10-5/year19Failure of Motor and Turbine-Driven Auxiliary Feedwater
Pumps and/or Operator Fails
to Control.1.07 x 10-5/yearThe change in incremental conditional core damage frequency (ICCDP) was calculatedas follows:ICCDF=  CCDF
SPAR  -  CDF base=  3.78 x 10
-4/year -  1.80 x 10
-5/year=  3.60 x 10
-4/yearGiven Assumptions 3.C through 3.G, the exposure time, representing the time that theperformance deficiency impacted the plant, for this analysis was 24 days.  Therefore,
the change in core damage frequency (CDFIntNR) caused by this finding, withoutapplying any recovery to the subject condition, and related to internal initiators was
calculated as follows:CDFIntNR=  ICCDF  *  EXP=  3.60 x 10
-4/year  *  (24 days  ÷  365 days/year)=  2.37 x 10
-5Given Assumption 3.H, the analyst determined that recovery credit for EmergencyDiesel Generator 1-01 would not be provided for any sequence that led to core damage
in less than 2 hours.  Using utilities in the SAPHIRE software to slice cutsets by basic
event, the analyst determined that 7.6 percent of all internal cutsets went to core
damage in less than 2 hours (P
SHORT).Given Assumption 3.I, the analyst applied a nonrecovery factor (P
NR) of 0.29 to all
A3-8Attachment 3remaining cutsets.  Therefore, the change in core damage frequency (CDFInternal)caused by this finding and related to internal initiators was calculated as follows:CDFInternal=  [CDFIntNR * P SHORT]  + [CDFIntNR * (1 - P SHORT) * P NR]=  [2.37 x 10
-5 * 0.076] + [2.37 x 10
-5 * (1 - 0.076) * 0.29]=  8.15 x 10
-6External Initiating EventsIn accordance with Manual Chapter 0609, Appendix A, Attachment 1, Step 2.2.5,"Screen for the Potential Risk Contribution Due to External Initiating Events," the analyst
assessed the impact of external initiators on each of the findings, because the Phase 2
SDP result provided a Risk Significance Estimation of 7 or greater.  The analyst
determined that, for the risk of an external initiator to be impacted by this performance
deficiency, the external event would have to cause a loss of offsite power that was not
accounted for in the internal events model.  Using the licensee's individual plant
evaluation for external events and Reference 1, the analyst determined that the
dominant sequences affected by the subject performance deficiency were from seismic
events, high winds, fire, and internal flooding events.A.Seismic Event InitiatorsAs discussed in Assumption 3.M, the analyst utilized the licensee's valuefor the affects on the risk of seismic events associated with a failed diesel
generator.  The incremental risk without recovery (ICCDPSeisNR) wascalculated as follows:ICCDPSeisNR=  CCDFSEISMIC * EXP=  2.1 x 10
-6/year * (24 days  ÷  365 days/year)=  1.38 x 10
-7Given Assumption 3.H and 3.Q, the analyst applied PSHORT in quantifyingthe change in risk from seismic events.Given Assumption 3.I, the analyst applied a nonrecovery factor (P
NR) of0.29 to the remaining portion of the risk from seismic events.  Therefore,
the change in core damage frequency (CDFSEISMIC) caused by thisfinding and related to seismic events was calculated as follows:CDFSEISMIC=  [ICCDPSeisNR * P SHORT] + [ICCDPSeisNR * (1 - P SHORT) * P NR]=  [1.38 x 10
-7 * 0.076] + [1.38 x 10
-7 * (1 - 0.076) * 0.29]=  4.75 x 10
-8
A3-9Attachment 3B.Internal Fire InitiatorsAs discussed in Assumption 3.N, the analyst utilized the licensee's valuefor the affects on the risk of internal fires associated with a failed diesel
generator.  The incremental risk without recovery (ICCDPFireNR) wascalculated as follows:ICCDPFireNR=  CCDFFIRE * EXP=  7.8 x 10
-6/year * (24 days  ÷  365 days/year)=  5.14 x 10
-7Given Assumption 3.H and 3.Q, the analyst applied PSHORT in quantifyingthe change in risk from internal fires.Given Assumption 3.I, the analyst applied a nonrecovery factor (P
NR) of0.29 to the remaining portion of the risk from seismic events.  Therefore,
the change in core damage frequency (CDFFIRE) caused by this findingand related to seismic events was calculated as follows:CDFFIRE=  [ICCDPFireNR * P SHORT]  + [ICCDPFireNR * (1 - P SHORT) * P NR]=  [5.14 x 10
-7 * 0.076] + [5.14 x 10
-7 * (1 - 0.076) * 0.29]=  1.77 x 10
-7C.Tornados and High Wind Initiators
As discussed in Assumption 3.O, the analyst utilized the licensee's valuefor the affects on the risk of high wind events associated with a failed
diesel generator.  The incremental risk without recovery (ICCDFWindNR)was calculated as follows:ICCDPWindNR=  CCDFWIND * EXP=  2.1 x 10
-5/year * (24 days  ÷  365 days/year)=  1.38 x 10
-6Given Assumption 3.H and 3.Q, the analyst applied PSHORT in quantifyingthe change in risk from high wind events.Given Assumption 3.I, the analyst applied a nonrecovery factor (P
NR) of0.29 to the remaining portion of the risk from high wind events.
Therefore, the change in core damage frequency (CDFWIND) caused bythis finding and related to seismic events was calculated as follows:
A3-10Attachment 3CDFWIND=  [ICCDPWindNR * P SHORT]  + [ICCDPWindNR * (1 - P SHORT) * P NR]=  [1.38 x 10
-6 * 0.076] + [1.38 x 10
-6 * (1 - 0.076) * 0.29]=  4.75 x 10
-7D.Internal Flooding Initiators
As discussed in Assumption 3.O, the analyst utiliz
ed the ratio determinedby the licensee's PRA for internal flooding to other initiators.  Given a
value of 1 percent, the change in core damage frequency (CDF FLOOD)caused by this finding and related to internal flooding was calculated as
follows:CDF FLOOD=  CDFInternal * P FLOOD=  8.15 x 10-6  * 0.01=  8.15 x 10
-8Total Change in Core Damage FrequencyGiven that each of the initiators in this analysis were treated to ensure that the finalprobabilities were independent of each other, the analyst determined that he total
change in core damage frequency (CDF) could be calculated by taking the sum ofeach independent change.  Therefore, the final Phase 3 result was calculated as
follows:CDF=  CDFInternal + CDFExternal=  CDFInternal + [CDFSEISMIC + CDFFIRE + CDFWIND + CDF FLOOD]=  8.15 x 10
-6 + [4.75 x 10
-8 + 1.77 x 10
-7 + 4.75 x 10
-7 + 8.15 x 10
-8]=  8.93 x 10
-6This result indicated that the finding was of low to moderate significance to the risk ofinternal initiating events.Large Early Release Frequency Contribution
In accordance with Manual Chapter 0609, Appendix A, Attachment 1, Step 2.2.6,
"Screen for the Potential Risk Contribution Due to LERF," the analyst assessed the
impact on the large early release frequency because the Phase 2 SDP result provided a
risk significance estimation of greater than 7.Using NRC Inspection Manual Chapter 0609 Appendix H, "Containment IntegritySignificance Determination Process," the senior reactor analyst determined that this was
a Type A finding (i.e., a finding that can influence the likelihood of accidents leading to
A3-11Attachment 3core damage that is also a LERF contributor).  For a pressurized water reactor with alarge, dry containment, like Comanche Peak Steam Electric Station, findings related to
inter-system loss-of-coolant accidents and steam generator tube ruptures have the
potential to impact LERF.Appendix H, Table 5.1, "Phase 1 Screening - Type A Findings at Full Power," providesthat station blackout scenarios and all other transients, including loss of offsite power
initiators, screen out from further evaluation.  These accident sequences are not
considered to be significant to LERF.  Therefore, the estimated LERF was calculatedto be less than 6.8 x 10
-7.  Because the LERF was less than the 1 x 10
-6 White/Yellowthreshold, the finding remains characterized as of low to moderate safety significance
(White).
}}

Revision as of 14:19, 12 July 2019

IR 05000445-07-008, on 12/4/07-1/24/08, Comanche Peak, Final Significance Determination for a White Finding and Notice of Violation, Special Inspection
ML080600164
Person / Time
Site: Comanche Peak Luminant icon.png
Issue date: 02/29/2008
From: Collins E
Region 4 Administrator
To: Blevins M
Luminant Generation Co
References
EA-08-028 IR-07-008
Download: ML080600164 (42)


See also: IR 05000445/2007008

Text

February 29, 2008EA-08-028Mike Blevins, Senior Vice President and Chief Nuclear Officer

Luminant Generation Company, LLC

ATTN: Regulatory Affairs

Comanche Peak Steam Electric Station

P.O. Box 1002

Glen Rose, TX 76043SUBJECT:FINAL SIGNIFICANCE DETERMINATION FOR A WHITE FINDING ANDNOTICE OF VIOLATION - COMANCHE PEAK STEAM ELECTRIC STATION -

NRC SPECIAL INSPECTION REPORT 05000445/2007008Dear Mr. Blevins:

On January 24, 2008, the U.S. Nuclear Regulatory Commission (NRC) completed its reviewsrelated to a Special Inspection at your Comanche Peak Steam Electric Station, Unit 1, facility. This Special Inspection Team was chartered to review the circumstances related to the failure

of Emergency Diesel Generator (EDG) 1-02 to start on November 21, 2007, and to evaluate the

actions taken in response to the problem. The NRC's initial evaluation satisfied the criteria in

NRC Management Directive 8.3, "NRC Incident Investigation Program," for conducting a special

inspection. The possibility that adverse generic implications were associated with the EDG

failure mechanism was the deterministic criterion met. Additionally, the result of the NRC's

initial conditional risk assessment associated with this degraded condition indicated that a

special inspection was warranted. The determination that the inspection would be conducted

was made by the NRC on November 30, 2007, and the inspection started on December 4,

2007.The inspection examined activities conducted under your license as they relate to safety andcompliance with the Commission's rules and regulations and with the conditions of your license.

The inspectors reviewed selected procedures and records, observed activities, and interviewed

personnel.The enclosed inspection report documents the inspection results, which were discussed onDecember 7, 2007, and again on January 10, 2008, with Mr. R. Flores and Mr. T. Hope,

respectively, and other members of your staff. On January 24, 2008, an exit meeting was held

with Mr. F. Madden, Director, Regulatory Affairs, and other members of your staff to convey the

Luminant Generating Company, LLC-2-final disposition of the inspection findings. Following a discussion of the preliminary safetysignificance of this finding during the exit briefing, Mr. Madden indicated that Luminant Power

does not contest the characterization of the risk significance of this finding, and that you have

declined to further discuss this issue at a Regulatory Conference or provide a written response.

Accordingly, the NRC is issuing the final significance determination for the inspection finding as

discussed below. On February 25, 2008, an additional exit meeting was held with Mr. T. Hope,

and other members of your staff to convey a revision to one of the inspection findings. This report documents one finding concerning a failure to satisfy Technical Specification (TS)Limiting Condition for Operation (LCO) 3.8.1 due to EDG 1-02 being in an inoperable conditionfollowing maintenance. Following the discovery of this condition, the TS required actions were

satisfied however, the time period between the occurrence of the condition and the discovery of

the condition exceeded the TS allowed outage time for the EDG. This finding has been

determined to be of low to moderate safety significance (White). This finding does not

represent an immediate safety concern because of the corrective actions you have taken.

These actions included restoring EDG 1-02 to an operable status, ensuring that all other EDGs

were not in a similar degraded condition, and curtailing painting activities pending the

implementation of suitable measures to prevent the recurrence of a similar condition.You have 30 calendar days from the date of this letter to appeal the NRC's determination ofsignificance for the identified White finding. Such appeals will be considered to have merit only

if they meet the criteria given in NRC Inspection Manual Chapter 0609, Attachment 2. In

accordance with the NRC Enforcement Policy, the Notice of Violation is considered an

escalated enforcement action because it is associated with a White finding.You are required to respond to this letter and should follow the instructions specified in theenclosed Notice when preparing your response.In addition, we will use the NRC Action Matrix to determine the most appropriate NRC responseto this issue, and we will notify you by separate correspondence of that determination.The report also documents one NRC-identified finding of very low safety significance (Green). This finding was determined to involve a violation of NRC requirements. However, because of

the very low safety significance and because it is entered into your corrective action program,

the NRC is treating the finding as a noncited violation (NCV) consistent with Section VI.A.1 of

the NRC Enforcement Policy. If you contest this NCV, you should provide a response within

30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear

Regulatory Commission, ATTN.: Document Control Desk, Washington, DC 20555-0001; with

copies to the Regional Administrator, U.S. Nuclear Regulatory Commission, Region IV, 611

Ryan Plaza Drive, Suite 400, Arlington, Texas 76011-4005; the Director, Office of Enforcement,

U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident

Inspector at the Comanche Peak Steam Electric Station.In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, itsenclosure, and your response (if any) will be available electronically for public inspection in the

Luminant Generating Company, LLC-3-NRC Public Document Room or from the Publicly Available Records (PARS) component ofNRC's document system (ADAMS). ADAMS is accessible from the NRC Web site at

http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room). To theextent possible, your response should not include any personal privacy, proprietary, or

safeguards information so that it can be made available to the public without redaction.Sincerely, /RA/Elmo E. CollinsRegional AdministratorDockets:50-445Licenses:NPF-87Enclosures:1.Notice of Violation

2.NRC Inspection Report 05000445/2007008

w/AttachmentsAttachment 1:Supplemental InformationAttachment 2:Special Inspection Charter

Attachment 3:Significance Determination Evaluationcc w/enclosures:Fred W. Madden, Director

Regulatory Affairs

Luminant Generation Company LLC

P.O. Box 1002

Glen Rose, TX 76043Timothy P. Matthews, Esq.Morgan Lewis

1111 Pennsylvania Avenue, NW

Washington, DC 20004Anthony Jones, Chief Boiler InspectorTexas Department of Licensing

and Regulation

Boiler Program

P.O. Box 12157

Austin, TX 78711Somervell County JudgeP.O. Box 851

Glen Rose, TX 76043

Luminant Generating Company, LLC-4-Richard A. Ratliff, ChiefBureau of Radiation Control

Texas Department of Health

1100 West 49th Street

Austin, TX 78756-3189Environmental and Natural Resources Policy Director

Office of the Governor

P.O. Box 12428

Austin, TX 78711-3189Brian AlmonPublic Utility Commission

William B. Travis Building

P.O. Box 13326

Austin, TX 78711-3326Susan M. JablonskiOffice of Permitting, Remediation

and Registration

Texas Commission on

Environmental Quality

MC-122

P.O. Box 13087

Austin, TX 78711-3087Environmental and Natural Resources Policy Director

Office of the Governor

P.O. Box 12428

Austin, TX 78711-3189Lisa R. Hammond, ChiefTechnological Hazards Branch

National Preparedness Division

FEMA Region VI

800 N. Loop 288

Denton, TX 76209

Luminant Generating Company, LLC-5-Electronic distribution:ROPreportsRIDSSECYMAILCENTERRIDSOCAMAILCENTERRIDSEDOMAILCENTERRIDSOEMAILCENTER

RIDSOGCMAILCENTERRIDSNRROD

RIDSNRRADIP RIDSOPAMAIL

RIDSOIMAILCENTERRIDSOIGMAILCENTER

RIDSOCFOMAILCENTERRIDSRGN1MAILCENTER

RIDSRGN2MAILCENTERRIDSRGN3MAILCENTER

RIDSNRRDIPMIIPBOEWEB ECollinsAHowellFuller - KSFC MaierVasquez - GMVD Furst, NSIR

Vegel - AXVChamberlain - DDCN Hilton, OE

Caniano - RJC1Powers - DAPJune Cai, OE

ACampbell - ACCCJohnson - CEJ1John Wray, OE

DLoveless - DPLDAllen - DBAASanchez - AAS1

Herrera - MSH3SSanner - ESSStarkey, OE - DRS

Mary Ann Ashley, NRRPaulk -CJPM Burrell, OE

Dricks - VLDWMaier - WAMR Barnes, OE

DPowers - DAPDPelton - DLP1SUNSI Review Completed: CEJ ADAMS: Yes G No Initials: CEJ

Publicly Available

G Non-Publicly Available

G Sensitive Non-Sensitive R:\_REACTORS\_CPSES\2007\CP2007-08 CHY.wpdADAMS ML080600164RIV:RI:DRP/ESRI:DRP/E

SRAC:DRP/ASES/ACESCHYoung:vlh;mjsAASanchezDPLoveless

CEJohnsonGMVasquez E-CEJ E-CEJ /RA//RA//RA/1/31/082/08/082/08/082/12/082/11/08D:DRSD:DRPC:ACESDRARARJCanianoDDChamberlainKSFullerATHowellEECollins /RA//RA//RA//RA//RA/2/11/082/21/082/14/082/22/082/28/08OFFICIAL RECORD COPYT=Telephone E=E-mail F=Fax

Enclosure 1NOTICE OF VIOLATIONLuminant Generation Company, LLC Docket No. 50-445Comanche Peak Steam Electric StationLicense No. NPF-87EA-08-028During an NRC inspection completed on January 24, 2008, a violation of NRC requirementswas identified. In accordance with the NRC Enforcement Policy, the violation is listed below:Unit 1 Technical Specification (TS) 3.8.1, "AC Sources - Operating," requires that whilethe plant is in Modes 1, 2, 3, or 4, two diesel generators (DGs) capable of supplying the

onsite Class 1E power distribution subsystem(s) shall be operable. For the condition of

one DG being inoperable, the required action is to restore the DG to an operable status

within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> and within 6 days from the discovery of the failure to meet the Limiting

Condition for Operation (LCO), or be in Mode 3 within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and Mode 5 within 36

hours.Contrary to the above, from November 1, 2007, through November 21, 2007, while theplant was in Mode 1, one of the two DGs capable of supplying the onsite Class 1E

power distribution subsystem(s) was inoperable, and action was not taken to either

restore the DG to an operable status within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> or be in Mode 3 within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and

Mode 5 within 36 hours4.166667e-4 days <br />0.01 hours <br />5.952381e-5 weeks <br />1.3698e-5 months <br />. Specifically, Emergency Diesel Generator (EDG) 1-02 was

made inoperable as a result of painting activities due to paint having been deposited and

remaining on at least one fuel rack in a location that prevented motion required to

support the operation of the EDG. This condition caused EDG 1-02 to fail to start during

a surveillance test on November 21, 2007. This violation is associated with a White significance determination process finding.Pursuant to the provisions of 10 CFR 2.201, Luminant Generation Company, LLC is herebyrequired to submit a written statement or explanation to the U.S. Nuclear Regulatory

Commission, ATTN.: Document Control Desk, Washington, DC 20555-0001 with a copy to the

Regional Administrator, Region IV, and a copy to the NRC Resident Inspector at the facility that

is the subject of this Notice, within 30 days of the date of the letter transmitting this Notice of

Violation (Notice). This reply should be clearly marked as a "Reply to a Notice of Violation;EA-08-028," and should include for each violation: (1) the reason for the violation, or, if

contested, the basis for disputing the violation or severity level; (2) the corrective steps that

have been taken and the results achieved; (3) the corrective steps that will be taken to avoid

further violations and (4) the date when full compliance will be achieved. Your response may

reference or include previous docketed correspondence, if the correspondence adequately

addresses the required response. If an adequate reply is not received within the time specified

in this Notice, an order or a Demand for Information may be issued as to why the license should

Enclosure 1-2-not be modified, suspended, or revoked, or why such other action as may be proper should notbe taken. Where good cause is shown, consideration will be given to extending the response

time.

If you contest this enforcement action, you should also provide a copy of your response, with

the basis for your denial, to the Director, Office of Enforcement, United States Nuclear

Regulatory Commission, Washington, DC 20555-0001. Because your response will be made available electronically for public inspection in the NRCPublic Document Room or from the NRC's document system (ADAMS), accessible from the

NRC Web site at http://www.nrc.gov/reading-rm/adams.html, to the extent possible, it shouldnot include any personal privacy, proprietary, or safeguards information so that it can be made

available to the public without redaction. If personal privacy or proprietary information is

necessary to provide an acceptable response, then please provide a bracketed copy of your

response that identifies the information that should be protected and a redacted copy of your

response that deletes such information. If you request withholding of such material, you mustspecifically identify the portions of your response that you seek to have withheld and provide in

detail the bases for your claim of withholding (e.g., explain why the disclosure of information will

create an unwarranted invasion of personal privacy or provide the information required by

10 CFR 2.390(b) to support a request for withholding confidential commercial or financial

information). If safeguards information is necessary to provide an acceptable response, please

provide the level of protection described in 10 CFR 73.21. Dated this 29

th day of February 2008.

U.S. NUCLEAR REGULATORY COMMISSIONREGION IVDockets:50-445Licenses:NPF-87

Report:05000445/2007008

Licensee:Luminant Generation Company, LLC

Facility:Comanche Peak Steam Electric Station, Unit 1

Location:FM-56, Glen Rose, Texas

Dates:December 4, 2007, through January 24, 2008

Team Leader:C. Young, P.E., Resident Inspector, Arkansas Nuclear One

Inspectors:A. Sanchez, Resident Inspector, Comanche Peak Steam Electric StationD. Loveless, Senior Reactor AnalystBranch Chief:C. Johnson, Chief, Project Branch ADivision of Reactor ProjectsApproved By:D. Chamberlain, Director Division of Reactor Projects

Enclosure 2-2-SUMMARY OF FINDINGSIR 05000445/2007008; 12/04/07 - 01/24/08; Comanche Peak Steam Electric Station (CPSES),Unit 1; Special Inspection in response to the failure of the Train B Emergency Diesel Generator

to start on demand on November 21, 2007.The report covered a 6-day period (December 4-7, 2007) of onsite inspection, with inofficereview through January 24, 2008, by a special inspection team consisting of two resident

inspectors and one senior reactor analyst. Two findings were identified, including one Green

noncited violation, and one White violation. The significance of most findings is indicated by its

color (Green, White, Yellow, or Red) using Inspection Manual Chapter 0609, "Significance

Determination Process." Findings for which the significance determination process does not

apply may be Green or be assigned a severity level after NRC management review. The

NRC's program for overseeing the safe operation of commercial nuclear power reactors is

described in NUREG-1649, "Reactor Oversight Process," Revision 4, dated July 2000.A.NRC-Identified and Self-Revealing FindingsCornerstone: Mitigating Systems

  • White. A violation of Unit 1 Technical Specification 3.8.1, "AC Sources -Operating," was identified for the licensee's failure to satisfy Limiting Condition

for Operation 3.8.1 in that painting activities conducted on the Unit 1 Train B

EDG 1-02 resulted in paint being deposited and left in a location that caused the

EDG to become inoperable. As a result, EDG 1-02 failed to start on demand

during the subsequent monthly surveillance test. Following the discovery of the

condition, the required actions were satisfied; however, the time period between

the occurrence of the condition and the discovery of the condition exceeded the

allowed outage time. This issue was entered into the licensee's corrective action

program as SMF-2007-03253.The finding was greater than minor because it was associated with the humanperformance attribute of the mitigating systems cornerstone, and it affected the

cornerstone objective to ensure the availability, reliability, and capability of

systems that respond to initiating events to prevent undesirable consequences.

The Phase 1 Worksheets in Manual Chapter 0609, "Significance Determination

Process," were used to conclude that a Phase 2 analysis was required because

the performance deficiency affected the emergency power supply system that is

a support system for both mitigating and containment barrier systems. Based on

the results of the Phase 2 analysis, the finding was determined to have low to

moderate safety significance (White). The senior reactor analyst determined

that a more detailed Phase 3 analysis was needed to fully assess the safety

significance. Based on the results of the Phase 3 analysis, the finding was

determined to have low to moderate safety significance (White). The Phase 1,

2, and 3 Significance Determination Process analyses associated with this

finding, including assumptions and limiting core damage sequences, is included

as Attachment 3 to this report. The cause of this finding was determined to have

a crosscutting aspect in the area of human performance associated with work

Enclosure 2-3-practices in that the licensee failed to provide adequate supervisory andmanagement oversight of work activities, including contractors, such that nuclear

safety is supported H.4(c). Specifically, the actions planned and taken to

assess and control the operational impact of the painting activities on the

functionality of the emergency diesel generator were not reflective of adequate

supervisory and management oversight of the activities (Section 2.1).*Green. The inspectors identified a noncited violation of Unit 1 TechnicalSpecification 5.4.1.a, "Procedures," for an inadequate alarm response

procedure. The inspectors determined that Procedure ALM-1302A, "Diesel

Generator 1-02 Panel," Revision 5, was inadequate in that it was ambiguous and

did not cause the responders to verify that the fuel racks were free as part of the

response actions to investigate the cause of the unit failing to start.

Consequently, the licensee failed to identify that the Unit 1 Train B Emergency

Diesel Generator 1-02 fuel racks were not free to move, which led to an

extended period of inoperability and a significant delay in diagnosing the cause

of the emergency diesel generator failure to start. This issue was entered into

the licensee's corrective action program as SMF-2007-03426.The finding was determined to be more than minor because it was associatedwith the procedure quality attribute of the mitigating systems cornerstone, and it

affected the cornerstone objective to ensure the availability, reliability, and

capability of systems that respond to initiating events to prevent undesirable

consequences. Using Manual Chapter 0609, "Significance Determination

Process," Phase 1 Worksheet, the finding was determined to have very low

safety significance (Green) because it was not a design or qualification

deficiency, did not represent a loss of safety function, did not represent an actual

loss of a single train for greater than its Technical Specification allowed outage

time, did not represent a loss of a non-Technical Specification Train of

equipment for greater than 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />, and did not screen as potentially risk

significant due to a seismic, flooding, or severe weather initiating event

(Section 2.2).B.Licensee-Identified Violations

None.

Enclosure 2-4-REPORT DETAILS1.0SPECIAL INSPECTION SCOPEThe NRC conducted a special inspection at Comanche Peak Steam Electric Station tobetter understand the circumstances surrounding the failure of the Unit 1 Train B

Emergency Diesel Generator (EDG) 1-02 to start on demand during a monthly

surveillance test on November 21, 2007. Following approximately 11 hours1.273148e-4 days <br />0.00306 hours <br />1.818783e-5 weeks <br />4.1855e-6 months <br /> of

troubleshooting, EDG 1-02 was restored to an operable status. In accordance with NRC

Management Directive 8.3, it was determined that the period of inoperability of the EDG,

both prior to and during the failure to start event, had sufficient risk significance to

warrant a special inspection. The initial incremental conditional core damage probability

associated with the assumed period of EDG inoperability was estimated to be

1.76 x 10-5. The possibility that adverse generic implications were associated with theEDG failure mechanism was the deterministic criterion met to warrant a special

inspection.

The team conducted the inspection in accordance with Inspection Procedure 93812,

"Special Inspection," and the inspection charter, which is included in this report as

Attachment 2. The special inspection team reviewed procedures, corrective action

documents, operator logs, and maintenance records for the EDG system. The team

interviewed various licensee personnel regarding the events that led up to and response

actions that followed the EDG failure, as well as design and operational characteristics

of the EDG and its support systems. The team reviewed the licensee's root cause

analysis report, past failure records, extent of condition evaluation, immediate and long

term corrective actions, and industry operating experience. A list of specific documents

reviewed is provided in Attachment 1.1.1Event SummaryOn November 21, 2007, at 10:20 a.m., EDG 1-02 failed to start on demand during amonthly slow start surveillance test. Prior to this, the last successful surveillance test on

EDG 1-02 was on October 24, 2007. The licensee's response to the failure to start is

described in Section 1.2 below. Troubleshooting efforts were ultimately successful, and

EDG 1-02 was restored to an operable status at 3:08 a.m. on November 22, 2007. The

failure was determined to be the result of fuel racks being stuck in the closed positions,

and not responding to a full open governor demand, thereby preventing sufficient fuel

from reaching the engine. The failure mechanism is described in Section 1.4 below.

The cause of the fuel rack binding was ultimately determined to be a drop of paint on a

fuel rack which prevented the rack from being able to move through the fuel pump

housing. The root and contributing causes of this failure are discussed on Section 1.3

below.Prior to the failed surveillance test on November 21, 2007, painting was conducted onand around EDG 1-02 and EDG 2-02 (Unit 2 Train B EDG). The painting activities

associated with EDG 1-02 began on October 15, 2007, and continued through

November 8, 2007.

Enclosure 2-5-Included below is a timeline that includes significant elements pertaining to this event.Date/TimeEventOctober 15, 2007Painting begins on EDG 1-02 on top of engine and around

heads.October 15, 2007 -November 1, 2007Painting activities continue on a daily basis.October 24, 2007Successful monthly slow start surveillance test of EDG 1-02. No painting is done on this day.October 29, 2007 -November 1, 2007Painting occurs around the 6L fuel pump.November 1, 2007Painting in locations that could have reasonably resulted instray paint/drops on fuel rack(s) is completed.November 21, 2007 3:28 a.m.EDG 1-02 declared inoperable due to bar over in preparationfor monthly surveillance test. 4:09 a.m.Bar over completed. Successful water roll via the air startsystem was completed. EDG 1-02 declared operable. 10:17 a.m.EDG 1-02 declared inoperable for monthly surveillance test.

10:20 a.m.EDG 1-02 failed to start on demand for monthly slow startsurveillance test. Operations personnel believed the EDG did

not roll. Troubleshooting commences. 4:49 p.m.Slow start attempt of EDG 1-02 resulted in EDG rolling up to90-100 rpm and failing to start. Troubleshooting continues. 6:02 p.m.Fast start attempt of EDG 1-02 resulted in a failure to start. Fuel racks were observed not to move in response to a

governor demand. 7:39 p.m.Fuel racks were manually stroked. Rack 6L was found to bestuck. Rack 2L moved approximately half of its travel range,

then became bound. Both racks were freed and stroked until

normal free range of motion was restored. 9:25 p.m.Walkdown inspections revealed residual paint on 6L, 4L, and4R fuel racks. Residue of paint on 6L was wiped away. 9:32 p.m.Successful start and run of EDG 1-02.

November 22, 2007 3:08 a.m.EDG 1-02 was declared operable.

Enclosure 2-6-1.2Licensee Response to the Failure of the EDG to StartThe inspectors evaluated the licensee's implementation of procedures (abnormal, alarm,troubleshooting, and normal operations) and Technical Specifications, reviewed plant

management's control and decision making actions, and reviewed the troubleshooting

and investigating activities that occurred following the Unit 1 Train B EDG failure to start

during the monthly surveillance test on November 21, 2007. The inspectors reviewed

corrective action documents, procedures, Technical Specifications, and operations logs.

The team performed system walkdowns and interviewed engineering, maintenance, and

operations personnel.The inspectors determined that, in general, the licensee responded to the event properlyand in accordance with plant procedures. Nuclear equipment operators (NEO) quickly

identified that the EDG 1-02 failed to start and immediately responded to the local EDG

alarm panel. The operations field support supervisor performed an inspection to look for

any obvious problems that could have caused the EDG to fail. NEOs noted that it did

not sound like a normal start, and assumed that a possible issue associated with the

starting air system had something to do with the failure to start. The licensee had

already declared the EDG inoperable prior to the attempted start in conjunction with the

surveillance test. In response to the Unit Failure To Start alarm on the local EDG alarm panel, NEOsperformed the steps of the local alarm panel procedure, Alarm Procedure

Manual ALM-1302A, "Diesel Generator 1-02 Panel," Revision 5, which instructed the

operations personnel to investigate the cause of the failure. This included checking for

proper operation and issues associated with the fuel racks, day tank, and the starting air

system. One of the applicable steps was to "check fuel racks free." This was

accomplished in accordance with the expectations of senior operations personnel by

visually verifying that there were no apparent conditions that would obstruct the motion

of the fuel racks. No abnormalities were identified at this time.The operations staff reviewed drawings and diagrams, interviewed the NEOs, andconsulted with meter and relay representatives, system engineering department

personnel, and the mechanical services department to develop a troubleshooting plan.

Due in large part to the testimony of the NEOs that the EDG did not even roll in

response to the start attempt, the troubleshooting plan focused on the starting air

system as the suspected cause of the failed start. The plan called for meter and relay

personnel to monitor various solenoids and relays during a subsequent slow start

attempt of the EDG. This attempt resulted in the EDG rolling up to 90-100 rpm, and

again failing to start. Indications now suggested that a fuel-related problem must exist,

and focus was shifted accordingly. A third attempt was performed with the EDG in a

fast start configuration. Again, the EDG failed to start. Observers noted that the fuel

racks did not move from their closed positions in response to the mechanical governor's

attempt to drive the fuel racks to the full open position. The licensee then attempted to

exercise the fuel racks and metering rods individually and discovered that two metering

rods (2L and 6L) were partially and fully bound, respectively. Licensee personnel

physically exercised the metering rods until they were free to move, and removed

evidence of paint that was found to be on the 6L metering rod by the fuel pump housing

interface. The licensee then performed a fourth attempt to start EDG 1-02, which was

Enclosure 2-7-successful. The EDG was fully loaded, and operations personnel completed thesurveillance testing. The EDG 1-02 was subsequently declared operable on the

morning of November 22, 2007 at 3:08 a.m. The inspectors determined that the initial troubleshooting plan was too narrowly focusedon finding an EDG starting air problem (despite a successful water roll via the air start

system that occurred earlier that morning), as opposed to pursuing all likely causes of a

failed start. If the focus of the response were broader, it is likely that the stuck metering

rod would have been discovered earlier, and the duration of EDG inoperability following

the failed start would have been reduced.Subsequently during the troubleshooting efforts, the joint engineering team developed aconfirm and refute matrix to process the results from troubleshooting. Possible causes

that were analyzed over the course of troubleshooting included:*Starting air receiver discharge valves mispositioned*Manual stop button mispositioned

  • Tachometers operational
  • Mechanical governor bound
  • Fuel supply to the engine
  • Main control board handswitch
  • Electronic governor not operating
  • Fuel racks not functioning**Determined to be the cause of the failure

As described in Section 1.3 below, the Unit 2 Train B EDG was also in the process ofbeing painted. Once the cause of EDG 1-02 inoperability was determined to be stuck

metering fuel rods, the operations staff inspected the Unit 2 Train B EDG and

determined that the same issue did not exist. Operations also inspected the Units 1

and 2 Train A EDGs and determined that the stuck metering rods issue did not exist.

The Unit 1 Train B EDG was the only EDG affected.1.3Root Cause and Corrective Action Assessment.1Root Cause AnalysisThe inspectors reviewed and assessed the licensee's root cause analysis for technique,accuracy, thoroughness, and corrective actions proposed and taken. The inspectors

reviewed the scope and processes used by licensee personnel to identify the root cause

for the failure of the Unit 1 Train B EDG to start during a monthly surveillance test. The

inspectors compared information gained through inspection to the event information and

assumptions made in the root cause reports. The inspectors interviewed licensee

personnel, reviewed logs, reviewed personal statements, and observed root cause team

meetings. The inspectors evaluated the licensee's extent of condition review and

common cause evaluation.

Enclosure 2-8-The licensee captured the EDG 1-02 failure to start problem in the corrective actionprogram as SMF-2007-03253, and performed a root cause analysis in response to

determine the cause of the failure. Evaluation techniques utilized by the licensee

included an Events and Causal Factors Chart and a Barrier Analysis. The result of

these efforts identified the most probable root cause of the failure to be a drop of paint

that was deposited and adhered to the 6L fuel rack in a location that prevented the rack

(along with all other fuel racks) from moving in the open direction in response to the

governor demand associated with an EDG start signal. This failure mechanism is

further discussed in Section 1.4 below. Although there was no documented evidence of

the actual paint drop, there was paint residue observed which remained in the subject

location following the manual manipulation and freeing of the stuck fuel rack during

troubleshooting. This residue was wiped off upon discovery.Additionally, the following four contributing causes to the failure were identified in thefinal root cause analysis:*Work practices of painters and other groups who performed daily inspectionsfailed to identify paint spatter and drops that should have been cleaned off

sensitive engine components.*The tools and techniques used by painters were not completely effective inpreventing paint spatter and drips.**Because the directions in alarm response procedure ALM-1302A were notspecific, the time period following the failure until the discovery of the cause of

the problem was extended.*The fuel control shaft break away force may have increased over time due towear and aging effects. This may have added to the force required to overcome

the adhesion of the paint drop.*This issue was also identified early in the inspection process by the inspectors and isfurther discussed in Section 2.2 below.The root cause team assessed that the engineering confirm/refute evaluation performedduring troubleshooting, along with the subsequent investigative actions outlined below,

were effective in considering and ruling out all other potential causes of the failure:*Electrical and control circuitry problems were investigated and ruled out. Due tothe initial reports that the field operator did not believe that the EDG even rolled

over, the root cause team investigated other possibilities that could have caused

the EDG not to have rolled, and still brought in the alarms that were received.

One viable possibility considered was a possible fault associated with the EDGStart/Stop hand switch in the control room. The hand switch in question was a

piece of original equipment. One of the corrective actions was to replace the

hand switch when the 6L fuel pump and metering rod was replaced after the

event. The switch was bench tested, disassembled, and inspected, and it was

determined that the switch not only functioned properly without signs of

degradation, but it would not be physically possible to have the switch

Enclosure 2-9-manipulated to send a stop signal to the diesel while an operator takes theswitch to the start position. The inspectors performed a visual inspection of the

switch internals and reviewed the testing methods and results. The inspectors

concluded that the EDG start/stop control switch would not have caused the

EDG failure to start on November 21, 2007.*The starting air system was examined and proven to be functional. Theinspectors confirmed this by performing system walkdowns. A water roll check

was performed satisfactorily.*The fuel day tank was inspected to ensure proper alignment and fuel quality.

  • Inspections of the joints that connect the fuel pump control shaft levers to thefuel racks were performed, and determined that none were exhibiting mechanical

binding. The inspectors confirmed this by performing a system walkdown.*The 6L fuel pump was replaced and sent to the vendor for testing, disassembly,and inspection. No abnormalities were identified, and internal binding of the

pump was determined not to be a cause of the event.*The capability of single paint drop to counter the force applied and prevent themotion of the fuel control shafts was assessed. A spare fuel pump was

subjected to a series of field tests to determine the force required to overcome

the adhesion of a drop of paint in the location that had been identified. The

results were consistent with the hypothesis that the force applied from the

mechanical governor could have been overcome by the presence of the paint

drop becoming wedged in the minimal clearance between the fuel rack and the

pump housing. Another pull test was done to confirm that a fuel rack exposed to

various combinations of dirt and grit would not require appreciably more pull

tension to operate.Aspects of organizational and programmatic effectiveness were also evaluated by theroot cause team, and confirmed by the inspectors. These included inadequate

supervisory and management involvement with the painting activities, work practices

employed during the job, and the less than comprehensive development of the

procedures and work packages associated with the activity.The extent of the condition that was determined to be the cause of the EDG 1-02 failurewas assessed by the root cause team. All other EDGs were thoroughly inspected to

verify that the same condition did not exist, particularly with the Unit 2 Train B EDG 2-02,

which had been similarly painted in September and October. All other EDGs were

verified to be free of the subject degraded condition. Emergency Diesel Generator 2-02

successfully passed its monthly surveillance test on November 28, 2007. The

inspectors reviewed the licensee's actions and concluded that the licensee's extent of

condition evaluation was adequate.The inspectors concluded, following interviews as well as a review of personalstatements made by the maintenance personnel, that the work practices of painters and

other work groups who performed daily paint clean-up inspections to identify paint

Enclosure 2-10-spatter and drops that needed to be cleaned off of sensitive engine components was avalid contributor to the event. The inspectors also determined that neither

documentation nor feedback from the inspections to the painters or operations

management regarding the results of those inspections was performed. The

communication of those results, to the right individuals, could have identified the need to

reinforce expectations, alter paint methods or barriers, or institute a stand down that

may have led to the prevention of the event. At a minimum, communication between

organizations (maintenance, inspection, operations, and management) was not as

strong as it could have been for this work on highly risk significant, safety-related

equipment.Along with the discussion above, the inspections that were performed as part of thepostpainting activities were agreed upon between operations and maintenance. Neither

the inspections nor any other applicable postmaintenance testing was specified by the

work order for performing the painting activities. Also, there was no discussion

concerning foreign materials control exclusion (FME) controls. FME has been a

significant issue with the licensee in the recent past, but no mention of this sensitivitywas made. The inspections that were performed were not documented anywhere as

having been done nor were any of the findings stemming from the inspections. The inspectors found that the licensee assembled an effective root cause team. Theroot cause team investigated every lead that was available to determine exactly why the

Unit 1 Train B EDG failed to start on November 21, 2007. The inspectors determined

that the scope, methods, and rigor associated with the root cause analysis were

appropriate and consistent with the safety significance of the problem, and that the

evaluation was successful in determining and addressing the most probable root and

contributing causes of this issue..2Corrective Action AssessmentThe inspectors evaluated the scope, adequacy, and timeliness of the licensee'scorrective measures that were both planned and implemented in response to the cause

of the EDG 1-02 failure. The inspectors concluded that the actions planned and taken

by the licensee were appropriate to address the degraded condition, to result in the

prevention of a future similar failure, and were consistent with the safety significance of

the event. Corrective actions to be taken prior to resuming painting activities include:*Revise Procedure MSM-G0-0220 used for painting to require a shiftlymanipulation of the fuel racks in addition to a visual inspection of components to

be free of paint spatter/drops*Verify the information contained in the painting pre-job briefing book to ensure itcontains all sensitive areas on the EDG that should not be painted*Revise Procedure MSM-G0-0220 to include pictures and other informationcontained in the painters' prejob briefing book used during EDG painting*Revise Procedure MSM-G0-0220 to provide for "as you go" inspections andcleaning when painting is done around sensitive components

Enclosure 2-11-*Include this event in prejob briefings for future activities to heighten sensitivity tothe potential effects of paint spatter/drops in areas that can bind mechanical

components or block air pathways*Improve tools and techniques used by painters to minimize drops and spatter. Also research available FME barriers that could be used to shield sensitive areasAdditional planned corrective actions include:

  • Develop a preventive maintenance activity to perform a fuel control shaft breakaway force test to monitor for potential degradation in the shaft linkage or

bearings*Revise alarm response Procedure ALM-1302A to remove ambiguity regardingchecking components for freedom of movement by providing specific instruction

to include a manual manipulation of the components1.4Scope of the Failure MechanismThe inspectors, through inspection and investigation, interviews of system engineers,reviews of EDG design documentation, and assessment of the licensee's root cause

analysis, developed a scope of the mechanism that was determined to be the root

cause of the EDG 1-02 failure. The fuel pump control racks (fuel metering rods) were

prevented from moving from their normal standby (closed) positions in response to a

governor demand by the presence of a drop of paint that had adhered to the fuel rack in

a location where the rack enters the housing of the fuel pump (with very minimal

clearance) when moving in the open direction. Since all fuel racks are mechanically

linked by the common fuel control shafts and cross shaft linkages, the motion of the

entire system in the open direction (back to the extensible link from the mechanical

governor) was inhibited by one fuel rack that was stuck in the standby (closed) position.

A torsion spring on the control shaft associated with each fuel pump control shaft lever

functions to allow continued motion of the system in the closed direction if one or more

individual fuel racks become bound. However, the feature does not provide this function

for system motion in the open direction, as in the response to an EDG start signal.1.5Event PrecursorsThe root cause of the EDG failure to start was determined to be paint that was inadvertently dropped onto a fuel pump metering rod. The inspectors reviewed

corrective action documents and interviewed system engineers in order to identify any

previous related issues that may have been precursors to the Unit 1 Train B EDG failure

to start. The inspectors reviewed all available documented issues dating back to 1999

that fell into each of the following two categories: (1) Previous similar or related EDG

failures, and (2) Previous issues involving equipment failures related to painting. The

inspectors determined that there had been no previous EDG or painting related issues

that may have been precursors to this event.

Enclosure 2-12-1.6EDG Maintenance and TestingThe inspectors reviewed the licensee's EDG Maintenance and testing programs. Theinspectors reviewed maintenance and testing records as well as the licensee's plans

and schedules related to preventive maintenance and testing of the EDGs. The

inspectors also interviewed several system engineers to gain an understanding of the

licensee's approaches and programs involving EDG maintenance and testing. The

inspectors determined that the licensee's EDG routine maintenance and testing

programs are adequate and that the licensee is following the program provisions.

However, the inspectors determined that these maintenance and testing practices for

painting activities were not adequate as discussed in Section 2.0.1.7Industry Operating Experience (OE)The inspectors reviewed the industry operating experience (OE) the licensee gainedthrough their normal review, as well as that which was referenced in the licensee's root

cause evaluation. The inspectors conducted interviews of licensee personnel, reviews

of pertinent OE materials discovered independently as well as with the assistance of the

NRC's Operating Experience Section, and an evaluation of actions taken by the licensee

in response to relevant OE. The specific documents reviewed during this review is listed

in Attachment 1 of this report.The inspectors determined that the licensee had appropriately reviewed andincorporated OE associated with the circumstances of the EDG failure, and that a failure

to incorporate applicable OE into station practices was not a contributing cause to the

EDG failure. The inspectors reviewed several items of OE, inspection reports, and

licensee event reports (LERs). The inspectors reviewed the licensee's responses to the

applicable cases. The licensee did have all of the OE in their OE review system, with

the exception of LERs. The licensee reviews industry OE that comes from INPO and

not specifically the LER database. It appeared that the licensee had accounted for all

available OE at the time that could have reasonably been obtained and reviewed.All of the OE pertaining to notification events of inoperable diesels due to paintingdescribed gross painting errors that resulted in inoperable diesel generators (e.g.,

inappropriate/movable components being painted). The licensee did take those events

into consideration when developing the work plan for painting of the EDGs in the

associated rooms. The licensee held meetings well in advance of the scheduled

painting window, ensured that operations and maintenance personnel were

communicating, and developed a painter's handbook that presented precautions as well

as clear photographs of the areas and components not to paint. The preparation was

adequate for the knowledge that the plant had on site at the time. The sensitivity that

one paint drop in a specific, unintended location could render the EDG inoperable was

not considered by the licensee in their preparation and conduct of the EDG painting

activities, but this was not a subject of previous OE.One item that was not specifically incorporated into the procedures for painting the EDGwas a specific postmaintenance test to be performed to prove operability. The

licensee's procedure described and recommended any of several postmaintenance

Enclosure 2-13-options, including visual inspections and equipment functionality tests. This procedureand its weaknesses were discussed as part of the root cause evaluation in Section 1.3.The licensee sent two of its employees (a system engineer and a painting supervisor)on a benchmarking trip prior to cleaning up, painting, and relamping the EDG Rooms.

The licensee employees were aware of the potential to make the EDG inoperable by

painting activities, but did not get enough information to be as sensitive as necessary for

their painting activities. After the failed EDG start, the licensee called the plants visited

during the benchmarking trip to ask more questions, and then discovered that one plant

had knowledge that very little paint or other foreign materials on the metering rods could

render the EDG inoperable. The licensee could have possibly obtained this information

if their staff were to have asked more probing questions, given the work that was

planned at the site. The inspectors concluded that the licensee was not fully effective in

addressing operating experience associated with painting impacts on emergency diesel

generator operability.1.8Potential Generic IssuesThe inspection team evaluated the circumstances surrounding the event and assessedthe root cause of the Unit 1Train B EDG failure to start. The team interviewed

numerous licensee personnel and reviewed industry operating experience as well as

NRC generic communications with the goal of identifying any potential generic issues

that should be addressed as a result of the event.The inspection team concluded that, while painting activities occur at all plants, there areno specific generic concerns associated with this instance of procedural compliance.

The licensee has also issued an action in the corrective action program to issue an OE

report to INPO for future reference.2.0SPECIAL INSPECTION FINDINGS2.1Painting Activities Result in Inoperability of EDGIntroduction: A White self-revealing violation of Unit 1 Technical Specification (TS)3.8.1, "AC Sources - Operating," was identified for the licensee's failure to satisfy TS

LCO 3.8.1 in that painting activities conducted on the Unit 1 Train B EDG 1-02 resulted

in paint being deposited and left in a location that caused the EDG to become

inoperable. As a result, EDG 1-02 failed to start on demand during the subsequent

monthly surveillance test. Following the discovery of the condition, the TS required

actions were satisfied; however, the time period between the occurrence of the condition

and the discovery of the condition exceeded the TS allowed outage time.Description: On October 15, 2007, the licensee commenced painting activities thatoccurred on and around EDG 1-02. A successful monthly slow start surveillance test

was performed on October 24, 2007. Painting activities continued through November 1,

2007. The inspectors reviewed Work Order (WO) 4-07-175968-00, which implemented

the painting activities on and around EDG 1-02 and specified that painting was to be

performed per the requirements of Procedure MSM-G0-0220, "General Plant Painting,"

Revision 2. The inspectors noted that the WO did not contain requirements for

Enclosure 2-14-postmaintenance testing of the EDG, and that Procedure MSM-G0-0220, "General PlantPainting," Revision 2, contained the following steps:NOTE: System engineer, operations, maintenance services or otherdepartments may provide useful guidance in determining appropriate protection

of equipment and post-painting functional testing.5.1.1.2 Painting conducted on equipment should be done in such a manner asto ensure paint does not bind components required to move. Prejob briefings,visual verification of postpainting operation, equipment functional testing or other

similar activities are recommended practices that should be employed when

painting equipment.Through interviews, the inspectors determined that representatives from themaintenance services, system engineering, maintenance, and operations departments

discussed plans for verifying at the end of each day that the EDG remained operable.

The above requirement and guidance of the general plant painting procedure was not

referenced in this discussion. It was decided that a senior operations department

personnel would perform a visual inspection at the end of each day to verify that

painting had not been done so as to affect the operability of the EDG. This plan was

understood and executed, but was not documented, nor were any inspection results

documented. Prejob briefs and postpainting inspections were focused on avoiding the

painting of components that were not supposed to be painted and were appropriate and

effective in that regard. However, appropriate sensitivity to the potential functional

impact of stray drop(s) of paint in sensitive location(s) was not emphasized.On November 21, 2007, EDG 1-02 failed to start on demand during its next monthlysurveillance test. Following approximately 11 hours1.273148e-4 days <br />0.00306 hours <br />1.818783e-5 weeks <br />4.1855e-6 months <br /> of troubleshooting, EDG 1-02 was

successfully started. This issue was entered into the licensee's corrective action

program as SMF-2007-003253-00. The licensee performed a root cause analysis to

determine the cause of the failure. The most likely cause of the failure was determined

to be a paint drop that had been deposited on the 6L fuel rack that caused the rack to

become stuck. This prevented motion of all 16 fuel racks, thereby preventing the EDG

from receiving sufficient fuel to run. Corrective actions planned and taken by the

licensee are discussed in Section 1.3 of this enclosure. Analysis: The performance deficiency associated with this finding involved thelicensee's failure to ensure that the assumed operability of safety-related equipment was

not affected by the performance of scheduled maintenance activities. Specifically,

painting was conducted on and around EDG 1-02 in such a manner that paint was

deposited and remained in a location that caused the EDG to become inoperable and

fail to start on demand during a subsequent surveillance test. Postpainting verification

of equipment functionality was inadequate. Consequently, the requirements of TS LCO 3.8.1.b and the associated required TS Actions B.4 and G.1 and 2 were not met. The

finding was greater than minor because it was associated with the human performance

attribute of the mitigating systems cornerstone, and it affected the cornerstone objective

to ensure the availability, reliability, and capability of systems that respond to initiating

events to prevent undesirable consequences. The Phase 1 Worksheets in Manual

Chapter 0609, "Significance Determination Process," were used to conclude that a

Enclosure 2-15-Phase 2 analysis was required because the performance deficiency affected theemergency power supply system that is a support system for both mitigating and

containment barrier systems. Based on the results of the Phase 2 analysis, the finding

was determined to have low to moderate safety significance (White). The senior reactor

analyst determined that a more detailed Phase 3 analysis was needed to fully assess

the safety significance. Based on the results of the Phase 3 analysis, the finding was

determined to have low to moderate safety significance (White). The Phase 1, 2, and 3

significance determination process analyses associated with this finding, including

assumptions and limiting core damage sequences, is included as Attachment 3 to this

report. The cause of this finding was determined to have a crosscutting aspect in thearea of human performance associated with work practices in that the licensee failed to

provide adequate supervisory and management oversight of work activities, including

contractors, such that nuclear safety is supported H.4(c). Specifically, the actions

planned and taken to assess and control the operational impact of the painting activities

on the functionality of the EDG were not reflective of adequate supervisory and

management oversight of the activities.Enforcement: Unit 1 Technical Specification (TS) 3.8.1, "AC Sources - Operating,"requires that while the plant is in Modes 1, 2, 3, or 4, two diesel generators (DGs)

capable of supplying the onsite Class 1E power distribution subsystem(s) shall be

operable. For the condition of one DG being inoperable, the required action is to restore

the DG to an operable status within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> and within 6 days from the discovery of the

failure to meet the Limiting Condition for Operation (LCO), or be in Mode 3 within 6

hours and Mode 5 within 36 hours4.166667e-4 days <br />0.01 hours <br />5.952381e-5 weeks <br />1.3698e-5 months <br />. Contrary to the above, from November 1, 2007,

through November 21, 2007, while the plant was in Mode 1, one of the two DGs capable

of supplying the onsite Class 1E power distribution subsystem(s) was inoperable, and

action was not taken to either restore the DG to an operable status within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> or be

in Mode 3 within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and Mode 5 within 36 hours4.166667e-4 days <br />0.01 hours <br />5.952381e-5 weeks <br />1.3698e-5 months <br />. Specifically, Emergency Diesel

Generator (EDG) 1-02 was made inoperable as a result of painting activities due to paint

having been deposited and remaining on at least one fuel rack in a location that

prevented motion required to support the operation of the EDG. This condition caused

EDG 1-02 to fail to start during a surveillance test on November 21, 2007. Following the

discovery of the condition on November 21, 2007, the licensee satisfied the TS required

actions by restoring the EDG to an operable status on November 22, 2007. This

violation is the subject of the enclosed Notice of Violation: VIO 05000445/2007008-01,

"Painting Activities Result in Inoperability of Emergency Diesel Generator."2.2Inadequate Alarm Response Procedure for EDG Failure to StartIntroduction: The inspectors identified a Green noncited violation of Unit 1 TechnicalSpecification 5.4.1.a, "Procedures," for an inadequate alarm response procedure. The

inspectors determined that Procedure ALM-1302A, "Diesel Generator 1-02 Panel,"

Revision 5, was inadequate in that it was ambiguous and did not cause the responders

to verify that the fuel racks were free as part of the response actions to investigate the

cause of the unit failing to start. Consequently, the licensee failed to identify that the

Unit 1 Train B EDG 1-02 fuel racks were not free to move, which led to an extended

period of inoperability and a significant delay in diagnosing the cause of the EDG failure

to start.

Enclosure 2-16-Description: On November 21, 2007, at 10:20 a.m., EDG 1-02 failed to start during aslow start monthly surveillance test. Field operators responded to the EDG local alarm

panel. Operators referenced the alarm response Procedure ALM-1302A, "Diesel

Generator 1-02 Panel," Revision 5, and reviewed the section for Alarm Window 6.6 "Unit

Failure To Start." A limited number of system malfunctions that could have caused the

failure to start were indicated. These included fuel rack or fuel oil day tank issues,

improper starting air alignment, failed timing chain, and a Governor malfunction.Operators implemented the "Operator Actions" section of the procedure, which includedactions to determine the cause of the unit failing to start. The first action indicated was

to "Check fuel racks free and in max fuel position." The field support supervisor (senior

reactor operator) believed that the appropriate action was to perform a visual inspection

of the fuel racks. The fuel racks were not in the "max fuel" position. The inspectors

later determined that, following the majority of postulated failed start scenarios, the fuel

racks would not be expected to remain in the "max fuel" position, even if they had

initially moved. In accordance with the operators' training, the expectation for

performing this step was to visually inspect the racks. However, the inspectors

determined that without observing them being in a position other than their normal

standby (closed) position, this visual check would not be sufficient to meet the intent of

the procedure step (i.e., to ensure that the racks were not stuck in the "no fuel" position,

which was a probable failure cause that was indicated earlier in the procedure). The

operator completed this procedure step, as well as subsequent steps for starting air

alignment, EDG day tank alignment, and fuel quality with no abnormalities identified.

Field operator actions were completed at 11:05 a.m. The licensee developed a troubleshooting plan and attempted two more starts of theEDG (both unsuccessful) before determining that the fuel racks and metering rods were

not responding to the Governor demand to open. At 7:39 p.m. the licensee exercised

the fuel racks and discovered that two of the metering rods were stuck, with one fully

stuck in the closed position and one which became partially stuck following some motion

in the open direction. Operations and maintenance performed followup inspections and

successfully started the EDG at 9:32 p.m. The diesel was declared operable following

the surveillance run and post run inspections on November 22, 2007 at 3:08 a.m.The inspectors concluded that the field operators performed the actions of the alarmresponse Procedure ALM-1302A, in accordance with station procedures and training,

and operations management's expectations. The inspectors further concluded that the

inadequacy of the alarm response procedure to give clear instruction and guidance to

ensure that the EDG fuel racks were verified to be free and not binding resulted in

missing an opportunity to identify the cause of the EDG failure to start in a timely

manner. This missed diagnosis not only led to a narrowly focused troubleshooting effort

by the licensee, but also allowed the EDG to remain unnecessarily inoperable for

approximately an additional 8.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br />.Analysis: The performance deficiency associated with this finding involved thelicensee's failure to adequately establish clear procedure guidelines to implement alarm

response Procedure ALM-1302A. This resulted in the licensee's failure to identify the

binding of the Unit 1 Train B EDG fuel racks and metering rods in a timely manner

following a failure to start. The finding was determined to be more than minor because

Enclosure 2-17-it was associated with the procedure quality attribute of the mitigating systemscornerstone, and it affected the cornerstone objective to ensure the availability,

reliability, and capability of systems that respond to initiating events to prevent

undesirable consequences. Using Manual Chapter 0609, "Significance Determination

Process," Phase 1 Worksheet, the finding was determined to have very low safety

significance (Green) because it was not a design or qualification deficiency, did not

represent a loss of safety function, did not represent an actual loss of a single train for

greater than its Technical Specification allowed outage time, did not represent a loss of

a non-Technical Specification train of equipment for greater than 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />, and did not

screen as potentially risk significant due to a seismic, flooding, or severe weather

initiating event.Enforcement: Unit 1 Technical Specification 5.4.1.a requires that written procedures beestablished, implemented, and maintained covering the procedures listed in Regulatory

Guide 1.33, "Quality Assurance Program Requirements," Revision 2, Appendix A,

Section 5, for Abnormal, Off-Normal, or Alarm Conditions. Contrary to the above, on

November 21, 2007, the licensee failed to adequately establish, implement, and

maintain a procedure for an alarm condition. Specifically, alarm response

Procedure ALM-1302A, "Diesel Generator 1-02 Panel," Revision 5, was not adequately

established and maintained, which resulted in the licensee's failure to recognize that the

EDG 1-02 fuel racks and metering rods were bound and caused the failure of the EDG

to start on November 21, 2007. Consequently, the EDG remained inoperable for

approximately 8.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> longer than necessary. Because the finding was determined to

be of very low safety significance and has been entered in the licensee's correction

action program as SMF-2007-003426, this violation is being treated as an NCV

consistent with Section VI.A of the Enforcement Policy: NCV 05000445/2007008-02,

"Inadequate Alarm Response Procedure for EDG Failure to Start."4OA6 Meetings, Including ExitOn December 7, 2007, and January 10, 2008, the results of this inspection werepresented to Mr. R. Flores, Site Vice President, and Mr. T. Hope, Regulatory

Performance Manager, respectively, and other licensee personnel who acknowledged

the findings. Additionally on January 24, 2008, the final results of this inspection were

presented to Mr. F. Madden, Director, Regulatory Affairs, and other members of the

licensee staff who acknowledged the findings. On February 25, 2008, an additional exit

meeting was conducted with Mr. T. Hope and other licensee personnel who

acknowledged the findings. The inspectors confirmed that no proprietary material was

retained during the inspection. ATTACHMENT 1: Supplemental InformationATTACHMENT 2: Special Inspection Charter

ATTACHMENT 3: Significance Determination Evaluation

Attachment 1A1-1SUPPLEMENTAL INFORMATIONKEY POINTS OF CONTACTLicensee PersonnelJ. Bain, System EngineerJ. Bales, Maintenance Services

T. Bennette, Operations

M. Blevins, Senior Vice President and Chief Nuclear Officer

H. Davenport, System Engineer

D. Davis, Performance Improvement Director

R. Flores, Site Vice President

D. Goodwin, Manager, Shift Operations

T. Hope, Manager, Regulatory Performance

M. Kanavos, Plant Manager

D. Kross, Director, Operations

S. Lakdawala, Corrective Action Program Manager

F. Madden, Director, Regulatory Affairs

D. McGaughey, Manager, Shift Operations

G. Merka, Regulatory Affairs

J. Meyer, Technical Support Manager

W. Morrison, Maintenance Smart Team Manager

J. O'Quinn, Maintenance

W. Reppa, System Engineering Manager

D. Scott, Root Cause Analyst

S. Smith, Director, System Engineering

R. Sorrell, System Engineer

T. Terryah, System Engineering Manager

T. Tigner, Programs Supervisor

B. Wagner, PROMPT Team

W. Williams, Maintenance Services

M. Wisdom, System Engineering

NRCD. Allen, Senior Resident InspectorLIST OF ITEMS OPENED, CLOSED, AND DISCUSSEDOpened05000445/2007008-01VIOPainting Activities Result in Inoperability of EmergencyDiesel Generator (Section 2.1)

Attachment 1A1-2Opened and Closed05000445/2007008-02NCVInadequate Alarm Response Procedure for EDG Failure toStart (Section 2.2)

Closed None Discussed NoneLIST OF DOCUMENTS REVIEWEDProceduresNUMBERTITLEREVISIONALM-1302ADiesel Generator 1-02 Panel5

MSM-G0-0002General Plant Painting2

MSM-P0-3374Emergency Diesel Generator Monthly Run RelatedInspections

3OPT-215-1Offsite Transmission Network Operability Data Sheet14OWI-104-28Plant Equipment Operator Diesel Generator 1-02Operating Log

13OWI-104-26Control Room Diesel Generator 1-02 Operating Log11MSM-G0-0216Protective Coatings23

MSM-G0-0217Maintenance Protective Coatings-Concrete0

MSM-G0-0218Maintenance Protective Coatings-Steel0

CMP-CV-1009Application of Protective Coatings to Carbon SteelSurfaces in the Containment and Radiation Areas

Outside of Containment

0ODA-102Conduct of Operations24ODA-401Control of Annunciators, Instruments, and ProtectiveRelays 9ODA-407Guidelines on Use of Procedures12

Attachment 1A1-3OPT-214AEmergency Diesel Operability Test19SOP-609Diesel Generator System17

STA-426Industry Operating Experience Program1

STA-692Protective Coatings Program0

TSP-503Emergency Diesel Generator Reliability Program3Smart FormsSMF-2007-03253SMF-2007-03426SMF-2007-03302SMF-2007-02319WOs4-07-1765224-07-176543

5-05-501230-AA

5-07-502391-AK4-07-1759684-07-176544

4-07-176582

4-07-1754924-07-1765454-95-091357-00

4-94-078722-00Miscellaneous InformationEvaluation EVAL-2007-003253-02-00, "Root Cause Analysis""Post-Work Test Guide," Revision 12

LER 07-004-00, "Emergency Diesel Generator Failed Surveillance Test Due to Paint on FuelInjector Control Linkage"TUElectric Office Memorandum, CPSES-9125952, October 10, 1991

TUElectric Office Memorandum, CPSES-9108929, April 3, 1991

TUElectric Office Mamorandum, CPSES-91000861, January 11, 1991

Technical Evaluation TE# SE-90-1814

Cooper-Enterprise Clearinghouse R4/RV4 Preventative maintenance Program (PMP) forNuclear Standby Applications, Revision 0Operations Guideline 3, Attachment 4, "Operations Department Alarm Response Expectations,"August 2006

Attachment 1A1-4CPNPP Operations Logs, November 21-22, 2007Amercoat 220, Waterborne Acrylic Topcoat Product Datasheet, circa 1999

Information NoticesIN 93-76, "Inadequate Control of Paint and Cleaners for Safety-Related Equipment"IN 91-46, "Degradation of Emergency Diesel Generator Fuel Oil Delivery Systems"NRC Inspection DocumentsInspection Procedure 93812, "Special Inspection," 7/18/2007

"Special Inspection Charter to Evaluate the Comanche Peak Steam Electric Station DieselGenerator Failure to Start Event," November 30, 2007NRC Inspection ReportsML073060511 (RBS)ML072040388 (DC Cook IR 05000316/2007004)LIST OF ACRONYMSADAMSagency document and management system

CFRCode of Federal RegulationsCPSESComanche Peak Steam Electric Station

EDGemergency diesel generator

FMEforeign material exclusion

INPOInstitute of Nuclear Power Operations

LERlicensee event report

NRCNuclear Regulatory Commission

OEoperating experience

PARSpublicly available records system

NEOnuclear equipment operator

SDPsignificance determination process

SMFsmart form

WOwork order

A2-1Attachment 2November 30, 2007MEMORANDUM TO:Cale Young, Resident Inspector, ANOAlfred Sanchez, Resident Inspector, CPSESFROM: Arthur T. Howell III, Director, Division of Reactor Projects AVegel for/RA/

SUBJECT:SPECIAL INSPECTION CHARTER TO EVALUATE THE COMANCHEPEAK STEAM ELECTRIC STATION DIESEL GENERATOR FAILURE

TO START EVENTA Special Inspection Team is being chartered in response to the Comanche Peak SteamElectric Station emergency diesel generator (EDG) failure to start event on November 21, 2007.

You are hereby designated as the Special Inspection Team members. Mr. Cale Young,

Resident Inspector, ANO, is designated as the team leader. The assigned SRA to support the

team is David Loveless.A.BasisOn November 21, 2007, Comanche Peak Unit 1 diesel generator, DG-102, failed to startduring the monthly surveillance test. After several failed attempts to start the diesel,

licensee engineers developed a trouble shooting plan to determine the cause of the

diesel failing to start. During the trouble shooting efforts, licensee personnel identified

that two fuel rack linkage/metering rods (L2 and L6) on DG-102 appeared to be binding.

Additional inspections indicated that there were very small signs of paint on the metering

rods for the L2 and L6 fuel pumps, but not enough to prevent movement. Painting

activities in all EDG rooms were suspended until further measures were taken to prevent

reoccurrence of this issue. During the trouble shooting activities, each individual fuel

pump was manually operated by maintenance personnel and all but two moved freely.

Maintenance personnel were able to manually move, and subsequently free, the L2 and

L6 metering rods. Operations personnel then performed the surveillance test

satisfactorily. Maintenance personnel verified that the metering rods on the remaining

EDGs had free movement of all fuel rack linkage/metering rods.During further investigation into when painting had occurred inside the EDG room, it wasdiscovered that the painters continued to paint in the diesel room after the last

successful surveillance test. This brings into question whether DG-102 would have

been able to perform its intended function if called upon from October 24 to

November 21, 2007.

A2-2Attachment 2This Special Inspection Team is chartered to review the circumstances related to thefailure of DG-102 to start, and to assess the effectiveness of the licensee's actions for

resolving these problems.B.ScopeThe team is expected to address the following:

1.Develop a chronology (time-line) that includes significant event elements.

2.Evaluate the licensee's response to the failure of the EDG to start. Ensure thatplant personnel responded in accordance with plant procedures and Technical

Specifications.3.Assess the licensee's root cause determination for the EDG failure, the extent ofcondition review, the common cause evaluation and corrective measures.

Evaluate whether the timeliness of the corrective measures are consistent with

the safety significance of the problem.4.Develop a complete scope of the failure mechanism identified by the licensee'sroot cause determination.5.Identify previous EDG issues that may have been precursors to the November 1,2007, event. Evaluate the licensee's corrective measures and extent of

condition reviews for those problems.6.Evaluate the licensee's EDG maintenance and testing programs. Verify that theprograms are adequate and that the licensee is following the program provisions.7.Evaluate pertinent industry operating experience that represents potentialprecursors to the November 21, 2007, event, including the effectiveness of

licensee actions taken in response to the operating experience.8.Determine if there are any potential generic issues related to the EDG failure atComanche Peak Unit 1. Promptly communicate any potential generic issues to

Region IV management.9.Collect data as necessary to support a risk analysis. Work closely with theSenior Reactor Analyst during this inspection.

A2-3Attachment 2C.GuidanceInspection Procedure 93812, "Special Inspection," provides additional guidance to beused by the Special Inspection Team. Your duties will be as described in Inspection

Procedure 93812. The inspection should emphasize fact-finding in its review of the

circumstances surrounding the event. It is not the responsibility of the team to examine

the regulatory process. Safety concerns identified that are not directly related to the

event should be reported to the Region IV office for appropriate action.The Team will report to the site, conduct an entrance, and begin inspection no later thanDecember 4, 2007. While on site, you will provide daily status briefings to Region IV

management, who will coordinate with the Office of Nuclear Reactor Regulation, to

ensure that all other parties are kept informed. If information is discovered that shows a more significant risk was associated with this issue, immediately contact Region IVmanagement for discussion of appropriate actions. A report documenting the results of

the inspection should be issued within 30 days of the completion of the inspection.This Charter may be modified should the team develop significant new information thatwarrants review. Should you have any questions concerning this Charter, contact me at

(817) 860-8148.

A3-1Attachment 3ATTACHMENT 3SIGNIFICANCE DETERMINATION EVALUATIONComanche Peak Steam Electric StationEDG Inoperability Caused By Painting ActivitiesSignificance Determination Basis

1.Phase 1 Screening Logic, Results, and AssumptionsIn accordance with NRC Inspection Manual Chapter 0612, Appendix B, "IssueScreening," the team determined that this finding represented a licensee performance

deficiency. The team then determined that the issue was more than minor because it

was associated with the equipment performance attribute and affected the mitigating

systems cornerstone objective to ensure the availability, reliability, or function of a

system or train in a mitigating system in that Emergency Diesel Generator DG-102

would not have started upon demand.The team evaluated this finding using the "SDP Phase 1 Screening Worksheet for theInitiating Events, Mitigating Systems, and Barriers Cornerstones," provided in Manual

Chapter 0609, Appendix A, "Determining the Significance of Reactor Inspection

Findings for At-Power Situations." For this finding, a Phase 2 estimation was required

because the performance deficiency affected the emergency power supply system that

is a support system for both mitigating and containment barrier systems.

2.Phase 2 Risk EstimationIn accordance with Manual Chapter 0609, Appendix A, Attachment 1, "User Guidancefor Phase 2 and Phase 3 Significance Determination of Reactor Inspection Findings for

At-Power Situations," the senior reactor analyst evaluated the subject finding using the

Risk-Informed Inspection Notebook for Comanche Peak Steam Electric Station, Units 1

and 2, Revision 2.01a. The following assumptions were made:a.The identified performance deficiency occurred some time between the last successful test on October 24, 2007, and the test failure that occurred on

November 21, 2007.b.In accordance with Manual Chapter 0609, Appendix A, Attachment 2, "SiteSpecific Risk-Informed Inspection Notebook Usage Rules," Rule 1.1, "Exposure

Time," the analyst determined the time frame over which the finding impacted

the risk of plant operations. Because the exact time of failure was unknown, an

exposure time of t/2 from the last valid test was used. This was 1/2 of the 28 days

between tests, or 14 days. Therefore, for the phase 2 analysis, the exposure

time used to represent the time that the performance deficiency affected plant

risk was between 3 and 30 days.

A3-2Attachment 3c.Table 2 of the risk-informed notebook requires that when a performancedeficiency affects the diesel generators, the following initiating event scenarios

are applicable: LOOP and LEAC. Therefore, the analyst utilized these

worksheets from the risk-informed notebook.d.According to the risk-informed notebook, Table 1, for a 3-30 day exposure, the initiating event likelihood should be 3 for a loss of offsite power and 5 for a loss

of offsite power with loss of one vital 6.9kV bus.e.The analyst gave no operator action credit as discussed in ManualChapter 0609, Appendix A, Attachment 1, Table 4, "Remaining Mitigation

Capability Credit." The requirements to have procedures in place and to have

trained the operators in recovery under similar conditions for such credit were not

met.The dominant sequences from the notebook were documented below:TABLE C.bFailure of Emergency Diesel Generator 102 to StartPhase 2 Sequences Initiating EventSequenceMitigating FunctionsResults Loss of Offsite Power 2 LOOP-AFW-FB 8

4 LOOP-EAC-REC5 6

7 LOOP-EAC-TDAFW 6 Loss of Offsite Power with Loss of One Vital 6.9 kV Bus 1 LEAC-PORV-HPR-MKRWST 8

3 LEAC-PORV-HPI 7Using the counting rule worksheet, the result from this estimation indicated thatthe finding was of low to moderate safety significance (WHITE). However, the

analyst determined that this estimate did not include a full coverage of the risk

related to the failure identified and that a better evaluation of the internal risk

would be necessary for fully assessing the risk related to external initiators.

3.Phase 3 AnalysisIn accordance with Manual Chapter 0609, Appendix A, the analyst performed a Phase 3analysis using the Standardized Plant Analysis Risk (SPAR) Model for Comanche Peak,

Revision 3.31, dated August 2006, to simulate the failed Diesel Generator 1-02.

Additionally, the analyst conducted an assessment of the risk contributions from external

initiators using insights and/or values provided by the licensee's probabilistic risk

assessment model, in the licensee's recent submittal for extension of completion times

for diesel generators (Reference 1), and simplified fire probabilistic risk assessment.Reference 1: Letter dated November 15, 2007, Blevins to U.S. NRC, Subject: "Comanche Peak Steam Electric Station (CPSES) Docket Nos. 50-445 and 50-446, Response to Request for Additional Information Related to Licence Amendment Request (LAR)06-009

,Revision to Technical Specification (TS) 3.8.1, 'AC Sources - Operating; Extension of Completion Times for Diesel Generators."

A3-3Attachment 3

AssumptionsTo evaluate the change in risk caused by this performance deficiency, the analyst madethe following assumptions:A.The vital batteries at Comanche Peak will deplete after approximately 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> of fullpostaccident loads without an operating battery charger, assuming that operators do

not take actions to shed unnecessary loads from the vital dc buses. This is the

value used in the licensee's probabilistic risk assessment.B.The Comanche Peak SPAR model, Revision 3.31, represents an appropriate tool forevaluation of the subject finding.C.The failure of Emergency Diesel Generator 1-02 was the result of binding of the fuelrack on at least one injection pump that was caused by painting activities on and

around the diesel.D.Emergency Diesel Generator 1-02 successfully started and loaded during asurveillance performed on October 24, 2007. The diesel failed to start during a

surveillance on November 21, 2007, because the fuel rack on at least one injection

pump was bound to the extent that the entire fuel rack assembly was unable to leave

the "no fuel" position.E.Painting activities in and around the Emergency Diesel Generator 1-02 engineended on November 1, 2007. Therefore, the conditions that caused the engine to

fail had to have been in place at that time for the root cause to be valid (See

Assumption C). F.The exposure time used for evaluating this finding should be determined inaccordance with Inspection Manual Chapter 0609, Appendix A, Attachment 2, "Site

Specific Risk-Informed Inspection Notebook Usage Rules." Attachment 2 discusses

the approach to establishing the exposure time that should be used for the

significance determination process. Step 1.1 states:"The exposure time used in determining the initiating event likelihood shouldcorrespond to the time period that the condition being assessed is reasonably

known to have existed. If the inception of the condition is unknown, then an

exposure time of one half of the time period since the last successful

demonstration of the component or function (t/2) should be used."G.The appropriate exposure time (EXP), representing the time that Emergency DieselGenerator 1-02 was not functional, for use in this evaluation is 24 days.The exact time at which the residual paint that caused the binding of the fuelracks occurred is unknown. However, it is reasonable to assume that the

condition existed after the completion of painting activities on November 1, 2007.

A3-4Attachment 3Therefore, in accordance with Assumption F, Emergency Diesel Generator 1-02would not have started upon demand for the 20 days November 1 through

November 21, 2007.Additionally, the inception of the condition could have occurred any time betweenthe last successful run of the machine on October 24 and the completion of the

painting activities on November 1. Therefore, in accordance with Assumption F,

Emergency Diesel Generator 1-02 would not have started upon demand for one

half of the period from October 24 through November 1, or for an additional 4-

day period.Based on these two arguments, the analyst determined that the appropriateexposure time was the sum of the 20 days that the machine was reasonably

assumed to have failed and one half the 8 day period that could have resulted in

the failure condition.H.Given the condition of the fuel rack and the interpretation by licensed operators ofannunciator response procedures, operators would not have been able to recover

Emergency Diesel Generator 1-02 prior to postulated core damage for sequences

less than 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />. Licensed operators stated that the annunciator response

procedure would not have directed operators to manipulate the fuel racks by hand

nor does it require operators to request maintenance personnel perform such a task.I.The appropriate nonrecovery probability for sequences longer than 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> is 0.29. The analyst conducted a human reliability analysis using the SPAR-H method to

determine an appropriate nonrecovery probability. To calculate this value, the

analyst used the following assumptions:a.The analyst assumed that nominal time was available for recovery diagnosis andaction. The licensee recovered the diesel in 11 hours1.273148e-4 days <br />0.00306 hours <br />1.818783e-5 weeks <br />4.1855e-6 months <br /> 11 minutes during

nonemergency conditions. Therefore, the analyst assumed that, if required,

recovery could have been reasonably performed within the 4 hour4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> coping period

plus extended boil down times.b.The analyst assumed that emergency response personnel would be under highstress during the diagnosis and recovery. This is based primarily on the belief

that recovery personnel would know that the consequences of the task would

represent a direct threat to plant safety.c.The complexity of this task was considered to be nominal. There is someambiguity in the diagnosis. However, there are only two fundamental paths in

diagnosis. The probability that the wrong path would be initially investigated is

taken into account in the performance shaping factor for procedure quality.d.Procedures for the diagnosis were incomplete. Solely following the proceduresavailable would not have led to recovery. The basic items to consider were

available in the annunciator response procedure, although it is not clear that this

procedure would have been governing and/or utilized by the recovery personnel.

A3-5Attachment 3e.All other performance shaping factors were considered nominal for obviousreasons.J.Emergency Diesel Generator 1-01 would not have failed from the same cause asEmergency Diesel Generator 1-02 because painting activities had not been

conducted on that diesel. Therefore, the analyst left the common cause failure

probability at it's nominal value.K.The nominal nonrecovery values used by the SPAR model are for the averagenonrecovery for either of two diesel generators. Therefore, given that recovery of

Emergency Diesel Generator 1-02 would be handled separately, the analyst

adjusted the generic nonrecovery value to account for only Emergency Diesel

Generator 1-01 being the only machine available for random failure recovery.L.The nominal likelihood for a loss of offsite power was unaffected by the subjectfinding.M.Evaluating the risk contribution of this finding related to seismic events isappropriately conducted by utilizing the licensee's assessment found in Reference 1.

The conditional core damage frequency (CCDFSEISMIC) given by the licensee was 2.1 x 10-6/year.N.The licensee's fire risk model is an appropriate tool for evaluation of the subjectfinding. The CCDF for fire (CCDFFIRE) provided by the licensee in Reference 1 was7.8 x 10-6/year.The analyst independently evaluated the risk change related to internal fires. These insights were then used to challenge and evaluate the results of the

licensee's model. In all cases, the licensee's model covered the scenarios

posed by the analyst and included a larger scope of fires than was feasible for

the analyst to evaluate.O.Traditionally, the initiation of most high wind events, including those that cause aloss of offsite power, are included in the licensee's PRA and/or the SPAR model.

However, the licensee's assessment in their individual plant evaluation for externalevents did not include events that damage other pieces of equipment that may affect

risk. As stated in Reference 1, the licensee estimated the CCDF for tornados

(CCDFWIND) given the failure of a diesel generator to be 2.3 x 10

-5/year.P.The best estimate of the risk contribution from the subject finding related to internalflooding is best evaluated using a ratio from the licensee's PRA as was discussed in

Reference 1. In their evaluation, the licensee stated that the risk from internal

flooding derived from their internal events PRA was approximately 1 percent (P

FLOOD)of the total plant core damage frequency.Q.The ratio of sequences going to core damage in the first 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> to those goingthrough battery depletion is the same for internal and external initiators. This

A3-6Attachment 3assumption permits the analyst to use the ratio from the internal events SPAR inapplying recovery to external initiators.R.The differences between the SPAR and the licensee's models were inconsequential. The analyst, in reviewing the differences between the models, determined that there

were several global differences including: the lack of random failure recovery for

diesel generators in the licensee's model and the lack of convolution integrals in the

SPAR model. However, the analyst determined that these differences were not of

consequence to this evaluation because the final results were within the same color

band. Internal Initiating EventsThe senior reactor analyst used the SPAR model for CPSES to estimate the change inrisk associated with internal initiators that was caused by the finding. Average test and

maintenance of modeled equipment was assumed and a cutset truncation of 1.0E-12

was used.Consistent with guidance in the RASP Handbook, including NRC document,"Common-Cause Failure Analysis in Event Assessment (June 2007)," and Assumptions

3.C, 3.G, and 3.K, the SRA modeled the condition by adjusting the following basicevents in the SPAR model: Basic Event

Original Value Conditional Value EPS-DGN-FS-1EG1 5.0 X 10

-3 TRUE EPS-XHE-XL-NR01H 7.72 X 10

-1 8.79 X 10

-1 EPS-XHE-XL-NR02H 6.48 X 10

-1 8.05 X 10

-1 EPS-XHE-XL-NR03H 5.56 X 10

-1 7.46 X 10

-1 EPS-XHE-XL-NR04H 4.84 X 10

-1 6.95 X 10

-1The SPAR baseline core damage frequency (CDF

BASE) was 1.80 x 10

-5/year. Theevaluation case for the above change set resulted in a conditional core damage

frequency (CCDF

SPAR) of 3.78 x 10

-4/year. The dominant core damage sequences weredocumented in the table below:

A3-7Attachment 3Initiating EventSequencePreponderant FailuresFrequencyLoss of OffsitePower20-03Failure of EDG 1-01 withBattery Depletion at 4 hour4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />s2.62 x 10-4/year20-06Failure of EDG 1-01 withBattery Depletion at 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />

combined with RCS Pump

Seal Failure .6.56 x 10-5/year20-45Failure of EDG 1-01 and theTurbine-Driven AuxiliaryFeedwater Pump with Core

Damage at 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />.2.37 x 10-5/year19Failure of Motor and Turbine-Driven Auxiliary Feedwater

Pumps and/or Operator Fails

to Control.1.07 x 10-5/yearThe change in incremental conditional core damage frequency (ICCDP) was calculatedas follows:ICCDF= CCDF

SPAR - CDF base= 3.78 x 10

-4/year - 1.80 x 10

-5/year= 3.60 x 10

-4/yearGiven Assumptions 3.C through 3.G, the exposure time, representing the time that theperformance deficiency impacted the plant, for this analysis was 24 days. Therefore,

the change in core damage frequency (CDFIntNR) caused by this finding, withoutapplying any recovery to the subject condition, and related to internal initiators was

calculated as follows:CDFIntNR= ICCDF * EXP= 3.60 x 10

-4/year * (24 days ÷ 365 days/year)= 2.37 x 10

-5Given Assumption 3.H, the analyst determined that recovery credit for EmergencyDiesel Generator 1-01 would not be provided for any sequence that led to core damage

in less than 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />. Using utilities in the SAPHIRE software to slice cutsets by basic

event, the analyst determined that 7.6 percent of all internal cutsets went to core

damage in less than 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> (P

SHORT).Given Assumption 3.I, the analyst applied a nonrecovery factor (P

NR) of 0.29 to all

A3-8Attachment 3remaining cutsets. Therefore, the change in core damage frequency (CDFInternal)caused by this finding and related to internal initiators was calculated as follows:CDFInternal= [CDFIntNR * P SHORT] + [CDFIntNR * (1 - P SHORT) * P NR]= [2.37 x 10

-5 * 0.076] + [2.37 x 10

-5 * (1 - 0.076) * 0.29]= 8.15 x 10

-6External Initiating EventsIn accordance with Manual Chapter 0609, Appendix A, Attachment 1, Step 2.2.5,"Screen for the Potential Risk Contribution Due to External Initiating Events," the analyst

assessed the impact of external initiators on each of the findings, because the Phase 2

SDP result provided a Risk Significance Estimation of 7 or greater. The analyst

determined that, for the risk of an external initiator to be impacted by this performance

deficiency, the external event would have to cause a loss of offsite power that was not

accounted for in the internal events model. Using the licensee's individual plant

evaluation for external events and Reference 1, the analyst determined that the

dominant sequences affected by the subject performance deficiency were from seismic

events, high winds, fire, and internal flooding events.A.Seismic Event InitiatorsAs discussed in Assumption 3.M, the analyst utilized the licensee's valuefor the affects on the risk of seismic events associated with a failed diesel

generator. The incremental risk without recovery (ICCDPSeisNR) wascalculated as follows:ICCDPSeisNR= CCDFSEISMIC * EXP= 2.1 x 10

-6/year * (24 days ÷ 365 days/year)= 1.38 x 10

-7Given Assumption 3.H and 3.Q, the analyst applied PSHORT in quantifyingthe change in risk from seismic events.Given Assumption 3.I, the analyst applied a nonrecovery factor (P

NR) of0.29 to the remaining portion of the risk from seismic events. Therefore,

the change in core damage frequency (CDFSEISMIC) caused by thisfinding and related to seismic events was calculated as follows:CDFSEISMIC= [ICCDPSeisNR * P SHORT] + [ICCDPSeisNR * (1 - P SHORT) * P NR]= [1.38 x 10

-7 * 0.076] + [1.38 x 10

-7 * (1 - 0.076) * 0.29]= 4.75 x 10

-8

A3-9Attachment 3B.Internal Fire InitiatorsAs discussed in Assumption 3.N, the analyst utilized the licensee's valuefor the affects on the risk of internal fires associated with a failed diesel

generator. The incremental risk without recovery (ICCDPFireNR) wascalculated as follows:ICCDPFireNR= CCDFFIRE * EXP= 7.8 x 10

-6/year * (24 days ÷ 365 days/year)= 5.14 x 10

-7Given Assumption 3.H and 3.Q, the analyst applied PSHORT in quantifyingthe change in risk from internal fires.Given Assumption 3.I, the analyst applied a nonrecovery factor (P

NR) of0.29 to the remaining portion of the risk from seismic events. Therefore,

the change in core damage frequency (CDFFIRE) caused by this findingand related to seismic events was calculated as follows:CDFFIRE= [ICCDPFireNR * P SHORT] + [ICCDPFireNR * (1 - P SHORT) * P NR]= [5.14 x 10

-7 * 0.076] + [5.14 x 10

-7 * (1 - 0.076) * 0.29]= 1.77 x 10

-7C.Tornados and High Wind Initiators

As discussed in Assumption 3.O, the analyst utilized the licensee's valuefor the affects on the risk of high wind events associated with a failed

diesel generator. The incremental risk without recovery (ICCDFWindNR)was calculated as follows:ICCDPWindNR= CCDFWIND * EXP= 2.1 x 10

-5/year * (24 days ÷ 365 days/year)= 1.38 x 10

-6Given Assumption 3.H and 3.Q, the analyst applied PSHORT in quantifyingthe change in risk from high wind events.Given Assumption 3.I, the analyst applied a nonrecovery factor (P

NR) of0.29 to the remaining portion of the risk from high wind events.

Therefore, the change in core damage frequency (CDFWIND) caused bythis finding and related to seismic events was calculated as follows:

A3-10Attachment 3CDFWIND= [ICCDPWindNR * P SHORT] + [ICCDPWindNR * (1 - P SHORT) * P NR]= [1.38 x 10

-6 * 0.076] + [1.38 x 10

-6 * (1 - 0.076) * 0.29]= 4.75 x 10

-7D.Internal Flooding Initiators

As discussed in Assumption 3.O, the analyst utiliz

ed the ratio determinedby the licensee's PRA for internal flooding to other initiators. Given a

value of 1 percent, the change in core damage frequency (CDF FLOOD)caused by this finding and related to internal flooding was calculated as

follows:CDF FLOOD= CDFInternal * P FLOOD= 8.15 x 10-6 * 0.01= 8.15 x 10

-8Total Change in Core Damage FrequencyGiven that each of the initiators in this analysis were treated to ensure that the finalprobabilities were independent of each other, the analyst determined that he total

change in core damage frequency (CDF) could be calculated by taking the sum ofeach independent change. Therefore, the final Phase 3 result was calculated as

follows:CDF= CDFInternal + CDFExternal= CDFInternal + [CDFSEISMIC + CDFFIRE + CDFWIND + CDF FLOOD]= 8.15 x 10

-6 + [4.75 x 10

-8 + 1.77 x 10

-7 + 4.75 x 10

-7 + 8.15 x 10

-8]= 8.93 x 10

-6This result indicated that the finding was of low to moderate significance to the risk ofinternal initiating events.Large Early Release Frequency Contribution

In accordance with Manual Chapter 0609, Appendix A, Attachment 1, Step 2.2.6,

"Screen for the Potential Risk Contribution Due to LERF," the analyst assessed the

impact on the large early release frequency because the Phase 2 SDP result provided a

risk significance estimation of greater than 7.Using NRC Inspection Manual Chapter 0609 Appendix H, "Containment IntegritySignificance Determination Process," the senior reactor analyst determined that this was

a Type A finding (i.e., a finding that can influence the likelihood of accidents leading to

A3-11Attachment 3core damage that is also a LERF contributor). For a pressurized water reactor with alarge, dry containment, like Comanche Peak Steam Electric Station, findings related to

inter-system loss-of-coolant accidents and steam generator tube ruptures have the

potential to impact LERF.Appendix H, Table 5.1, "Phase 1 Screening - Type A Findings at Full Power," providesthat station blackout scenarios and all other transients, including loss of offsite power

initiators, screen out from further evaluation. These accident sequences are not

considered to be significant to LERF. Therefore, the estimated LERF was calculatedto be less than 6.8 x 10

-7. Because the LERF was less than the 1 x 10

-6 White/Yellowthreshold, the finding remains characterized as of low to moderate safety significance

(White).