IR 05000528/1993033
| ML20057C705 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 09/03/1993 |
| From: | Morrill P NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION V) |
| To: | |
| Shared Package | |
| ML20057C703 | List: |
| References | |
| 50-528-93-33, 50-529-93-33, 50-530-93-33, NUDOCS 9309290282 | |
| Download: ML20057C705 (17) | |
Text
,
.
,
,
.
U. S. NUCLEAR REGULATORY COMMISSION l
'
. REGION V
'
i Report Nos.
50-528/93-33, 50-529/93-33, and 50-530/93-33 Licensee Arizona Public Service Company P. O. Box 53999, Station 9012 Phoenix, AZ 85072-3999 Facility:
Palo Verde Nuclear Generating Station, Units 1, 2, and 3
,
Location:
Wintersburg, Arizona Dates:
August 2 - 6, 1993 Inspectors:
G. Heyer, Chief, PWR Operations Section, Region I G. Johnston, Senior Licensing Examiner, Region V
-
Lynch, Contractor SEA Inc.
'
,
k-3-T3 Approved by:
'
,
P. Morrill, Chief Date Signed
-
Operations Section l
Insoection Summary Auaust 2 - 6. 1992 (Recort Nos. 50-528/93-33. 50-529/93-33 and 50-530/93-331 Areas Inspected:
,
The inspection was conducted to determine the root cause of recent poor operator performance during requalification examinations. The failure rate of t
crews and operators during the September 1992 and March 1993 requalification examinations was higher than expected. The inspectors interviewed operators, instructors, training management, and operations management, lhe team also observed portions of simulator training associated with emergency operating procedure (EOP) usage and reviewed selected records.
Gen (ral Conclusions and Specific Findinas The inspectors found indications of organizational ineffectiveness in
the slow resolution of E0P useability issues, unclear expectations and standards for operators, extensive resource allocation to enable
'
coordination between departments, frequent changes in direction within the operations training department, and weak corrective actions following the 1992 requalification examinations.
The inspectors concluded that the high failure rates on the 1992 and
!
1993 examinations were caused by an increase in performance standards
'
for the operators, without a corresponding increase in operator
performance.
-,
9309290282 930903 PDR ADOCK 05000S28 G
. -
.-
.
...
q
,
.
,
,
'
.
The inspectors disagreed with APS's conclusion that the major cause of
the high failure rate on job performance measures (JPMs) was lack of
~
training and concluded that higher standards for JPM performance was the
-
.
root-cause.
?
The organization did not make difficult decisions (such as the approach'
-
on emergency operating procedures [EOPs)), or implement effective corrective actions (such as the corrections made in response _to the APS October,1992 analysis of the September 1992 requalification
,
examinations). The progress that occurred required extensive i
coordination and consensus-building efforts.
The inspectors found that the distributed authority of the licensee's
-
organization hindered the corrective actions necessary to improve performance.
No significant safety matters, violations, deviations, or open items were identified during this inspection.
,
.
l s
i
,
'
i P
j
!
'
-.
i
-.
=-.
_
_
.
DETAILS 1.
Persons Contacted
,
erizona Public Services Company
- W. Conway, Executive Vice President, APS J. Levine, Vice President, Nuclear Production
- E. Simpson, Vice President, Nuclear Support
- J. Bailey, Assistant Vice President, Engineering and Projects
- E. Firth, General Manager, Nuclear Training
- R. Gouge, Director, Plant Support
- C. Seaman, Director, Quality Assurance
- R. Nunez, Manager, Operations Training
- M. Baughman, Supervisor, Licensed Operator Continuing Training
- M. Clyde, Manager, Unit 3 Operations
- J. Dennis, Manager, Operations Standards
- M. Se11and, Senior Advisor, Operations Standards
- R. Couquot, Supervisor, Quality Audits and Monitoring Others
- R. Henry, Salt River Project Site Representative.
- J. Draper, Southern California Edison Site Representative U. S. Nuclear Reaulatory Commission
- P. Morrill, Chief, Operations Section, Region V
- G. Meyer, Team Leader, Chief, Operator Licensing Section (PWR), Region I
- G. Johnston, Senior Licensing Examiner, Region V
- J. Lynch, Contractor, SEA Inc.
The inspectors interviewed and held discussions with other licensee and contractor personnel during the course of the inspection.
- Denotes those attending the exit meeting on November 6, 1992.
2.
Scope The objective of this inspectio7 was to assess the root cause for weak performance in the licensed operator requalification program at Palo Verde Units 1, 2 and 3.
The Arizona Public Service Company (APS)
conducted requalification examinations in August / September 1992 and the NRC conducted requalification examinations in March 1993. During these examinations, failure rates were higher than industry averages.
APS failed 32% of the operators and NRC failed 22% of the operators (two of nine crews) during simulator examinations. NRC examinations in October 1989, September 1991, and March 1993 demonstrated declining performance from 1989 to 1993 (in terms of pass-fail percentages).
3.
Methodoloav
,
,
.t The inspection team prepared for the inspection during the week prior to the inspection. The preparations included review and analysis of NRC and
'
APS reports starting in October 1989.
Three NRC requalification
'
examination reports, two Systematic Assessment of Licensee Performance (SALP) reports, three APS reviews of requalification performance, a NRC l
team inspection report of emergency operating procedures (E0Ps), a follow-up E0P team inspection report, the Augmented Team Inspection (AIT)
report of the March 1993 Unit 2 steam generator tube rupture event, a NRC human performance study of a Unit 3 feedwater pump trip event, and a NRC Diagnostic Evaluation Team (DET) review of Palo Verde in December 1989 i
were reviewed. The inspectors analyzed the findings from these documents using the Human Performance Investigation Process (HPIP) and Management Oversight and Risk Tree (MORT) techniques.
Based on the above reviews,
,
interview questions and topics (protocols) were developed for operations and training personnel. The protocols for discussion included training effectiveness, requalification failures, E0Ps, and morale.
.
During the onsite inspection week, the inspectors used the interview protocols to interview licensed operators, instructors, supervisors, and Also further discussions with supervisors and man;gers, managers.
observation of simulator training, and reviews of records were conducted.
4.
Specific Inspection Findinas
,
As shown in the diagram below, APS has five parties involved in the operations functional area, i.e., three unit Operations Departments, an Operations Standards Department (responsible for procedures), and an Operations Training Department. The operations functional area includes unit operations, procedures (including E0Ps), and operator training.
,
PRESIEENT P RDt W CTION I
I I
I
'
otti"!.
.iti:!.
J,.i"!.
i::'k
- i"!!'i
.I UNIT f UNjT 3 UNIT 5 PL ANT SUPPORT M41 CLEAR T R AIN8W G W#hA0tR M AN AQt R MANAQtR W AN AGtR M AN AGE R
'
orIR$ sons opEi$Ne
[EicNo eENTARe'e'
ERAIN fuS
"
i
.
-
._
_......_
.__
.
P
3 During the interviews, observations, and record reviews the inspectors determined that several areas provided indication'o'f organization
'
problems in resolving issues related to the licensed operator performance Those areas included the resolution of on requalification examinations.
E0P useability problems, establishment of consistent expectations and standards for operators, coordination and direction for operator training, and effective corrective actions.
Each of these areas appeared
,
'
to be affected by a weak organizational capability to coordinate Each of activities and reach a consensus for action to address problems.
,
these areas is discussed in more detail in paragraphs 4.a through 4.d below.
The inspectors' major observation related to operator requalification training was that the APS organizational structure (described above)
resulted in operator training requirements being decided by consensus.
Because the five managers responsible for implementation of operations functional activities have similar authority, but report to five separate Only at the general managers there was a lack of centralized authority.
Vice President, Nuclear Production, level do the five parties report to a Since the Vice President did not become involved in the single person.
day-to-day resolution of generic operational training issues, resolution of such issues occurred when one party established a position and built a consensus with the other parties.
Resolution of E0P Useability Issues
!
a.
The symptom-based emergency operating procedures were developed in response to the Three Mile Island accident and have been in use for about ten years. Most reactor facilities have addressed and resolved E0P issues such that the resulting E0Ps provide for the effective, timely actions to address reactor events.
(See Section 6 for a
,
'
discussion of these E0P issues) The Palo Verde E0Ps, although adequate, have had continued problems which caused operators to have During this difficulties when implementing the procedures.
inspection, APS management stated that an E0P revision was being validated that would be implemented in September and would address some of the useability issues. They also stated that an additional
revision was planned for March 1994 to address remaining E0P issues.
The inspectors concluded that significant E0P procedure problems were still being corrected. The inspectors observed that the Palo Verde organization structure has training, operations, and procedure authority spread over five managers. The inspectors found this structure resulted in reduced accountability and incentive to address difficult problems related to E0P useability.
,
'
b.
Excectations and Standards for Operators (1) September 1992 APS Administered Requalification Examination
The high failure rate (32%) during the 1992 APS requalification examination resulted from higher standards for operator
,
-.
.
-
-
.
.
-
,
,
-
,
.
i
performance, and from operator confusion on E0P rules of usage.
There were three levels of procedural guidance in the E0Ps; (1)
the action and guidance columns in the E0Ps, (2) guidance contained in the E0P rules of usage in administrative
,
procedures 40DP-9AP05 " Emergency Operating Procedure Technical
-
Guidelines," (referred to as AP05), (3) 40DP-9AP06 " Emergency Operations Technical Guidelines," (refstred to as AP06).
In l
some instances the guidance was inconsistent prior to the examination, and licensee expectations for procedural compliance with all the guidance was unclear.
.
Specifically, one issue was whether operators were expected to
.
!
comply with the detailed guidance contained in AP05 and AP06.
!
Ti.e Training department built a consensus with the Unit Operations managers and_0perations Standards that established that AP05 and APOS guidance would be followed. When that
,
standard was used to evaluate operators, a high failure rate
resulted. After four weeks of the six weeks of examinations, the standards regarding compliance with AP05 and AP06 guidance,
,'
were relaxed and the failure rate dropped in half.
i The inspectors concluded that management expectations and
,
i standards for operators were frequently unclear and inconsistent due to having all parties involved in setting and
,
enforcing the standards. The parties appeared to have varying
,
expectations on operator performance, and extensive consensus-i building efforts were required in setting standards.
Further, i
any effort to increase standards represented a very difficult task due to the need to get agreement from all parties involved. The inspectors noted several instances in which the operations training department had taken the lead in establishing expectations and increasing standards.
,
Specifically, prior to the September 1992 requalification t
examinations the operations training department asked j
Operations Standards and the Unit Operations supervisors for
,
specific criteria for evaluating the operators use of the E0Ps.
>
For the March 1993 requalification examinations operations training sought to set specific performance (JPM) criteria for
,
the operators during Job Performance Measures. Subsequently, a j
higher failure rate for JPMs than expected occurred.
The inspectors concluded that not consistently agreeing on expectations and communicating them to the operators prior to the September 1992 APS requ,: 'ication examinations, was the
!
i primary cause of the inability to implement the higher standards related to AP05 and AP06 guidance.
It appeared that
!
the parties had not worked together effectively. The inspectors also concluded that the higher standards were l
reasonable. The conclusion the inspectors drew from the nature of the standards was that consistency, and not unreasonable standards, appeared to be the cause of most of the operators difficulties.
i
-.
-
- - -
.. -.
.
.
--
..
_. -
- -...
-
-.
-
.
5 (2) March 1993 NRC Requalification Simulator Examinations
,
The inspectors observed that the October 1992 APS review of the
,
high failure rate during the September 1992 APS requalification
'
examinations found weaknesses in JPM performance, E0P usage,
-
secondary plant operations, and understanding of electrical power supplies. The licensee concluded that continued emphasis on E0P training, primarily on command and control, combined with clarification of E0P rules of usage would adequately address the identified problems. Although additional training was completed, similar operator weaknesses were identified during the 1993 NRC requalification examination in March 1993.
The inspectors observed that some of the poor operator performance in March 1993, was due to crews not completing some procedural steps earlier than listed in the functional recovery procedure (FRP). The ability to pull forward (i.e., before the
procedure requires) procedural steps to perform mitigating actions at more opportune times was important to good crew performance.
Pulling forward procedures was not identified by the licensee in October 1992, as an area which needed special
emphasis to the operators.
Based on the interviews of operators and instructors, the progress of operators in training on the E0Ps was slow.
Instead of progressing through transients of increasing
,
difficulty, the operators have continued to train on the E0Ps j
with routine scenarios.
During subsequent training cycles, operators have often repeated previous training to better understand the required E0P actions.
Consequently, challenging scenarios were infrequently encountered during normal
requalification training. As a result, the operators appeared to be poorly prepared for the more challenging NRC requalification test scenarios in March 1993.
The inspectors concluded that higher than expected failure rate on simulator scenarios during the 1993 NRC requalification examination resulted from hi her standards for operator g
performance (i.e., bringing forward procedural steps) and more complex scenarios than the operators had previously
,
experienced.
Numerous operators stated that the difficulty of
,
the scenarios was notably above that which they had experienced
,
in training.
l (3) March 1993 NRC Requalification Job Performance Measures i
Examinations The inspectors determined that the higher than expected failure rate on job performance measures (JPMs) during the 1993 NRC l
requalification examination resulted from higher expectations
for operator performance and not from lack of training. A June 27, 1993 APS letter to the NRC addressing the examinations
.,
.
- - -
,, -
-,,.,
,
,
.
--
,,,
.
- -_
-
-
..
.
,
.
.
.
t
,
stated, "The JPM examination failures are attributed to the
,
minimization of JPM performance during requalification training
,
two years ago in favor of more dynamic simulator time and work on soft competencies such as communications and teamwork.
Further, on-shift performance of JPMs was not substituted for the reduction of JPM performance in training." The letter i
described the corrective actions, which involved emphasis on.
training.
Based on interviews with operators the inspectors found that JPM training had been minimal for numerous years and that the JPM pass rate had been acceptable during this period. Some operators described the JPMs as easy tasks to complete that were always passed. Some operators stated that they had failed JPMs but had been unaware of that failure until after the JPM
administration.
This was an indication of a lack of comprehens,on of the standards required to successfully complete the task more than a lack of understanding the task.
<
Also, during the interviews the inspectors found no indication
that the JPM failures related to lack of understanding or knowledge that training would be expected to address.
'
A review of the failures established that for a significant number of JPMs failed, there was a misunderstanding by the examinee of the task to be accomplished.
Strict adherence by the evaluators to the letter of the task statements in the JPMs combined with the examines' misunderstanding of the task being required appeared to be the cause of the failures.
Accordingly, the inspectors concluded that the JPM _ failures resulted from higher standards applied by the evaluators than the operators were accustomed to.
,
(4) Other Findings Related to the Requalification Program Based on interviews with operators, a year ago there was frequent, strong emphasis on strict procedural compliance. The failures of operators in the 1992 APS requalification
'
examinations based on poor compliance with APO5 and AP06
'
guidance, resulted in a cookbook approach to procedures, (i.e.,
the procedures were followed without taking additional actions
'
permitted, but not required by the procedures, which would provide more effective mitigation).
Apparently, the E0Ps had been developed based on very detailed procedural steps and a philosophy of rigid adherence to them.
As a result, the procedures attempted to cover all cases and had become complex, lengthy, and cumbersome.
In interviews,
,
many operators expressed difficulty in finding the right procedure, or the right part of a very lengthy procedure, to deal with plant situations. As a consequence, a cautious operator, to avoid making an incorrect decision, would go through a procedure, one step at a time, and follow it exactly
[
i i
-.
.
__
_
_.
. _ -.
. _ - _ _ _.
-
.
_
.
>
,
as written.
For example, since the FRP allowed steps to be
brought forward, but did not require that' action, the most
!
,
conservative approach appeared to be to not carry any steps
]
forward. The inspectors observed that this strategy was adequate, but would not optimize plant recovery during real
events or demonstrate the ability to deal with challenging
>
L scenarios during examinations.
It appeared to the inspectors that the implicit expectation that procedures were to be
'
followed without taking additional allowable actions which would provide more effective mitigation may have been a factor in the weak performance during the 1993 HRC requalification
,'
examinations and the Unit 2 steam generator tube rupture event.
l The operators commented that emphasis on E0P procedural
compliance had evolved since the tube rupture such that a more
sensible approach existed. This approach combined good procedural compliance with the operators thinking about which actions the event response should involve. Also, operators were being allowed to use a process to move procedural steps forward within the E0PS if the steps were also consistent with r
the steps and actions in effect.
r i
The inspectors concluded that the licensed operators have been subjected to frequent changes in the standards and expectations for performance in the requalification program. This affected the results of the September 1992 and March 1993 requalification
-
examinations.
l Coordination and Direction for Trainina and Operations Standards c.
Decisions regarding Palo Verde operations training were made by the three Unit Operations Departments, Operetor Training, and Operations Standards. The inspectors noted that because of the committee-like
organizational structure used to make decisions, additional resources and efforts were needed to accomplish adequate
coordination.
i Specifically, each Unit Operations Department was staffed with a
licensed senior reactor operator (SRO) as a training coordinator to interface with Operations Training.
A training liaison SRO was assigned to Operations Training for the purposes of coordination
,
i between Operations Training and the Operations Departments at each
'
?
unit.
Further, supervisors and managers from the unit Operations
,
Departments and Operations Training held an interface meeting
'
!
following each seven week training cycle to review performance, coordinate expectations, and address future training needs. Tr.e
,
inspectors viewed this a high resource commitment by APS.
!
l In addition, a single point of contact approach had been established
'
to enable Operations Training to get a timely resolution from operations management on issues and concerns on procedures, policy
'
and expectations. The approach involved a designated Unit l
l
!
\\
--_
-. _ _ - _
_ _
-__
-
.
- -
-
-
-
.
.
-
...
-
_.
-
.
.
.
8 Operations Manager, who would rcpresent the three operations managers regarding timely decisions. The desig'nated single point of
.
contact rotated among the three unit managers on a yearly basis.
l This approach was successful only-to the extent that the single
point of contact made the effort to build a consensus among the t
three managers. However, lacking that consensus, his decision represented only one of the-three managers, which was less than a majority and limited the effectiveness of the decision.
The resources and time commitment to coordinate Operations and
.,
Operations Standards were extensive. Six licensed' operator t
positions were assigned to coordination in Operations Standards.
During the inspection, the six personnel assigned to fill these positions included one non-licensed operator and one licensed
,
reactor operator who worked in Operations Standards part time.
i Operation Standards also used the single point of contact for operations management and held weekly meetings with the unit l
i operations managers.
.
A two year cycle training program had been approved and was in place; however, the training plan had been frequently change? or.
substantially modified to respond to events and management directions. As a result, the instructors were not sure as to their future job assignments and schedule. They had been frequently-called on to make short term changes to the training they were to i
conduct. Training appeared to have sacrificed long range planning to focus on specific requalification issues as the issues arose, r
A related indication of the organizational burden appeared to be the J
uncertain method by which training needs were determined.
In the
!
interviews a range of answers were given_for who determined the training to be given. Operations and training personnel were
generally enthusiastic regarding their ability to provide inputs for needed training, but were uncertain as to how decision making
,
regarding training needs was accomplished. This seemed to reflect
-
the committee-like nature of decision making, (i.e., good communication but weak decision-making).
Instructors, training coordinators, and training supervisors I
perceived a lack of long range planning and plan execution in the training program. They felt that the training program had been l
,
reactive instead of proactive, and that the operations training
'
program has suffered from frequent reactive. changes.
The inspectors concluded that training coordination and direction
'
,
had suffered from frequent reactive changes and a lack of clear i
direction. However, the efforts of individuals within the five organizations involved appeared to have resulted in good l
communications.
g i
e n
.,,.
..o
,,
n.
-
r_
--
__ _
_..__
_
__.
_ _.
.
_ _.
.
l L
.
.
.
.
d.
Corrective Actions The inspectors noted that the October 1992 APS ' review'of the high j
failure rate during the September 1992 APS requalification examinations found weaknesses in JPM performance, E0P usage, secondary plant operations, and understanding of electrical power supplies. These areas remained largely uncorrected anc were
identified as weaknesses during the 1993 NRC requalification examination in March 1993. During interviews, the operations and
'
training managers stated that after the September 1992 examinations they had determined corrective actions.
They concluded that continued emphasis on E0P training, primarily on command and control, combined with clarification of E0P rules of usage, would
adequately address the identified problems.
The inspectors also determined that corrective actions following the 1993 NRC requalification examinations included better cooperation between operations and training regarding JPMs, in that JPMs were reviewed and emphasized with extra operators on shift, training instructors interfaced with on shift operators on JPMs, and some review of JPMs was occurring during requalification training.
The inspectors concluded that effective corrective actions from the September 1992 requalification examinations were not effectively
<
i addressed due to the focus of the organization on training on the E0Ps. This focus was apparently caused by a lack of consensus on
what other corrective actions would be most effective.
t i
5.
Root Cause Conclusion
'
c The inspectors concluded that the high failure rates on the September 1992 and March 1993 examinations were caused by higher standards for the.
'
operators combined with a lack of corresponding improvement in operator performance.
In September 1992, the higher standards increased performance expectations for the operators, by imposing close adherence l
to E0P technical guidelines that had not been required previously.
In
.
!
March 1993, the operators were subjected to increased simulator scenario difficulty, for which they were not prepared during training.
6.
Emeraency Operatina Procedures The inspectors determined that the E0Ps presented significant useability
-
concerns for the operators. The inspectors made this determination based on interviews with operators and instructors, and watching operators use the E0Ps during training on the simulator. The inspectors noted that the i
i complex nature of the Palo Verde E0Ps was such that they had worked well only under the best conditions. Good E0P performance occurred when a
,
crew had been together for a significant length of time, had participated in extensive training, and experienced straight-forward scenarios during
!
simulator training and evaluation.
,
!
<
Y
---
- - - - - - -
--
,w
,
s----
,,,,,,.
,-
,
. -,,,.,
~
!
-
i
.
'
i Problems Caused by Poor E0P Useability a.
,
During interviews, operators, instructors, and' managers almost uniformly stated their belief that the NRC requalification failures had occurred due to shuffled crews, i.e., operators that had not worked together and had trained together only for a short time.
,
(Shuffling the members of crews had been done to accommodate the
schedule required for examining operators to meet their license renewal requirement.) The inspectors found that shuffling adversely i
affected the performance of the crews by disrupting the extensive
'
teamwork necessary to use Palo Verde E0Ps.
However, the inspectors concluded that if the E0Ps could have been implemented clearly and
uniformly, the crew shuffling would not have had the drastic impact on operator performance that occurred. There appeared to be little recognition on the part of APS personnel that E0P useability also appeared to be part of the operators' performance deficiencies.
.
The inspectors observed that the complexity of the E0Ps required
.
'
extensive teamwork, which is best achieved through extensive training as a crew. The ability to pull forward procedural steps
(i.e., before the procedure requires) to perform mitigating actions was crucial to good crew performance.
Procedural steps often needed to be accomplished earlier than normally encountered within the
'
procedure to provide timely event mitigation actions. Otherwise the crew would have a more complicated situation when the steps that were not pulled forward were encountered.
Therefore, teamwork in
,
i deciding alternate approaches, and communication among the crew members regarding such alternatives, appeared to be essential to the
<
pulling forward process and the eventual success of a crew.
.
The licensee recently determined that high intensity training as a i
crew, involving two weeks of simulator time and extensive classroom
!
discussions, was the solution to improve the operators' E0P
!
performance. However, the inspectors viewed the high intensity training as an indication of the poor useability of the E0Ps, in
'
that extraordinary training appeared to be necessary to overcome the complexity of the E0Ps. This training approach was developed by APS based on the success of the remediation training given to the two failed crews following the March 1993 NRC requalification examinations.
In May 1993 the NRC examiners noted dramatically improved crew performance during the retake examinations. During interviews, operators from these crews were enthusiastic in their
.
!
support for the three week remediation training effort. The remediation training was similar to the high intensity training.
The licensee has decided to implement the high intensity training l
i for all of the crews during the two Fall cycles of requalification training.
The inspectors found that some of the operator delays during the Unit 2 steam generator tube rupture event were attributable to E0P useability issues. The AIT inspection report noted that while the l
crew appeared to recognize the tube rupture and took timely
-
-
-
,-
.
'
.
mitigating actions within the appropriate abnormal operating procedure (A0P), the isolation strategies within the E0Ps were
,
prolonged.
'
The interviews of operators and instructors showed that progress of the operators in training on the E0Ps had been slow.
Specifically, instead of progressing through transients of increasing difficulty, the operators continued to struggle with the E0Ps on routine scenarios and have had to repeat some aspects of previous training to better understand the E0P actions. However, the operators and their managers almost uniformly commended the quality.of instruction and the professionalism of the instructors. The inspectors found the slow progress of operators to be ar.other indication of a problem caused by E0P useability issues.
,
During the interviews the operators uniformly complained about the numerous revisions to the E0Ps and the two additional planned i
revisions. Despite an effort during requalification training which presented a synopsis of the upcoming September 1993 E0P changes to
-
encourage acceptance by the operators, the inspectors found little acceptance or enthusiasm by the operators.
The inspectors found that a factor limiting APS's ability to address the relatively large amount of training needed to adequately use the complex Palo Verde E0Ps was the availability of simulator time. A consequence of the large staff of operators for the three units and only one simulator has been that the time available for weekly
training session on the simulator is about ten hours for each crew, I
slightly less than the industry norm of twelve to sixteen hours for each weekly training session. APS managers indicated that a second simulator had been procured and was expected to be operational by the end of 1993.
i b.
Specific E0P Useability Issues While the inspectors did not review the E0Ps in detail, some aspects regarding useability were apparent to the inspectors based on interviews, observation of simulator scenarios, and reviews of previous NRC inspections. These useability issues are addressed as follows.
(1) The safety function status checks (initial phase) prior to transitioning to a recovery procedure appeared to take too
,
long. Typically the initial phase involved approximately 8
)
minutes during which the control room supervisor (CRS), an SRO, in parallel with the two board reactor operators (R0s),
independently checked the status of safety functions and
,
analyzed the symptoms of the event. The inspectors noted the
'
CRS's independent verification of status was at the expense of a delay in event analysis and mitigating actions. This delay was a poor trade-off for initiating early actions to mitigate an event.
For example, the inspectors noted an adverse impact
.
......
.
- _
_
.. _ -
. _
s
.
~
'
12-
'
of this delay during a loss of control air scenario. The control room staff was initially progressi,ng in an abnormal
>
operating procedure (A0P) addressing the mitigating actions, when a reactor trip occurred. The three operators were then
<
unable to continue with these actions for approximately 10 minutes while the E0P safety function status checks and I
diagnosis were completed.
Little communication occurred during the initial phase because communicating disrupted the others while they were completing their status checks.
(2) The E0Ps were very long and complex. The inspectors noted that the E0Ps took approximately two feet of shelf space, compared to a third of that space for Westinghouse E0Ps or ten flow-charts for General Electric E0Ps.
Further, knowledge of extensive rules of usage in APOS and AP06 were required in
,
-
addition to the E0Ps. The burden of this material appeared to slow the operators' understanding and extend the training. The length also appeared to reflect the complexity and cumbersome nature of the E0Ps.
(3) The format of the procedures did not support ease of use. The E0Ps had a two column format, a column of action steps and a column of detailed guidance for each action step.
However,
-
with this format an operator had to read both columns to get the information to execute a given action step. Also, this approach seemed to have more conditional steps, (i.e., if...,
then....) which appeared to demand greater concentration for the operators to fully comprehend.
(4) Transitioning within the E0Ps appeared to be complicated, and entry to the cumbersome functional recovery procedure (FRP)
predominated when any ambiguity existed.
Some instructors stated that the operators experienced considerable difficulty in progressing through the Diagnostic Logic Tree structure and meeting the entry conditions for the proper recovery procedure.
They stated that this resulted in a group of operating crews transitioning to different procedures on a similar scenario and that this confusion tended to force entry into the FRP.
Further, instructors stated that the operators tended to get bogged down within the FRP.
The result of being bogged down was that the operators could only focus on the actions within the FRP to the exclusion of other actions.
The inspectors concluded that the E0Ps impeded good performance by the licensed operators. The efforts by the licensee to address the useability of the E0Ps to date had focused principally on the training of the operators in the use of E0Ps rather than critically assessing the
E0Ps useability.
_
.
7.
Evaluation of APS Corrective Actions APS management stated their plan was to implement a revision to their E0Ps in September 1993, which would include the CE Owners Group Emergency Operating Procedures Guidelines methodology in their 41EP-lE001
" Emergency Operations" procedure.
Specifically, the revision was intended to streamline the initial phase by (1) having the CRS monitor the R0s actions and not independently verify status and (2) to convert the Diagnostic Logic Tree to a flowchart. This revision was to be covered with all of the crews during '.he planned two week high intensity
' raining (described in Section 6.a).
Also, APS management stated their p,:ns to implement another revision to their E0Ps in March 1994 to adress revisions to the FRP that would more closely conform to the CE Owners' Group guidelines.
The inspectors did not review the proposed revisions or evaluate the ability of these revisions to correct the identified E0P useability issues.
The inspectors made the following observations regarding tne corrective action process:
1.
Decision-making personnel in operations and operations standards appeared to have had limited exposure to other CE facilities performing their own E0Ps.
Based on the extended history of E0P concerns by NRC at Palo Verde, this raised question of the extent to which E0P alternatives had been seriously considered by the licensee. The inspectors noted that numerous instructors had extensive E0P experience at other facilities, some being CE designs, but this experience had not been factored into major scope or content E0P decisions. Also, recent licensee efforts to review other facilities' E0Ps appeared to have been sponsored by Training.
2.
The inspectors questioned whether the planned high intensity training was appropriate given the planned E0P revision in March 1994.
3.
The inspectors noted that it was unclear who had responsibility for the format and philosophy for the revised E0Ps.
The inspectors concluded that the licensee, despite the expenditure of considerable resources, had yet to implement corrective actions that adequately addressed identified E0P useability and training issues.
8.
Other Observations During the inspection the inspectors made the following observations that were not directly related to the above findings and conclusions:
A systems approach to training appeared to be in place and
functioning satisfactorily.
e
'
.
~
- Two biennial manipulations required by APS 15-DP-0TR52, Licensed
Operator Continuing Training - Training Program Description, were not included in the two-year training plan in 'effect. They were:
-BOS: Loss of turbine cooling water or loss of cooling water to an l
individual component.
I-808: Mispositioned CEA or CEAs (CEA drop)
APS training management stated that a review of the two-year-
,
training plan would be conducted to ensure that the requirements of 15DP-0TR52 were met prior to completion of the two year cycle.
!
9.
Palo Verde Unit 2 Restart E0P Short Term Corrective Actions The inspector reviewed the E0Ps to determine if appropriate short term corrective actions had been completed.
The following items were verified:
A note was added to Radiation Monitor System flowchart for the
'
operator to consider past and present radiation monitor conditions.
A step was added to the Radiation Monitor System flowchart for the
-
operators to consider a high alarm or abnormal rising trend on the condenser vacuum exhaust radiation monitor.
Additional checks were added to procedure 42EP-R003 " Steam Generator
Tube Rupture" and the FRP for identifying a SG with a tube rupture based on differential feedwater flow between SGs to maintain level.
The checks for a steam generator tube rupture (SGTR) within the FRP
were changed to be continuously applicable.
10. Exit Meetina The inspectors met with the APS representatives listed in Section 1 on
'
August 6, 1993 at the corporate offices in Phoenix.
In addition, a
preliminary exit meeting was held with Mr. J. Levine, Vice President,
.
Nuclear Production, or August 5,1993 due to his unavailability on the planned exit meeting date.
During these.neetings, the team leader
,
summarized the scope of the inspection activities and reviewed the inspection findings and conclusions as described in this report.
The inspection team leader emphasized that there had been five parties i
involved in resolving issues involving E0Ps and operator training on the E0Ps. This appeared to be a burden to resolving problems, as it required a consensus to be built before action could take place. The inspectors
,
concluded that marginal requalification performance was not attributable
,
only to Training but all five departments were responsible. The l
l
!
...
-
.
..-
.
,
.
.
.
__.
.-.
. _ _.
.. - -
-.
!
.
i
,
'
'
organizational structure had not provided the incentive to make difficult l
'
decisions such as the approach on E0Ps, and to implement effective corrective actions. The results that had been achie'ved had occurred under the burdens of extensive coordination and consensus building
~
efforts. APS management responded that they would review the
.
implications of the above conclusions on their plans ~ for E0Ps and training.
The f acility licensee did not identify as proprietary any of-the materials provided to or reviewed by the inspectors during the
'
inspection.
i t
!
i
i
!
l i
f L
.
k
.
!
!
!
.
'l
\\
!
>
. - ~ -
-
,
-. -
-.
.
. -
. -