ML041330373
| ML041330373 | |
| Person / Time | |
|---|---|
| Site: | Arkansas Nuclear  |
| Issue date: | 07/23/2003 |
| From: | Operations Branch IV |
| To: | Entergy Operations |
| References | |
| Download: ML041330373 (9) | |
Text
S:\\DRS\\OB\\EXAMS\\ANO\\UNIT1\\January 2004 Exam\\Lynns package\\draft exam comments\\Appendix_E_Admin_JPM_comments.wpd
- Page 1 -
APPENDIX E - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX JPM#
1.
Dyn (D/S) 2.
LOD (1-5)
- 3. Attributes
- 4. Job Content Errors 5.
U/E/S 6.
Explanation (See below for instructions)
IC Focus Cues Critical Steps Scope (N/B)
Over-lap Job-Link Minutia A.1.a-R S
1 X
X U
Does not seem to test anything except ability to do simple arithmetic on a few numbers. Perhaps having applicant gather data from plant displays would add relevance to this JPM. A 45 minute validation time seems quite excessive (5 minutes is more appropriate).
A.1.a-S S
3 E
Should we tell the applicant how many errors there are? Also, wouldnt it be better to have the errors hide a real safety concern, rather than create a false safety concern? Also, in cue, change...math... to...arithmetic...
A.1.b-R/S S
1 X
X U
Task seems way too easy; perhaps same task could be done on a more complicated/safety-related system that also requires recognition of TS LCO entry conditions to make it more worthwhile.
A.2-R S
2 E
Initial conditions should read: Supplement 5 of 1104.036 is completed through step 2.6"; it reads incorrectly on both page 2 and page 3 of the JPM.
A.2-S S
3 X
U Should have the applicant identify the TS requirements associated with failing the Surveillance test. Also, identifying the need to enter the TS clock on step 2.1.4 should be a critical task by itself (applicant should not pass the JPM if they fail to identify a TS requirement)
A.3-R/S S
4 X
U On page 2 of 4, the Initial Conditions read:...determine the individual stay times and another AO without exceeding the annual administrative dose limit...; it appears that the statement should read:...stay times FOR YOURSELF and another AO...; also, the key should establish a range of acceptable answers rather than exact numbers for calculations. Also, how was the DAC stay time arrived at?
A.4-R S
2 E
The word training is misspelled in question #1 in two places. An answer key should be included in the JPM. The answer to Question
- 2 is not given in the reference associated with this JPM. Try describing the audible alarm and ask the applicant to recognize the difference between a SAE alarm and other alarms (Fire, etc..).
A.4-S S
3 E
Not sure the plant announcement should be a critical task; perhaps the JPM should be ended after the classification?
Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
1.
Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically an system reconfiguration or realignment.
2.
Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3.
Check the appropriate box when an attribute weakness is identified:
The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
The JPM does not contain sufficient cues that are objective (not leading).
All critical steps (elements) have not been properly identified.
Scope of the task is either too narrow (N) or too broad (B).
Excessive overlap with other part of operating test or written examination.
4.
Check the appropriate box when a job content error is identified:
Topics not linked to job content (e.g., disguised task, not required in real job).
Task is trivial and without safety significance.
5.
Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6.
Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).
S:\\DRS\\OB\\EXAMS\\ANO\\UNIT1\\January 2004 Exam\\Lynns package\\draft exam comments\\Appendix_F_Category_B_JPM_comments.wpd
- Page 1 -
APPENDIX F - REGION IV OPERATING TEST JOB PERFORMANCE MEASURE QUALITY REVIEW MATRIX JPM#
1.
Dyn (D/S) 2.
LOD (1-5)
- 3. Attributes
- 4. Job Content Errors 5.
U/E/S 6.
Explanation (See below for instructions)
IC Focus Cues Critical Steps Scope (N/B)
Over-lap Job-Link Minutia B.1.a D
2 S
B.1.b D
3 X
E Steps 1 and 5 should not be a critical task (verify...). Steps 13 and 14 should be critical.
B.1.c D
3 X
E Top of page 3 of 6 lists too many steps as critical (typo). Shouldnt step 5 be critical? Step 7 should not be critical. Should allow operator to complete Supp-1 steps 3.2 and 3.3 and document that PCV-1000 is INOP.
B.1.d D
3 E
Step 2:...Suction form... should be...Suction from...;
B.1.e D
4 S
B.1.f D
2 X
U Too short and simple; have applicant start EDG1 and then load rather than having the EDG1 already running as an initial condition.
B.1.g D
1 X
X U
Task is way too easy; theres essentially no discriminatory value whatsoever; perhaps having the applicant place a channel in manual bypass would be more appropriate.
B.2.a S
3 E
There are two step 12's listed in the JPM - one of them should be promoted to 13. Cover sheet says this is to be performed in Simulator, but cue sheet suggests that it will be simulated in plant. It is not clear why this is considered an Alternate path JPM; why does would D-03A Output Voltage drifting to zero after opening both output and input breakers cause concern?
B.2.b S
3 X
E Cover sheet indicates JPM will be performed on Simulator, but it looks like an in-plant JPM to me. For cues, dont tell the applicant he got the results he desired, just tell him what his senses would tell him and let him judge whether thats what he expected or not. Find out what the right cues are during validation and include them in the JPM.
B.2.c S
2 X
E Cover sheet indicates JPM will be performed on Simulator, but it looks like an in-plant JPM to me. See cuing comment above.
Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating tests. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comment and explain the issue in the space provided.
1.
Determine whether the task is dynamic (D) or static (S). A dynamic task is one that involves continuous monitoring and response to varying parameters. A static task is basically an system reconfiguration or realignment.
2.
Determine level of difficulty (LOD) using established 1-5 rating scale. Levels 1 and 5 represent inappropriate (low or high) discriminatory level for the license being tested.
3.
Check the appropriate box when an attribute weakness is identified:
The initiating cue is not sufficiently clear to ensure the operator understands the task and how to begin.
The JPM does not contain sufficient cues that are objective (not leading).
All critical steps (elements) have not been properly identified.
Scope of the task is either too narrow (N) or too broad (B).
Excessive overlap with other part of operating test or written examination.
4.
Check the appropriate box when a job content error is identified:
Topics not linked to job content (e.g., disguised task, not required in real job).
Task is trivial and without safety significance.
5.
Based on the reviewers judgment, is the JPM as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
6.
Provide a brief description of problem in the explanation column. Provide conclusion on whether JPM SET criteria satisfied (i.e., number/distribution of safety functions, A.3 and A.4 integrated with parts B/C, Admin topics per section meet ES).
S:\\DRS\\OB\\EXAMS\\ANO\\UNIT1\\January 2004 Exam\\Lynns package\\draft exam com ments\\Appendix_N_Category_C_exam_comments.wpd
- Page 1 -
APPENDIX N - REGION IV OPERATING TEST SCENARIO REVIEW MATRIX Scen Set 1
ES 2
TS 3
Crit 4
IC 5
Pred 6
TL 7
L/C 8
Eff 9
U/E/S 10 Explanation (See below for instructions) 1 X
X X
E ES-301-5 not filled out adequately to determine if each applicant will have the opportunity to demonstrate competence on the required evolutions. This form should reflect the planned number of events each applicant is expected to be able to get credit for, which means the form should be filled out for each scheduled combination (ie: one form to reflect RO applicants who will be the CBOR for scenario 1 and the CBOT for scenario 3, then again for RO applicants who will be CBOT for scenario 1 and CBOR for scenario 3, etc...); the evolution-type count will likely be different for each combination, and each combination reflected in the schedule should be evaluated on this form to ensure all combinations will receive the required number of evolutions during the exam.
ES-301-4 list of actual attributes: there are 6 total malfunctions, only 1 EOP (1202.010) is used (initial RX Trip EOP does not count -
Appendix D C.2.f), and zero EOP contingencies. Start time for Events 4/5 should be cued rather than timed (T=25) so that SRO has time to demonstrate TS competency. Also, TS required actions are not listed, but should be.
2 X
X X
E See first comment about ES-301-5 above. ES-301-4 list of actual attributes: there are 7 total malfunctions, only 1 EOP (1202.007) is used (initial RX Trip EOP does not count - Appendix D C.2.f), and 1 EOP contingency (1202.008). Is there a danger that the CRS will trip the plant during event 5/6 and prevent the RO from getting his needed Reactivity manipulation during event 7/8? Critical Task for throttling CV2627 should be tied to some other plant parameter (ie: prior to exceeding a certain SG Level, or prior to exceeding a certain amount of cooldown or something like that); otherwise, he could overfill the generator and challenge the primary cooldown limits and still meet the critical task if he hit 280 gpm eventually - the question to ask is: what is critical? Is 280 gpm critical, or is some other issue really the critical issue with uncontrolled feeding of the SG? What makes it urgent? Start time for Event 5 should be cued rather than timed (T=27) so that SRO has time to demonstrate TS competency. Also, TS required actions are not listed, but should be.
3 X
X X
X E
See first comment about ES-301-5 above. There is no equipment listed OOS (ES-301-4 #1) - try putting EDG OOS since it is on the other two as well. ES-301-4 list of actual attributes: there are only 2 abnormal events (1203.012D and 1203.015), only 1 EOP (1202.010) is used (initial RX Trip EOP does not count - Appendix D C.2.f), and 1 EOP contingency (1202.002). Start time for Event 6 should be cued rather than timed (T=30) so that SRO has time to demonstrate TS competency. Also, TS required actions are not listed, but should be. As above, Critical tasks need to be tied to some measurably urgency for success (ie: close CV1009 prior to reaching a pressure of ____ or a level of _____, etc.).
Instructions for Completing Matrix This form is not contained in or required by NUREG-1021. Utilities are not required or encouraged to use it. The purpose of this form is to enhance regional consistency in reviewing operating test scenario sets. Additional information on these areas may be found in Examination Good Practices Appendix D. Check or mark any item(s) requiring comm ent and explain the issue in the space provided.
1.
ES: ES-301 checklists 4, 5, & 6 satisfied.
2.
TS: Set includes SRO TS actions for each SRO, with required actions explicitly detailed.
3.
Crit: Each manipulation or evolution has explicit success criteria documented in Form ES-D-2.
4.
IC: Out of service equipment and other initial conditions reasonably consistent between scenarios and not predictive of scenario events and actions.
5.
Pred: Scenario sequence and other factors avoid predictability issues.
6.
TL: Time line constructed, including event and process triggered conditions, such that scenario can run without routine examiner cuing.
7.
L/C: Length and complexity for each scenario in the set is reasonable for the crew mix being examined, such that all applicants have reasonably similar exposure and events are needed for evaluation purposes.
8.
Eff: Sequence of events is reasonably efficient for examination purposes, especially with respect to long delays or interactions.
9.
Based on the reviewers judgment, is the scenario set as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
10.
Provide a brief description of problem in the explanation column
ES-401 Written Examination Form ES-401-9 (R8, S1)
Review Worksheet Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 1
F 2
E B is correct 2
F 2
S 3
F 2
S 4
F 2
S 5
H 3
S 6
H 3
S 7
H 3
S 8
H 2
S 9
H 2
S 10 F
2 S
Instructions
[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]
1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
2.
Enter the level of difficulty (LOD) of each question using a 1 - 5 (easy - difficult) rating scale (questions in the 2 - 4 range are acceptable).
3.
Check the appropriate box if a psychometric flaw is identified:
The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).
The answer choices are a collection of unrelated true/false statements.
More than one distractor is not credible.
One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).
4.
Check the appropriate box if a job content error is identified:
The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).
The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
The question requires reverse logic or application compared to the job requirements.
5.
Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).
6.
Based on the reviewers judgment, is the question as written (U)nacceptable (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?
7.
At a minimum, explain any U ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 2
Form ES-401-9 (R8, S1)
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 11 H
2 S
12 H
3 S
13 F
2 X
E Distractors B and C weak 14 H
3 E
Use component name in distractor i.e. P-7A or EFW pump may cause confusion with main feed pumps 15 H
3 X
E The answer should focus on indication for interpretation. Answer provides cue as it is the only one that does not provide such information.
16 F
2 S
17 H
3 E
Needs big edit of stem. Focus on emergency boration requirement, not on elements of actions.
18 H
3 E
Break out for each distractor/answer what is (1), (2), and (3) 19 F
2 X
E Distractors A and C are potentially correct if a candidate makes an assumption.
20 H
2 S
21 H
2 S
22 F
2 U
Classic GFES question.
23 H
3 E
Question structure doesnt match format of distractors/answer.
24 F
2 X
U Piggyback is most useful on medium to small breaks. Large break usually refers to classic D-E-G break, Large break will usually bring RCS pressure down below 150 rapidly, obviating use of piggyback for low BWST level.
25 H
2 E
Format problem. See 18.
26 H
3 E
Stem edits, past tense.
27 H
2 S
28 F
3 S
29 F
3 E
Edits of distractors/answer needed. Should be sentences, more uniform in structure, and punctuated.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 30 F
2 E
Stem indicates that OTSG level would be higher, very unlikely due to nature of OTSG design charachteristics and ICS at the indicated leak rate.
31 H
3 E
Format. See 18.
32 F
2 S
33 F
2 S
34 F
2 S
35 H
3 S
36 H
3 S
37 H
3 S
38 F
2 X
U Essentially very simple T/F structure with unrelated distractors 39 H
3 S
40 H
3 S
41 H
3 S
42 H
3 S
43 F
2 U
GFES level. Need to make question operational i.e. How loss of compensation would affect an event.
44 F
2 S
45 F
2 S
46 F
2 S
47 F
3 S
48 H
3 S
49 H
3 S
50 H
2 X
U All are correct by Note. Pose question in operational terms.
51 F
2 S
52 H
3 S
53 F
2 S
54 H
3 S
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 55 H
2 S
56 F
2 S
57 F
2 S
58 H
2 X
E A and C distractors appear to be unrelated to annunciator. D is a throwaway unrelated.
59 F
2 E
Question should be operationally oriented.
60 H
3 S
61 F
2 X
E Trick question. Pose more operationally.
62 H
2 X
X U
Major problem in structure. Cue provided in stem. Distractors unrelated to event in progress 63 H
3 X
E B is poor in credibility. C is also weak.
64 F
3 U
Question has weak job link. Focus should be on operator ability to monitor and predict.
65 F
2 S
66 F
2 X
U 67 H
3 S
68 F
2 S
69 H
2 S
Make sure that distractor choices are not just random, but clearly reflect common errors.
70 F
2 X
X U
No clear job link. May be minutia, as calculations are not likely routine.
71 H
3 S
72 H
3 E
Format of question.
73 F
2 E
Format.
74 F
2 U
Conflicts with question 34 75 F
2 S
76 F
2 E
Format.
77 F
2 X
E A is a weak distractor.
78 H
5 U
Overly difficult. Unless simplification can be accomplished.
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 79 F
2 S
80 F
2 E
Use Instr. # in stem to avoid confusion.
81 F
2 E
Is D correct?
82 F
2 S
83 F
3 S
84 F
3 S
85 F
2 S
86 F
2 S
87 F
2 S
88 F
2 S
89 F
3 S
90 H
3 S
91 F
3 S
92 F
2 S
93 F
2 S
94 F
3 S
95 F
2 E
Question format fix.
96 F
2 S
97 F
2 S
98 H
3 S
99 H
2 S
100 H
3 S
101 H
3 X
S 102 H
3 X
S 103 F
2 X
S 104 F
3 X
S
Q#
1.
LOK (F/H) 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other 6.
U/E/S 7.
Explanation Stem Focus Cues T/F Cred.
Dist.
Partial Job-Link Minutia
- /
units Back-ward Q=
K/A SRO Only 105 H
2 X
X E
The distractors as posed are easy to eliminate.
106 H
3 X
S 107 H
2 X
S Does instrument present an area outside of interlocks?
108 F
2
?
S Overlap?
109 F
2 X
S 110 F
2 X
S 111 F
2 X
S This K/A probably should have a high LOD 112 F
2 X
X E
A is a weak distractor.
113 F
2 X
S 3.0.3 entry makes this a F LOD of 2 rather than H LOD of 4 114 F
3 X
U While the process you describe is correct. Very unlikely to be done in this fashion since Engineering would do the 50.59.
115 F
1 X
X U
Look for a stronger question. Weak low LOD.
116 F
2 X
S 117 F
2 X
S 118 H
3 X
S 119 F
2 X
S 120 F
2 X
S 121 H
3 X
S 122 H
2 X
S 123 H
3 X
S Question is OK. However applicants are likely to ask question during exam.
124 H
2 X
S 125 F
2 X
S