ML21193A312
| ML21193A312 | |
| Person / Time | |
|---|---|
| Site: | Calvert Cliffs  |
| Issue date: | 06/16/2021 |
| From: | NRC Region 1 |
| To: | Exelon Generation Co |
| Shared Package | |
| ML20261H327 | List: |
| References | |
| EPID L-2021-OLL-0028 | |
| Download: ML21193A312 (7) | |
Text
ES-401 Written Examination Review Worksheet Form ES-401-9 Q
- 1.
LOK (F/H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist Partial Job-Link Minutia
- /Units Backward Q -
K/A SRO Only 1
H 2
B E
Added approximately to distractors 2
H 2
N E
Added full 3
H 2
B E
Added noun name to EOP 5 4
F 2
B E
Removed lowest temperature in stem and removed caps on Bleedoff. added correct AOP implemented added words to D explanation 5
H 3
B S
6 H
3 B
E Overlaps with question 29. Rewrote question.
7 H
3 B
S 8
H 3
B S
Question is not significantly modified. Will be called a bank question Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:
- 1.
Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.
- 2.
Enter the level of difficulty (LOD) of each question a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.
- 3.
Check the appropriate box if a psychometric flaw is identified:
Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).
Cues: The stem or distractors contain cues (e.g., clues, specific determiners, phrasing, length).
T/F: The answer choices are a collection of unrelated true/false statements.
Cred. Dist.: The distractors are not credible; single implausible distractors should be repaired, and more than one is unacceptable.
Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem).
- 4.
Check the appropriate box if a job content flaw is identified:
Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).
Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).
- /Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).
Backward: The question requires reverse logic or application compared to the job requirements.
- 5.
Check questions that are sampled for conformance with the approved K/A and those K/As that are designated SRO-only. (K/A and license-level mismatches are unacceptable.)
- 6.
Enter questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of Form ES-401, Section D.2.f.
- 7.
Based on the reviewers judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of ditorial enhancement, or (S)atisfactory?
- 8.
At a minimum, explain any U status ratings (e.g., how the Appendix B psychometric attributes are not being met).
ES-401 2
Form ES-401-9 Q
- 1.
LO K
(F/
H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist Partial Job-Link Minutia
- /Units Backward Q -
K/A SRO Only 9
H 3
N E
Changed distractor D to different EOP Changed EOP-8 to EOP updated plausibilities 10 F
2 N
E Added safety related to description of busses 11 H
3 N
S 12 F
3 N
S 13 H
2 B
E Question contains a none of the above distractor in D.
Rewrote distractors.
14 F
3 N
E D stands out being the only Unit 2 component. Changed Distractor D to 11.
15 F
2 B
S 16 H
3 B
S 17 F
3 B
S 18 F
2 B
S 19 H
3 B
E Added noun name to EOPs 20 H
2 N
S 21 H
3 B
S 22 F
3 M
S 23 H
2 N
E Reworded stem to say termination criteria 24 F
2 N
S 25 F
3 B
S
ES-401 3
Form ES-401-9 Q
- 1.
LO K
(F/
H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist Partial Job-Link Minutia
- /Units Backward Q -
K/A SRO Only 26 F
3 X
N U
A2 and C2 implausible. Made distractors more plausible:
changed alpha to neutron removed RCS leak and added in CRUD burst 27 H
3 B
S 28 H
3 N
S 29 H
2 B
S Overlaps with question 6. Question 6 was rewritten 30 H
3 B
E Edited stem to cooldown to Mode 5 31 H
3 N
E Removed stops opening from all 4 distractors 32 F
2 N
S 33 H
3 B
S 34 F
3 N
S 35 H
2 B
S 36 H
3 B
E Subscripted Pvar 37 H
3 B
E Added RPS to pretrips to all distractors 38 H
2 B
S 39 H
3 B
S 40 H
3 B
S 41 H
2 N
S
ES-401 4
Form ES-401-9 Q
- 1.
LO K
(F/
H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist Partial Job-Link Minutia
- /Units Backward Q -
K/A SRO Only 42 H
3 B
S 43 H
2 B
E Added space in 950 psia 44 F
3 B
S 45 H
3 N
S 46 H
2 N
S 47 F
2 N
S 48 F
3 B
S Rewrote in a format more like 2x2 49 H
2 B
E Capitalize Mode 50 F
3 B
S 51 H
3 M
E Change distractor C to shifts to condenser:
52 F
2 B
S 53 F
3 B
E Small editorial change to spacing in stem - removed a space in D 54 F
2 B
E Change B2 and D2 to Fire Main with/without hoses changed distractor to DI water vice condensate 55 F
3 N
S 56 H
3 N
S 57 F
3 B
S 58 F
3 B
E Made all Thot and Tcolds subscripts
ES-401 5
Form ES-401-9 Q
- 1.
LO K
(F/
H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist Partial Job-Link Minutia
- /Units Backward Q -
K/A SRO Only 59 H
3 B
S 60 F
3 N
E Added per AOP 6F to stem 61 H
2 B
S 62 H
3 N
S 63 F
3 N
E changed 20% to 8%
64 F
3 B
S 65 H
3 B
E Added 10 inches below setpoint and continuing to lower 66 F
3 B
S 67 H
3 N
S 68 F
3 B
S 69 F
2 B
S 70 F
3 B
S 71 F
3 B
S 72 F
1 B
U LOD1. Question was rewritten 73 F
1 N
U Too easy LOD1. Rewrote question to be more operationally valid and challenging to test continuous step concept.
74 F
2 B
S 75 F
3 N
S
ES-401 6
Form ES-401-9 Q
- 1.
LO K
(F/
H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist Partial Job-Link Minutia
- /Units Backward Q -
K/A SRO Only 76 H
3 B
S 77 F
3 B
S 78 H
3 N
S 79 H
2 B
E added will be directed by the Unit Supervisor, SRO Justification 80 H
3 N
S 81 H
3 N
S 82 H
3 N
S 83 F
2 N
E Changed stem wording, added SRO justification to strengthen question to be SRO level 84 F
3 B
S 85 H
3 N
U B2 and C2 implausible. Picked a new distractor from EOP-8 86 H
3 B
S 87 H
2 X
B U
K/A mismatch, this tests reportability as written. Question rewritten - Made question a 2x2 with actions per procedure to correct issue 88 H
3 N
S 89 H
3 X
N U
Not SRO level as written. added SRW/SW pressures to the stem changed distractor to selecting the action in the appropriate AOP 90 F
2 B
E Edited stem to remove whether or not statements are true 91 F
3 B
E changed to will the Unit Supervisor direct
ES-401 7
Form ES-401-9 Q
- 1.
LO K
(F/
H)
- 2.
LOD (1-5)
- 3. Psychometric Flaws
- 4. Job Content Flaws
- 5. Other
- 6.
Source (B/M/N)
- 7.
Status (U/E/S)
- 8. Explanation Stem Focus Cues T/F Cred.
Dist Partial Job-Link Minutia
- /Units Backward Q -
K/A SRO Only 92 H
3 N
S 93 H
3 B
S 94 F
2 B
S 95 F
2 N
S 96 F
3 B
S 97 H
3 B
S 98 F
2 N
E Edited stem to remove following groups 99 F
3 B
E Edited stem to ensure that this is a transition to an EOP (SRO knowledge) 100 F
3 X
B U
K/A mismatch. Edited question and selected new K/A.