ML20274A092

NRC Comments on Draft Licensee-Developed Written Exam (Folder 2)

ML20274A092
Person / Time
Site:	Calvert Cliffs
Issue date:	02/07/2020
From:	Division of Reactor Safety I
To:	Exelon Generation Co
Shared Package
ML19105A101	List: ML19157A096 ML20057E559 ML20057E560 ML20274A066 ML20274A083 ML20274A092 ML20274A094 ML20274A105 ML20274A136 ML20279A331 ML21025A073 ML21025A143 ML21025A414
References
CAC 000500, ES-401-9
Download: ML20274A092 (7)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

Source (B/M/N)

7.

Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /Units Backward Q -

K/A SRO Only 1

H 1

N U

LOD1, replaced question 2

F 3

B E

Removed resulting in a reactor trip from stem 3

H 3

B S

4 H

3 N

S 5

H 3

N S

Changed stem to have applicant diagnose Loss of Offsite power instead of being told that here is a loss of offsite power. P13001 is lost 6

H 3

N S

7 H

3 N

S 8

H 3

X B

U A2 and C2 implausible and cued answer in stem..Rewrote stem to in the CCW system instead of into the CCW system.

Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOD) of each question a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.

3.

Check the appropriate box if a psychometric flaw is identified:

Stem Focus: The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

Cues: The stem or distractors contain cues (e.g., clues, specific determiners, phrasing, length).

T/F: The answer choices are a collection of unrelated true/false statements.

Cred. Dist.: The distractors are not credible; single implausible distractors should be repaired, and more than one is unacceptable.

Partial: One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem).

4.

Check the appropriate box if a job content flaw is identified:

Job Link: The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

Minutia: The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).

1. /Units: The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

Backward: The question requires reverse logic or application compared to the job requirements.

5.

Check questions that are sampled for conformance with the approved K/A and those K/As that are designated SRO-only. (K/A and license-level mismatches are unacceptable.)

6.

Enter questions source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of Form ES-401, Section D.2.f.

7.

Based on the reviewers judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8.

At a minimum, explain any U status ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

Source (B/M/N)

7.

Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /Units Backward Q -

K/A SRO Only 9

H 2

X N

U A2 and C2 are implausible - Heater bank 2 handswitch is in OFF as stem indicated, so they cant auto energize.

Rewrote stem to describe a power supply failure to Prop Heater Bank #2 10 F

3 B

E Removed will mitigate the ATWS from stem Rewrote distractor C to change borate amount to more plausible number (2000 ppm) 11 F

2 B

S 12 F

2 B

E Changed stem to what sensor is used to differentiate 13 F

3 B

E Removed RO is verifying natural circulation Question is Low Cog, not Hi Cog 14 H

3 B

S 15 H

3 N

S 16 H

2 N

S 17 F

3 N

S 18 H

3 B

E Added to distractors B and D to have a heat removal strategy.

19 F

2 N

E Will change 4th bullet to describe one CEA (CEA #1) continues to withdraw 20 F

2 N

S 21 H

3 N

S 22 H

3 N

S 23 H

2 N

S 24 H

3 N

S

ES-401 3

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

Source (B/M/N)

7.

Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /Units Backward Q -

K/A SRO Only 25 F

3 N

S 26 H

3 N

S 27 F

2 N

S 28 H

3 B

E Shortened to 2 cases, and removed Column B 29 H

2 N

S 30 F

1 N

S 31 H

3 B

S 32 F

2 N

S 33 F

2 B

S 34 H

2 B

S 35 F

2 N

S 36 H

2 N

S 37 F

2 B

S 38 H

2 N

E Add has/ to have in part 2 of the stem.

39 H

3 B

S 40 H

3 N

E Removed second to last sentence in stem stating With the given conditions..

Separate distractors into two numbered parts.

ES-401 4

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

Source (B/M/N)

7.

Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /Units Backward Q -

K/A SRO Only 41 H

3 N

S 42 F

3 N

S 43 H

3 X

B E

Added to stem per Caution in OI-8E 44 F

3 B

S 45 H

3 N

S 46 F

2 N

S 47 F

3 B

S 48 F

2 N

S 49 H

3 B

S 50 F

3 B

S 51 H

2 B

S 52 F

2 B

S 53 H

3 B

S 54 F

2 N

S 55 H

3 B

S 56 H

3 N

E Wrote out PZR 57 H

2 B

S

ES-401 5

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

Source (B/M/N)

7.

Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /Units Backward Q -

K/A SRO Only 58 H

2 N

E Added what the failure was. Edited plausibility statement.

59 H

3 M

S 60 H

3 B

S 61 H

3 B

S 62 H

2 M

S 63 F

3 N

S 64 H

3 N

S 65 F

3 N

S 66 F

2 N

E Removed first sentence in stem 67 F

3 N

S 68 F

2 B

E Added details to stem about RO and what his responsibilities are to better match KA 69 F

2 F

S 70 F

2 B

S 71 H

3 B

S 72 F

3 B

S 73 F

2 B

S 74 F

3 N

S

ES-401 6

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

Source (B/M/N)

7.

Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /Units Backward Q -

K/A SRO Only 75 F

2 B

S 76 H

3 N

S 77 H

1 X

N E

The TS is provided as a reference, as well as which TS to use is listed in the stem. Question is borderline DLU and as a result, LOD1.

78 F

2 B

E Change 150% to 125% to make more plausible in B2 and C2 79 H

3 B

E A2 and B2, made chargers plural 80 H

3 B

S 81 H

3 N

S 82 H

3 B

S 83 H

3 N

E A1 and D1 are implausible since stem says what is the expected effect. A1 and D1 describe no effect.

Added if any to effect in stem 84 F

2 N

S 85 H

3 B

S 86 F

3 N

E Added LCO(s) so it doesnt imply plural only.

87 F

3 N

S 88 H

3 N

S 89 F

3 N

S 90 H

2 N

S

ES-401 7

Form ES-401-9 Q

1.

LOK (F/H)

2.

LOD (1-5)

3. Psychometric Flaws

4. Job Content Flaws

5. Other

6.

Source (B/M/N)

7.

Status (U/E/S)

8. Explanation Stem Focus Cues T/F Cred.

Dist Partial Job-Link Minutia

1. /Units Backward Q -

K/A SRO Only 91 H

1 N

E Part 2 of stem (was) cues that there is only one TS required component inop, therefore C and D are implausible.

Added were to part 2 of stem.

92 H

2 B

S 93 H

2 B

S 94 F

2 N

E Changed warehouse workers to equipment operator in stem 95 F

3 N

E Removed first sentence in stem Removed capitalized audits in stem 96 H

3 N

S 97 F

2 N

S 98 F

3 N

S 99 F

2 N

S 100 F

3 N

E Removed first sentence in stem