ML21239A082

NRC Comments on Draft Licensee-Developed Written Exam (Folder 2)

ML21239A082
Person / Time
Site:	Peach Bottom
Issue date:	05/18/2021
From:	Todd Fish Exelon Generation Co
To:	NRC Region 1
Shared Package
ML20259A337	List: ML21237A502 ML21237A507 ML21237A514 ML21237A524 ML21237A535 ML21239A082 ML21348A054 ML21348A065
References
EPID L-2021-OLL-002
Download: ML21239A082 (7)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q

1. LOK LOO

6. Source

7. Status

8. Explanation (F/H)

(1-5)

Stem Cues T/F Cred.

Partial Job-Minutia

1. /Units Backward Q-KIA SRO Only (B/M/N)

(U/E/S)

Focus Dist Link 1

H 2.5 N

s 2

H 3.0 N

s 3

H 3.0 N

s 4

F 2.5 N

s 5

H 2.8 N

s 6

F 3.0 N

s 7

H 3.0 N

s 8

H 2.5 N

s Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts:

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty {LOO) of each question a 1 (easy) to 5 (difficult); questions with a difficulty between 2 and 4 are acceptable.

3.

Check the appropriate box if a psychometric flaw is identified:

"Stem Focus": The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

"Cues": The stem or distractors contain cues (e.g., clues, specific determiners, phrasing, length).

"T/F": The answer choices are a collection of unrelated true/false statements.

"Cred. Dist.": The distractors are not credible; single implausible distractors should be repaired, and more than one is unacceptable.

"Partial": One or more distractors are partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by the stem).

4.

Check the appropriate box if a job content flaw is identified:

"Job Link": The question is not linked to the job requirements (i.e., the question has a valid KIA but, as written, is not operational in content).

"Minutia": The question requires the recall of knowledge that is too specific for the closed-reference test mode (i.e., it is not required to be known from memory).

#/Units": The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

"Backward": The question requires reverse logic or application compared to the job requirements.

5.

Check questions that are sampled for conformance with the approved KIA and those KIAs that are designated "SRO-only." (KIA and license-level mismatches are unacceptable.)

6.

Enter question's source: (B)ank, (M)odified, or (N)ew. Verify that (M)odified questions meet the criteria of Form ES-401, Section D.2.f.

7.

Based on the reviewer's judgment, is the question, as written, (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8.

At a minimum, explain any "U" status ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 2

Form ES-401-9

1.

2.

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q

LOK LOO

6. Source

7. Status 8, Explanation (F/H)

(1-5)

Stem Cues T/F Cred.

Partial Job-Link Minutia

1. /Units Backward Q-K/A SRO Only (B/M/N)

(U/E/S) ~

Focus Dist 9

H 3.0 N

s 10 F

2.5 N

s 11 F

2.5 N

s 12 F

3.0 N

s 13 F

2.5 B

s 14 H

3.0 B

s 15 H

3.0 N

s 16 H

3.0 B

s 17 H

2.5 B

s 18 F

3.0 N

s 19 H

2.8 N

E Update picture to clarify provided indications.

20 H

3.0 N

s RRC and SO actions are 21 F

3.0 X

N u

too similar; two possible correct choices. Revised auestion.

22 F

1.0 X

N u

LOD=1. Replaced KIA and Q.

23 F

3.0 N

s 24 H

3.0 B

s 25 H

3.0 N

s

ES-401 3

Form ES-401-9

1.

2.

6. Source

7. Status Q

LOK LOO

3. Psychometric Flaws

4. Job Content Flaws

5. Other (B/M/N)

(U/E/S)

8. Explanation

• -**n

., ~,

26 F

2.5 N

s 27 F

2.5 B

s 28 H

2.5 N

s 29 H

2.5 N

s 30 H

2.5 N

s 31 F

3.0 B

s 32 H

3.0 N

s 33 H

2.5

/

N s

34 F

3.0 N

s 35 H

3.0 N

s 36 F

3.0 N

s 37 F

2.5 N

s 38 H

3.0 N

s 39 H

3.0 N

s 40 F

2.5 N

s 41 F

2.5 N

s 42 F

3.0 N

s 43 F

3.0 N

s

ES-401 4

Form ES-401-9

1.

2.

6. Source

7. Status Q

LOK LOD

3. Psychometric Flaws

4. Job Content Flaws

5. Other (B/M/N)

(U/E/S)

8. Explanation

,r.. "

'A ~,

• 44 H

2.5 N

s 45 H

3.0 B

s 46 H

3.0 B

s 47 H

3.0 N

u Overlap with 071.

Replaced KIA and Q.

48 F

2.5 N

s 49 F

2.5 N

s 50 H

2.5 N

s 51 F

2.5 N

s 52 H

3.0 B

s 53 F

2.5 N

s 54 H

3.0 N

s 55 F

2.5 B

s

• 56 F

3.0 N

s C and D not credible for 57 F

3.0 X

N u

severity of event.

Replaced KIA and Q.

58 H

2.5 N

s 59 F

3.0 N

s 60 H

3.0 B

s LOD=5. Answer requires 61 F

5 X

N u

detailed procedure memorization bevond

ES-401 5

Form ES-401-9

1.

2.

6. Source

7. Status Q

LOK LOD

3. Psychometric Flaws

4. Job Content Flaws

5. Other (B/M/N)

(U/E/S)

8. Explanation

'A r, immediate actions.

Replaced KIA and Q 62 H

3.0 N

s 63 F

2.5 N

s 64 F

2.5 N

s 65 H

2.5 N

s 66 H

3.0 N

s 67 H

3.0 N

s 68 H

3.0 N

s 69 H

2.5 V

N s

70 F

2.5 B

s 71 H

3.0 N

s 72 H

3.0 N

s 73 H

3.0 B

s 74 H

3.0 N

s 75 H

2.5 N

s 76 H

1.0 X

X N

u LOD=1. DLU, given TS.

Replaced KIA and Q 77 H

3.0 N

s 78 F

2.5 N

s

ES-401 6

Form ES-401-9

1.

2.

6. Source

7. Status Q

LOK LOD

3. Psychometric Flaws

4. Job Content Flaws

5. Other (8/M/N)

(U/E/S)

8. Explanation 79 H

3.0 N

s 80 H

3.0 8

s 81 H

3.0 N

s 82 H

3.0 N

s 83 H

2.5 N

s 84 H

3.0 N

s 85 H

3.0 N

s 86 H

3.0 N

s 87 H

1.0 X

8 u

Not SRO-only. Low LOD.

Rewrote Q Question provides info (resin injection or fuel 88 H

3.0 X

X 8

u failure), thus no SRO diagnosis/prediction needed. Replaced KIA and Q 89 H

3.0 8

s 90 H

3.0 N

s 91 H

3.0 N

s 92 H

3.0 N

s 93 H

3.0 N

s 94 H

3.0 8

s 95 H

3.0 8

s

ES-401 7

Form ES-401-9

1.

2.

6. Source

7. Status Q

LOK LOD

3. Psychometric Flaws

4. Job Content Flaws

5. Other (B/M/N)

(U/E/S)

8. Explanation

. ~,

LOD=1. Q requires RO 96 H

1.0 X

N u

knowledge. Replaced KIA and Q.

97 F

3.0 B

s 98 H

2.5 N

s 99 H

3.0 N

s 100 H

3.0 N

s