ML080700355

2008-02 - Draft Written Exam Comments

ML080700355
Person / Time
Site:	Arkansas Nuclear
Issue date:	02/29/2008
From:	Garchow S Operations Branch IV
To:	Entergy Operations
References
50-368/08-301
Download: ML080700355 (8)

Text

Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1.

Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2.

Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3.

Check the appropriate box if a psychometric flaw is identified:

The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

The answer choices are a collection of unrelated true/false statements.

The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4.

Check the appropriate box if a job content error is identified:

The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

The question requires reverse logic or application compared to the job requirements.

5.

Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6.

Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7.

Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8.

At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

ES-401 Written Examination Review Worksheet Form ES-401-9

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 1

F 3

U B

U S

B, CR, 41.5, NRC 2002 - Other distractors could be considered correct if SG pressure is not specified and the condensate pumps are running. Also, D could be argued to be correct in that 95% SG level would provide and adequate heat sink.

The S/G pressure was be added.

2 H

F 4

U B

U S

D, CR, 41.10, NRC 2003 - Suggest changing B to lower PZR level for added credibility. KA mismatch in that the question does not test on a major component or control.

This question was rewritten.

3 H

3 U

B U

S B, CR, 41.10, NRC 1998 - Also, should not ask a question for the primary reason, this is like best answer. D could also be argued to be correct.

Following a discussion with the facility, this question is ok as written. The word primary will be deleted from the stem.

4 F

2 U

B U

S A, CR, 41.3, NRC 1998 - KA mismatch. The answer to this question deals with knowledge regarding medium or small break LOCAs, not LBLOCA. At least, the knowledge does not deal with pumps DURING a LBLOCA.

Following a discussion with the facility, this question is acceptable as is.

5 F

2 U

B U

S B, CR, 41.10, NRC 1998/2005 - Notes need to be corrected: D is listed as B. This is NOT a modified question because one condition in the stem was not changed. That makes it a bank question from the last two NRC exams. This is a KA mismatch

ES-401 2

Form ES-401-9

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation because the question does not test on the reason for the response but tests on the expected response itself.

This question was rewritten.

6 H

2 B

S B, CR, 41.4 7

H 3

N S

A, CR, 41.5 8

H 4

U N

U S

C, CR, 41.4 - This is a KA mismatch because the question does not test on the reasons for the response but the response itself.

This question was revised.

9 H

4 B

S A, CR, 41.7, NRC Exam 2007 10 H

2 B

S D, CR, 41.7 11 H

3 B

S A, CR, 41.5 12 H

3 B

S C, CR, 41.10, NRC 2002 13 H

2 U

B U

S C, CR, 41.5, NRC 1998 - Add the SG pressure to the stem so that the condensate system could not be considered as a SG feed option making more than one answer correct.

Upon further review of the question, it is ok as written.

14 L

2 U

B U

S D, CR, 41.8, NRC 1998 - This question does not test on the effect of battery discharge rates on capacity. Instead it tests on procedural requirements.

This question was revised.

15 H

2 U

B U

S B, CR, 41.10 - This question does not test on the function or purpose of major components or controls but instead tests on operator actions.

This question was revised.

16 H

3 N

S B, CR, 41.4 17 H

3 B

S C, CR, 41.6 18 H

F 3

U B

U S

B, CR, 41.4 - This question does not test on the reasons for the response but the response itself.

This question was revised.

19 H

3 B

S C, CR, 41.7, NRC 2002 20 F

2 N

S A, CR, 41.2 21 F

2 U

B U

S C, CR, 41.10, NRC 1998 - Answer A is also correct since 5.3 Hg exceeds 7.0 Hg.

Upon further review of the question, it is ok as is.

ES-401 3

Form ES-401-9

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 22 F

3 B

S B, CR, 41.9, NRC 2002 23 F

3 N

S B, CR, 41.9 24 H

3 U

U B

U S

A, CR, 41.10, NRC 2002 - This question does not test on the availability of main or aux feed as required by the KA. Also, answer B could be argued as being correct.

Distractor B was revised.

25 H

3 N

S D, OR, 41.10 26 F

2 B

S D, CR, 41.5, NRC 1998 27 H

3 N

S D, CR, 41.10 28 H

2 N

S D, CR, 41.7 29 H

2 U

B U

S C, CR, 41.10 - This does not match the KA because the question does not address the predict the impact part of the KA.

This question was revised.

30 H

3 N

S A, CR, 41.4 31 F

2 B

S A, CR, 41.7 32 H

2 N

S C, OR, 41.5 33 H

3 N

S A, CR, 41.4 34 H

3 B

S D, CR, 41.4 35 H

3 N

S C, CR, 41.10 36 H

3 B

S B, CR, 41.7 37 H

3 N

S C, CR, 41.10 38 H

F 3

U N

U S

C, CR, 41.5 - This is a KA mismatch because the question does not involve the RPS as required by the KA.

This question was revised.

39 H

2 B

S A, CR, 41.7 40 H

3 B

S D, CR, 41.7 41 H

3 U

B U

S A, CR, 41.8 - This KA is focused on the cooling for the containment spray pump itself and not the containment spray process flow.

This question was reivised.

42 H

3 N

S D, CR, 41.5

ES-401 4

Form ES-401-9

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 43 H

3 B

S B, CR, 41.4 44 H

3 B

S D, CR, 41.4 45 F

3 N

S A, CR, 41.4 46 H

3 M

S B, CR, 41.4 47 H

3 N

S C, CR, 41.7 48 H

4 M

S C, CR, 41.7 49 F

2 N

S C, CR, 41.7 50 H

3 U

B U

S B, CR, 41.7, NRC 2002 - This is a KA mismatch because it does not test on the purpose of the load sequencer.

After discussing with the facility, this question is ok as is.

51 H

3 M

S B, CR, 41.11, NRC 2003 52 H

F 3

U B

U S

B, CR, 41.7 - This is a KA mismatch because it does not test on the electrical power supply to the pump motor.

This question was revised.

53 F

2 U

N U

S D, CR, 41.4 - This is a KA mismatch because it does not test on the IAS.

This question was revised.

54 H

3 U

B U

S D, CR, 41.9 - This is a KA mismatch because it does not test on containment changes based on operating containment controls.

This question was revised.

55 F

2 N

S B, CR, 41.9 56 H

2 B

S C, CR, 41.6 57 H

3 B

S D, CR, 41.2, NRC 1998 58 F

3 B

S D, CR, 41.9, NRC 2002 59 F

2 B

S B, CR, 41.9 60 H

3 U

B U

S A, CR, 43.7, NRC 2000 - This is an SRO only question because it tests on refueling interlocks that are outside the scope of the control room operator.

This question was revised.

61 H

3 M

S C, CR, 41.7, NRC 1998 62 F

2 B

S A, CR, 41.4

ES-401 5

Form ES-401-9

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation 63 F

3 N

S C, CR, 41.13 64 F

2 B

S D, CR, 41.11, NRC 2002 65 H

3 E

M E

S B, CR, 41.7 - This question is unnecessarily complex.

This question is ok as written.

66 F

2 B

S C

67 H

3 U

U B

U S

C, CR, 41.5 - This question is U because all four distractors will raise Tave and choosing the best method is not acceptable. Second it does not test on the function or purpose of a major component.

This question was revised.

68 F

2 B

S B, CR, 41.10 69 F

3 B

S D, CR, 41.10 70 F

2 U

N U

S A, CR, 41. This question doesnt test knowledge of processes for determining reactivity effects per KA.

After discussing with the facility, this question is acceptable as is.

71 H

2 M

E S

D, CR, 41.10; the second A distractor should be a D; The typo was corrected.

72 F

3 U

U B

U S

A, CR, 41.10, NRC 2003; this question asks what action is NOT taken (backward -

not operationally valid); doesnt match KA (question is about environmental releases vice excess rad levels and personnel exposures);

This question was revised.

73 F

2 B

S C, CR, 41.10, NRC 1998; 74 F

2 U

B U

S D, CR, 41.10, NRC 2002, B & C are too similar and not credible; This question was revised.

75 H

3 B

S A, CR, 41.10, NRC 2005 76 H

3 M

S B, CR, 43.5 77 H

3 M

S A, CR, 43.5 78 F

3 B

S B, CR, 43.3, NRC 2002 79 H

3 N

S D, CR, 43.5; 80 F

4 M

S C, CR, 43.5; NRC 2006; 81 H

3 U

B U

A, CR, 43.5; B is also correct;

ES-401 6

Form ES-401-9

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation S

This question was revised.

82 H

3 E

M E

S A, CR, 43.5; change stem to...required RCS pressure criteria...;

83 H

3 N

E S

C, CR, 43.5; in A, change six to 24; in B, change six to two; 84 F

3 N

S C, CR, 43.5; 85 H

3 X

B E

S B, CR, 43.2; This was submitted as a modified question, but its really still bank; C is not credible as written (if C then D also - it needs to be distinguished from D); also, change doors to door in C; 86 H

3 N

S B, CR, 43.5 87 H

3 N

S D, OR, 43.2 88 H

2 U

M U

S C, CR, 43.5; doesnt test knowledge of the E-Plan in connection with MFW per KA; This question was revised.

89 H

3 N

E S

D, OR, 43.5; references provided did not support correct answer (no documentation to support the 60 minute NRC notification requirement);

90 F

3 U

U B

U S

B, CR, 43.4, NRC 2002; C is also a correct answer; A is not credible in that it is not a measure to prevent an accidental release per the stem; in is misspelled in D; Following a discussion with the facility, this question is ok as is.

91 H

2 U

U M

U E

S A, OR, 43.5; direct lookup; in the stem, Radiation Monitors is... is a grammatical error; The question will be revised to correct the grammatical error, otherwise it is ok.

92 H

3 U

N U

S C, OR, 43.5; challenge to containment criteria seem to be a judgment call which would allow some candidates to argue that a GE is appropriate for this case (making C and D correct);

This question was revised.

93 H

3 U

M U

S D, CR, 43.5; it looks like the actions in either C or D are the same and candidate are not generally required to have the attachments from a normal operating procedures memorized.

The question is ok as written.

94 F

2 U

U B

U S

D, CR, 41.10; this question only requires knowledge of the 55.53 condition, which is RO knowledge; the stem and correct answer have not been significantly modified from the original; B is also correct in that it meets the shift manning and exceeds (and therefore meets) the requirements of 55.53;

ES-401 7

Form ES-401-9

3. Psychometric Flaws

4. Job Content Flaws

5. Other Q#

1.

LOK (F/H)

2.

LOD (1-5)

Stem Focus Cues T/F Cred.

Dist.

Partial Job-Link Minutia

1. /

units Back-ward Q=

K/A SRO Only

6.

B/M/N

7.

U/E/S

8.

Explanation This question was revised.

95 F

2 U

U B

U S

A, CR, 43.5; question has not been significantly modified per ES-401 D.2.f; second half of answer is two narrow since no system temperature is given in the stem (there may be no correct answer listed depending on RCS temperature), last, this is an RO question since it is testing on a system limit/caution. Suggest a question using tech spec basis.

This question was revised.

96 F

2 U

B U

S C, CR, 41.10; this appears to be RO knowledge since it is associated with an RO lesson plan.

The NOTES section of the question was revised to reflect why it is an SRO only question.

97 H

3 N

S A, CR, 43.2; 98 H

2 B

E S

D, CR, 43.5, NRC 2002, it should be an RWP in A and a specific RWP in B.

99 H

3 U

N U

S C, CR, 43.4 - Based on the notes, it looks like terminating the release is the correct answer - which is B (step 4.17.5 of 2104.022);

This question was revised to reflect the correct answer.

100 H

3 U

N U

S A, CR, 41.10; this is RO LOK - this KA is not suitable for an SRO-only question; This question was revised.

B= 47 F= 26 E= 01 M= 06 H= 49 (65%)

U= 25 (33%)

RO TOTALS:

N= 22 (29%)

Additional Notes: 65% exceeds the 50 - 60% required band for higher cognitive questions.

B= 8 F= 7 E= 5 M= 7 H= 18 (64%)

U= 11 (44%)

SRO TOTALS:

N= 10 (40%)

Additional Notes:

GENERAL COMMENT

S:

1.

Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

2.

Chief Examiner comments are indicated in blue in the explanation section.

3.

Average difficulty is 2.68 on the RO exam and 2.8 on the SRO exam. The average difficulty on the RO exam is lower than the typical average (2.8 - 3.0) for other similar exams.

ES-401 8

Form ES-401-9

4.

The 10CFR55.41/43 distribution is:

41.1 = 0 41.8 = 2 43.1 = 0 41.2 = 2 41.9 = 6 43.2 = 3 41.3 = 1 41.10 = 19 43.3 = 1 41.4 = 13 41.11 = 2 43.4 = 2 41.5 = 9 41.12 = 0 43.5 = 16 41.6 = 2 41.13 = 1 43.6 = 0 41.7 = 16 41.14 = 0 43.7 = 0

5.

The answer distribution is RO / SRO:

a = 17 / 7 b = 19 / 5 c = 20 / 7 d = 19 / 6

6.

There are 3 / 4 (RO/SRO) questions with attachments provided.

7.

There are 30 questions (40%) on the RO exam that have been used in previous ANO-2 licensing exams. This is a magnitude greater than what has been seen on any other exam in this reviewers experience. Only 1 of the 30 was used on the previous 2 NRC exams. There are 5 repeat NRC exam questions on the SRO exam with one of those used on the previous 2 exams.

8.

The 16 questions on the SRO exam from 10 CFR 55.43.5 (assessment of facility conditions and procedure selection) is higher than the typical 9-12 seen for this knowledge topic.