ML082750585

09-Draft Written Exam Comments

ML082750585
Person / Time
Site:	Arkansas Nuclear
Issue date:	09/05/2008
From:	Brian Larson NRC Region 4
To:
References
Download: ML082750585 (8)

Text

ES-401 Written Examination Review Worksheet Form ES-401-9 AN1-2008-09 Instructions [Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.] 1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level. 2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable). 3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much ne edless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem). 4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with ques tion in gallons).

$ The question requires reverse logic or application compared to the job requirements. 5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable). 6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f. 7. Based on the reviewer

=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory? 8. At a minimum, explain any A U@ ratings (e.g., how the Appendix B psychometric attributes are not being met).

Attachment 8 Page 1 of 8 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred. Dist. Partial Job-Link Minutia#/ units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 1 F 3 B S 2 H 3 N S 3 H 3 B S Change CBOT to ATC in stem. Accept as is 4 H 3 N S Need to walk through procedure to get to throttling criteria. Done. Ok as is.

5 H 3 N S 6 H 3 X X B E Repeat from 2005 retake; appears to be a collection of T/F statements (no need for stem); only B & C deal with "reasons" per the KA; remove name from History Not T/F statements as stem is required. Added "reasons" to A & D.

7 H 3 X X B U 'C' could also be considered correct; doesn't really test knowledge of the "operational implications of a loss of DHR" per KA (this question just asks for actions to prevent possible loss of DHR); reworded stem 8 H 3 X X N S Should 'C' be "select LT-1002 on C04"?

No. Is 40" a clear indication of a failed ES-401 2 Form ES-401-9 AN1-2008-09 Attachment 8 Page 2 of 8

3. Psychometric Flaws 4. Job Content Flaws 5. Other OBDI 202 - IOLE Process Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation channel? Yes doesn't test ability to determine Pzr Level during Pzr Pressure Malfunctions per KA; accept as is 9 H 2 B S Repeat from 2007; 10 F* 2 X N E change stem to "...EOP, the fourth RCP must be secured before what specified temperature and why?" References not sufficient to justify answer accept as is 11 H 3 X B* S changes are not sufficient to warrant "modified" label; changed to Bank change question to address knowledge, not action; accept as is C is not credible since the stem already indicates that there is EFW flow to both OTSGs.

Accept as is 12 H 3 X N U this is not RO-level (55.43(b)(2)); new question 13 H 3 N S 14 H 3 N S 15 F 3 N E In the question, 'B' says "...causing excessive RCS makeup," but in the "answer" section it says "...minimizing the impact to RCS makeup." And the Notes says the result is swapping inventory control to HPI block valve- Which is correct? Rewrote question to include position of both valves and updated notes 16 H 3 X B U doesn't test knowledge of REASONS for use of dc control power for D/G's per KA; added reasons 17 F 3 X X X M U This answer is obvious and cued from the stem; doesn't test ability to monitor/operate components served by IA to minimize drain on system per KA. Rewrote stem 18 H 3 N E Should the stem say "RCS pressure has increased to 120 psig..."?

Fixed Change the stem to say "Under these conditions, what is the preferred..." Done 19 H 3 B E procedure says power must be <40% for rod transfer, but stem says power is at 40%;

is this a problem? Changed to 39%

20 F 2 N S 21 F 3 B* S changes are not sufficient to warrant "modified" label; Changed to Bank 22 F 3 B S 23 H 3 N S 24 H 3 X X B S This is really an SRO-level question (55.43(b)(5)); entry condition to procedure is RO knowledge doesn't test knowledge of REASONS per KA; accept as is 25 F* 3 X B E The answer is keyed to procedure entry condit ions, not "level at which consideration would be given to commencing a shutdown;" stem should be modified to refer to ES-401 3 Form ES-401-9 AN1-2008-09 Attachment 8 Page 3 of 8 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation entry conditions.

Accept as is 26 H 3 X N E shouldn't the stem say "Cooldown rate is 70 degrees F/hr..." ? fixed 27 H 3 X B U doesn't test knowledge of the REASONS per KA; added reasons 28 F 3 X N U doesn't test knowledge of affects of RCP coastdown on RCS parameters per KA; also, couldn't validate correct answer; procedure says "if no RCPs are operating, use lowest." Lowest what? The procedure also says "when both DHR systems are in service, use the lower DH cooler outlet temperature for plotting cooldown rate; however, we only have one DHR system in operation for this question, and it is not explicitly stated that the DH cooler outlet is used for cooldown in this condition. Also, TS 3.4.3 Figures 1 & 2 both indicate in their respective Note-2 that the DHR RETURN temperature is to be used for HUR/CDR calculations. New question 29 H 3 B S 30 H 3 B S 31 F 3 B S 32 H* 2 N E change stem to "...loss of Decay Heat Re moval..." or simila r phrasing (strictly speaking, a "loss of Decay Heat" would eliminate the need for the DHR system and make this question moot); fixed 33 F 2 N S 34 F 2 N E Eliminate first sentence of stem; fixed 35 F 3 N E is 'D' supposed to read "C-28A &B" or "E-28A &B"? (see "notes" explaining answer) fixed notes 36 H 2 B E this is a very easy question and a missed opportunity to ask a more substantive question on the KA; I recommend rewriting with more meat; revised notes and reworded stem 37 H 3 X X B U 'B' and 'C' are not very credible; doesn't test knowledge of effect of Loss/Malfunction of bypass block circuits per KA; changed stem 38 H 3 B S Repeat from 2005 retake; 39 F 2 N S 40 F 2 N S 41 H 3 N S ES-401 4 Form ES-401-9 AN1-2008-09 Attachment 8 Page 4 of 8 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 42 H 3 N* E The "source" says "direct" but question "history" says it's new. Fixed history.

43 H 4 B S 44 H* 3 X B E 'A' is not credible; MFPs have setback feature - revised notes - accept as is 45 H 3 X X N U doesn't test knowledge of the effect of a loss/malfunction of AFW has on SG per KA; revised stem also, it's not clear from the references provided why the feed rate on A SG should be 6"/minute - how is that determined? Revised answers based on calculation 46 H* 2 N S explanations for 'B' and 'C' don't make sense to me.

Accept as is 47 H 4 N S 48 F* 3 X X X B U this question doesn't really test what the KA is asking for and, based on the procedure, it looks like a new KA is needed to test something more operationally relevant; rewrote question 49 F 3 X N S is this really RO knowledge (55.43(b)(2))? RO lesson plan - accept as is 50 H 3 X B E 'A' is not credible when the stem gives the discharge pressure of 20 psig; accept as is 51 H 3 B S Repeat from 2005; doesn't count 52 H* 3 B S 53 F 3 B S 54 F 2 N S 55 F 2 X N U 'B', 'C', & 'D' are all correct for the way the question is worded; should change stem question to "Under these conditions, what actions shoul d be considered to help protect..." and should replace 'D'; revised stem 56 H 3 B S 57 F 3 N S 58 F* 3 B S 59 F 2 N S 60 F 3 N S 61 H 3 X M E I do not see a procedural direction to place the assembly in the storage rack and I do not see a prohibition to putting it back in the SFP; revised answers ES-401 5 Form ES-401-9 AN1-2008-09 Attachment 8 Page 5 of 8 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 62 F 2 X N S neither 'A' or 'B' are credible; accept as is 63 H 4 B S 64 H 3 N S 65 H 2 X B S doesn't test ability to predict/monitor rad levels associated with ARM system controls per KA. Background levels have changed - accept as is.

66 F 2 B E what is "eB change number"? It is not mentioned in the reference provided; electronic database - common language at AN1 67 F 3 B S Repeat from 2007; 68 F 2 X N E do Work Order Instructions often refer to the procedure lineup section for restoration?

If so, 'C' would be a reasonable correct answer. Operators are trained to use op procedures - accept as is 69 F 2 B S 70 F 2 B S 71 F 2 N E only need to know the TEDE value to get this correct; recommend having two options with 2000mrem and two with 5000mrem; fixed 72 F 3 N S 73 H 3 B S Repeat from 2005; doesn't count 74 H 3 N S 75 F 2 N S 76 H 2 X N E recognizing reactor trip criteria does not discriminate at the SRO level; changed power level - 43.b.5 77 H 3 X X N U are SRO's supposed to know all these action times by memory? If they are only supposed to know the 1-hour actions from memory, then none of the other distractors are credible (since they all have more than 1-hour actions).

Removed time from all answers 78 F 3 X X N U it is not clear from the references what is meant by "exceeding the facilities' licensed limit for PTS;" I see where the procedure talks of "invoking" PTS limits, but I do not see where it says what those "invoked" limits will be; also, isn't knowledge of CDR limits in the RTs RO-level knowledge? Revised to include both PTS limits and TS 79 H 2 X B U with the TS provided, anyone who can read can eliminate 'A' and 'B' as not credible ES-401 6 Form ES-401-9 AN1-2008-09 Attachment 8 Page 6 of 8 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation (since they do not match any action in the TS); also, 'B' grammar needs to be cleaned up; also, the correct answer probably should also include the "and" part of action E. Revised A & B, added the "and" part to C & D 80 H 4 X X N E change "...have actions to..." to "...should be used to..."; revised stem doesn't test ability to verify alarms are consistent with plant conditions per KA; accept as is 81 H 3 X N E change "...contains specific steps to..." to "...should be used to..." fixed 82 H 3 X N S this is testing an understanding of thermodynamic plant response and knowledge of trip setpoints - that's RO-level knowledge. TS bases - accept as is 83 H 3 M E "notes" section was not updated to reflect the Modifi ed version of the question. fixed 84 H 3 B E Repeat from 2005; change "...contains mitigating actions..." to "...should be used..."

fixed History section states this question was used on the 2005 RO exam - if so, is it SRO only knowledge?

History was wrong - corrected to reflect SRO exam 85 H 3 X N U 'B' (establish 40-60) satisfies the procedure (establish <100) and therefore could also be considered correct; Accept as is with enhancements 86 H 3 N S 87 H 4 B E Repeat from 2005; it would be better to include the action in the choices rather than give the action and just ask for the correct procedure; accept as is 88 H 3 X X N E answer is obvious (none of the distractors address the RB pressure); doesn't test knowledge of immediate actions per KA; should probably add an action to the choices (along with procedure selection) and then do two answers with ESAS and two with some other credible procedure; accept as is - applicant must use given info to eliminate distractors change "...contains specific actions to mitigate..." to "...should be used to mitigate..."

fixed 89 H 3 X N E since one only has recognize the cause of the condition, this really seems like RO knowledge; should probably 2 causes and 2 procedures in the choices instead of both being different in all 4 options; fixed 90 H 3 X N U same comment as #89; RO system knowledge (recognizing how the valves fail) is all that is needed to answer this. Added procedure strategy 91 H* 3 N S 92 F 2 X X N U if SRO's (and RO's) are only required to know 1-hour TS from memory, then none of the other distractors are credible and this might really be just RO knowledge; revised to eliminate "times" - changed D ES-401 7 Form ES-401-9 AN1-2008-09 Attachment 8 Page 7 of 8 OBDI 202 - IOLE Process

3. Psychometric Flaws 4. Job Content Flaws 5. Other Q# 1. LOK (F/H) 2. LOD (1-5) Stem Focus Cues T/F Cred.

Dist. Partial Job-LinkMinutia#/

units Back-wardQ= K/A SROOnly 6. B/M/N 7. U/E/S 8. Explanation 93 H 3 N E change "...contains actions for..." to "...should be used for..."

done 94 F 2 X N E 'A' & 'C' are not credible because they include 'B' (i.e., if 'A' or 'C' were correct, 'B' would also be correct, therefore neither 'A' nor 'C' could logically be the correct answer - believing that either were correct would be believing that there are 2 correct answers) revised stem and corrected D 95 H 3 X B U doesn't test knowledge of SRO's responsibilities during fuel handling per KA; added actions required to stem 96 F 2 X N U doesn't test knowledge of work order requirements per KA; revised question 97 F 2 M S 98 F 3 M S 99 H 3 X B S reactor trip criteria = RO-knowledge; accept as is 100 H 3 X N E 'B' & 'C' are not credible for this circumstance; Revised B, accept C as is B= 35 F=33 (44%) E=18 M= 2 H=42 (56%) U=10 RO TOTALS:

N= 38 S=47 B= 5 F=6 (24%) E=11 M= 3 H=19 (76%) U=8 SRO TOTALS:

N= 17 S=6

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

2. Chief Examiner comments are indicated in blue. 3. Average difficulty is __

2.73 ___ on the RO exam and

___

on the SRO exam.

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 43.1 = 41.2 = 43.2 = 41.3 = 43.3 = 41.4 = 43.4 = 41.5 = 43.5 =

ES-401 8 Form ES-401-9 AN1-2008-09 Attachment 8 Page 8 of 8 OBDI 202 - IOLE Process 41.6 = 43.6 = 41.7 = 43.7 = 41.8 = 41.9 = 41.10 = 41.11 = 41.12 = 41.13 = 41.14 =

5. The answer distribution is: RO / SRO A = B = C = D = 6. There are 3 questions with attachments provided.