ML082750585

09-Draft Written Exam Comments

ML082750585
Person / Time
Site:	Arkansas Nuclear
Issue date:	09/05/2008
From:	Brian Larson NRC Region 4
To:
References
Download: ML082750585 (8)

Text

ES-401 AN1-2008-09 Written Examination Review Worksheet Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 1 F 3 B S 2 H 3 N S 3 H 3 B S Change CBOT to ATC in stem. Accept as is 4 H 3 N S Need to walk through procedure to get to throttling criteria. Done. Ok as is.

5 H 3 N S Repeat from 2005 retake; appears to be a collection of T/F statements (no need for 6 H 3 X X B E stem); only B & C deal with reasons per the KA; remove name from History Not T/F statements as stem is required. Added reasons to A & D.

C could also be considered correct; doesnt really test knowledge of the operational 7 H 3 X X B U implications of a loss of DHR per KA (this question just asks for actions to prevent possible loss of DHR); reworded stem 8 H 3 X X N S Should C be select LT-1002 on C04? No. Is 40 a clear indication of a failed Instructions

[Refer to Section D of ES-401 and Appendix B for additional information regarding each of the following concepts.]

1. Enter the level of knowledge (LOK) of each question as either (F)undamental or (H)igher cognitive level.

2. Enter the level of difficulty (LOD) of each question using a 1 B 5 (easy B difficult) rating scale (questions in the 2 B 4 range are acceptable).

3. Check the appropriate box if a psychometric flaw is identified:

$ The stem lacks sufficient focus to elicit the correct answer (e.g., unclear intent, more information is needed, or too much needless information).

$ The stem or distractors contain cues (i.e., clues, specific determiners, phrasing, length, etc).

$ The answer choices are a collection of unrelated true/false statements.

$ The distractors are not credible; single implausible distractors should be repaired, more than one is unacceptable.

$ One or more distractors is (are) partially correct (e.g., if the applicant can make unstated assumptions that are not contradicted by stem).

4. Check the appropriate box if a job content error is identified:

$ The question is not linked to the job requirements (i.e., the question has a valid K/A but, as written, is not operational in content).

$ The question requires the recall of knowledge that is too specific for the closed reference test mode (i.e., it is not required to be known from memory).

$ The question contains data with an unrealistic level of accuracy or inconsistent units (e.g., panel meter in percent with question in gallons).

$ The question requires reverse logic or application compared to the job requirements.

5. Check questions that are sampled for conformance with the approved K/A and those that are designated SRO-only (K/A and license level mismatches are unacceptable).

6. Enter question source: (B)ank, (M)odified, or (N)ew. Check that (M)odified questions meet criteria of ES-401 Section D.2.f.

7. Based on the reviewer=s judgment, is the question as written (U)nsatisfactory (requiring repair or replacement), in need of (E)ditorial enhancement, or (S)atisfactory?

8. At a minimum, explain any AU@ ratings (e.g., how the Appendix B psychometric attributes are not being met). Page 1 of 8 OBDI 202 - IOLE Process

ES-401 AN1-2008-09 2 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only channel? Yes doesnt test ability to determine Pzr Level during Pzr Pressure Malfunctions per KA; accept as is 9 H 2 B S Repeat from 2007; change stem to ...EOP, the fourth RCP must be secured before what specified 10 F* 2 X N E temperature and why? References not sufficient to justify answer accept as is changes are not sufficient to warrant modified label; changed to Bank change 11 H 3 X B* S question to address knowledge, not action; accept as is C is not credible since the stem already indicates that there is EFW flow to both OTSGs. Accept as is 12 H 3 X N U this is not RO-level (55.43(b)(2)); new question 13 H 3 N S 14 H 3 N S In the question, B says ...causing excessive RCS makeup, but in the answer section it says ...minimizing the impact to RCS makeup. And the Notes says the 15 F 3 N E result is swapping inventory control to HPI block valve Which is correct? Rewrote question to include position of both valves and updated notes doesnt test knowledge of REASONS for use of dc control power for D/Gs per KA; 16 H 3 X B U added reasons This answer is obvious and cued from the stem; doesnt test ability to monitor/operate 17 F 3 X X X M U components served by IA to minimize drain on system per KA. Rewrote stem Should the stem say RCS pressure has increased to 120 psig...? Fixed Change 18 H 3 N E the stem to say Under these conditions, what is the preferred... Done procedure says power must be <40% for rod transfer, but stem says power is at 40%;

19 H 3 B E is this a problem? Changed to 39%

20 F 2 N S 21 F 3 B* S changes are not sufficient to warrant modified label; Changed to Bank 22 F 3 B S 23 H 3 N S This is really an SRO-level question (55.43(b)(5)); entry condition to procedure is RO 24 H 3 X X B S knowledge doesnt test knowledge of REASONS per KA; accept as is 25 F* 3 X B E The answer is keyed to procedure entry conditions, not level at which consideration would be given to commencing a shutdown; stem should be modified to refer to Page 2 of 8 OBDI 202 - IOLE Process

ES-401 AN1-2008-09 3 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only entry conditions. Accept as is 26 H 3 X N E shouldnt the stem say Cooldown rate is 70 degrees F/hr... ? fixed 27 H 3 X B U doesnt test knowledge of the REASONS per KA; added reasons doesnt test knowledge of affects of RCP coastdown on RCS parameters per KA; also, couldnt validate correct answer; procedure says if no RCPs are operating, use lowest. Lowest what? The procedure also says when both DHR systems are in service, use the lower DH cooler outlet temperature for plotting cooldown rate; 28 F 3 X N U however, we only have one DHR system in operation for this question, and it is not explicitly stated that the DH cooler outlet is used for cooldown in this condition. Also, TS 3.4.3 Figures 1 & 2 both indicate in their respective Note-2 that the DHR RETURN temperature is to be used for HUR/CDR calculations. New question 29 H 3 B S 30 H 3 B S 31 F 3 B S change stem to ...loss of Decay Heat Removal... or similar phrasing (strictly 32 H* 2 N E speaking, a loss of Decay Heat would eliminate the need for the DHR system and make this question moot); fixed 33 F 2 N S 34 F 2 N E Eliminate first sentence of stem; fixed is D supposed to read C-28A &B or E-28A &B? (see notes explaining answer) 35 F 3 N E fixed notes this is a very easy question and a missed opportunity to ask a more substantive 36 H 2 B E question on the KA; I recommend rewriting with more meat; revised notes and reworded stem B and C are not very credible; doesnt test knowledge of effect of Loss/Malfunction 37 H 3 X X B U of bypass block circuits per KA; changed stem 38 H 3 B S Repeat from 2005 retake; 39 F 2 N S 40 F 2 N S 41 H 3 N S Page 3 of 8 OBDI 202 - IOLE Process

ES-401 AN1-2008-09 4 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 42 H 3 N* E The source says direct but question history says its new. Fixed history.

43 H 4 B S 44 H* 3 X B E A is not credible; MFPs have setback feature - revised notes - accept as is doesnt test knowledge of the effect of a loss/malfunction of AFW has on SG per KA; revised stem also, its not clear from the references provided why the feed rate on A 45 H 3 X X N U SG should be 6/minute - how is that determined? Revised answers based on calculation 46 H* 2 N S explanations for B and C dont make sense to me. Accept as is 47 H 4 N S this question doesnt really test what the KA is asking for and, based on the 48 F* 3 X X X B U procedure, it looks like a new KA is needed to test something more operationally relevant; rewrote question 49 F 3 X N S is this really RO knowledge (55.43(b)(2))? RO lesson plan - accept as is 50 H 3 X B E A is not credible when the stem gives the discharge pressure of 20 psig; accept as is 51 H 3 B S Repeat from 2005; doesnt count 52 H* 3 B S 53 F 3 B S 54 F 2 N S B, C, & D are all correct for the way the question is worded; should change stem 55 F 2 X N U question to Under these conditions, what actions should be considered to help protect... and should replace D; revised stem 56 H 3 B S 57 F 3 N S 58 F* 3 B S 59 F 2 N S 60 F 3 N S I do not see a procedural direction to place the assembly in the storage rack and I do 61 H 3 X M E not see a prohibition to putting it back in the SFP; revised answers Page 4 of 8 OBDI 202 - IOLE Process

ES-401 AN1-2008-09 5 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 62 F 2 X N S neither A or B are credible; accept as is 63 H 4 B S 64 H 3 N S doesnt test ability to predict/monitor rad levels associated with ARM system controls 65 H 2 X B S per KA. Background levels have changed - accept as is.

what is eB change number? It is not mentioned in the reference provided; 66 F 2 B E electronic database - common language at AN1 67 F 3 B S Repeat from 2007; do Work Order Instructions often refer to the procedure lineup section for restoration?

68 F 2 X N E If so, C would be a reasonable correct answer. Operators are trained to use op procedures - accept as is 69 F 2 B S 70 F 2 B S only need to know the TEDE value to get this correct; recommend having two options 71 F 2 N E with 2000mrem and two with 5000mrem; fixed 72 F 3 N S 73 H 3 B S Repeat from 2005; doesnt count 74 H 3 N S 75 F 2 N S recognizing reactor trip criteria does not discriminate at the SRO level; changed 76 H 2 X N E power level - 43.b.5 are SROs supposed to know all these action times by memory? If they are only supposed to know the 1-hour actions from memory, then none of the other distractors 77 H 3 X X N U are credible (since they all have more than 1-hour actions). Removed time from all answers it is not clear from the references what is meant by exceeding the facilities licensed limit for PTS; I see where the procedure talks of invoking PTS limits, but I do not 78 F 3 X X N U see where it says what those invoked limits will be; also, isnt knowledge of CDR limits in the RTs RO-level knowledge? Revised to include both PTS limits and TS 79 H 2 X B U with the TS provided, anyone who can read can eliminate A and B as not credible Page 5 of 8 OBDI 202 - IOLE Process

ES-401 AN1-2008-09 6 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only (since they do not match any action in the TS); also, B grammar needs to be cleaned up; also, the correct answer probably should also include the and part of action E. Revised A & B, added the and part to C & D change ...have actions to... to ...should be used to...; revised stem doesnt test 80 H 4 X X N E ability to verify alarms are consistent with plant conditions per KA; accept as is 81 H 3 X N E change ...contains specific steps to... to ...should be used to... fixed this is testing an understanding of thermodynamic plant response and knowledge of 82 H 3 X N S trip setpoints - thats RO-level knowledge. TS bases - accept as is 83 H 3 M E notes section was not updated to reflect the Modified version of the question. fixed Repeat from 2005; change ...contains mitigating actions... to ...should be used...

84 H 3 B E fixed History section states this question was used on the 2005 RO exam - if so, is it SRO only knowledge? History was wrong - corrected to reflect SRO exam B (establish 40-60) satisfies the procedure (establish <100) and therefore could also 85 H 3 X N U be considered correct; Accept as is with enhancements 86 H 3 N S Repeat from 2005; it would be better to include the action in the choices rather than 87 H 4 B E give the action and just ask for the correct procedure; accept as is answer is obvious (none of the distractors address the RB pressure); doesnt test knowledge of immediate actions per KA; should probably add an action to the choices (along with procedure selection) and then do two answers with ESAS and 88 H 3 X X N E two with some other credible procedure; accept as is - applicant must use given info to eliminate distractors change ...contains specific actions to mitigate... to ...should be used to mitigate... fixed since one only has recognize the cause of the condition, this really seems like RO 89 H 3 X N E knowledge; should probably 2 causes and 2 procedures in the choices instead of both being different in all 4 options; fixed same comment as #89; RO system knowledge (recognizing how the valves fail) is all 90 H 3 X N U that is needed to answer this. Added procedure strategy 91 H* 3 N S if SROs (and ROs) are only required to know 1-hour TS from memory, then none of 92 F 2 X X N U the other distractors are credible and this might really be just RO knowledge; revised to eliminate times - changed D Page 6 of 8 OBDI 202 - IOLE Process

ES-401 AN1-2008-09 7 Form ES-401-9

1. 2. 3. Psychometric Flaws 4. Job Content Flaws 5. Other 6. 7. 8.

Q# LOK LOD (F/H) (1-5) Stem Cues T/F Cred. Partial Job- Minutia #/ Back- Q= SRO B/M/N U/E/S Explanation Focus Dist. Link units ward K/A Only 93 H 3 N E change ...contains actions for... to ...should be used for... done A & C are not credible because they include B (i.e., if A or C were correct, B would also be correct, therefore neither A nor C could logically be the correct 94 F 2 X N E answer - believing that either were correct would be believing that there are 2 correct answers) revised stem and corrected D doesnt test knowledge of SROs responsibilities during fuel handling per KA; added 95 H 3 X B U actions required to stem 96 F 2 X N U doesnt test knowledge of work order requirements per KA; revised question 97 F 2 M S 98 F 3 M S 99 H 3 X B S reactor trip criteria = RO-knowledge; accept as is 100 H 3 X N E B & C are not credible for this circumstance; Revised B, accept C as is B= 35 F= 33 (44%) E= 18 RO TOTALS: M= 2 H= 42 (56%) U= 10 N= 38 S= 47 B= 5 F= 6 (24%) E= 11 SRO TOTALS: M= 3 H= 19 (76%) U= 8 N= 17 S= 6

GENERAL COMMENT

S:

1. Bank questions are indicated by B; Modified are indicated by M; New questions are indicated by N.

2. Chief Examiner comments are indicated in blue.

3. Average difficulty is __ 2.73 ___ on the RO exam and ___ on the SRO exam.

4. The 10CFR55.41/43 distribution is: RO / SRO 41.1 = 43.1 =

41.2 = 43.2 =

41.3 = 43.3 =

41.4 = 43.4 =

41.5 = 43.5 = Page 7 of 8 OBDI 202 - IOLE Process

ES-401 AN1-2008-09 8 Form ES-401-9 41.6 = 43.6 =

41.7 = 43.7 =

41.8 =

41.9 =

41.10 =

41.11 =

41.12 =

41.13 =

41.14 =

5. The answer distribution is: RO / SRO A=

B=

C=

D=

6. There are 3 questions with attachments provided. Page 8 of 8 OBDI 202 - IOLE Process