ML120480078
| ML120480078 | |
| Person / Time | |
|---|---|
| Site: | Robinson  |
| Issue date: | 02/10/2012 |
| From: | NRC/RGN-II |
| To: | |
| Mark Bates | |
| Shared Package | |
| ML120470109 | List: |
| References | |
| Download: ML120480078 (52) | |
Text
ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 1 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only
GENERAL COMMENT
S B= Bank / M=Modified / N=New / F=Fundamental Level (I.E. Memory) / H=Higher Cognitive Level (I.E. C/A)
Test what operators are required to do by procedure
- not what they will do. Generally, extra information that does not serve a purpose, should be deleted from the stem and answer choices.
Note: For Bank questions, at a minimum change the location of the correct answer.
Answer choices should only contain the information that makes it unique. Several questions contained the flaw of having too many items in the answer choices.
For each Bank question, list the bank from which it was taken AND list any previous NRC exams on which it appeared.
I just want to ensure that the bank was not limited to recently administered NRC exams and that it was not limited to a one
-for-one KA match (all questions meet several KAs). NUREG
-1021, ES-401, Page 8 of 33, states, "If the bank contains more than one question that fits a specific KA statement, randomly select from among the available questions unless there is an appropriate basis for selecting a specific question (e.g., higher cognitive level, better discrimination validity, more operationally oriented, or site
-specific priority)."
I think 6 questions may have been repeated from the 2007 NRC exam
- This still needs to be verified. If this is the case, some of them may need to be modified. Simply addressing some of the comments throughout the exam may resolve the issue.
RO EXAM 1 007EG2.4.11 B H 1 2 X U S Is it operationally valid to provide a reactor trip in the stem without providing the power level at the time of trip? This seems a bit artificial, like you are withholding obvious information just to try to make the question work.
Addressed. MAB 29NOV2011
"Not tripped" does not appear to be plausible. When all turbine stop valves are closed, then the turbine is effectively tripped, regardless of what the basis document states.
Addressed. MAB 29NOV2011 2 008AK2.03 N H 2 X E S What other meaningful pressure values can be used other than 1715? Your chosen value seems so low that it becomes not plausible.
It does not make ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 2 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only sense that the PORV would be allowed to remain open until well after your reactor trip setpoint of 1844 psig and at the point of SI.
This question will be satisfactory if you can address this plausibility concern with two of the distractors. This question will be considered an E due to the fix being relatively easy, although technically two answer choices are affected.
2185 psig may be a better choice to consider
- at least it is prior to a reactor trip and SI. Addressed. MAB 29NOV2011 3 009EK2.03 B H 2 S No comments.
4 011EK1.01 N F 2 U S This question will not be counted toward the number of unsat questions. The KA requires testing Large Break LOCA knowledge. I am not sure how natural circulation cooling and reflux boiling impact a LBLOCA. The question was written for a very small LOCA
- a break size for which NC cooling / reflux boiling actually makes sense. I think a new KA is required. We can discuss further if necessary, otherwise you will need to write a question to the replacement KA.
Addressed. MAB 29NOV2011 NEW KA: 011EK2.02 (IR=2.6)
Addressed. MAB 29NOV2011 5 015AK2.08 N H 2 S Previously reviewed an d approved. No further comments.
6 022AA2.03 B H 1 2 x x x x U S Question is backward logic, which presents some of the following concerns. Addressed. MAB 29NOV2011 How do you know what HAS caused the alarm?
Could a different failure cause 113B to fail closed?
Would it be more appropriate to test what a possible, or viable, single failure could cause the alarms?
Addressed. MAB 29NOV2011
The second part of "B" and "C" is not needed
- it is just extra information. Those two choices should just state that the air line has been completely severed. Delete ", causing the valve to fail CLOSED."
Addressed. MAB 29NOV2011 How does letdown have any impact on the two stated alarms? These alarms appear to be only applicable to the makeup. "D" does not appear to be plausible. Discuss with licensee to enhance understanding of plausibility.
Addressed. MAB 29NOV2011
How would the charging pu mp suction impact the two stated alarms?
ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 3 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only Discuss plausibility of "A
". Also, since the applicant is forced to assume failures, is there a failure or combination of failures that could make "A" potentially correct.
Addressed. MAB 29NOV2011 7 025AK3.01 N F 2 E S Discussion with previous CE noted.
This question appears to be (F)undamental LOK, not (H)igher. Memory item for the reasons for doing the steps. Discuss.
Addressed. MAB 29NOV2011
No further comments. Intent of the KA is considered to be met.
8 027AA1.05 N H 2 S No comments.
9 029EG2.4.49 B F 2 x E S In the correct answer, why would the applicant push the GV Down and GV Fast buttons if the turbine manual pushbutton was successful?
Are the applicants forced to make an assumption here? NUREG
-1021 states that applicants are not to make assumptions, therefore they could conclude the turbine trip was successful.
OK - CE read incorrectly.
The question statement asks for the next (singular) required action, yet the correct answer is a list of several actions. Same comment can be applied to other answer choices where more than one action is provided.
Addressed. MAB 29NOV2011
What immediate actions have already bee n performed? The stem does not provide this information. This forces the applicant to make assumptions as to where they are in the procedure. This creates multiple correct answers.
Reviewed - OK. MAB 29NOV2011
The question wording and presentation needs to be tightened to ensure one and only one correct answer.
Reviewed - OK. MAB 29NOV2011 10 038EK1.04 B F 2 E S The licensee should produce supporting documentation for flow returning to the core via the hot leg. The provided documentation only states that reflux boiling will occur, but does not define that reflux boiling is entirely limited to returning flow to the core via the hot leg.
Discussions and further documentation satisfied CE
- OK. MAB 29NOV2011 11 054AA1.04 N F 2 x E S The procedure has the operator start at least one SI pump and open both PORVs. Will opening one PORV provide adequate cooling, which is what the question is asking? No supporting documentation is provided to ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 4 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only support that two PORVs are required for adequate cooling. This question will be rated as satisfactory if sufficient documentation can show one and only one correct answer.
Supporting documentation reviewed
- OK. MAB 29NOV2011 12 055EG2.1.31 B F 2 S Previously reviewed and approved. No further comments.
13 056AK1.03 N H 1 2 x E S Plausibility needs to be enhanced by using 1785 psig without converting to psia. For example, 61.85 F would be the resultant SCM when 1785 psia is used in conjunction with 558F.
Addressed. MAB 29NOV2011 Also, does your temperature display have a mechanism for notifying an operator that the T/C reading is not reliable and should not be used (I.E. turns a different color)? If so, I would like this to be used along with failing to convert the 1785 psig to psia.
No. MAB 29NOV2011 In summary, make the answer choices similar to the following convention:
A. Lowest P and Highest Temp B. Lowest P without converting to psia and Highest Temp C. Lowest P and Lowest Temp (or preferably a higher invalid T/C temp) D. Lowest P without converting to psia and Lowest Temp (or preferably a higher invalid T/C temp)
Addressed. MAB 29NOV2011 The plausibility can be fixed with a small effort, therefore this question is rated as an "E".
Addressed. MAB 29NOV2011 14 057AA2.04 N F 2 S No comments.
15 058AA1.03 N H 2 X E S Why are so many columns used? Generally good question writing practice would suggest that only the info needed to make 4 unique answer choices should be used. With that stated, can the answer choices be limited to only the first and last column (SI Actuation and Exciter Field Breaker)?
The extra information typically just weakens the plausibility of the distractors by providing additional ways to disqualify those distractors. Addressed. MAB 29NOV2011 16 065AA2.05 B F 2 x E S IA pressure is 83 psig. Therefore, I am having difficulty understanding how MFR valves would be affected. Typically MFR valves will control at most plants at pressures much lower than 83 psig (maybe even down to 50 or 60 psig).
Addressed. MAB 29NOV2011 ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 5 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only Add "to" prior to the blank.
Addressed. MAB 29NOV2011 "C" is not plausible. Everything in the stem points to instrument air as being the problem (Instrument Air Alarms are annunciating). So if IA is the problem, taking manual control does not have much credibility.
Addressed. MAB 29NOV2011 17 W/E 05EK3.1 N H 2 E S In a loss of heat sink scenario, how long would it take to get to 25% wide range SG levels
- assuming EOC, long operating run, trip from full power? After this time has elapsed, what would pressurizer level likely be? Is 12% a reasonable pressurizer level for these circumstances?
I s this situation operationally valid?
Addressed. MAB 29NOV2011 18 W/E 12EK3.3 N H 1 2 x U S Why would locally throttling AFW flow be plausible when nothing in the stem would cause an applicant to doubt control of AFW from the control room. Some complication needs to be added to the stem to make A(2) and B(2) plausible.
There may be several ways to accomplish this
- like failing a power supply that has no impact, but would add credibility.
Addressed. MAB 29NOV201 1 19 005AK2.02 B L 2 E S Add to the question stem, "in accordance with AOP
-001." Addressed. MAB 29NOV2011 20 028AA2.04 B H 2 x U S Why are three columns of information provided in the answer choices? Generally good question writing practice would suggest that only the info needed to make 4 unique answer choices should be used. Suggest deleting the last column because it does not help to distinguish the answer choices. Because of this knowledge required by the KA is not needed to answer the question. I can answer the question with or without that third column just by knowing how the two level indicators respond.
Addressed. MAB 29NOV2011 To try to fix the question, I would suggest breaking the reference leg on the LT that is not selected for control. Test how that LI responds with a broken reference leg and then test how the "C" charging pump speed controller responds. By doing this, the KA is met and the previous comment would also be addressed.
Addressed. MAB 29NOV2011 21 032AA2.09 B H 2 S Suggest deleting the extra words in the answer choices unless they are necessary to remain. I.E. A(2) pulse height discriminator; B(2) detector.
Addressed. MAB 29NOV2011 22 033AG2.2.44 B H 2 S No comments.
ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 6 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only 23 060AK1.02 N F 2 x U S Why is all the peripheral information provided to start out the question? Delete all of the unnecessary information.
Window dressing that has no material impact does not help match the KA. The only thing it does is add more for the applicant to read. In this question you are asking for parts of the definition of a DAC and an ALI
. I do not think that only testing knowledge of those two definitions is testing the operational implications part of the KA.
Addressed. MAB 29NOV2011 Suggestion: Any operational implication could be tested. This could include procedural actions based on exceeding an exposure limit, etc.
You do not need to test every aspect of the list provided in the KA, but you do need to test the operational implications.
Addressed. MAB 29NOV2011 24 067AA1.06 N H 2 x E S "C" is not plausible. "A" is a subset of "C"; therefor e, if "C" was correct, then "A" would also be correct. A guessing man would always choose the smaller of answers "A" and "C".
Addressed. MAB 29NOV2011 "D" is not plausible. "B" is a subset of "D".
Addressed. MAB 29NOV2011 Suggest adding words in parentheses to "A" and "B" stating that placing control room in pressurization mode is not required. This will address the subset issue.
Addressed. MAB 29NOV2011 Question was rated as "E" due to the simplicity of the needed corrections, even though there were two distractors that were not plausible, Addressed. MAB 29NOV2011 25 076AK2.01 N H 2 E S No documentation was supplied that supports both R
-11 and R-12 having elevated readings. The question is likely satisfactory, but to ensure the technical accuracy, some documentation needs to be supplied to show that both of these would alarm.
Documentation reviewed. OK MAB 29NOV2011 26 W/E 08EG2.4.2 N F 2 E S Currently the question stem asks for what conditions will meet the entry conditions. This causes the two choices with 320F to not be plausible, because if 320F meets the criteria, then 290F will obviously also meet the criteria. This can be addressed by testing specifically what the procedure (Status Tree) states for P.1 entry criteria. I.E. The CSFST for CSF
-4 states that -. Addressed. MAB 29NOV2011 ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 7 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only 27 W/E 10EK3.1 N H 2 S No comments.
28 003A1.07 N H 2 x E S Appendix E of NUREG
-1021 directs the applicants to not assume anything. The question directly contradicts the Appendix E direction by stating that they are required to assume. Is there a credible pressure control malfunction that results in a linear pressure decrease?
Addressed. MAB 29NOV2011 Suggestion: Discard the math and just test the values.
This may also allow the stem to be simplified. Testing the values for when actions need to occur and then testing what those actions are will meet the KA.
Addressed. MAB 29NOV2011 29 004K5.30 B H 2 X? U S Some supporting information was supplied with the question, but no justification for plausibility was written in the question analysis. The correct answer makes perfect sense to me, but I am struggling to see plausibility in the other answer choices. I placed a question mark on the question rating because no justification was supplied, so maybe there is something I am missing. Plausibility will need to be discussed to see if the question is acceptable.
Addressed. MAB 29NOV2011 Suggestion: Test controller manipulation or controller response. How does the operator adjust letdown pressure if being controlled in manual and how would the operator adjust temperature.
Addressed. MAB 29NOV2011 Suggestion: How does the plant respond to a failure of valves or controllers for the bypass around the RHR heatexchanger? Then how does the letdown controller respond (demand increases or decreases)?
Addressed. MAB 29NOV2011 30 004K6.14 N H 2 S Previously reviewed and approved. No further comments.
31 005K3.01 N H 1 2 x x U S In the stem
- how many RCPs have been secured? Would it be better to state that "all" RCPs have been secured?
OK after reading. MAB 29NOV2011 Do procedures allow more than one RHR pump to be in operation? The stem states that the operating RHR pump trips. Does this leave open the possibility that an RHR pump is still operating?
Addressed. MAB 29NOV2011 Delete all the extra unnecessary stuff in the answer choices
- the only effect it has is to reduce plausibility. Just include the amount of ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 8 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only information needed to make 4 unique answer choices. The answer choices should look something like:
A. RCS pressure will rise; RCS temperature will rise B. RCS pressure will lower; RCS temperature will rise C. RCS pressure will rise; RCS temperature will lower D. RCS pressure will rise; RCS temperature will lower Addressed. MAB 29NOV2011 There may be an overlap issue with Q 29 (004K5.30). Knowledge from Q29 includes knowing that an increase in temperature will result in an increase in pressure during solid operating conditions. This question tests what will happen to RCS pressure when your cooling pump is tripped (I.E. temp increase). I understand the argument that the pump has tripped and this could impact pressure
- but there still may be some overlap / double jeopardy issues.
Addressed. MAB 29NOV2011 No explanation was provided to help me understand why a temperature decrease is plausible
. Currently I do not see plausibility in temperature lowering when my cooling pump trips and stops moving coolant. Therefore, unless convinced otherwise, "C" and "D" are not plausible.
Addressed. MAB 29NOV2011 32 006K5.10 B H 2 x E S Do procedures ever allow terminating SI when termination criteria are not met? If so, then I will view "C" as plausible, if not, then "C" will not be plausible and will need to be modified or replaced.
OK after discuss ions. MAB 29NOV2011 33 007A1.01 N F 2 E S Why is all the stuff at the beginning of the question provided? What importance or function does it have? Would this be the same question if it started with Which ONE (1) of the following?
Addressed. MAB 29NOV2011 Considering the above comment, is this a (F)undamental LOK vs. a (H)igher LOK? It seems like the applicant just needs to recognize the setpoint and know the basis, which would be a lower cognitive question.
Addressed. MAB 29NOV2011 34 007K3.01 N H 2 x U S Analysis of answer choice "A": What would PRT pressure do if CL Inj RV failed open? Your analysis states that the high PRT pressure alarm wi ll not come in
- is this true?
Addressed. MAB 29NOV2011 None of the second set of conditions is needed for this question. Delete all of it unless there is a reason that it needs to stay. Remember that adding unnecessary information does not help match a KA, etc. If it is not needed ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 9 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only to answer the question, then it has no impact on the KA match.
Addressed. MAB 29NOV2011 Modification Idea: I would consider deleting the PRT Hi Temp alarm to make the source the CL inj RV and test whether sump level will go up and whether R-2 will rise.
Addressed. MAB 29NOV2011 Are there any differences in indications between a PORV failing open and a Safety Valve failing open?
Addressed. MAB 29NOV2011 The question states, "assuming no operator action." Appendix E disallows assumptions. Simply state that no operator actions have taken place, rather than telling the applicants to make an assumption. I would suggest doing a word search on the entire exam and try to remove the word assume wherever possible.
There is no harm in stating that no operator actions have occurred, but this is also part of the rules for taking the test as stated in Appendix E.
Addressed. MAB 29NOV2011 35 008G2.4.50 N H 2 x U Test what the operator is "required" to do
- not what he will do. Do a search of the entire exam (I know the word will appears elsewhere). Change the second part of the question prefix to: APP
-002-E5 requires the operator to _______________.
Make similar changes throughout the exam. We must test what operators are required to do because who knows what they will do.
Addressed. MAB 29NOV2011 Distractors A(2) and B(2) are not plausible.
From a common sense perspective, a safety related pump has no cooling during a surveillance test (non-emergency), so it only makes sense to protect that pump by stopping it. There is not much plausibility here.
Addressed. MAB 29NOV2011 36 010K1.03 B H 2 S No comments.
37 010K4.01 B F 2 x E S Is "A" arguably correct? Will water continually pass through the spray nozzle? Could the nozzle be considered full? How is full defined for the spray nozzle? We need to ensure that "A" is not a correct answer choice. If needed, it will be replaced. We can discuss if you have the documentation that supports it as being incorrect. I have concerns because the mechanism for reducing thermal stresses is to have a continuous stream of water. Are we then testing the definition of what a full spray nozzle is to differentiate between "A" and "D"?
Addressed. MAB 29NOV2011 ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 10 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only 38 012A3.05 N H 1 2 X U S "C" and "D" not plausible because they are not mutually exclusive answer choices. Knowing that there cannot be two correct answers makes these two choice not plausible.
- How are "C" and "D" different? N
-42 is bypassed for both answer choices and power is above P
-10 for both answer choices. N
-42 being bypassed would play into both answer choices, thereby creating a subset issue. These two answer choices are not mutually exclusive. Suggest making "D" simply: "Reactor will not trip." Then replace "C" to another credible mechanism that could trip the reactor.
Addressed. MAB 29NOV2011 39 013K2.01 B F 2 S No comments.
40 022A3.01 B H 2 S No comments.
41 022A4.05 N F 2 S No comments.
42 026K1.01 N F 2 U S The second part of the answer choices re confusing. It appears that the "A" RHR pump will in fact supply suction to allow operation of both spray pumps. It is true that the "B" spray pump does not have power, but its operation is not disallowed due to the RHR supply.
It also strikes me as strange to see "operation" underlined here. Seeing that underlined may, in fact, validate my comment here in that the RHR supply has no impact on the operation of the Spray Pump. This ambiguity could lend itself to more than one correct answer because the RHR supply does allow operation of both spray pumps.
Addressed. MAB 29NOV2011 This question appears to test two knowledge items: (1) the power supply to the "B" Spray Pump, and (2) the position of SI
-844A&B. Q Modified. MAB 29NOV2011 43 026K3.02 N H 1 2 x U S RCS water is always acidic. PWRs use boric acid as one of the two primary means of reactivity control that are available to licensed operators.
So when faced with a situation where chemicals cannot be added due to the Add Tank Outlet failing closed, why would an operator believe that the sump water would be caustic? If Robinson also uses TSP baskets in the sump or something like that, then there may be some plausibility for the water to be caustic.
Addressed. MAB 29NOV2011 I also struggle in seeing plausibility in the gas binding.
Addressed. MAB 29NOV2011 44 039K4.06 N F 2 S No comments.
ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 11 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only 45 059A2.04 M F 2 S Previously reviewed and approved. No further comments.
46 059A3.02 N F 2 S No comments.
47 061K5.01 B H 2 E S It looks like this question was used on a recent NRC exam (ILT 11
-1). What methodology was used to select this question? How did you ensure that you randomly selected among all of your bank questions that met this KA? OK MAB 29NOV2011 No comments on the question itself.
48 06 1K6.02 B H 2 S No comments.
49 062A2.1 0 N H 2 S No comments.
50 063A1.01 N F 2 U S The question statement does not address both parts of the question
- it only asks for the time limitations. The question should solicit everything being asked for in the fill in the blank and answer choices.
Addressed. MAB 29NOV2011 C(1) and D(1) are not plausible. You have provided the applicants with two kinds of loads to be shed, both of which would reduce load on the batteries. It is not credible that the applicant would choose to shed an inverter when the other choice is a LOW priority load.
Addressed. MAB 29NOV2011 51 064K6.07 N F 2 E S What benefit does it add to test their ability to extrapolate to a point in time that coincides with a certain value? This is at least the third question that has done this. Does your air compressor raise pressure at a linear rate of 1 psig/min? If not, then this is not an operationally valid question. Also, would you expect an operator to know the rate at which a compressor will charge your EDG air receivers and then calculate the time that will support 8 cold starts?
Is this testing meaningful information?
Addressed. MAB 29NOV2011 I see no supporting documentation that supports the air receivers raising pressure at 1 psig/min.
Addressed. MAB 29NOV2011 Suggest just testing the various pressures (100, 210, 216, and 220).
Addressed. MAB 29NOV2011 52 073G2.2.12 N H 2 E Add to the question statement that you are asking the question with the ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 12 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only S CKT TEST pushbutton depressed iaw OST
-924-2. I.E. Given the above conditions, which one of the following completes the statement?
Addressed. MAB 29NOV2011 53 076G2.4.18 N H 2 E S ONLY does not work in this case. Will PATH
-1 have the operators close ANYTHING else? More precise language needs to be used here. I understand why you tried to use ONLY, because just closing 16A, without also closing 16B, will not isolate SW to the turbine building. By wording it in this fashion, there is also an argument that there is no correct answer because closing 16C is not the only valve that PATH
-1 will direct to be closed. Addressed. MAB 29NOV2011 54 078K2.02 N F 2 S No comments.
55 103A4.09 B F 2 S It looks like this question was used on a recent NRC exam (ILT 11
-1). How was this question selected ES-401: If the bank contains more than one question that fits a specific K/A statement, randomly select from among the available questions unless there is an appropriate basis for selecting a specific question (e.g., higher cognitive level, better discrimination validity, more operationally oriented, or site
-specific priority).
OK MAB 29NOV2011 No further comments.
56 001A3.05 N H 2 S No comments.
57 011K2.02 N H 2 S No comments.
58 01 5K6.01 N H 2 x E S "D" is not plausible. The interplay between the first and second half does not make sense. If the lower section fails low, then all (or most) power would be indicated in the top. Therefore making the delta high. Even if an applicant has a misconception that the delta was an absolute value, the number would still be high, not low.
Addressed. MAB 29NOV2011 "A" also has minimal credibility.
Addressed. MAB 29NOV2011 The alarm in question should be referred to by its official designation and name as you have on other questions. What is the official designation for the Section Deviation Alarm?
Addressed. MAB 29NOV2011 59 016K3.09 B H 1 2 x U S "A" and "C" are not plausible. We are evaluating high dP between steam lines. Therefore, if the same failures, whether it be high or low, occur on ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 13 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only each steam line, then how does that impact differences between the steam lines? Addressed. MAB 29NOV2011 60 028A4.02 N F 2 S No comments.
61 041G2.4.11 N H 2 E S "D" does not appear to be plausible. No justification is supplied for its plausibility. Would BYPASS TAVG INTERLOCK ever be used at power with higher RCS temperatures?
Addressed. MAB 29NOV2011 62 071K1.06 B F 2 S No comments.
63 072K5.01 N H 2 x U S Why is the second part of the answer choices included? Just with part (1) alone, you have provided 4 unique answer choices, therefore none of part (2) is needed to arrive at the correct answer. Only information that is needed to make that answer choice unique should be included. Extra information only acts to lower the credibility of the distracters by providing additional ways to disqualify that distractor. Knowledge of the KA must be required to arrive at the correct answer in order to meet the KA. In this case, all I need to know is what is detected and how does the instrument display it. Operational implications needs to be tested to meet the KA.
Addressed. MAB 29NOV2011 64 079K4.01 N H 2 S No comments.
65 086A1.05 N F 2 x x E S The supporting documentation suggests that the MDFP may not start until 95 or 96 psig. The documentation suggests that the MDFP could start anywhere between 95 and 105 psig. Does this mean that it is possible that someone could argue that "A" is an alternate correct answer?
Addressed. MAB 29NOV2011 "C" is not plausible. I am having trouble believing that someone would think that an engine would be started before an electrical motor would be used. Addressed. MAB 29NOV2011 66 G2.1.15 N F 2 x x U S OPS-NGGC-1000, Section 9.17.5.1 (f) states that for site
-specific standing instructions may use a database OR Attachment 8 (which could be placed in a book)
. The question appears to pertain to site
-specific standing instructions
- therefore, could "A" be an alternate correct answer? Is there a procedure contradiction here? Does this warrant a procedure change request? I would think that most standing instructions would be site specific; therefore (f) would be the applicable section to follow.
Addressed. MAB 29NOV2011 ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 14 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only "B" is not plausible. I understand what you have used for plausibility justification, but it still does not seem credible that someone could pull a procedure off the shelf and begin to use it
- not knowing that it has been altered by a standing order. It is only logical to believe that the procedure itself must be changed to ensure that a task is performed correctly.
Addressed. MAB 29NOV2011 67 G2.1.17 B F 1 2 x U S "C" is not plausible. This is a plant announcement
- an announcement to the entire plant. When an announcement is made to the entire plant, would it be credible that everyone be required to respond?
Addressed. MAB 29NOV2011 "D" also is not plausible.
Nothing in the stem even indicates that the TSC and EOF have been activated. Even if the EOF was activated
- it is located off site. Why would there be a requirement for the announcement to be in a continuous do
-loop until an off
-site acknowledgement is made for an announcement?
Addressed. MAB 29NOV2011 For a plant announcement, what mechanism exists for three way communications? Three way communications typically is done only one to one or in small groups.
Addressed. MAB 29NOV2011 I also noted that this question was used on the 2007 NRC exam.
OK. MAB 29NOV2011 68 G2.1.27 B F 2 x x E S The wording of this question does not sound right to me. The design function of the ICCM is a SB LOCA?
I think what you are trying to test is that the ICCM is designed to monitor for inadequate core cooling during a SB LOCA. I think some modification to the wording of the question and/or answer choices is necessary.
Addressed. MAB 29NOV2011 "C" is not plausible. A steam line break will cool the core
- cooling will not be inadequate.
Addressed. MAB 29NOV2011 69 G2.2.6 B F 1 2 x U S What is PI
-2089 at Robinson?
Modified. MAB 29NOV2011 "B" is not plausible. The procedure in question is an AOP. Therefore, the plant is in an abnormal condition
- potentially on a backshift. Is it credible to think that the procedure owner would be required to approve the temporary procedure change?
ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 15 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only Addressed. MAB 29NOV2011 "C" is not plausible for similar reasons. How long does the normal procedure change process take? The plant is in an abnormal condition. Is it reasonable that a typo would prevent the crew from taking care of the plant? Addressed. MAB 29NOV2011 70 G2.2.7 N F 2 x E S This type of information is usually tested on the SRO portion of the exam. Is this RO required knowledge at Robinson? Is there an RO learning objective for this information? If this question appears in its current form in the final submittal, then it will be considered as RO knowledge at Robinson. Licensee assures that this is RO knowledge at HBR. OK MAB 29NOV2011 No comments on the material aspects of the question.
71 G2.3.13 B F 1 2 x U S Nuclear Shift Manager does not appear to contain plausibility for a LHRA entry approval. A normal evolution is occurring and the issue is with high radiation levels. Suggest replacing the approval part. What about equipment verifications in a high dose area
- are their alternate methods that can be u sed to maintain does ALARA?
Addressed. MAB 29NOV2011 Question was on the ILC
-09 NRC Exam.
OK MAB 29NOV2011
How was this question selected? Have the licensee walk through in detail, the selection process used for this question.
OK MAB 29NOV2011
NUREG-1021, ES-401, Page 8 of 33, states, "If the bank contains more than one question that fits a specific KA statement, randomly select from among the available questions unless there is an appropriate basis for selecting a specific question (e.g., higher cognitive level, better discrimination validity, more operationally oriented, or site
-specific priority)."
OK MAB 29NOV2011 72 G2.3.4 B F 2 E S Why are three columns of information included?
Only include information in the distractors that is needed to make the answer choices unique. In this case you can completely delete the first column (Extremities) and you still have 4 unique answer choices. The only purpose the extra information serves is to reduce the plausibility of distractors by providing more ways to disqualify the distractors.
ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 16 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only Addressed. MAB 29NOV2011 73 G2.4.25 B F 2 S Previously reviewed and approved.
74 G2.4.39 2007 NRC Exam B F 2 S No comments.
75 G2.4.9 N H 2 E S How would the operators know that there were no indications that the leak was in the RHR system? No sump/rad alarms in aux bld would not tell the entire story? I just want to ensure that the conditions provided are operationally valid to place them in the right procedure location to support the correct answer.
Addressed. MAB 29NOV2011 SRO EXAM 76 038EG2.4.11 N H 2 S No comments.
77 054AG2.4.11 N H 2 X? X? E S Why is 2385 MWth plausible? What documents refer to 2385 MWth? No documentation was provided to justify its plausibility. (I am assuming 102% of rated?) There may be two acceptable options: (1)To enhance plausibility, consider making UFMs out of service and iterate on 2346 MWth and 2300 MWth. OR (2) provide supporting documentation for the plausibility of 2385 MWth.
Addressed. MAB 29NOV2011 Why is OP-105 an incorrect choice? I see no explanation or documentation with the question that supports OP
-105 being incorrect. Consider iterating on (1) AOP
-10 contains steps to reduce power. AND (2) AOP-10 does not contain steps to reduce power.
Addressed. MAB 29NOV2011 The last sentence in the analysis of answer choice B does not appear to be correct. It states that 2385 MWth is correct
- yet the correct answer has 2346 MWth as correct.
Addressed. MAB 29NOV2011 Knowledge of max licensed power limits is RO knowledge
- after all the ROs are operating the plant.
Addressed. MAB 29NOV2011 78 055EA2.01 B H 2 x x U S KA/SRO-only: The KA is not met at the SRO level. EPP
-1 is a major EOP as defined in the SRO guidance. RO knowledge can be used to know that EPP-1 does not apply. Knowing failure modes for a valve is RO ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 17 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only knowledge.
Addressed. MAB 29NOV2011 Could e-plan be used to hit the KA at the SRO level?
Addressed. MAB 29NOV2011 How does the question map through the SRO guidance document?
Addressed. MAB 29NOV2011 79 058AA2.03 N H 2 x U S SRO-only: If the plant conditions do not result in an SI, then using RO knowledge, the applicant would conclude that EPP
-7 is not a valid answer choice. Then system response knowledge is the only thing needed to analyze the second part of the answer choices also.
Addressed. MAB 29NOV2011 How does the question map through the SRO guidance document?
Addressed. MAB 29NOV2011 80 W/E 04EG2.4.3 N H 2 S Previously worked with the licensee to develop a satisfactory question.
81 W/E 05EA2.1 N H 2 x U S SRO-only: Red path procedure entry is RO knowledge. Mitigating strategy is RO knowledge. No SRO level procedure selection is tested in the question. Part of the H.1 strategy is to limit heat input to the RCS
- I.E. stop RCPs. Testing details of a procedure is not SRO knowledge. Details of a procedure could only be SRO only knowledge if that information is needed to make a procedure selection. In this case, stopping an RCP is not a procedure selection
- it is a single action that also happens to be part of the overall mitigating strategy.
Addressed. MAB 29NOV2011 How did this question map through the SRO guidance document?
Addressed. MAB 29NOV2011 82 W/E15EG2.1.32 N H 2 E S Prior to the STA reporting to the control room, who is responsible for monitoring safety function status trees? Where are these responsibilities defined in administrative procedures? At most plants, monitoring of safety functions is RO required knowledge. Does Robinson have an SRO
-learning objective for knowing the hierarchy of safety functions?
Addressed. MAB 29NOV2011 I see no supporting documentation to justify the second part of the answer choices. Before this question can be rated as satisfactory, I will need to see documentation that justifies plausibility and incorrectness of distractors as well as the correctness of the answer. The basis for the prioritization potentially could be used to justify SRO
-only because the guidance document is silent on EOP basis information.
ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 18 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only Addressed. MAB 29NOV2011 83 037 AG2.4.41 N H 1 2 x E S Do Robinson procedures actually allow a SG to be isolated prior to shutting down? If a SG is isolated at 100% power, how would the plant respond? My guess is that isolating the C SG at 100% power would result in a trip. Therefore, the first part of C and D are not plausible unless the licensee can provide sufficient justification for plausibility.
Addressed. MAB 29NOV2011 Have the licensee show the leakrate calc that results in greater than 77 gpm leakage. Addressed. MAB 29NOV2011 84 076AA2.05 N H 2 S No comments.
85 W/E 09EA2.1 B H 2 ? ? ? S The display of information is a little confusing. The second to last bullet states that it is 15 minutes after the trip. Is the last bullet also 15 minutes after the trip? The second bullet lists conditions after the trip, yet the third bullet list conditions at the time of trip. To clarify, consider providing sets of conditions at various times
- like INITIAL CONDITIONS, then CURRENT CONDITIONS.
Addressed. MAB 29NOV2011 In this case, I believe that procedure selection is being tested in that direction for controlling temperature is found in both procedures. The applicant must know more than just one action and the applicant must know more than the mitigation strategy. Also, more than just entr y conditions for the AOP and major EOPs are needed.
OK MAB 29NOV2011 The KA must be met at the SRO level. The KA requires selection of procedures for natural circ. How is the procedure selection in the form of temperature control associated with natural circ. It seems like the procedure selection may just be associated with a loss of air. Discuss with licensee to enhance my understanding.
OK MAB 29NOV2011 86 003A2.03 N H 2 S Previously worked with the licensee to develop a satisfactory question. 87 007A2.05 N H 2 S Previously worked with the licensee to develop a satisfactory question.
88 022G2.4.50 N H 2 ? x ? U S The question tests what "should" be done. Why would it be wrong to stop HVH-3 when it has high vibes? Would it be possible for me to find an admin procedure that would allow an operator to secure a piece of equipment to protect it against further damage? The concern here is having one and only one correct answer. Why would C really be wrong?
ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 19 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only Suggestion: Reword the first part of the question: To reset the High Vibration Alarm, APP
-002-A7 directs: stopping HVH
-3 prior to resetting the vibration alarm vs. resetting the high vibration alarm without stopping HVH-3. Addressed. MAB 29NOV2011 Does a motor calculation error make the "B" CV Pump inoperable? Does it depend on the specifics of the calculational error? How does your procedures define OOS? Can a piece of equipment simply be off and be considered OOS?
Addressed. MAB 29NOV2011 Does the High Vibe alarm make HVH
-3 inoperable? The question is set up to leave HVH
-3 running, therefore, would it be wrong to assume operable until an operability recommendation could be produced by engineering? If it is clearly inoperable, then why would an operator try to restart them as stated in two of the distractors?
Addressed. MAB 29NOV2011 89 059A2.03 N H 2 E S Why is the second part of the distractors incorrect? I do not see any documentation that supports Supplement G being incorrect. Discuss with licensee to ensure that there is only one correct answer.
Addressed. MAB 29NOV2011 H.3 is a yellow path. The question does not include a reference for the applicant. Is yellow path info required closed book knowledge at Robinson? Addressed. MAB 29NOV2011 90 064G2.4.46 N H 2 S Previously worked with the licensee to develop a satisfactory question.
91 011G2.2.38 N H 2 S No comments.
92 045A2.11 N H 2 S No comments.
93 029A2.01 N H 2 E S Add "in accordance with Tech Specs" to the end of the first question statement.
Addressed. MAB 29NOV2011 Add "in accordance with ODCM" to the end of the first question statement.
Addressed. MAB 29NOV2011 94 G2.1.31 N H 2 S Previously worked with the licensee to develop a satisfactory question.
95 G2.1.41 N F 1 x X? U RO knowledge can be used to disqualify AOP
-013. AOP entry conditions ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 20 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only 2 S are RO required knowledge. ROs also likely move fuel in the spent fuel pool at various times, where AOP
-013 would still be applicable.
Addressed. MAB 29NOV2011 ROs are licensed to move fuel. Union contracts, etc., may allow contractors to do it, but those contracts are not permanent restrictions. Does Robinson require fuel movers (whether they be ROs or contract help such as Westinghouse) to know where to place an assembly when level is lowering? I would want to see the learning objectives for fuel movers to ensure that this is not required knowledge. If fuel movers are not required to know this, then it may be OK to allow fuel placement requirements to be SRO-only knowledge.
Addressed. MAB 29NOV2011 Placing an assembly in its original location is always correct - it was OK for the assembly to be there prior to lifting it, so it is fairly basic to understand that it would also be OK to put it back in the same spot. This causes B(1) and D(1) to be not plausible.
Addressed. MAB 29NOV2011 96 G2.2.35 N H 2 E S Suggest changing A(2) and B(2) to "Allowed". I do not think conditionalizing the answer choices adds plausibility.
Addressed. MAB 29NOV2011 97 G2.2.42 N H 2 x x U S A reference is required to be provided to the applicant if testing greater than one hour Tech Spec actions. I think the reference will cause the Mode 3 entry piece to not be plausible.
Addressed. MAB 29NOV2011 Also, the KA does not appear to be met at the SRO level. The Tech Spec entry is the part that meets the KA, but that is RO knowledge.
Addressed. MAB 29NOV2011 Suggestion: write a question where the PORV is on its backup nitrogen supply. I would expect the RO to know that the PORV will work, but it would be the SRO that would be required to know whether the PORV was operable. This is just one idea
- there may be others.
Addressed in another way. MAB 29NOV2011 98 G2.3.4 B H 2 x x U S The KA is not met at the SRO level. Radiation exposure limits are being tested at the basic rad worker (or RO) level. To hit the KA at the appropriate level, I would suggest testing emergency dose approval authorization for saving life or equipment. This is a common SRO question for this KA.
Addressed. MAB 29NOV2011 ES-401, Rev. 9 Written Exam Review Worksheet Form ES-401-9 Robinson 20 11-302 Page 2 1 of 21 Q# K/A# B M N L O K L O D Psychometric Flaws Content Flaws U E S Comment Explanation Stem Focus Cues T/F 1 Non Cred Dist >1 Non Cred Dist Partial Min B/W Q= K/A SRO Only The E-Plan classification is not related to the KA.
Addressed. MAB 29NOV2011 99 G2.4.30 N H 2 E S Does Robinson have a learning objective that supports asking this question in a closed book format? I am asking this to ensure that Robinson Training and Operations Management all agree that this is closed book knowledge at your facility. This question appearing in its current form in your Final Submittal will be viewed as the licensee officially agreeing that this is closed book knowledge for SROs at Robinson and therefore acceptable to appear in that manner on their NRC exam.
Addressed. MAB 29NOV2011 100 G2.4.43 N F 2 x E S This question tests what the HOO will do. Maybe the licensee should not test what the HOO might do? The HOO is not required to follow your OMM procedures. Would it not be possible that the HOO could ask if you were ready for the code, or wanted the code, prior to providing it? Maybe they would do that to ensure you were ready to receive the information. Would that be wrong?
Suggestion: Test where the code is stored or whether or not the code is safeguards information, or something of that nature. The second part of the answer choices are OK. Just the first part needs some work. There appears to be enough info in the OMM procedure to test something that will ensure only one correct answer to a further extent.
Addressed. MAB 29NOV2011
ROBINSON 2011
-302 FINAL SAMPLE PLAN The final sample plan is the combination of the Form ES
-401-4 and the Draft Sample Plan.