ML102500676
Text
Issue Date: 03/23/09 1 0307
NRC INSPECTION MANUAL DIRS MANUAL CHAPTER 0307 REACTOR OVERSIGHT PROCESS SELF-ASSESSMENT PROGRAM
0307-01 PURPOSE
The Reactor Oversight Process (ROP) is a regulatory framework t hat includes licensee performance indicator (PI) data, NRC inspecti on activity and determination of inspection finding significance, and licensee performanc e assessment. The ROP self-assessment program evaluates the overall effectiveness of the ROP through its su ccess in meeting its preestablished goals and intended outcomes.
0307-02 OBJECTIVES
02.01 To establish the processes for colle cting information and dat a to support the ROP self-assessment program.
02.02 To establish a process for objectively evaluating the effectiv eness of the ROP in achieving the goals of being objective, ri sk-informed, understandable, and predictable as well as the applicable agency performance goals listed in the NRC
=s Strategic Plan.
02.03 To provide timely, objective in formation to inform program planning and to develop recommended improvements to the ROP. 02.04 To inform the Commission, NRC senior management, and the public of the results of the ROP self-assessment program, including any conclusions and resultant improvement actions.
0307-03 DEFINITIONS
03.01 Audit. A periodic examination and checking of selected records or activities to verify their correctness or compliance with predetermined standards.
03.02 Effectiveness. The ability to achieve the desir ed outcomes of an activity, program, or process. A program is considered effectiv e if it is meeting it s goals and achieving the intended outcomes. Ensuring that NRC actions ar e effective (e.g., high quality, efficient, timely, and realistic) is one of the organizational excellence objectives in the NRC's Strategic Plan and is used to evaluate the ov erall success of the ROP. Overall ROP effectiveness is measured each year through its self-assessment program using predefined Issue Date: 03/23/09 2 0307 metrics and performance evaluations of key progr am areas in accordance with this manual chapter.
03.03 Objective. A desired program attribute in which decisions are based on factual information and uninfluenced by emotion, surmise , or personal prejudice. Being objective is one of the four program goals of the ROP used to evaluate its overall success and effectiveness.
03.04 Openness. The ability to perform in a manner that is accessible and unobstructed in order to maintain public confidence. T he goal to ensure openness explicitly recognizes that the public must be in formed about, and have a reasonable opportunity to participate meaningfully in, the NRC
=s regulatory processes.
Ensuring openness in our regulatory process is one of the organizational excellenc e objectives in the NRC's Strategic Plan and is used to evaluate the overall success and effectiveness of the ROP.
03.05 Operating Plan. A management tool used to ensur e that planning, budgeting, and performance management are performed in an integrated and balanced manner. It includes a summary of NRR programs, projects, ac tivities, and other items to be measured throughout the year and the metrics which will be used to monitor them. Programs and activities are aligned under the five agency Strategic Goals.
03.06 Predictable. A desired program attribute in which more than one individual can follow the same defined process and arrive at t he same conclusion in a consistent manner (i.e., repeatable). Being predictable is one of t he four program goals of the ROP used to evaluate its overall success and effectiveness.
03.07 ROP Inspectable Area Lead. Person assigned responsibility to oversee and manage the use of individual baseline inspec tion procedures or attachments to those procedures.
03.08 ROP Program Area Lead. Person assigned responsibility to oversee and manage the associated programs for the major elements of the Reactor Oversight Process. The ROP elements are the performanc e indicator (PI) program, t he inspection program, the significance determination process (S DP), and the assessment process.
03.09 Risk-Informed. An approach to decision-making in which risk insights are considered along with other factors (such as engineering judgment, safety limits, redundancy, and diversity) to better focus lic ensee and regulatory attention on issues commensurate with their importance to health and safety. Being risk-informed is one of the four program goals of the ROP used to eval uate its overall success and effectiveness.
03.10 Survey. A sampling or partial collection of facts, figures, or opinions taken and used to approximate or indicate what a complete collection and analysis might reveal. The NRC utilizes internal and external surveys to gather stakeholder feedback and gauge stakeholder satisfaction with t he effectiveness of the ROP.
03.11 Understandable. A desired program attribute in which the process and its results are clear and written in plain English.
Being understandable is one of the four program goals of the ROP used to evaluate it s overall success and effectiveness.
Issue Date: 03/23/09 3 0307 0307-04 RESPONSIBILITIES AND AUTHORITIES
04.01 Director, Office of Nucl ear Reactor R egulation (NRR). Oversees and manages the Reactor Oversight Process.
04.02 Director, Office of Research (RES). Provides support and data as requested by the Director, NRR.
04.03 Director, Office of Nuclear Secu rity and Incident Response (NSIR). Provides support and data as requested by the Directo r, NRR, for the Security and Emergency Preparedness cornerstones.
04.04 Regional Administrators. Provide data to support t he ROP self-assessment program as requested by the Director, NRR.
04.05 Director, Division of Risk Assessment (DRA). Provides data to support the ROP self-assessment program as di rected by the Director, NRR.
04.06 Director, Division of Inspection and Regional Support (DIRS). a. Oversees the implementation of the ROP self-assessment program.
- b. Develops policies for t he ROP self-assessment program.
- c. Issues the annual ROP self-assessment report.
- d. Issues status reports to the Deputy Regional Administrators.
04.07 Chief, Performance Assessment Branch. a. Develops program guidance and procedures for the ROP self-assessment program. b. Ensures data from all s ources are collected and consoli dated to facilitate analysis.
- c. Recommends and implem ents improvements to the ROP self-assessment program. d. Monitors the effectiveness of corrective actions and improvements to the ROP that are developed in response to self-assessment findings.
- e. Develops the annual ROP self-assessment report.
04.08 ROP Program Area Leads. a. Collect self-assessment data each cal endar quarter for a ssigned program area (e.g., PI, inspection, SDP, and assessment).
Issue Date: 03/23/09 4 0307 b. Collect and analyze self-a ssessment data for the previous year, and develop the annual program evaluation for assigned program area.
04.09 ROP Inspectable Area Leads. a. Collect data and user experience for assigned inspectable areas and summarize the information for the annual self-assessment report.
- b. Annually review and evaluate the impl ementation of assigned inspection procedures.
NOTE: A complete listing of current ROP Pr ogram Area Leads and Inspectable Area Leads is available through the A points of contact
@ link from the ROP Digital City Web page.
0307-05 DISCUSSION
The ROP is the NRC's primary means of assu ring that commercial nuclear power plants are operated safely , securely, and in accordance with applicable regulations. The ROP consists of inspections, performance indica tors, significance determination processes, assessment, and enforcement. It is important that the ROP be per iodically evaluated and improved when necessary to ensure continued achievement of its specified goals and intended outcomes.
As noted in IMC 0308, "Reactor Oversight Proce ss Basis Document," the objectives of the staff in developing the various components of the ROP were to provide tools for inspecting and assessing licensee performance in a manner t hat was more risk-informed, objective, predictable, and understandabl e than the previous oversight processes. Accordingly, t he goals of the ROP include the four specific program goals of being objective, risk-informed, understandable, and predictable, as well as the applicable organizational excellence objectives (e.g., openness and effectiveness) from the NRC
=s Strategic Plan for Fiscal Years 2008-2013. Each of these ROP goals s upport the NRC's mission and characterize the manner in which the agency intends to achieve its strategic goals of safety and security:
to ensure adequate protection of public health and safety and the environment, and to ensure adequate protection in the secure use and management of radioactive materials.
The six ROP goals are summarized below:
Objective - decisions are based on fa ctual information and uninfluenced by emotion, surmise, or personal prejudice.
Risk-informed - risk insights are consider ed along with other factors (such as engineering judgment, safety limits, redundan cy, and diversity) to better focus licensee and regulatory attention on issues commensurate with their importance to health and safety.
Understandable - the process and its results are clear and written in plain English.
Predictable - more than one individual c an follow the same defined process and arrive at the same conclusion in a consistent manner (i.e., repeatable).
Issue Date: 03/23/09 5 0307 Open - the NRC appropriately informs and invo lves stakeholders in the regulatory process. Effective - NRC actions are high quality, efficient, timely, and realistic, to enable the safe and beneficial use of radioactive materials.
The NRC utilizes a planning, budgeting, and performance management (PBPM) process and program-level operating plans, which incl ude performance measures and targets, to directly ensure that the performance goals of t he Strategic Plan are properly assessed and that key program outputs and outcomes are met. The PBPM process and associated operating plans are the primar y means of determining whether the strategic performance goals are being met. The ROP se lf-assessment program is not meant to replicate or replace this activity; however, many of t he ROP self-assessment pr ogram metrics are the same as or similar to measures and criteria of the PBPM.
The ROP self-assessment process utilizes pr ogram evaluations and performance metrics to determine its success in meeting the goal s and intended outcomes of the ROP. The level of effectiveness of the ROP is determi ned by considering whet her the program goals are met and the intended outcomes are achiev ed. The intended outco mes of the ROP, which help form its basis and are incorporated in to the various ROP processes, include to successfully:
$ Monitor and assess licensee performance
$ Identify performance issues th rough NRC inspection and licensee PIs
$ Determine the significance of identified performance issues
$ Adjust resources to focus on significant performance issues
$ Evaluate the adequacy of corrective actions for performance issues
$ Take necessary regulatory actions for significant performance issues
$ Communicate inspection and assessment results to stakeholders
$ Make program improvements based on stakeholder feedback and lessons learned
Periodically, the ROP self-assessment program collects information from various sources, including the Reactor Program System (RPS), the inspection progr am, the PI program, other industry-level indicators, periodic independent audits, stakeholder surveys, public comments, and other stakeholder interactions.
Based on this information, the success of the ROP's major program areas (PIs, ins pection program, significance determination
process, and assessment) is assessed. Thes e program area evaluati ons align directly with, and fulfill the intent of, the planned program evaluations fo r the ROP as stipulated in Appendix B to NRC's Strategic Plan for Fisca l Years 2008-2013. In addition, the ROP's overall effectiveness is assessed and recommendations for improvement are made.
05.01 Performance Metrics. A set of performance metri cs associated with each of the program areas of the ROP was developed to assess performance with respect to the seven
goals of the ROP mentioned above. A detailed description of these performance metrics is contained in Appendix A. The four primary RO P program areas includ e the PI program, the inspection program, the significance determination process, and the assessment program.
These program specific metrics are desi gnated as the PI, IP, SDP, and AS metrics respectively.
Issue Date: 03/23/09 6 0307 Metrics of a more general nature are also monitored and analyzed, primarily using stakeholder feedback, to gauge overall performanc e of the ROP. In addition, general metrics were developed to provide the basis for the annual evaluations of the ROP resource expenditures, resident inspecto r demographics and staffing, the training and qualifications program, and the regulatory impact analysis. These more general metrics are designated as the O (overall) metrics.
Each metric in Appendix A includes its definition, the criteria to determine whether it is met, the organization responsible for gathering the data, and a cross-refe rence to those goals each metric is intended to support. Each lead organization outside of NRR/DIRS has been provided with instructions summarizing the data elements and metrics they are requested to support, the periodicity the data is needed, and any specific instructions necessary to clarify the scope of the data.
For example, the regional counterparts have been provided with a detailed data collection and submittal form detailing the specific data elements needed to support each performance metric for wh ich the regions have lead responsibility for data collection. The performance metri cs will be reviewed as part of the annual ROP self-assessment process to eval uate their efficiency and effectiveness in providing a useful
assessment of the ROP. Me trics may be added, deleted, or modified as necessary to provide a meaningful management tool.
05.02 Data Collection. NRR/DIRS has the overall respon sibility for data collection. A variety of methods are used to collect data regarding the performance of the ROP. These methods include data from the RPS, inte rnal and external stakeholder surveys, independent audits, responses to Federal Register notices, and information collected via program document reviews.
In addition, RES, NSIR, the regional offices, and other NRR branches are tasked to provide data. To the extent possible, data collection is from agency databases and the need for ad hoc, manually developed data is minimized. Since the self-
assessment program is relying heavily on t he quality of the data contained in the RPS database, it is imperative that the regions ensure the accu racy and timeliness of the RPS data. As part of the annual metric review, NRR/DIRS will evaluate the need to modify or add permanent automated systems to obtain needed metric info rmation to minimize the burden on the staff.
With the exception of stakeholder surveys and responses to Federal Register notices, data is collected quarterly. Data reporting is comp leted within 45 calendar days of the end of the quarter under review. Internal and external stakeholder surveys or Federal Register notices to collect stakeholder feedback are issued at least biennially. Also, periodic equipment trending reports issued by RES are re viewed to identify additional insights into ROP performance.
- a. ROP inspectable area leads remain c ognizant of the impl ementation of their assigned procedures. Throughout the year, they collect feedback forms written against their assigned areas, they participate in industry meetings to gain insights into the industry's perceptions of their areas, and they visit regions and sites to perform or observe their inspections in t he field and to discuss their areas with the inspectors and regional managers. The purpose for the ROP inspectable area leads to perform or observe inspections is (1) to assess the adequacy of the inspection procedure for possible improvem ents to its scope, focus, and guidance, (2) to assess the adequacy of the ROP program guidance, and (3) to collect Issue Date: 03/23/09 7 0307 comments on the ROP from inspectors and licensees. This programmatic self-assessment is not intended to audit the perfo rmance of the regions in implementing the ROP.
Each year, the inspectable area leads su mmarize the insights gained, significant issues with, and major changes to their a ssigned areas. The summary is given to the ROP program area lead responsible fo r the inspection program in time to support the inspection program evaluati on for the annual self-assessment report.
In addition, a more detailed review and rea lignment of inspection resources will be performed at least biennially in accordanc e with Appendix B to this Chapter. The focus of this effort is to adjust exis ting inspection resources to improve the effectiveness of the inspection program in identifying significant licensee performance deficiencies.
- b. ROP program area leads remain cognizant of the implementation of their assigned programs. Throughout the y ear, they collect feedback forms written against their assigned areas, they visit regions and si tes to discuss their areas with the inspectors and regional managers, and they par ticipate in industry meetings to gain insights into the industry's perceptions of their areas. The program area leads collect self-assessment metric data for their areas each calendar quarter.
At the end of each year, t he ROP program area leads co llect metric data and other insights into their areas and analyze the dat a for the previous year. The analyses form the basis for the program evaluat ions and are included in the annual self-assessment report on the ROP.
05.03 Data Analysis and Recommendation Development. NRR/DIRS has the overall responsibility for analyzing program data and developing recommended improvements to the ROP. Data analysis consists of comparing performance metric data with preestablished criteria and writi ng a determination of its meaning or programmatic impact.
For example, criteria for acceptable ROP performance have been identified for each performance metric as detailed in Appendix A. Thus a favorable comparison of data to criteria would indicate t he ROP met the process goals and objectives, and likely, no programmatic changes would be recommended. However, for an unfavorable comparison more analysis is required to determine c ausal factors and develop recommended process improvements.
The analysis of data also includes evaluati ng the feedback forms, t he results of audits conducted on various aspects of the ROP, comments collected fr om internal and external stakeholders, and any other in sights gained by ROP inspectable area leads and ROP program area leads.
Due to their direct experi ence with the inspection and over sight programs gained through their implementation of the procedures, the regions should be consulted during the data analysis and recommendation development proce ss to ensure the regional insights are incorporated into the change process.
05.04 ROP Self-Assessment Reports. There are several types of periodic ROP self-assessment reports that serve diffe rent purposes as described below:
Issue Date: 03/23/09 8 0307 a. Periodic Reports. As noted above, metric data are collected and analyzed on a quarterly basis, as applicable. Periodi c reports may be issued as deemed necessary to address particular issues of concern resulting fr om the quarterly data and analysis.
- b. Annual Performance Metric Report. NRR/DIRS develops an annual ROP performance metric report after the conclu sion of each calendar year. The overall summary report must discuss any metrics t hat did not meet their preestablished criteria, the staff
=s analysis of the reasons for not meeting the criteria, and any actions taken or planned to change the progr am or improve its implementation.
The report may briefly discuss any other si gnificant lessons from the analyses of the metrics, even if the lesson is related to a metric that did meet its criteria. The report will also identify any metrics not counted during t he previous year and the reasons for that. The analysis may be incl uded in a separate report or may be incorporated into the annual Commi ssion paper discussed in 05.04.c.
- c. Annual Commission Paper. As directed by the Commission, the staff presents the results of its annual self-assessment of the ROP in a Commission paper. The Commission paper is written to support the Agency Action Review Meeting (AARM) and the Commission briefing on AARM results that follows the review meeting. This paper typically addresses any lessons learned from the previous year, effectiveness evaluations of any major c hanges made to the ROP, the status of issues discussed in the Commission paper fr om the previous year, and any other significant issues affecting the ROP.
The annual Commission paper includes evaluati ons of the four key program areas of the ROP: the PI program , the inspection program, t he SDP, and the assessment program. In addition, the staff typica lly includes discussions and assessments of ROP communication and inspector traini ng activities, ROP self-assessment and independent evaluations (including regulator y impact), ROP resources, and other topic areas, as warranted. The self-a ssessment focuses on the effectiveness of recent significant program changes, the st rengths and weaknesses of the program, and additional planned actions to improve pr ogram effectiveness. The paper also presents the staff
=s overall conclusions as to whether the ROP has been successful in meeting the goals and int ended outcomes of the ROP. The staff also highlights those areas of the ROP that warrant focus in the upcom ing year based on the self-assessment results and lessons learned.
The paper typically includes several attachments with additional detail to support the staff
=s assessment and conclusions.
- d. Consolidated Response to External Survey. To address a prior concern that the staff had been unresponsive to the exter nal stakeholder survey comments, NRR/DIRS prepares a conso lidated response to the external survey. This response consolidates all comments received in response to the survey by question and provides a comprehensive response to each question. The annual ROP self-assessment Commission paper, the annual RO P performance metric report, and the consolidated response to the external survey are posted to the ROP Web page and sent along with an acknowledgment letter to each survey respondent.
05.05 Customized Audits of the ROP. After each annual ROP cycle, NRR/DIRS may use Issue Date: 03/23/09 9 0307 the insights gained from the self-assessment to develop topics for audits that delve more deeply into those aspects of t he ROP that show indications of weaknesses or areas for future development. The topics may be sugges ted by an analysis of the metrics, an analysis of the feedback forms, audits of inspection reports, survey responses, or
information gathered at counterpart meetings.
NRR/DIRS develops an audit plan that tailors the audit to each region and identifies the attributes to be verified and associated standards. The audits can verify consist ency of program impl ementation among the regions, verify an aspect of the program over all four regions, or focus on specific areas within one or two regions. The audits are generally conducted by NRR/DIRS staff, who may ask for assistance from other branches or the regions if a particular expertise is needed.
END
Appendix A, Reactor Oversight Process Self-Assessment Metrics
Appendix B, ROP Realignment Process
, Revision History for IMC 0307
Issue Date: 03/23/09 Att1-1 0307 ATTACHMENT 1 Revision History For IMC 0307 Commitment Tracking Number Issue Date Description of Change Training Needed Training Completion
Date Comment Resolution
Accession
Number N/A 12/12/02 Revised significantly to include a more detailed discussion of the role of inspectable
and program area leads, the annual review of
the baseline inspection program, and other
aspects of the self-assessment program. The
specific metrics for these roles were added to
Appendix A.
None N/A N/A N/A 12/12/03 Revised to provide greater detail for documenting the results of the annual
inspection procedures reviews, and some
metrics in Appendix A were modified to better
align with the operating plan metrics and other
program commitments.
None N/A N/A N/A 01/14/04 Based on a decision at the DRP/DRS counterpart meeting hel d on December 17-18, 2003, metric IP-5 was revised to change the
inspection report timeliness to 45 calendar
days for all inspection reports, with exception
of reactive inspection reports, which will stay
at 30 days.
None N/A N/A Issue Date: 03/23/09 Att1-2 0307 N/A 02/20/06 Revised to support the new safety performance measures of the NRC
=s Strategic Plan, to better define the ROP goals and
intended outcomes, and to consolidate and
clarify several of the performance metrics.
Completed 4 year historical CN search.
None N/A ML060110235
N/A
11/28/06 Revised to measure the effectiveness of the safety culture enhancements to the ROP, to
clarify expectations regarding the resident
demographics and staffing metrics, and to
include a discussion of the consolidated
response to external survey questions.
None N/A
N/A 01/10/08 CN 08-002 Revised to eliminate and consolidate several metrics, to separate Appendix A from the base
IMC to serve as a stand-alone document, and
to summarize and link to Appendix B on the
ROP realignment process.
None N/A ML073510410 W200800299 03/23/09 CN 09-010
Revised to address the Commission SRM
dated June 30, 2008, to reflect the recently
issued Strategic Plan for FY 2008 - 2013, and
to reincorporate the security cornerstone in the
ROP self-assessment process, and some
metrics were revised for clarification purposes
while others were removed to eliminate
redundancy or unnecessary burden.
None N/A ML090300620