ML19311C785

Observation Report Final Memo and Report

ML19311C785
Person / Time
Issue date:	01/22/2020
From:	Robert Pascarelli NRC/NRR/DRA/APLA
To:	Mike Franovich NRC/NRR/DRA
References
Download: ML19311C785 (19)

Text

January 22, 2020 MEMORANDUM TO:

Michael X Franovich, Director Division of Risk Assessment Office of Nuclear Reactor Regulation FROM:

Robert Pascarelli, Chief

/RA/

Probabilistic Risk Assessment Branch A Division of Risk Assessment Office of Nuclear Reactor Regulation

SUBJECT:

UNITED STATES NUCLEAR REGULATORY COMMISSION AUDIT ON NEWLY DEVELOPED METHOD PEER REVIEW GUIDANCE - PILOT PEER REVIEWS OBSERVATIONS REPORT BACKGROUND The industry is enhancing the peer review guidance of Probabilistic Risk Assessments (PRAs).

The enhanced guidance will be documented in Nuclear Energy Institute (NEI) 17-07, Performance of PRA Peer Reviews Using the ASME/ANS PRA Standard, (ML17341A548).

The U.S. Nuclear Regulatory (NRC) staff plans to endorse NEI 17-07 with comments and conditions, as necessary, in the planned update to Regulatory Guide 1.200, An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities,(ML090410014).

In NEI 17-07, the industry proposed that newly-developed methods (NDM) for PRAs be peer reviewed using the same framework and processes that guide current peer reviews of a plant-specific PRA. A summary of results from such NDM peer reviews could be provided to the NRC staff prior to using these NDMs in plant-specific PRAs. The industry has proposed a definition for NDMs in draft PWROG-19027-NP, Newly Developed Method Requirements and Peer Review, (ML19273A417). Industry has also proposed high level requirements (HLRs) and detailed supporting requirements (SRs) against which NDMs should be peer reviewed.

On May 21 and 22, 2019, and June 13, 2019, the NRC staff observed three pilot applications of industrys NDM peer review process and associated guidance. The NRC Staff observed the on-site peer review discussions between the peer reviewers and the method developers. NRC staff also had access via a SharePoint site to the NDM methods peer review guidance documents, the method reports, the method developers self-assessments, the resulting peer review reports, and various associated documentation.

Enclosure:

NRC Audit CONTACT: Stephen C. Dinsmore, NRR/DRA 301-415-8482

OBJECTIVE The pilot program included NRC staff observation (i.e., audits) of three NDM reviews with the following objectives;

1. Are the NDM HLRs and SRs adequate for determining the technical acceptability of NDMs for PRAs?

2. Are there, or should there be, any differences in the process guidance and reporting because of inherent differences between current peer reviews confirming the proper application of a method versus the proposed peer reviews of the acceptability of an NDM?

3. Should there be any difference in the process guidance and reporting requirements because oversight activities instead of licensing amendment reviews may be used to assure that licensees are implementing the NDM peer reviews in accordance with staff expectations?

SUMMARY

Based on the NRC staff observations from the pilots as described in Enclosure 1 the, NRC staff in the Division of Risk Assessment in the Office of the Nuclear Reactor Regulation reached several conclusions as discussed in detail in the Enclosure. These include; The framework, process, and requirements provide a well-structured approach for review of newly developed methods. The intention is that if all the SRs and HLRs are met the method should be considered technically acceptable.

Investigating and demonstrating the technical acceptability of NDMs has several key differences in comparison to current peer reviews of the licensees PRA so that additional or augment peer review guidance will be useful and necessary.

The reporting and timing of NDM peer review results to be provided to the NRC Staff is different than the current peer review results for which the reporting is limited to unresolved facts and observations (F&Os) submitted to the NRC Staff in support of individual licensing action requests.

One specific change to the guidance for the NDM peer reviews is the proposal that a NDM should not be used until all the SRs are met. All three pilot peer reviews had a number of SRs that were deemed not met and multiple F&Os that had not been resolved when this memo was issued. Therefore, the pilot reviews have been not completed and additional observations may arise when the NRC observes the completion of the NDM reviews and the final reports.

Additionally, the NRC also intends to periodically conduct audits of a licensees implementation of the NDM peer review process, as well as review a sampling of the final peer review reports.

ML19311C785 NRR-106 OFFIC E NRR/DRA/APLA NRR/DRA/APLB NRR/DRA NRR/DRA/APLA NAME SDinsmore MReisi-Fard SWeerakkody RPascarelli DATE 1/15/2020 1/17/2020 1/17/2020 1/22/2020

UNITED STATED NUCLEAR REGULATORY COMMISSION OBSERVATIONS OF NEWLY DEVELOPED METHOD PEER REVIEW GUIDANCE - PILOT PEER REVIEWS OBSERVATION REPORT

1.0 INTRODUCTION

The industry is enhancing the peer-review guidance of Probabilistic Risk Assessments (PRAs).

The enhanced guidance will be documented in Nuclear Energy Institute (NEI) 17-07, Performance of PRA Peer Reviews Using the ASME/ANS PRA Standard. The U.S. Nuclear Regulatory (NRC) staff plans to endorse NEI 17-07, with comments and conditions, as necessary, which would then allow industry to use NEI 17-07 to support peer reviews of PRAs.

When completed and endorsed by the NRC, NEI 17-07 will consolidate and replace the current peer review guidance in NEI 05-04, NEI 07-12, and NEI 12-13. Similar to previous peer review guidance such as in NEI 05-04, NEI 17-07 provides detailed process and documentation guidance on peer-reviewing a PRA against the American Society of Mechanical Engineers (ASME)/American Nuclear Society (ANS) PRA Standard. Results of these peer reviews will be available to the US Nuclear Regulatory Commission (NRC) staff in headquarters and the regional staff, if needed, to verify that peer-reviews are conducted in accordance with NRC endorsed processes and requirements.

In NEI 17-07, the industry proposed that newly-developed methods (NDM) for PRAs be peer-reviewed using the same framework and processes that guide current peer reviews of a plant-specific PRA. A summary of results from such NDM peer reviews could be provided to the NRC staff prior to using these NDMs in plant-specific PRAs. Industry proposed high level and detailed supporting requirements for NDMs as well as a definition for NDM in draft Pressurized-Water Reactor Owners Group (PWROG-19027-NP), Newly Developed Method Requirements and Peer Review for use as pilots. Changes were made to this early draft after three pilot applications (discussed below) and Revision 0 was issued in September 2019. Revision 0 includes changes that address some of the observations in this report but NRC staff positions, as applicable, on Revision 0 will be included in the next revision of RG 1.200 and not discussed in this observation report.

The ASME\\ANS PRA Standard includes Parts subdivided into Technical Elements. Technical Elements are further defined with high level requirement (HLR) and supporting requirements (SRs). The ASME/ANS PRA Standard and peer review guidance provide the structure, process, and criteria which, combined with the peer reviewers PRA expertise, results in the examination of the various PRA technical elements. The outcome of the PRA peer review are conclusions on whether each SR is adequately addressed (e.g., Met or Met at Capability Category II), or identification of the relevant inadequacy and a proposed a resolution in a fact &

observation (F&O) assigned to the SR. The current proposed NDM SRs do not differentiate between the Capability Categories and are, therefore, Met or Not Met.

ENCLOSURE

The industry has proposed developing a new Technical Element with new HLRs and SRs that would be used by peer review teams to review NDMs. The NDM peer review will assess the NDM against the high level and supporting requirements to determine whether they are adequately addressed or identify the relevant inadequacies and propose a resolution in a F&O.

The HLR and SR framework for NDMs is consistent with that in the NRC endorsed ASME/ANS PRA Standard. The expectation is that this new NDM Technical Element will be incorporated into a future edition of the ASME/ANS PRA Standard.

Use of a process that requires peer reviewers to review NDMs against requirements that are acceptable to the NRC staff is essential to maintain public safety since future inputs and insights from PRA models that may incorporate these NDMs will be used by licensees to self-approve changes to key operational parameters. NRC staff approved risk-informed that use PRA results generated by the licensee in the future without a regulatory review include changes to surveillance intervals, special treatment requirements, fire protection program features, and technical specification completion times. Therefore, both NRC staff and industry representatives have and will expend significant efforts to develop and pilot definition of NDMs and the SRs and HLRs used to review the reasonableness of NDMs. Consequently, on May 21 and 22, 2019, and June 13, 2019, the NRC staff observed three trial applications of the proposed NDM peer review process.

2.0

SUMMARY

OF NRC STAFF OBSERVATIONS As part of the pilot applications, the NRC Staff observed the on-site peer review discussions between the peer reviewers and the method developers. NRC staff also had access via a SharePoint site to the NDM methods peer review guidance documents, the method reports, the method developers self-assessments, the resulting peer review reports, and various associated documentation.

The NRC staff observations had three general objectives: (1) are the NDM HLRs and SRs adequate for determining the technical acceptability of NDMs for PRAs to the peer-reviewers, (2) are there, or should there be, any differences in the process guidance and reporting because of inherent differences between current reviews conforming the proper application of a method versus the proposed reviews of the acceptability of a method, and (3) should there be any difference in the process guidance and reporting requirements because oversight activities instead of the licensing amendment reviews may be used to assure that licensees are implementing NDM peer-review methods in accordance with staff expectations.

With respect to item (1), the staff observation on the adequacy of the NDM HLRs and SRs indicated that the general framework seems reasonable and workable. The NRC staff also observed that specific improvements in the HLRs and SRs have been made based on the pilot applications and therefore, the NDM HLRs and SRs represent the best available information.

With respect to item (2), the NRC staff developed the following general observations with respect to whether there are, or should be, any differences in the process guidance to reflect any difference between NDM review versus the current per reviews (item 2 above):

1. The framework, process, and requirements provide a consistent and well-structured approach for review of newly developed methods.

2. The NRC staff has frequently encountered peer review results where confusion caused by weak and incomplete documentation prevented a conclusion or obscured a technical

weakness. During the NDM pilots, the NRC staff observed that documentation weaknesses would be especially challenging in NDM reviews because weak

3. documentation could lead to cascading effect with potentially significant problems for licensees who plan to incorporate the NDMs to their PRAs.

4. Peer review of the technical acceptability of NDMs has several key differences in comparison to current peer-reviews of the licensees PRA such as the need for non-PRA subject matter experts, additional resources to support detailed investigation of the NDM, detailed review of all SRs instead of sampling, and the need to resolve all F&Os with subsequent NDM peer reviews and not plant specific resolutions.

5. Current F&Os can be dispositioned by licensees with plant and PRA specific resolutions. Disposition of F&Os from NDM peer reviews should only be possible with a subsequent NDM peer review to ensure consistent and well-defined technical acceptability and avoid one method developing into multiple alternatives.

6. Similar to the current peer review process, it is intended that, by meeting all the SRs under all the HLRs, the NDM will satisfy the intent of the HLRs and therefore the method should be technical acceptable.

7. Plant specific peer review reports are not publicly available, but aspects of the methodology reports and methodology peer review report, should be publicly available.

With respect to Item (3), the NRC staff had initially proposed that, in addition to discussion and conclusion about each NDM HLR and SR, the NDM peer review team should also reach and document an overall conclusion about the acceptability of the method. The pilot application peer review reports included such conclusion, but the NRC staff observed that such conclusions were less useful than the sum of the individual conclusions about all the HLRs and SRs, and did not provide any additional information. This issue, however, will not pose a concern if (a) all peer review F&Os on NDMs will be closed prior to any licensee incorporating them into their models, and (b) peer-review reports will be available to NRC to support its regulatory decisions relating to licensing or inspections.

As a result of the pilots, the industry reviewers and the NRC staff identified some improvements to the HLR and SRs to (1) ensure that the technical acceptability of NDM is thoroughly investigated, and (2) provide sufficient information to allow NRC staff to understand the bases for the determination of the technical acceptability of the NDM by the peer reviewers. Additional experience may lead to updates to the HLR and SRs much as the current ASME/ANS PRA Standard HLR and SRs have been updated over time.

The NRC staff will use the NDM peer review pilot observations to inform additional work in the area of NDMs in PRAs for regulatory applications and decisions.

3.0 OVERVIEW OF THE PILOT APPLICATIONS On May 21 and 22, 2019, and June 13, 2019, the NRC staff observed three pilot applications of industrys NDM peer review process and associated guidance. The May reviews included a 2 day onsite review while the June review was a single day on-site review. On-site review means that the review team met at one location with the method developer(s). All reviews also included off-site work (both pre-and post-on-site) review work that was not observed by the

NRC staff. For the two May 2019 pilots, the on-site reviews were conducted at the Westinghouse offices in Rockville, Maryland. The June 2019 review was held at the Jensen Hughes offices in Rockville, Maryland.

All three method development documents as well as peer-review reports were available to the NRC staff. The documents include the method developers Self-Assessment Basis for proposing that each SR is met, the corresponding peer review teams Basis for Assessments explaining why the peer review team assigned the capability category (CC) to each SR, and all F&Os.

The three pilots represented a reasonable range of NDMs as discussed below. Each of the three pilot reviews concluded that some SRs were not met and provided F&Os.

The NRC staffs observations were used to support the ongoing NRC staff comments on updates to the NDM HLRs and SRs as well as the guidance in NEI 17-07.

3.1 NDM Pilot applications 1 and 2 DATES: May 21 and 22, 2019 LOCATION: Westinghouse offices in Rockville, Maryland NRC STAFF OBSERVERS Michael Levine, NRR/DRA Shilp Vasavada, NRR/DRA Mehdi Reisi Fard, NRR/DRA Anders Gilbertson, RES/DRA Mary Drouin, ADM Sunil Weerakkody, NRR/DRA Stephen Dinsmore, NRR/DRA Observations from NDM #1:

PWROG-18026-P, Revision 0-B, Component Reliability Data Issues and Strategies - PA-RMSC-1494, October 2018 The method proposed to reallocate observed diesel failures between the current three bins, fails-to-start, fails-to-load-run, and fails-to-run. The data re-evaluation for the diesels was one of several components for which observed data was re-evaluated in larger report (PWROG-18026-P). It was unclear to peer reviewers which sections of the larger report were applicable to this method and which were not. Several SR Bases for Assessment and F&Os indicated that the larger report lacked specific sections that address SRs requiring clear descriptions of assumptions and limitations related specifically to diesel failure data and data interpretation.

In this pilot, a Self-Assessment Basis was only done at the HLR level because of the limited scope of this NDM. Although the SR on Scope and Purpose was judged met, the peer reviewers SR Basis for Assessment did not summarize the scope or the purpose. The basis simply referred to the clearly defined scope in Section 6 of the related reports but then continued that the purpose should have been documented more clearly. The peer review

team then assigned a F&O to this met SR which stated that it does not appear that the scope meets the purpose.

Two additional SR Basis for Assessments on applicability and limitations were similarly brief and uninformative, effectively stating that the data was diesel data and therefore sufficiently obvious applicable to the method without linking the proposed calculations to the characteristics of diesel failure data collected. These two SRs were also assigned met with, however, F&Os assigned that reflected confusion about why the new data interpretation was an improvement (i.e., purpose) and when the new failure values should be used (i.e., limitations).

Other F&Os (e.g., 5-5) question how the diesel failure data characteristics are related to the final diesel failure mode estimates, but the peer review team did not appear to consider whether an independent evaluation of the interpretation of the diesel failure data should be part of the peer review effort as was proposed during the third pilot application.

Although referencing a F&O to a met SR is allowed, the guidance in NEI 05-04 states that such findings [are] typically for non-systematic discrepancies that the PRA peer review team judges require correction. It appears the review team was assigning the status of the SRs on general engineering reasonableness conclusions (e.g., diesel data can be used for diesels) but there is no clear explanation and justifications in the PWROG-18026-P report, and the review team did not have time nor access to the detailed data and workings of the method to develop the justifications. In addition to assigning F&Os to met SRs, most F&Os were referenced to more than one SRs. Assigning F&Os to two or sometimes three SRs is also observed in the current peer reviews, but this is also an exception and not as prevalent as appeared in the pilot. It is unclear whether the relatively extensive use of F&Os for met SRs, and multiple SRs associated with F&Os is a reflection of improvements needed in the definitions of the SRs, or inherent in the use or HLRs and SRs to review newly developed methods.

The peer review guidance specifies that the analysis owners (in this case the method developers) should provide a self-assessment of all the HLRs and SRs to the peer review team.

Although limited to HLRs, this self-assessment was the only one of the observed self-assessment basis that did not rely primarily on simply repeating the guidance. For example, instead of simply repeating that the scope is clearly definedand in line with the purpose the self-assessment states the purpose of this NDM is to provide a method for calculating EDG fail-to-run failure rate based on an understanding of the load-run data as demand events rather than one-hour run data. Also, instead of the data are relevant the self-assessment states that data was collected from the INPO ICES Database for 236 EDGs at 95 sites across the US gathered in the 10-year period 2006 to 2015.

Observation from NDM#2:

PWROG-18027-NP, Revision 0-b, General Screening Criteria for Loss of Room Cooling in PRA Modeling, PA-RMSC-1391, May 2019 The method proposed a process to characterize the impact of loss of room cooling in the PRA.

This method was described in a dedicated report, although there were several different relatively independent tasks to the method (e.g., calculating time to reach temperatures, selecting a new damage temperature, selecting a temperature/damage curve) and not all tasks were equally well described and justified. The self-assessment for this method was performed for all SRs and the information provide for some SRs included references back to specific sections in the method documentation when available.

Two of the NDM SR peer reviewers Basis for Assessment noted that expertise that is not normally found in a PRA team was required, in this case the evaluation of acceptability of interpretation and alternative use of environmental qualification data to justify and/or change time to failure at elevated temperatures. Discussions with the peer review team indicated that the team eventually decided it was not necessary to have this expertise in order to understand the method. Instead, the review team requested, and was provided, evidence that the method developers interacted with EQ subject matter (SM) experts. Based on an understanding that the method developer relied on appropriate SM experts, the peer review team decided that SR NMA-B4 (assumptions based on engineering judgment have a documented rational backed by appropriate expertise) and SR NMA-D4 (data is relevant to its intended use) were met. The NRC staff notes that the peer review team identified the SRs that required SM experts, but while the NRC staff had anticipated that all parts of the methods be reviewed, the peer review team decided that understanding the method was sufficient. Relying on the method developers SM experts to assign met to these SRs does not seem to satisfy the expectation that all important assumption and data be independently reviewed during the review of an NDM. This SR Basis for Assessment indicates that the peer review teams may, when necessary, recognize the need for additional expertise. NRC staff believes that this recognition should be formalized and clarification of what should be reviewed by SM experts, as opposed to understand by PRA practioners, should be included in the guidance and subsequent documentation requirements.

Both the self-assessment and the peer review writeups for the room cooling method describe the many assumptions noted throughout the method and number of assumptions. However, there was no summary of assumptions leaving the number, extent and impact of the assumptions unclear. Also, the NMA-B1 (assumptions have a valid technical basis) assigns a met to NMA-B1 but the Basis for Assessment proceeds to state that there is little discussion on the basis that support the assumptions. The method report, the self-assessment, and the peer review reports did not provide confidence that the peer review team, or the method developers, had documented and considered all the assumptions that the method included and their corresponding technical bases.

This pilot illustrated that when an NDM constituted multiple somewhat independent tasks, ensuring that every SR is fully addressed for every individual task can be challenging. Similar to the EDG method review above, the peer review team assigned a met to most SRs but then referenced a F&O to almost all the met SRs.

3.2 NDM Pilot Application 3 DATE: June 13, 2019 LOCATION: Jensen Hughes offices in Rockville, Maryland NRC STAFF OBSERVERS Nick Melly, RES/DRA JS Hyslop, NRR/DRA Shilp Vasavada, NRR/DRA Mike Levine, NRR/DRA Adrienne Brown, NRR/DRA Sunil Weerakkody, NRR/DRA Mehdi Reisi Fard, NRR/DRA Brian Metzger, NRR/DRA

De (Wesley) Wu, NRR/DRA Observations from NDM#3:

FPRA-Approach-Within-a-Cabinet-Fire-Damage-Scenarios (Partial Damage) - June 12 2019-ver-c.docx.

The method proposed to reduce the frequency for the loss of all functions within an electrical cabinet following an electric fire igniting in the cabinet. The method was described in a very brief document and self-assessment developed to support the peer review. Additional documentation was subsequently developed and used by the peer review team. The NRC staff did not observe these subsequent documents and interactions and therefore has no observations on them.

One industry peer reviewer was very experienced in fire PRA and fire modeling, and participates in the development of fire PRA guidance documents and training to the nuclear industry. The other reviewer was very experienced in fire protection engineering and had experience in the development of new fire PRA guidance and was responsible for maintenance of fire PRAs for a fleet of plants. The NRC observers together were experienced in fire PRA, fire modeling, and fire protection engineering and applications of these fields. The NRC staff observers noted that the reviewers were very familiar with the current PRA method being replaced by this new method.

Due to their experience, the staff concluded that these peer reviewers were also SM experts in fire protection and fire PRA. As such they were familiar with the characteristics of fires growth phenomena, fire effect on components, and the strengths and weakness in the observed fire data upon which the new method was developed. The staff observed that the peer reviewers examined some of the results of the data review and requested (and eventually received) more information on the criteria used to evaluate the data. However, the peer review team apparently did not review the fire data used to develop the method either before the peer review or during the development of the peer review report.

An illustration of the possible complexity of NDM reviews conducted according to pre-defined resources (i.e., fixed team size and time) is that the NRC staff noted that additional steps should be performed if the peer review was to be considered a complete and independent review of the method and its results. The peer reviewers and NRC staff observations identified that data analysis SRs needed to be updated to ensure the technical basis is explicit and unambiguous.

The peer review team also recommended that at least two analysts apply these criteria to all the data to ensure consensus. Thus, this method demonstrated the complexity and follow-up required for some peer reviews.

This peer review team assigned not met to more SRs than the peer review of other two NDMs that staff observed, perhaps due to the limited documentation. But, as observed above, a number of met SRs were nevertheless assigned an F&O. The lack of substantive documentation contributed to many of the not met SRs. The fire PRA and fire modelling subject matter experts also provided specific resolutions to identified weakness.

3.3 Conduct of the Reviews The on-site review of each of the three methods generally started with the method developers providing a description of the method and responding to questions by the peer review teams.

The peer review teams would then continue to review the documents and provide further questions to the method develops. Eventually the peer review team worked through the SRs sometimes on their own, as determined by the lead reviewer, and sometimes as a team. There was a final meeting establish consensus between the two reviewers and a final presentation of the results to the method developers.

The NRC staffs observation of the NDM peer review process as summarized above found that it was similar to the normal peer review process with the following observed differences:

Consistent with the general peer review guidance, two reviewers were assigned to each NDM. A single-person peer review of NDMs would be inappropriate.

The NRC staff notes that the structure of the ASME Standard (parts, elements, HLRs, and SRs) supports peer reviewer expertise related to, for example, all HLRs in any given element because they are technically related, and this supports not requiring more than two peer reviewers. During these pilots, discussions among the reviewers, the method developers and the staff indicate that sometimes more than two peer reviewers might be necessary ensure that expertise is available in all aspects of an NDM to fully address all different assumptions.

Normal peer reviews evaluate whether the methods were understood and applied correctly by the PRA practioners to the as-built, as-operated plant. However, NDM peer reviews should be familiar with the strengths and limitations of the underlying engineering assumptions (i.e., be subject matter experts) in order to evaluate the method itself. The necessity of SM experts to judge the adequacy of the method is recognized in the current draft guidance documents but may require additional clarification.

During current peer reviews, reviewers are focused on the implementation of a method as described in the PRA documentation, and confirming that description in the PRA is consistent with the reviewers experience with the method. Much of the discussion that does occur between the licensee personnel and the reviewers centers on where the description of the evaluation is located in the voluminous PRA documents, and where the plant specific information supporting the method during the development of the PRA are documented. The SRs Basis for Assessment are then dispositioned by identifying where the evaluation inputs and outputs are documented in the PRA. However, the NRC staffs observations of the pilots revealed that an NDM may be supported by a relatively short report and the discussion between the reviewers and the method developer(s) tended to be focused on explanation of the proposed method, how the method as proposed should be implemented in a PRA, where any supporting methods and data were obtained, and whether the associated assumptions are reasonable and applicable to the method as it would be applied at a plant. All missing documentation would need to be developed and would generally require a subsequent NDM peer review(s) before all SRs would be met. This lacking documentation is, as discussed elsewhere in this report, the cause of many of the finding F&Os. Peer reviews may help identify but will not compensate for inadequate documentation.

During current peer reviews, discussions about detailed steps or assumption in the methods tend to refer to the actual steps as applied in the PRA that are described in the detailed documentation in the PRA under review and the experience of the peer review

and licensee personnel from previous applications of the method. During the NDM review, the discussions concentrated on the sources of data and technical descriptions and justification of assumptions required by the method.

Current peer review recommendations for resolving technical deficiencies provided in F&Os tend to involve issues that, based on the reviewers experience, were not consistent with previous applications of the method. F&Os developed during the NDM review were often directed toward how the NDM might be better documented, changed, or improved but without specific recommendations on how to change the NDM. For example, the room temperature method seemed to use a 24-hour mission time as a simple, fixed value as the end point in the room temperature calculations whereas the review teams noted that safe and stable is the actual end point and additional evaluation related to appropriate mission time would be more consistent with state-of-of the practice. Another example is an F&O about the implications of moving early observed failures from the Fails to Start and Load data to the Failure to Start data but retaining any run time in the observed run-time data should be further evaluated.

4.0 BASIS FOR ASSIGNING SRS AS MET - DEVELOPMENT AND REPORTING The peer review guidance in the ASME standard and the NEI documents include requirements that the self-assessment provided by a licensee to the peer review team include their assessment of compliance for each ASME/ANS SR with references to those portions of their PRA documentation that demonstrate the appropriate degree of compliance. These self-assessments basis are provided for each HLR and SR to be reviewed and, as required, refer to the reports, sections, and tables in the PRA documentation that illustrate how the licensees evaluation is consistent with the SRs requirements.

The peer review team also provides its Basis for Assessment for each SR which generally will include the self-assessments references and may add comments by the peer review team. If no F&O is written for an SR, the Basis for Assessment provide the only documentation of the evaluation of the PRA against each SR. The report templates in NEI 05-04 requires that peer review report include summaries of the results of the review for each technical element (i.e.,

HLR) within the scope of the review. The guidance also provides example peer review results where a Summary of Assessment is provided for each SR but there is no guidance about what the peer review team should include in this assessment.

Current self-assessment and subsequent Basis for Assessments in the peer review reports are not submitted to the NRC to support the review of risk-informed LARs. Only the F&Os and proposed resolutions are submitted. The NRC staff sometimes accesses the self-assessments and peer review reports during audits of risk-informed application. Past and current evaluations of self-assessment reports indicate that the self-assessment follow the guidance and refer to very specific reports, sections, and tables in the thousands of pages of PRA documentation to support the reported conclusion. The peer review reports Summary of Assessment generally repeat the self-assessments references and add peer review comments and conclusions supporting the assigned capability category.

The NRC staff experience with NRC staff review of NDMs is that method developers often rely on their inherent knowledge of the subject and generally do not systematically and comprehensively document, for example, the full set of assumptions and limitations. This issue is further complicated by the, often numerous, assumptions required to support PRA methods.

The first two pilot methods were supported by fairly comprehensive reports. One of the observed self-assessments was limited to assessment against the HLRs because of the limited

scope of the NDM. Although the associated peer review Basis for Assessment repeatedly stated that different issues were described, very few specific report sections were referenced.

The second self-assessment was similar to the current self-assessment because specific sections and sub-sections of the method report were referenced. The self-assessment for the third peer review was not available.

The NRC staff has limited experience with reviewing the Basis for Assessments because the peer review reports are not submitted to the NRC. It is still unclear what will be submitted to the NRC following an NDM peer review. The observation that all F&Os should be resolved before a method is used would, with the current reporting process, result in no peer review documentation being submitted which contradicts the intent that the NRC is provided with a meaningful report prior to use of the NDM. Insofar as the self-assessment and associated Basis for Assessments refer to specific sections and sub-sections of, in this case, method descriptions, this information would be useful. However, inadequate documentation is, and continues to be, a weakness in the large complex PRA studies. Based on NRC staff experience in reviewing NDMs, inadequate documentation will be also be an issue in NDMs.

The Bases for Assessment developed by the NDM peer review teams provides an opportunity for the team to demonstrate a thorough understanding of the proposed method, and to document their concurrence with, or disagreement with, the NDM. Providing guidance that will result in the NDM peer review process developing clear and technically adequate documentation about how well an NDM meets the associated criteria will require additional work on how the self-assessments and associated Basis for Assessments are evaluated and reported.

5.0 FACTS AND OBSERVATIONS - DEVELOPMENT AND REPORTING During the review of an SR, if the peer reviewers identify any issues/problems that impact the capability of the PRA (or NDM) to satisfy the SR, they will document these problems using an F&O. Peer reviewers experience in writing F&Os is evident in the pilot applications where, as opposed to the relatively uninformative Bases for Assessment discussions on met SRs, the F&Os described the issue in reasonable detail and include a suggested resolution.

Current F&Os can be resolved (e.g., corrected) for a specific risk-informed application by a licensee or dispositioned as having no impact on the application. The NRC staffs observations of the three pilots noted that it is unlikely that licensees would be able to develop and implement a plant specific correction to an observed weakness in an NDM because such methods consist of a number of assumptions and evaluations and any resolution of an individual assumption or evaluation may have implications throughout the method. In addition, the licensees would not have a detailed understanding (e.g., subject matter experts) of the NDM to even determine the appropriate resolution. Furthermore, licensee disposition or resolution of F&Os for NDMs would negatively impact the efficiency of NDM peer review process. Therefore, resolution of F&Os from NDM peer reviews by the method developers before application in a plant-specific PRA would appropriate.

Consequently, it may be necessary to perform more than one round of peer reviews before an adequate set of documentation exits to support an NDM. The staff and industry are still discussing whether an Appendix X close out or a focused scope peer review would be the most appropriate. Most likely, the same guidelines for current peer reviews can be used, i.e., and

upgrade that changes the method may require a focused scope peer review while filling in missing documentation may only require an Appendix X closure.

6.0 INDEPENDENCE AND EXPERTISE OF PEER REVIEW TEAMS All three peer review teams were independent of the method authors and utilized the available time with a questioning review. The reviewers were able to understand generally how the methods were developed and how they should be applied. The peer reviewers were very competent in general PRA methodologies and applications.

As discussed above, the NRC staff observed that there may be a variety of expertise and Sm experts who should be involved in an NDM review. Insufficient SM expertise on an NDM peer review team will invalidate the expectation that the NDM peer review will perform an independent review of the compilation of analyses, tools, assumptions, and data that are used to support the NDM. In some cases, the need for SM experts should be obvious before the review while in others the issue may only become apparent during the review.

Guidance of how to expand the peer review team to include subject matter experts as required, and how to appropriately reflect these experts input into the final peer review report is needed.

It would appear that the initial self-assessment by the method developers should be able to identify when, and what type of, subject matter experts would be required. It is unclear if subject matter experts should be included in addition to the PRA peer reviewers, or if they could replace PRA peer reviewers. Replacing PRA peer reviewers may weaken the confidence in the peer review results that is provided when two PRA reviewers independently agree with the conclusion (i.e., the Basis for Assessment).

7.0 RESOURCE REQUIREMENTS TO REVIEW DIFFERENT METHODS Focused scope peer reviews have been focused on a related set of technical SRs. Each SR summarizes a relatively narrow subset of well-defined tasks. The current technical SRs were developed through several revisions of the ASME PRA standard to fit within a review framework that anticipated two experienced reviews spending a limited amount of time to determine that the SRs were met, or met at a given capability category as applicable. The tasks defined in SRs for an NDM review must instead summarize more general tasks because the SRs must be capable of guiding and documenting the evaluation any method for developing all PRA inputs and all PRA models. Therefore the complexity, expertise and time requirements to review newly developed methods can vary greatly between different methods and, correspondingly, the resources required to review different methods can vary greatly. The available guidance does not appear to recognize and address these different resource requirements.

With respect to room-cooling, a standard computer code to calculate room temperature appears to be part of the method well understood by PRA engineers, but the use of accelerated aging environmental qualification tables to increase the failure temperature above 150F was accepted by the peer review team as a viable option without further guidance on how, and what expertise is needed, to concur with the assumption based on technical knowledge. Several F&Os summarized issues associated with convoluting probability of failure at a given temperature with the probability of reaching that temperature (called the interference method). Resolution of these F&Os appear to require further clarification and documentation and it is unclear if this task should be stripped from the method or otherwise how these F&Os would affect use of the NDM.

The NRC staffs observations of all three pilots revealed that two reviewers and the allotted time may not be sufficient to review all aspects of a newly developed method. In the case of the diesel failure modes, the reviewers did not have the time, access to the data, nor perhaps detailed knowledge of diesel operation to determine whether to agree or disagree with the new binning of the data beyond a general understanding of the change in assumptions from the original data reviews and the proposed data reviews. With respect to postulated fire damage, staff also noted that the review team did not have the time, the access to the data, nor the criteria for damage to components within the cabinet that was used. It is unclear to what extent the reviewers should provide an independent review of the proposed evaluation. Current peer reviews allow for a sampling review, where only some of the SRs in an element are reviewed in detail the judgement is made that the sampling demonstrated that the appropriate process was applied. Therefore, the industry participants postulated that the data analysis only needed to be a sample of the data. However, sampling is valid only if clear guidelines exist so that the observed samples demonstrate that the guidelines are properly applied. Even for these relatively straightforward, data re-evaluations, it appeared that clear guidelines may be difficult to develop and were not available, e.g., an F&O states that the data selection process which will serve as the technical basis and therefore define the limitations of the NDM need to be documented. The NRC staff observed that NDM reviews can involve a more detailed review of the technical bases, including the data, compared to sampling. NRC staff observations informed the updates to NEI 17-07.

8.0 SUPPORTING REQUIREMENT DESCRIPTIONS The review teams determined how well the NDM SRs had been fulfilled for each newly developed method. The description of the draft HLRs and SRs in the NDM peer review were, in some cases, vague and/or so broad that they overlapped. As noted above, there appeared to be relatively many met SRs that were then assigned an F&O, and relatively many F&Os referenced multiple SRs.

The pilot applications were conducted to evaluate the proposed NDM HLRs and SRs. The HLRs and SRs in the 2009 version of the ASME Standard were developed through literally hundreds of peer reviews and subsequent evaluations. It is anticipated that the SRs (and perhaps the HLRs) will be improved and developed through additional pilot applications, or following experience in performing and reporting the NDM reviews.

All participants felt that an important and very useful addition to the review and the documentation would be the identification of the current ASME Standard requirements that would be affected by implementation of the newly developed method. It became apparent that a focused scope peer review of the method implementation into a plant-specific PRA could use this information to define the scope of the focus scope implementation peer review.

9.0 DEFINITION OF THE NDM Peer-reviewers encountered challenges with the definition of NDM. Specifically, an NDM was defined as one that is not current state-of-practice or a consensus method. However, during the peer-review there were disagreement with respect what constitutes state-of-practice or consensus methods. This lack of definition affected these peer reviews insofar as these (and most) NDMs primarily combine or extend current state-of-practice and consensus methods making the parts of the methodology that should be peer reviewed hard to identify, isolate, and evaluate. The observations of the peer reviewers and NRC staff from the three pilots informed the development of a definition of NDMs.

10.0 OVERALL CONCLUSION ON A NEWLY DEVELOPED METHOD REVIEW PROCESS GUIDANCE The current peer review process has been established and is applied to characterize the technical acceptability of a PRA by characterizing how all the tasks in a PRA have been addressed. A peer review does not accept or reject the PRA or any parts of the PRA. The F&O development and resolution process is well developed and has proven to be very effective at identifying and eventually resolving weakness in PRAs. F&Os should also be valuable for improving NDMs although subject matter experts may be needed both to develop and to resolve F&Os.

The Self-Assessment and peer review Basis for Assessment for every SR have evidently been successfully used by industry to provide a clear reference to where the various PRA tasks are described in a PRA. Only F&Os have been submitted for review by the NRC Staff and therefore the Staff has no experience in using the Self-Assessments or the Basis for Assessment as a basis for determining the technical acceptability of the PRA or, in this case, the NDM that was reviewed. If, as currently anticipated, F&Os need to be resolved before an NDM is used, only the Self-Assessment and Basis for Assessment would be readily available to be provided to the NRC Staff. This may not be sufficient to support the initial reasonableness check nor any subsequent evaluation of the acceptability of an NDM in a proposed or on-going PRA application. Identification of what documentation will be provided to the NRC at what stage of NDM development and use will be further evaluated.

For some NDMs, the need for additional expertise (beyond PRA expertise) was observed by the NRC staff as an added complexity of NDM peer reviews and guidance is needed on when and how to include address this additional expertise should be brought into the process. Many NDMs are expected to characterize failure probabilities caused by unique and variable magnitude hazards, e.g., fires, temperatures, wind speeds, water levels. Both the frequency/magnitude characteristics of the hazard and the impact of the hazard magnitude on SSC failure probability may need to be characterized. Many NDMs will only be applicable within an envelope comprised of a range of hazard magnitudes and/or SSC characteristics. After the envelope and associated process has been defined (i.e., the method), it is expected that PRA analyst can properly apply the process to specific locations and SSCs and can also support peer reviews of these analyses as needed. However, the applicable envelope and process to develop PRA inputs from plant characteristics may need to be developed and justified using specialized phenomenological expertise and material and/or structural expertise respectively.

Meaningful review and acceptance of these NDMs can only be performed by engineers with equivalent expertise.

One final conclusion that is apparent from the pilots and the discussions in this observation report is that NDM reviews are much more complex and consequently resource intensive in comparison to traditional peer reviews (i.e., reviews about the proper implementation of methods) and industry should be prepared that these reviews will be resource intensive and may often require several sequential peer reviews before completion.

Detailed Summary of observations

1) The framework, process, and requirements provide a well-structured approach for review of newly developed methods

a. Development, documentation, reporting and resolution of F&Os is well established

and is an integrated component of PRA Technical adequacy reviews.

b. The Self-Assessment and the Basis for Assessment developed for and during the peer review have not been used in licensing actions but will most likely be relied on for NDM reporting and review, particularly if the process requires all F&Os to be resolved before the NDM is used. Current reliance in the Basis for Assessment on references to where documentation supporting a met designation can be found may need to be augmented to include some technical information when the only reporting available in the peer review report on an NDM are the Basis for Assessments.

2) NRC staff has frequently encountered peer review results where confusion caused by weak and incomplete documentation prevented a conclusion or obscured a technical weakness. During the NDM pilots, the NRC staff observed that documentation weaknesses would be especially challenging in NDM reviews because weak documentation could lead to cascading effect with potentially significant problems for licensees who plan to incorporate the NDMs to their PRAs.

a. Many F&Os in current LAR reviews are directly referenced to documentation SRs even though the weaknesses identified by those F&Os have technical implications.

b. Many F&Os in current LAR review are written for technical SRs but which the licensee dispositions as being documentation issues.

c. Documentation issues with NDM are more important because they could mean that the method itself is incorrect or incomplete, or, even if the method is correct, could lead to confusion and mis-application of the method.

d. Documentation weakness for NDM should be more formally resolved than for current peer review documentation issues.

3) Review of the technical acceptability of NDMs has several key differences in comparison to current peer-reviews of the licensees PRA such as the need for non-PRA subject matter experts, additional resources to support detailed investigation of the NDM, detailed review of all SRs instead of sampling, and the need to resolve all F&Os with subsequent NDM peer reviews and not plant specific resolutions.

a. The PRA standard specifies that peer reviewers shall be knowledgeable (by direct experience) of the specific methodology, code, tool, or approach that was used.

PRA engineers can determine whether any PRA methods they are knowledgeable of have been properly applied. However, the review of newly developed methods may require subject matter experts to evaluate the appropriateness of approximations and assumptions such as those related to material properties, electrical and process system and subsystem characteristics, and reasonable expectations of equipment performance under unusual circumstances.

b. The time required to review how well PRA tasks using developed methods have been implemented is relatively short and well understood while the time required to develop an understanding and characterize a new method could vary greatly.

c. Current peer reviews allow completing a review after a sampling of the SRs support a judgement that the element was done properly. NDM peer review should evaluate each of the relatively few SRs to ensure the full method has been evaluated.

d. Appropriate resolution of deficiencies or findings in the implementation of PRA methods may be determined by PRA engineers while, as with the initial reviews, subject matter experts may be required to resolve F&Os for newly developed methods, which could include changes to the basic steps in the method.

4) Current F&Os can be dispositioned by licensees with plant and PRA specific resolutions. Disposition of F&Os from NDM peer reviews should only be undertaken by the method developer to ensure consistent and well-defined technical acceptability and avoid one method developing into multiple alternatives.

a. F&Os developed using the current peer review guidance are often dispositioned by each licensee in a LAR. Acceptable LAR specific disposition include conclusions that the F&O resolution would have no impact on the application, the SRs is only a documentation weakness, or the identified weakness has been resolved with changes to the PRA.

b. NDM F&Os may be quite complex consistent with the complexity of developing a NDM and resolution may require expertise and resources that are best utilized in a one-time, industry wide resolution.

c. Some F&Os may be resolved with alternative technical modifications to the method or its assumptions and licensee specific resolutions may result in multiple variations, each of which would need to be reviewed by the NRC staff when used in a LAR.

5) Similar to the current peer review process, it is intended that, by meeting all the SRs under a given HLR, the NDM will satisfy the intent of the HLRs and therefore that the method should be technical acceptable.

a. None of the three pilot NDM reviews were completed such that all SRs were met and there were no F&Os. Part of the pilot review guidance requested that the peer review team to make an overall conclusion on the acceptability of the method, but the overall conclusions, as discussed above, included cautions about the as yet incomplete reviews and therefore are not useful. How much this depended on the observation that no NDM reviews were completed so that no F&Os remain, versus a reluctance by the peer reviewers to make declaratory statements concerning the entire proposed method is unknown.

b. Conversely, the peer review teams appeared to be able to make declaratory statements that SRs had been met when they so concluded, even if there appeared sometimes to be a limited basis for these statements.

c. Logically, if all SRs are met, the NDM will have met all the guidelines it was reviewed against and if the basis for each met is adequately described, the NDM should be acceptable and the associated documentation should be complete.

6) Plant specific peer review reports are not publicly available, but aspects of the methodology reports should be publicly available. If performed systematically and

comprehensible, the NDM peer review reports (or series of reports if multiple reviews are need) should provide valuable information that to support the demonstration of the acceptability of any NDM that successfully satisfies all the SRs.