ML20080N774: Difference between revisions

From kanterella
Jump to navigation Jump to search
(StriderTol Bot insert)
 
(StriderTol Bot change)
 
(One intermediate revision by the same user not shown)
Line 18: Line 18:
History, Status, Commentary and Challenges Nathan Siu Senior Technical Adviser for PRA Analysis U.S. Nuclear Regulatory Commission Expanded version of a presentation originally developed for CRIEPI/NRRC and OECD/NEA Workshop on the Proper Treatment of Uncertainties in Reactor Safety Assessment March, 2020
History, Status, Commentary and Challenges Nathan Siu Senior Technical Adviser for PRA Analysis U.S. Nuclear Regulatory Commission Expanded version of a presentation originally developed for CRIEPI/NRRC and OECD/NEA Workshop on the Proper Treatment of Uncertainties in Reactor Safety Assessment March, 2020


Foreword On December 19, 2019, the Nuclear Risk Research Center (NRRC) of the Japan Central Research Institute of Electric Power Industry (CRIEPI) and the Organization for Economic Cooperation (OECD) Nuclear Energy Agency (NEA) invited the author to participate in a workshop on the improvement and enhancement of risk-informed decision making (RIDM) processes in reactor safety assessment. The workshop, titled A Workshop on the Proper Treatment of Uncertainties in Reactor Safety Assessment, was to be held on May 26-27, 2020 in Tokyo, Japan. At the request of the workshop organizers, the authors talk was to be titled Technology for the Treatment of Uncertainties: History, Status, and Some Challenges. On March 12, due to travel restrictions arising from the covid-19 pandemic, the author was directed to withdraw from the workshop. The following slides are an expanded version of the talk the author was planning on presenting.
2 Foreword On December 19, 2019, the Nuclear Risk Research Center (NRRC) of the Japan Central Research Institute of Electric Power Industry (CRIEPI) and the Organization for Economic Cooperation (OECD) Nuclear Energy Agency (NEA) invited the author to participate in a workshop on the improvement and enhancement of risk-informed decision making (RIDM) processes in reactor safety assessment. The workshop, titled A Workshop on the Proper Treatment of Uncertainties in Reactor Safety Assessment, was to be held on May 26-27, 2020 in Tokyo, Japan. At the request of the workshop organizers, the authors talk was to be titled Technology for the Treatment of Uncertainties: History, Status, and Some Challenges. On March 12, due to travel restrictions arising from the covid-19 pandemic, the author was directed to withdraw from the workshop. The following slides are an expanded version of the talk the author was planning on presenting.
2


Outline
3 Outline
* Framework for discussion             tech*nol*o*gy, n. the sum of
* Framework for discussion
  - Parameter Uncertainties           techniques, skills, methods, and
- Parameter Uncertainties
  - Model Uncertainties               processes used in the production of
- Model Uncertainties
  - Completeness Uncertainties       goods or services or in the accomplishment of objectives, such as
- Completeness Uncertainties
  - Communication scientific investigation. [Wikipedia]
- Communication
* Current state of practice
* Current state of practice
* History                             In this talk:
* History
technology {methods, models,
* Commentary and challenges tech*nol*o*gy, n. the sum of techniques, skills, methods, and processes used in the production of goods or services or in the accomplishment of objectives, such as scientific investigation. [Wikipedia]
* Commentary and challenges            computational tools, guidance, data}
In this talk:
3
technology {methods, models, computational tools, guidance, data}


What are we talking about?
4 DISCUSSION FRAMEWORK What are we talking about?
DISCUSSION FRAMEWORK 4


Discussion Framework Context for Treatment of                 P{XlC,H}
5 Context for Treatment of Uncertainties: Risk-Informed Decisionmaking (RIDM)
Uncertainties: Risk-Informed subjective                          knowledge Decisionmaking (RIDM)               proposition conditions Adapted from NUREG-2150 5
P{XlC,H}
subjective proposition conditions knowledge Discussion Framework Adapted from NUREG-2150


Discussion Framework Parameter, Model, and Completeness Uncertainty:
6 Parameter, Model, and Completeness Uncertainty:
A Practical Categorization mod*el, n. a M (Model of the World):         representation of reality created with a specific Scope, structure              objective in mind.
A Practical Categorization M (Model of the World):
i: Parameters                  A. Mosleh, N. Siu, C. Smidts, and C. Lui, Model Uncertainty: Its Characterization and
Scope, structure i: Parameters
: Universe                      Quantification, Center for Reliability Engineering, University of Maryland, College Park, MD, 1995. (Also NUREG/CP-0138, 1994)
: Universe Known Unknowns Unknown Unknowns Discussion Framework mod*el, n. a representation of reality created with a specific objective in mind.
PRA models for NPPs
A. Mosleh, N. Siu, C. Smidts, and C. Lui, Model Uncertainty: Its Characterization and Quantification, Center for Reliability Engineering, University of Maryland, College Park, MD, 1995. (Also NUREG/CP-0138, 1994)
* Typically an assemblage of sub-models with parameters
PRA models for NPPs Typically an assemblage of sub-models with parameters Implicitly include issues considered but not explicitly quantified
* Implicitly include issues considered but not explicitly Known Unknowns                    quantified Unknown Unknowns 6


Discussion Framework Parameter, Model, and Completeness Uncertainty:
7 Parameter, Model, and Completeness Uncertainty:
A Practical Categorization mod*el, n. a M (Model of the World):     representation of reality created with a specific Scope, structure          objective in mind.
A Practical Categorization M (Model of the World):
i: Parameters                A. Mosleh, N. Siu, C. Smidts, and C. Lui, Model Uncertainty: Its Characterization and
Scope, structure i: Parameters
: Universe                  Quantification, Center for Reliability Engineering, University of Maryland, College Park, MD, 1995. (Also NUREG/CP-0138, 1994)
: Universe Known Unknowns Unknown Unknowns Discussion Framework mod*el, n. a representation of reality created with a specific objective in mind.
PRA models for NPPs
A. Mosleh, N. Siu, C. Smidts, and C. Lui, Model Uncertainty: Its Characterization and Quantification, Center for Reliability Engineering, University of Maryland, College Park, MD, 1995. (Also NUREG/CP-0138, 1994)
* Distinctions are not necessarily crisp
PRA models for NPPs Distinctions are not necessarily crisp Regardless of allocation to categories, need to consider in characterization of uncertainties
* Regardless of allocation to categories, need to consider Known Unknowns                    in characterization of Unknown Unknowns                uncertainties 7


Discussion Framework Parameter Uncertainty: An Example Flood Height Date (ft)
8 Parameter Uncertainty: An Example
* Parameter of interest: frequency of flooding ()                                                       3/19/1936            36.5
* Parameter of interest: frequency of flooding ()
* Prior state-of-knowledge: minimal                                                                     6/1/1889             34.8 10/16/1942             33.8
* Prior state-of-knowledge: minimal
* Evidence: 10 events over 1877-2017 (140 years)                                                        10/1/1896             33.0 11/6/1985             30.1
* Evidence: 10 events over 1877-2017 (140 years)
* Posterior state-of-knowledge:                                                                          9/8/1996             29.8 1/21/1996             29.4 05 = 0.040/yr Probability Density prior                                                                                          11/25/1877             29.2 50 = 0.069/yr 95 = 0.11/yr                            4/27/1937             29.0 posterior mean = 0.071/yr                          6/23/1972             27.7 return period = 12 yr 0.00           0.05   0.10           0.15             0.20       0.25         0.30
* Posterior state-of-knowledge:
: Flood Frequency (/yr)                                   1880   1900   1920   1940     1960     1980   2000 8
Date Flood Height (ft) 3/19/1936 36.5 6/1/1889 34.8 10/16/1942 33.8 10/1/1896 33.0 11/6/1985 30.1 9/8/1996 29.8 1/21/1996 29.4 11/25/1877 29.2 4/27/1937 29.0 6/23/1972 27.7 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Probability Density
: Flood Frequency (/yr) 05 = 0.040/yr 50 = 0.069/yr 95 = 0.11/yr mean = 0.071/yr prior posterior Discussion Framework return period = 12 yr 1880 1900 1920 1940 1960 1980 2000


Discussion Framework Model Uncertainty:
9 Hurricane Andrew: 8/22/1992, 1200 UTC (about 2 days before FL landfall)
Hurricane Example Hurricane Andrew: 8/22/1992, 1200 UTC (about 2 days before FL landfall)
Plot adapted from University of Wisconsin-Milwaukee (https://web.uwm.edu/hurricane-models/models/archive/)
Plot adapted from University of Wisconsin-Milwaukee (https://web.uwm.edu/hurricane-models/models/archive/)
9
Model Uncertainty:
Hurricane Example Discussion Framework


Discussion Framework Completeness Uncertainty:
10 https://en.wikipedia.org/wiki/Hurricane_Irma#/media/File:Irma,_Jose_and_Katia_2017-09-07.png Completeness Uncertainty:
Multiple Hurricane Example (A Known Unknown)
Multiple Hurricane Example (A Known Unknown)
Turkey Point Katia Irma Jose 10  https://en.wikipedia.org/wiki/Hurricane_Irma#/media/File:Irma,_Jose_and_Katia_2017-09-07.png
Irma Jose Katia Discussion Framework Turkey Point


Discussion Framework Risk Communication (Internal)
11 Risk Communication (Internal)
Adapted from NUREG-2150 Other Considerations
Discussion Framework Other Considerations Current regulations Safety margins Defense-in-depth Monitoring Quantitative Qualitative Adapted from NUREG-2150
* Current regulations
* Safety margins
* Defense-in-depth
* Monitoring Quantitative Qualitative 11


What do people do now?
12 CURRENT STATE-OF-PRACTICE What do people do now?
CURRENT STATE-OF-PRACTICE 12


Current State of Practice State-of-Practice: Parameter Uncertainties
13 State-of-Practice: Parameter Uncertainties
* Treatment involves Estimation (including expert elicitation)
* Treatment involves Estimation (including expert elicitation)
Propagation
Propagation
* Straightforward mathematics and mechanics
* Straightforward mathematics and mechanics
* Some practical challenges 13
* Some practical challenges Current State of Practice


Current State of Practice State-of-Practice:                                                                                 Hurricane Andrew Model Uncertainties 8/22/1992, 1200 UTC Adapted from University of Wisconsin-Milwaukee (https://web.uwm.edu/hurricane-models/models/archive/)
14 State-of-Practice:
Model Uncertainties
* Important to acknowledge and treat (in context of decision)
* Important to acknowledge and treat (in context of decision)
* Multiple approaches
* Multiple approaches
    - Consensus model
- Consensus model
    - Sensitivity analysis
- Sensitivity analysis
    - Weighted alternatives (e.g., SSHAC)
- Weighted alternatives (e.g., SSHAC)
    - Output uncertainties Adapted from V.M. Andersen, Seismic Probabilistic Risk Assessment Implementation Guide, EPRI 3002000709, Electric Power Research Institute, M.H. Salley and A. Lindeman, Verification and Palo Alto, CA, December 2013 Validation of Selected Fire Models for Nuclear Power Plant Applications, NUREG-1824 Supplement 1/EPRI 3002002182, November 2016.
- Output uncertainties Current State of Practice Hurricane Andrew 8/22/1992, 1200 UTC Adapted from University of Wisconsin-Milwaukee (https://web.uwm.edu/hurricane-models/models/archive/)
14
Adapted from V.M. Andersen, Seismic Probabilistic Risk Assessment Implementation Guide, EPRI 3002000709, Electric Power Research Institute, Palo Alto, CA, December 2013 M.H. Salley and A. Lindeman, Verification and Validation of Selected Fire Models for Nuclear Power Plant Applications, NUREG-1824 Supplement 1/EPRI 3002002182, November 2016.


Current State of Practice State-of-Practice:                                       NUREG-1855 Rev. 1 (2017)
15 State-of-Practice:
Completeness Uncertainties                               Options:
Completeness Uncertainties
* Progressive analysis
* Potential concerns
* Potential concerns                                           (screening, bounding, conservative, detailed)
- Known gaps (missing scope)
    - Known gaps (missing scope)
* Scenario categories
* Change scope of risk-
* Scenario categories                                   informed application
* Contributors within categories
* Contributors within categories
    - Unknown gaps                                           RG 1.174 Rev. 3 (2019)
- Unknown gaps
    - Heuristics/biases
- Heuristics/biases
* Excessive amplification (fear of the dark)
* Excessive amplification (fear of the dark)
* Excessive discounting (out of sight, out of mind)
* Excessive discounting (out of sight, out of mind)
* Treatment
* Treatment
    - Analysis guidance
- Analysis guidance
    - Additional analysis/R&D
- Additional analysis/R&D
    - Risk-informed decisionmaking 15
- Risk-informed decisionmaking NUREG-1855 Rev. 1 (2017)
Options:
Progressive analysis (screening, bounding, conservative, detailed)
Change scope of risk-informed application RG 1.174 Rev. 3 (2019)
Current State of Practice


Current State of Practice State-of-Practice: Internal Risk Communication
16 State-of-Practice: Internal Risk Communication
* Often implicit (focus on mean values)
* Often implicit (focus on mean values)
* Various graphic displays
* Various graphic displays
* Includes story as well as numbers Likelihood Class 5 (10-5/yr)   4 (10-4/yr)     3 (10-3/yr)     2 (10-2/yr)   1 (10-1/yr)
* Includes story as well as numbers Current State of Practice Documents and Presentations (Flatland)
A   Marginal     Undesirable Undesirable           Critical       Critical Documents and          Interactive Severity Class Presentations        Discussion                        B   Marginal       Marginal       Undesirable Undesirable         Critical (Flatland)      (Storytelling)                    C   No Action     Marginal       Marginal       Undesirable Undesirable D   No Action     No Action       Marginal         Marginal     Undesirable E   No Action     No Action       No Action         Marginal     Marginal 16
Interactive Discussion (Storytelling)
Likelihood Class 5 (10-5/yr) 4 (10-4/yr) 3 (10-3/yr) 2 (10-2/yr) 1 (10-1/yr)
Severity Class A
Marginal Undesirable Undesirable Critical Critical B
Marginal Marginal Undesirable Undesirable Critical C
No Action Marginal Marginal Undesirable Undesirable D
No Action No Action Marginal Marginal Undesirable E
No Action No Action No Action Marginal Marginal


How did we get here?
17 A BRIEF HISTORY How did we get here?
A BRIEF HISTORY 17


History A Series of Challenges and Responses Modern Applications Expansion Across Industry Early PRAs Hanford to WASH-1400 1940 1950     1960     1970   1980     1990       2000   2010       2020 18
18 A Series of Challenges and Responses 1940 1950 1960 1970 1980 1990 2000 2010 2020 Hanford to WASH-1400 Early PRAs Expansion Across Industry Modern Applications History


History From Hanford to WASH-1400 Technical Challenges: 1) Quantifying accident probability
19 TMI-2 From Hanford to WASH-1400 SGHWR analysis WASH-740 For more information: T.R. Wellock, A Figure of Merit: Quantifying the Probability of a Nuclear Reactor Accident, Technology and Culture, 58, No. 3, July 2017, pp. 678-721.
: 2) Means to communicate risk WASH-740                                                                                             Hanford AEC/NRC Credible Accident UKAEA Estimates:
Credible Accident System reliability studies Recommend:
not in the generation
accident chain analysis Hanford AEC/NRC UKAEA Technical Challenges: 1) Quantifying accident probability
                                                              -  OpE (pessimistic) of the ACRS members                  -  Decomposition present                          (optimistic)
: 2) Means to communicate risk not in the generation of the ACRS members present Farmer Curve WASH-1400 Estimates:
Recommend:                                                                  Farmer Curve                      WASH-1400 accident                                              System chain              System reliability            reliability                    SGHWR analysis                    studies                    studies                      analysis 1950                        Windscale        1960                                                1970                              TMI-2 1980 For more information: T.R. Wellock, A Figure of Merit: Quantifying the Probability of a Nuclear Reactor Accident, 19      Technology and Culture, 58, No. 3, July 2017, pp. 678-721.
OpE (pessimistic)
Decomposition (optimistic)
History Windscale 1950 1960 1970 1980 System reliability studies


History WASH-1400 Uncertainties (Level 1)
20 WASH-1400 Uncertainties (Level 1)
WASH-1400: it is reasonable to believe that the                 WASH-1400 Uncertainties (Estimated*)
WASH-1400: it is reasonable to believe that the core melt probability of about 5x10-5 per reactor-year predicted by this study should not be significantly larger and would almost certainly not exceed the value of 3x10-4 which has been estimated as the upper bound for core melt probability.
core melt probability of about 5x10-5 per reactor-year predicted by this study should not be significantly larger and would almost certainly not exceed the value of 3x10-4 which has been estimated as the upper 5th            50th              95th Surry mean bound for core melt probability.
Risk Assessment Review Group (NUREG/CR-0400):
Peach Bottom Risk Assessment Review Group (NUREG/CR-0400):
We are unable to define whether the overall probability of a core melt given in WASH-1400 is high or low, but we are certain that the error bands are understated. We cannot say by how much.
We are unable to define whether the overall             1.E-05                            1.E-04                      1.E-03 CDF (/ry) probability of a core melt given in WASH-1400 is high or low, but we are certain that the error bands are   *Based on data from Tables V 3-14 (PWR) and 3-16 (BWR) of Appendix V, assuming distributions are lognormal; median values are somewhat higher understated. We cannot say by how much.                than reported in Section 7.3.1 of the Main Report.
1.E-05 1.E-04 1.E-03 CDF (/ry)
20
WASH-1400 Uncertainties (Estimated*)
Surry Peach Bottom 5th 50th 95th mean
*Based on data from Tables V 3-14 (PWR) and 3-16 (BWR) of Appendix V, assuming distributions are lognormal; median values are somewhat higher than reported in Section 7.3.1 of the Main Report.
History


History Some Early Developments and PRAs Challenges: 1) Filling known gaps (completeness uncertainty)
21 TMI-2 Chernobyl Some Early Developments and PRAs Challenges: 1) Filling known gaps (completeness uncertainty)
: 2) Clarifying meaning: models and results Biblis Sizewell
: 2) Clarifying meaning: models and results Clinch River (LMFBR)
Limerick Millstone Seabrook (full scope)
Fleming
(-factor)
Zion (full scope)
TMI-1 (full scope)
Oconee (full scope) 1980 1985 1975 Apostolakis (subjective probability)
Forsmark Koeberg
(~WASH-1400)
Super Phénix (FBR DHR)
AIPA (HTGR)
USDOE NRC US Industry International Other Notable Kaplan/
Garrick (risk)
History EC/JRC Benchmarks (systems, CCF, HRA)
RSSMAP/IREP Sizewell
(+DI&C)
Indian Point (full scope)
Oyster Creek
(+seismic)
Biblis
(+aircraft)
(+aircraft)
(+DI&C)                                                    USDOE Clinch River        Oyster Creek                                                                                    NRC (LMFBR)
NUREG/CR-2300
Indian Point
(+seismic)
(full scope)
US Industry AIPA            Forsmark                                                                                                International Limerick (HTGR)            Koeberg                          Zion Millstone                                        Other Notable
(~WASH-1400)                    (full scope)
Seabrook Super                                                                    (full scope)
Phénix                                  RSSMAP/IREP (FBR DHR)                                                                                                            TMI-1 Oconee (full scope)
Apostolakis                      Kaplan/                            (full scope)
Fleming                      (subjective                    Garrick                                                  EC/JRC Benchmarks
(-factor)                  probability)                    (risk)        NUREG/CR-2300                           (systems, CCF, HRA) 1975                                        TMI-2    1980                                                  1985 Chernobyl 21


History Sample Level 1 Results Display 22
22 Sample Level 1 Results Display History


History Sample Results - Sub-Model Uncertainty Effect Effects of fire model (COMPBRN) uncertainty on fire growth time N. Siu, "Modeling Issues in Nuclear Plant Fire Risk Analysis," in EPRI Workshop on Fire Protection in Nuclear Power Plants, EPRI NP-6476, J.-P. Sursock, ed., August 1989, pp. 14-1 through 14-16.
23 Sample Results - Sub-Model Uncertainty Effect History Effects of fire model (COMPBRN) uncertainty on fire growth time N. Siu, "Modeling Issues in Nuclear Plant Fire Risk Analysis," in EPRI Workshop on Fire Protection in Nuclear Power Plants, EPRI NP-6476, J.-P. Sursock, ed., August 1989, pp. 14-1 through 14-16.
23


History Sample Results - Model Uncertainty (User Effect)
24 Sample Results - Model Uncertainty (User Effect)
Early core melt, containment cooling Early core melt, no containment cooling Damage State Frequency (/yr), Review 10-4                                                                         Late core melt, containment cooling Late core melt, no containment cooling Containment bypass Steam generator tube rupture Direct containment failure 10-6 Internal Events                                                                                External Events 10-8                                                                                                1.E-03                                                                                        1.E-03 1.E-04                                                                                        1.E-04 1.E-05                                                                                        1.E-05 1.E-06 Review 1.E-06 Review 1.E-07                                                                                        1.E-07 10-10                                                                                              1.E-08                                                                                        1.E-08 1.E-09                                                                                        1.E-09 1.E-10                                                                                        1.E-10 1.E-11                                                                                        1.E-11 1.E-11   1.E-10   1.E-09   1.E-08   1.E-07   1.E-06   1.E-05   1.E-04   1.E-03 10-10          10-8            10-6            10-4 1.E-11   1.E-10   1.E-09   1.E-08   1.E-07   1.E-06   1.E-05   1.E-04   1.E-03 Original                                                                                         Original Damage State Frequency (/yr), Original Data source: G.J. Kolb, et al., Review and Evaluation of the Indian Point Probabilistic Safety Study, NUREG/CR-2934, December 1982.
Damage State Frequency (/yr), Review Damage State Frequency (/yr), Original 10-10 10-8 10-6 10-4 10-10 10-8 10-6 10-4 Early core melt, containment cooling Early core melt, no containment cooling Steam generator tube rupture Containment bypass Direct containment failure Late core melt, containment cooling Late core melt, no containment cooling 1.E-11 1.E-10 1.E-09 1.E-08 1.E-07 1.E-06 1.E-05 1.E-04 1.E-03 1.E-11 1.E-10 1.E-09 1.E-08 1.E-07 1.E-06 1.E-05 1.E-04 1.E-03 Original Review Internal Events 1.E-11 1.E-10 1.E-09 1.E-08 1.E-07 1.E-06 1.E-05 1.E-04 1.E-03 1.E-11 1.E-10 1.E-09 1.E-08 1.E-07 1.E-06 1.E-05 1.E-04 1.E-03 Original Review External Events Data source: G.J. Kolb, et al., Review and Evaluation of the Indian Point Probabilistic Safety Study, NUREG/CR-2934, December 1982.
24                                        (ML091540534)
(ML091540534)
History


History Severe                  Expansion Across Industry (US)
25 Chernobyl 9/11 Expansion Across Industry (US)
Accident Policy              Technical challenges: 1) Characterizing the fleet (variability)
Technical challenges: 1) Characterizing the fleet (variability)
Statement                                            2) Developing confidence for mainstreaming RIDM Safety Goal                                        PRA Policy                        NRC Policy                                        Statement Statement                                                                            US Industry GL 88-20 GL 88-20         Supplement 4                         NUREG-1560               NUREG-1742 NUREG-1150   NUREG-1150 (draft)   (final)            1982      ASP Plant Class Models     SPAR Models IPEEEs IPEs 1985 Chernobyl            1990                            1995                      2000 9/11 25
: 2) Developing confidence for mainstreaming RIDM 1985 1990 2000 1995 GL 88-20 GL 88-20 Supplement 4 NUREG-1560 NUREG-1742 NUREG-1150 (final)
NUREG-1150 (draft)
Severe Accident Policy Statement Safety Goal Policy Statement PRA Policy Statement ASP Plant Class Models 1982 SPAR Models History NRC US Industry IPEs IPEEEs


History NUREG-1150 Estimated* Uncertainties (Level 1)
26 NUREG-1150 Estimated* Uncertainties (Level 1)
Model Uncertainty Model Uncertainty
Model Uncertainty Model Uncertainty
  *Notes: totals shown in this
*Notes: totals shown in this 1)
: 1) NUREG-1150 does not aggregate the hazard-specific results. The totals shown are rough estimates assuming that the NUREG-1150 distributions are lognormal.
NUREG-1150 does not aggregate the hazard-specific results. The totals shown are rough estimates assuming that the NUREG-1150 distributions are lognormal.
26 2) The WASH-1400 distributions are based on data from Tables V 3-14 (PWR) and 3-16 (BWR) of Appendix V, assuming that the distributions are lognormal. The median values are somewhat higher than reported in Section 7.3.1 of the Main Report
2)
The WASH-1400 distributions are based on data from Tables V 3-14 (PWR) and 3-16 (BWR) of Appendix V, assuming that the distributions are lognormal. The median values are somewhat higher than reported in Section 7.3.1 of the Main Report History


History IPE/IPEEE - Variability Across Fleet Internal Events + Internal Floods                                                                      Total 40                                                                                 40 BWR                                                                          BWR PWR                                                                           PWR 30                                                                                30 Number                                                                        Number 20                                                                                20 10                                                                                10 0                                                                              0 1x10-6   3x10-6   1x10-5   3x10-5   1x10-4   3x10-4   1x10-3                 1x10-6   3x10-6   1x10-5   3x10-5   1x10-4   3x10-4   1x10-3 CDF (/ry)                                                                  CDF (/ry) 27
27 IPE/IPEEE - Variability Across Fleet 0
10 20 30 40 Number BWR PWR CDF (/ry) 1x10-6 3x10-6 1x10-5 3x10-5 1x10-4 3x10-4 1x10-3 Internal Events + Internal Floods 0
10 20 30 40 Number BWR PWR CDF (/ry) 1x10-6 3x10-6 1x10-5 3x10-5 1x10-4 3x10-4 1x10-3 Total History


History The Modern Era (US)
28 9/11 The Modern Era (US)
Technical challenges: 1) RIDM issues (e.g., realism, heterogeneity, aggregation)
Technical challenges: 1) RIDM issues (e.g., realism, heterogeneity, aggregation)
SECY-98-144                                            2) Post-Fukushima issues (e.g., external hazards)
: 2) Post-Fukushima issues (e.g., external hazards)
: 3) New/advanced reactors (e.g., conduct of operations)
: 3) New/advanced reactors (e.g., conduct of operations)
RG 1.174 NUREG-2150 ASME PRA                                       NRC Risk-Standard       NTTF Request                  US Industry Informed for Information ROP                                    NUREG-1855        (Reevaluations) 10 CFR 50.48(c)
NUREG-1855 History Fukushima RG 1.174 ASME PRA Standard 10 CFR 50.48(c)
NFPA 805 (Fire Protection)                           NFPA 805 LARs (Fire Protection)
(Fire Protection)
Risk-Informed ROP NFPA 805 NUREG-2150 NTTF Request for Information (Reevaluations) 2000 2010 2020 2005 2015 NRC US Industry SECY-98-144 Risk-Informed License Amendment Requests (LARs)
SAMAs (Life Extension)
SAMAs (Life Extension)
Risk-Informed License Amendment Requests (LARs)
SPAR Models NFPA 805 LARs (Fire Protection)
SPAR Models 2000  9/11              2005                      2010    Fukushima        2015                  2020 28


History Variability in Recent Results (Level 1) 0.35 0.30 Population Mean:
29 Variability in Recent Results (Level 1)
4.7x10-5 0.25 Fraction of Plants 0.20 0.15 0.10 Lowest                                    Highest Reported:                                Reported:
History 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35
0.05        3.5x10-6                                 1.3x10-4 0.00
-6.0
                                  -6.0        -5.5       -5.0     -4.5     -4.0         -3.5   -3.0 1E-6                     1E-5               1E-4                 1E-3 CDF (per reactor year) 29
-5.5
-5.0
-4.5
-4.0
-3.5
-3.0 1E-6 1E-5 1E-4 1E-3 CDF (per reactor year)
Fraction of Plants Highest Reported:
1.3x10-4 Lowest Reported:
3.5x10-6 Population Mean:
4.7x10-5


History Variability in Results - Comparison with IPE/IPEEE 1E-3 0.001 0.50 NFPA 805 Total CDF (IPE + IPEEE) 0.40 Fraction of PRAs IPE/IPEEE 0.30 1E-4 0.0001 0.20 0.10 0.00 1   2         3   4       5   6       7   8         9   10 0.01       0.1           1           10           100       1000 1E-5 0.00001 1E-5 1.00E-05             1E-4 1.00E-04         1E-3 1.00E-03                                   Fire CDF/Internal Events CDF Total CDF (Recent LARs) 30
30 Variability in Results - Comparison with IPE/IPEEE History 0.00 0.10 0.20 0.30 0.40 0.50 1
2 3
4 5
6 7
8 9
10 NFPA 805 IPE/IPEEE 0.01 0.1 1
10 100 1000 Fire CDF/Internal Events CDF Fraction of PRAs 0.00001 0.0001 0.001 1.00E-05 1.00E-04 1.00E-03 Total CDF (IPE + IPEEE)
Total CDF (Recent LARs) 1E-5 1E-4 1E-3 1E-5 1E-4 1E-3


Where might we do better and how?
31 COMMENTARY AND CHALLENGES Where might we do better and how?
COMMENTARY AND CHALLENGES 31


Commentary and Challenges An Important Note
32 An Important Note
* Challenges regarding the treatment of uncertainty in PRA and RIDM exist for non-probabilistic approaches as well; the PRA/RIDM approach acknowledges these challenges explicitly.
* Challenges regarding the treatment of uncertainty in PRA and RIDM exist for non-probabilistic approaches as well; the PRA/RIDM approach acknowledges these challenges explicitly.
* The following slides are not a critique of the overall PRA/RIDM philosophy - they should be viewed in the framework of continuous improvement.
* The following slides are not a critique of the overall PRA/RIDM philosophy - they should be viewed in the framework of continuous improvement.
32
Commentary and Challenges


Commentary and Challenges A Changing World
33 A Changing World
* Evolving situation*
* Evolving situation*
    - market forces
- market forces
    - new nuclear technologies
- new nuclear technologies
    - new analytical methods and data
- new analytical methods and data
    - new professionals
- new professionals
* Increased reliance on risk models, characterization of uncertainties
* Increased reliance on risk models, characterization of uncertainties
      *See Applying the Principles of Good Regulation as a Risk-Informed Regulator, 33 October 15, 2019 (ADAMS ML19260E683)
*See Applying the Principles of Good Regulation as a Risk-Informed Regulator, October 15, 2019 (ADAMS ML19260E683)
Commentary and Challenges


Commentary and Challenges: Parameter Uncertainties Reminder: Parameter Uncertainties and Mean Values
34 Reminder: Parameter Uncertainties and Mean Values Commentary and Challenges: Parameter Uncertainties Mean = 7.6 x 10-5 /yr 95th = 2.6 x 10-4 /yr 50th (Median) = 3.9 x 10-5 /yr probability density function frequency (/yr)
* Mathematically defined probability density function Mean
Mean 0
* Affected by tail
* Does not correspond to 0
50th (Median) = 3.9 x 10-5 /yr                                         a specific percentile Mean = 7.6 x 10-5 /yr 95th = 2.6 x 10-4 /yr frequency (/yr) 34


Commentary and Challenges: Parameter Uncertainties Parameter Uncertainties: Challenges
Mathematically defined Affected by tail Does not correspond to a specific percentile
* Quantification generally required, diverse views on value added                             2015 Industry-wide estimates from: https://nrcoe.inl.gov/resultsdb/AvgPerf/
 
* Service Water Pumps: 2 failures in 16,292,670 hours
35 Parameter Uncertainties: Challenges
* Quantification generally required, diverse views on value added
* Technical challenges:
* Technical challenges:
* Normally Running Pumps: 225 failures in 59,582,350 hours Standby Pumps (1st hour operation): 48 failures in 437,647 hours
- Effect of data pre-processing
    - Effect of data pre-processing
* Selection
* Selection Probability Density Function Service Water
* Interpretation
* Interpretation                                                                                                                     Normally Running
- Effect of analysis shortcuts
    - Effect of analysis shortcuts                                                                                                           Standby (Normalized)
* Standard prior distributions
* Standard prior distributions
* Simplified expert elicitation
* Simplified expert elicitation
* Independence assumption 1.E-09      1.E-08        1.E-07        1.E-06        1.E-05      1.E-04        1.E-03
* Independence assumption
    - Ensuring correspondence with actual                                                                   Failure Rate (/hr) state-of-knowledge
- Ensuring correspondence with actual state-of-knowledge
* Basic events (micro)
* Basic events (micro)
* Overall results (macro) 35
* Overall results (macro)
Commentary and Challenges: Parameter Uncertainties 1.E-09 1.E-08 1.E-07 1.E-06 1.E-05 1.E-04 1.E-03 Probability Density Function (Normalized)
Failure Rate (/hr)
Service Water Normally Running Standby 2015 Industry-wide estimates from: https://nrcoe.inl.gov/resultsdb/AvgPerf/
Service Water Pumps: 2 failures in 16,292,670 hours Normally Running Pumps: 225 failures in 59,582,350 hours Standby Pumps (1st hour operation): 48 failures in 437,647 hours


Commentary and Challenges: Model Uncertainties Model Uncertainties - Commentary
36 Model Uncertainties - Commentary
* Model uncertainties can be large; importance depends on decision Hurricane Irma: 9/8/2017, 0000 UTC
* Model uncertainties can be large; importance depends on decision
* Some practical approaches (e.g., consensus                                               (about 2 days before FL landfall) models, deterministic screening) can understate uncertainties                                                 Outer
* Some practical approaches (e.g., consensus models, deterministic screening) can understate uncertainties
* Subjective probability framework =>                           prediction is closest
* Subjective probability framework =>
    - Need to include user effect                             to actual course
- Need to include user effect
    - Raises question regarding fundamental meaning of weighted average approaches                         Plot adapted from University of Wisconsin-Milwaukee (https://web.uwm.edu/hurricane-models/models/archive/)
- Raises question regarding fundamental meaning of weighted average approaches
* Model output uncertainty approach is appealing but care is needed in implementation 36
* Model output uncertainty approach is appealing but care is needed in implementation Commentary and Challenges: Model Uncertainties Plot adapted from University of Wisconsin-Milwaukee (https://web.uwm.edu/hurricane-models/models/archive/)
Hurricane Irma: 9/8/2017, 0000 UTC (about 2 days before FL landfall)
Outer prediction is closest to actual course


Commentary and Challenges: Model Uncertainties Model Uncertainty User Effects: HRA Example 1 NRC, SPAR-H        INL, SPAR-H Same method, different teams NRI, CREAM       NRI, DT+ASEP Same team, different methods All teams, all methods A Bye, et al., International HRA Empirical Study, NUREG/IA-0216, August 2011.
37 Model Uncertainty User Effects: HRA Example 1 Same method, different teams Same team, different methods All teams, all methods NRI, CREAM NRI, DT+ASEP NRC, SPAR-H INL, SPAR-H A Bye, et al., International HRA Empirical Study, NUREG/IA-0216, August 2011.
37
Commentary and Challenges: Model Uncertainties


Commentary and Challenges: Model Uncertainties Model Uncertainty User Effects: HRA Example 2 1.0E+0 Human Error Probability (HEP)
38 Model Uncertainty User Effects: HRA Example 2 HFE 2A HFE 1C HFE 1A HFE 3A HFE 1B Decreasing difficulty Human Error Probability (HEP) 1.0E+0 1.0E-1 1.0E-2 1.0E-3 1.0E-4 1.0E-5 ASEP Team 1 ASEP Team 2 SPAR-H Team 1 SPAR-H Team 2 CBDT & HCR/ORE Team 1 CBDT & HCR/ORE Team 2 CBDT & HCR/ORE Team 3 ATHEANA Team 1 ATHEANA Team 2 Empirical 95th Percentile Empirical 5th Percentile Adapted from NUREG-2156 Commentary and Challenges: Model Uncertainties
ASEP Team 1 1.0E-1                                                 ASEP Team 2 SPAR-H Team 1 SPAR-H Team 2 1.0E-2 CBDT & HCR/ORE Team 1 CBDT & HCR/ORE Team 2 1.0E-3                                                  CBDT & HCR/ORE Team 3 ATHEANA Team 1 1.0E-4                                                  ATHEANA Team 2 Empirical 95th Percentile Empirical 5th Percentile 1.0E-5 HFE 2A  HFE 1C  HFE 1A  HFE 3A    HFE 1B Decreasing difficulty Adapted from NUREG-2156 38


Commentary and Challenges: Model Uncertainties Challenges: Quantification of Model Output Uncertainty Time (s)    Experiment (K)    DRM (K)
39 Challenges: Quantification of Model Output Uncertainty
* Bayesian methods                                                                                           180            400            450 Data                    360            465            510
* Bayesian methods
    - Framework consistent with overall PRA                                                                 720            530            560
- Framework consistent with overall PRA
    - Early approaches used in past PRAs                                                                   840            550            565
- Early approaches used in past PRAs
    - Can address practical issues (e.g., non-                                                                             Temperature (K) homogeneous data)*                                                                                               Assume        Assume Non-Percentile  Homogeneous      Homogeneous Output Uncertainty
- Can address practical issues (e.g., non-homogeneous data)*
* Challenges include                                                                                                     Data              Data 1st          415.2          372.8
* Challenges include
    - Uncertainties in unmeasured parameters                                                               5th          437.5          400.7
- Uncertainties in unmeasured parameters
    - Sub-model limits of applicability                                                                     50th          457.1          470.5
- Sub-model limits of applicability
    - Representativeness of computed results                                                               95th          479.7          559.4 99th          509.1          608.7
- Representativeness of computed results Time (s)
      *See E. Droguett and Ali Mosleh, Bayesian methodology for model uncertainty using model performance data, 39      Risk Analysis, 28, No. 5, 1457-1476, 2008.
Experiment (K)
DRM (K) 180 400 450 360 465 510 720 530 560 840 550 565
*See E. Droguett and Ali Mosleh, Bayesian methodology for model uncertainty using model performance data, Risk Analysis, 28, No. 5, 1457-1476, 2008.
Commentary and Challenges: Model Uncertainties Temperature (K)
Percentile Assume Homogeneous Data Assume Non-Homogeneous Data 1st 415.2 372.8 5th 437.5 400.7 50th 457.1 470.5 95th 479.7 559.4 99th 509.1 608.7 Data Output Uncertainty


Commentary and Challenges: Completeness Uncertainties Completeness Uncertainty It would                cease to be a
40 Completeness Uncertainty
* Sources danger if we could define it.
* Sources
  - Known gaps (missing scope)                                                               - Sherlock Holmes
- Known gaps (missing scope)
  - Unknown gaps                                                                       (The Adventure of the Copper Beeches)
- Unknown gaps
* Concerns Car Wont Start
* Concerns
  - Excessive amplification (Fear of the dark)
- Excessive amplification (Fear of the dark)
  - Excessive discounting (availability heuristic:
- Excessive discounting (availability heuristic:
Out of sight, out of mind)                       Battery Charge Insufficient Fuel System Defective Other Engine Problems All Other Problems Starting System          Ignition System        Mischievous Acts Defective                Defective              Of Vandalism B. Fischhoff, P. Slovic, S. Lichtenstein, Fault trees: Sensitivity of estimated failure probabilities to problem representation, Journal of Experimental Psychology: Human Perception and Performance, 4(2), May 1978, 330-344.
Out of sight, out of mind)
40
It would cease to be a danger if we could define it.
Sherlock Holmes (The Adventure of the Copper Beeches)
Commentary and Challenges: Completeness Uncertainties B. Fischhoff, P. Slovic, S. Lichtenstein, Fault trees: Sensitivity of estimated failure probabilities to problem representation, Journal of Experimental Psychology: Human Perception and Performance, 4(2), May 1978, 330-344.
Car Wont Start Battery Charge Insufficient Starting System Defective Ignition System Defective Mischievous Acts Of Vandalism All Other Problems Fuel System Defective Other Engine Problems


Commentary and Challenges: Completeness Uncertainties Known Gaps (Known Unknowns)
41 Known Gaps (Known Unknowns)
* Broad scenario categories Rationale                                 Common Example(s)
* Broad scenario categories
Out of scope                             security/sabotage, operation outside approved limits Low significance (pre-analysis judgment) external floods (many plants pre-Fukushima)
* Contributors within categories
Appropriate PRA technology* unavailable   management and organizational factors PRA not appropriate                     software, security
*Technology = {methods, models, tools, data}
* Contributors within categories Category                                 Example(s)
Rationale Common Example(s)
External hazards                         multiple coincident or sequential hazards Human reliability                       errors of commission, non-proceduralized recovery Passive systems                         thermal-hydraulic reliability 41      *Technology = {methods, models, tools, data}
Out of scope security/sabotage, operation outside approved limits Low significance (pre-analysis judgment) external floods (many plants pre-Fukushima)
Appropriate PRA technology* unavailable management and organizational factors PRA not appropriate software, security Category Example(s)
External hazards multiple coincident or sequential hazards Human reliability errors of commission, non-proceduralized recovery Passive systems thermal-hydraulic reliability Commentary and Challenges: Completeness Uncertainties


Commentary and Challenges: Completeness Uncertainties Unknown Unknowns: You Say Tomto Model
42 Unknown Unknowns: You Say Tomto Model Known Unknowns Unknown Unknowns
* Explicit or implicit?
* Explicit or implicit?
* Extent of coverage?                   Viewpoint Precise classification is Known                                  important only if it affects:
* Extent of coverage?
Unknowns
* Understanding
* Known by whom?
* Known by whom?
* Known when?
* Known when?
* Time from idea to theory to PRA implementation?
Viewpoint Precise classification is important only if it affects:
* Understanding
* Communication
* Communication
* Time from idea to theory
* Decision making Commentary and Challenges: Completeness Uncertainties
* Decision making to PRA implementation?
Unknown Unknowns 42


Commentary and Challenges: Completeness Uncertainties Unknown Unknowns: A Demonstrated Problem?
43 Unknown Unknowns: A Demonstrated Problem?
Then (a surprise?)
Model Known Unknowns Unknown Unknowns Then (a surprise?)
Now (treated in current PRAs?)
Now (treated in current PRAs?)
Browns Ferry fire (1975) - a long-recognized hazard; not in draft Model  WASH-1400 but routinely treated now TMI (1979) - precursors include Davis-Besse (1977); operator EOCs not in models; current recognition and some explorations Chernobyl (1986) - precursor at Leningrad (1975); non-routine test Known Unknowns during shutdown in any LPSD analyses?
Browns Ferry fire (1975) - a long-recognized hazard; not in draft WASH-1400 but routinely treated now Chernobyl (1986) - precursor at Leningrad (1975); non-routine test during shutdown in any LPSD analyses?
Blayais flood (1999) - external floods often screened at time; current recognition, multi-hazard under development Maanshan HEAF/SBO (2001) - HEAF phenomenon known, in any PRAs at time? Now included as an initiator; smoke effect?
TMI (1979) - precursors include Davis-Besse (1977); operator EOCs not in models; current recognition and some explorations Blayais flood (1999) - external floods often screened at time; current recognition, multi-hazard under development Maanshan HEAF/SBO (2001) - HEAF phenomenon known, in any PRAs at time? Now included as an initiator; smoke effect?
Davis-Besse RPV corrosion (2002) - RPV failure analyses focused on Unknown Unknowns crack propagation; M&O failure not in PRAs Fukushima Daiichi (2011) - precursors: Blayais (1999), Indian Ocean (2004), hazard under review at time; PRA models under development 43
Davis-Besse RPV corrosion (2002) - RPV failure analyses focused on crack propagation; M&O failure not in PRAs Fukushima Daiichi (2011) - precursors: Blayais (1999), Indian Ocean (2004), hazard under review at time; PRA models under development Commentary and Challenges: Completeness Uncertainties
 
44 Illuminating Uncertainties: From Lampposts to Search Beacons Wheres the goat???
Commentary and Challenges: Completeness Uncertainties


Commentary and Challenges: Completeness Uncertainties Illuminating Uncertainties: From Lampposts to Search Beacons Wheres the goat???
45 What Can We (PRA R&D) Do?
44
* Continue to develop technology to address known gaps
- Risk-informed prioritization
- Fully engage appropriate disciplines
- Take advantage of general computational and methodological developments
* Facilitate re-emphasis on searching
- Demonstrate efficiency and effectiveness with current tools (e.g., MLD, HBFT) vs.
checklist/screening
- Develop improved tools (including OpE mining)
Event (NUREG/CR-4839), 1992 Aircraft impact Avalanche Coastal erosion Drought External flooding Extreme winds and tornadoes Fire Fog Forest fire Frost Hail High tide, high lake level, or high river stage


Commentary and Challenges: Completeness Uncertainties What Can We (PRA R&D) Do?
Commentary and Challenges: Completeness Uncertainties
* Continue to develop technology to address                      Event (NUREG/CR-4839), 1992 known gaps                                                    Aircraft impact Avalanche
    - Risk-informed prioritization                              Coastal erosion
    - Fully engage appropriate disciplines                      Drought External flooding
    - Take advantage of general computational and                Extreme winds and tornadoes methodological developments                                Fire
* Facilitate re-emphasis on searching                            Fog Forest fire
    - Demonstrate efficiency and effectiveness with              Frost Hail current tools (e.g., MLD, HBFT) vs.                        High tide, high lake level, or high checklist/screening                                        river stage
    - Develop improved tools (including OpE mining) 45


Commentary and Challenges: Internal Risk Communication Sources of Breakdowns: Risk Communication Between Risk Managers and Public*
46 Sources of Breakdowns: Risk Communication Between Risk Managers and Public*
* Differences in perception of information
* Differences in perception of information
    - Relevance
- Relevance
    - Consistency with prior beliefs
- Consistency with prior beliefs
* Lack of understanding of underlying science
* Lack of understanding of underlying science
* Conflicting agendas
* Conflicting agendas
* Failure to listen
* Failure to listen
* Trust
* Trust Commentary and Challenges: Internal Risk Communication
    *J.L. Marble, N. Siu, and K. Coyne, Risk communication within a risk-informed regulatory decision-making environment, International 46  Conference on Probabilistic Safety and Assessment (PSAM 11/ESREL 2012), Helsinki, Finland, June 25-29, 2012. (ADAMS ML120480139)
*J.L. Marble, N. Siu, and K. Coyne, Risk communication within a risk-informed regulatory decision-making environment, International Conference on Probabilistic Safety and Assessment (PSAM 11/ESREL 2012), Helsinki, Finland, June 25-29, 2012. (ADAMS ML120480139)


Commentary and Challenges: Internal Risk Communication Risk Information: Inherently Complex
47 Risk Information: Inherently Complex Hyperdimensional
* Hyperdimensional
- Scenarios
  - Scenarios                                                     Will somebody find me a
- Likelihood
  - Likelihood                                                    one-handed scientist?!
- Multiple consequence measures Heterogeneous
  - Multiple consequence measures
- Qualitative and quantitative
* Heterogeneous                                                                - Senator Edmund Muskie
- Multiple technical disciplines Dynamic
  - Qualitative and quantitative                                                      (Concorde hearings, 1976)
- System changes (e.g., different operational modes, effects of decisions)
  - Multiple technical disciplines I. Flatow, Truth, Deception, and the Myth of the One-Handed Scientist,
- Changing information (learning, adding/discounting data)
* Dynamic                                                        October 18, 2012. Available from:
- New applications (and contexts)
https://thehumanist.com/magazine/november-december-
Uncertain
  - System changes (e.g., different operational modes, effects  2012/features/truth-deception-and-the-myth-of-the-one-handed-of decisions)                                                scientist
- Sparse or non-existent data
  - Changing information (learning, adding/discounting data)
- Outside range of personal experience Will somebody find me a one-handed scientist?!
  - New applications (and contexts)
- Senator Edmund Muskie (Concorde hearings, 1976)
* Uncertain
I. Flatow, Truth, Deception, and the Myth of the One-Handed Scientist, October 18, 2012. Available from:
  - Sparse or non-existent data
https://thehumanist.com/magazine/november-december-2012/features/truth-deception-and-the-myth-of-the-one-handed-scientist Commentary and Challenges: Internal Risk Communication
  - Outside range of personal experience 47


Commentary and Challenges: Internal Risk Communication and the World is changing
48 and the World is changing
* Experiences, knowledge
* Experiences, knowledge
* Information content and delivery preferences
* Information content and delivery preferences
* Comfort with analytics, risk, probability
* Comfort with analytics, risk, probability
* Mobility Language is not merely a tool for human communication; language is itself a means by which the realities of the world are divided and viewed.
* Mobility Source: https://www.nrc.gov/reading-rm/doc-collections/commission/slides/2019/20190618/staff-20190618.pdf Commentary and Challenges: Internal Risk Communication Language is not merely a tool for human communication; language is itself a means by which the realities of the world are divided and viewed.
                                              - P.S. Dull, 1978 Source: https://www.nrc.gov/reading-rm/doc-collections/commission/slides/2019/20190618/staff-20190618.pdf 48      P.S. Dull, A Battle History of the Imperial Japanese Navy (1941-1945), Naval Institute Press, Annapolis, MD, 1978
- P.S. Dull, 1978 P.S. Dull, A Battle History of the Imperial Japanese Navy (1941-1945), Naval Institute Press, Annapolis, MD, 1978


Commentary and Challenges: Internal Risk Communication Addressing Complexity (and Escaping Flatland)
49 Addressing Complexity (and Escaping Flatland)
* Tufte model: use rich displays and reports, encourage user                 Continuing Challenges to explore
* Tufte model: use rich displays and reports, encourage user to explore
      - Promotes active involvement of decision maker
- Promotes active involvement of decision maker
* Target audience(s)
- Increases general trust?
                                                                                  - Heterogeneous
* A graduated technical approach to assist?
      - Increases general trust?                                                 - Changing
Interface Interaction Mode Hyperlinked dashboards, reports Manual Video AI assist Visual immersion Multisensory immersion Time Commentary and Challenges: Internal Risk Communication Target audience(s)
* A graduated technical approach to assist?                                     - Constrained resources Interface                         Interaction Mode
- Heterogeneous
* Schema
- Changing
                                                                                  - No standards:
- Constrained resources Schema
          - Hyperlinked dashboards, reports - Manual                                currently an art
- No standards:
          - Video                            - AI assist                          - Solutions being Time                                                                                developed
currently an art
          - Visual immersion                                                        intuitively; no scientific testing
- Solutions being developed intuitively; no scientific testing Continuing Challenges
          - Multisensory immersion 49


Commentary and Challenges: Internal Risk Communication From Static to Interactive Dashboard to Sci-Fi?
50 Graphic adapted from https://www.flickr.com/photos/83823904@N00/64156219/
M. Korsnick, Risk Informing the Commercial Nuclear Enterprise, Promise of a Discipline: Reliability and Risk in Theory and in Practice, University of Maryland, April 2, 2014.                                                 Graphic adapted from https://www.flickr.com/photos/83823904@N00/64156219/
(permission CC-BY-2.0)
(permission CC-BY-2.0) 50
From Static to Interactive Dashboard to Sci-Fi?
M. Korsnick, Risk Informing the Commercial Nuclear Enterprise, Promise of a Discipline: Reliability and Risk in Theory and in Practice, University of Maryland, April 2, 2014.
Commentary and Challenges: Internal Risk Communication


Closing Remarks
51 Closing Remarks
* RIDM, enabled by PRA, provides a practical approach to safety-related decisionmaking under uncertainty
* RIDM, enabled by PRA, provides a practical approach to safety-related decisionmaking under uncertainty
* Appropriate application of RIDM requires                   !
* Appropriate application of RIDM requires appropriate characterization and communication of uncertainties, supported by technology
appropriate characterization and communication Many calculations bring of uncertainties, supported by technology       success; few calculations
* Moving forward: bold exploration or avoidance?
* Moving forward: bold exploration or avoidance?   bring failure. No calculations at all spell disaster!
Jason/(Momotar) or Pandora/
Jason/ (Momotar) or Pandora/                     - Sun-Tzu (The Art of War)
(Urashima Tar)?
(Urashima Tar)?
51
Many calculations bring success; few calculations bring failure. No calculations at all spell disaster!
- Sun-Tzu (The Art of War)


Acknowledgments The author gratefully acknowledges helpful suggestions by G. Apostolakis, A. Mosleh, and M. Cheok on presentation structure, approach, and content, and technical information provided by M. Kazarians and J. Nakoski.
52 Acknowledgments The author gratefully acknowledges helpful suggestions by G. Apostolakis, A. Mosleh, and M. Cheok on presentation structure, approach, and content, and technical information provided by M. Kazarians and J. Nakoski.  
52


ADDITIONAL SLIDES 53
53 ADDITIONAL SLIDES


Reasonable Assurance of Adequate Protection USAEC/USNRC UKAEA Staff                      UKAEA AEA (Amended) adequate protection to the                             Farmer Curve health and safety of the public Atoms for                                  NRC Letter AEC Chairman                             Peace Conf.                            reasonable assurance recognize every possible event               UKAEA call for                        of adequate protection assure that the probability of a             comprehensive mishap is satisfactorily low               safety assessment                MIT Proposal for Atomic Energy Act (AEA)                                  AEC Staff                                  reactor risk study protect health and                              Credible Accident minimize danger                                                                    TRG Report 1949(R)
54 Reasonable Assurance of Adequate Protection 1940 1950 1960 1970 Atomic Energy Act (AEA) protect health and minimize danger AEA (Amended) adequate protection to the health and safety of the public AEC Chairman recognize every possible event assure that the probability of a mishap is satisfactorily low AEC Staff Credible Accident MIT Proposal for reactor risk study UKAEA Staff Farmer Curve TRG Report 1949(R)
WASH-740                                   SGHWR analysis 1940                      1950                                  1960                                1970 54
SGHWR analysis WASH-740 Atoms for Peace Conf.
UKAEA call for comprehensive safety assessment NRC Letter reasonable assurance of adequate protection USAEC/USNRC UKAEA


Parameter Uncertainties:
55 Parameter Uncertainties:
Some Historical Results Industry results from: Garrick, B.J., Lessons learned from 21 nuclear plant probabilistic risk assessments, Nuclear Technology, 84, No. 3, 319-339(1989).
Some Historical Results Industry results from: Garrick, B.J., Lessons learned from 21 nuclear plant probabilistic risk assessments, Nuclear Technology, 84, No. 3, 319-339(1989).
55


Uncertainty Reduction - Perspective Depends on Scaling 56
56 Uncertainty Reduction - Perspective Depends on Scaling


Early Views on Completeness
57 Early Views on Completeness W. F. Libby (Acting Chairman, AEC) - March 14, 1956 response to Senator Hickenlooper: it is incumbent upon the new industry and the Government to make every effort to recognize every possible event or series of events which could result in the release of unsafe amounts of radioactive material to the surroundings and to take all steps necessary to reduce to a reasonable minimum the probability that such events will occur in a manner causing serious overexposure to the public. [Emphasis added]
* W. F. Libby (Acting Chairman, AEC) - March 14, 1956 response to Senator Hickenlooper: it is incumbent upon the new industry and the Government to make every effort to recognize every possible event or series of events which could result in the release of unsafe amounts of radioactive material to the surroundings and to take all steps necessary to reduce to a reasonable minimum the probability that such events will occur in a manner causing serious overexposure to the public. [Emphasis added]
* L. Silverman (Chairman, ACRS) - {{letter dated|date=October 22, 1960|text=October 22, 1960 letter}} to AEC Chairman John A. McCone: We believe that a searching analysis which is necessary at this stage [reactor siting approval] should be done independently by the owner of the reactor [Emphases added]
* L. Silverman (Chairman, ACRS) - October 22, 1960 letter to AEC Chairman John A. McCone: We believe that a searching analysis which is necessary at this stage [reactor siting approval] should be done independently by the owner of the reactor [Emphases added]
57


ACRS Concerns with WASH-1400 Methodology*
58 ACRS Concerns with WASH-1400 Methodology*
Topic                                                       Signature Events[1]                         Post-WASH-1400 Accident initiator quantification                                                                       Extensive treatment: fires, earthquakes Fukushima (Presumably external events)                                                                          Inconsistent treatment: floods Atypical reactors                                           Fermi 1 [2]                                 Multiple PRAs for non-LWRs Many design and operational improvements identified Design errors                                              [3]                                          by PRAs; database includes events involving design problems Multiple methods emphasizing importance of context; Operator error quantification                              TMI-2 still an active area of development Consequence modeling                                       Chernobyl, Fukushima                         Continuing, evolutionary improvements (MACCS)
Topic Signature Events[1]
Improved hardware database; fits and starts with Data                                                        Many HRA; extreme natural hazards a continuing challenge
Post-WASH-1400 Accident initiator quantification (Presumably external events)
Fukushima Extensive treatment: fires, earthquakes Inconsistent treatment: floods Atypical reactors Fermi 1 [2]
Multiple PRAs for non-LWRs Design errors
[3]
Many design and operational improvements identified by PRAs; database includes events involving design problems Operator error quantification TMI-2 Multiple methods emphasizing importance of context; still an active area of development Consequence modeling Chernobyl, Fukushima Continuing, evolutionary improvements (MACCS)
Data Many Improved hardware database; fits and starts with HRA; extreme natural hazards a continuing challenge
*ACRS letter to Congressman Udall re: adequacy for estimating likelihood of low probability/high consequence events (Dec. 16, 1976)
*ACRS letter to Congressman Udall re: adequacy for estimating likelihood of low probability/high consequence events (Dec. 16, 1976)
Table Notes:
Table Notes:
: 1. Events whose key characteristics (for the given topic) might not have been captured by a WASH-1400 vintage analysis.
1.
58  2.
Events whose key characteristics (for the given topic) might not have been captured by a WASH-1400 vintage analysis.
2.
Fermi 1 had limited fuel melting. However, without an analysis, it isnt clear if a WASH-1400 vintage analysis would have captured this scenario.
3.
3.
Fermi 1 had limited fuel melting. However, without an analysis, it isnt clear if a WASH-1400 vintage analysis would have captured this scenario.
Design weaknesses have played a role in multiple events. More detailed review is needed to determine if: a) these are errors, and b) if they would have been missed by a WASH-1400 vintage analysis.
Design weaknesses have played a role in multiple events. More detailed review is needed to determine if: a) these are errors, and b) if they would have been missed by a WASH-1400 vintage analysis.


Empirical Experience Accidents                           Some Significant* U.S. Precursors Year Plant(s) Precursor?         Year    Plant(s)        Notes 1979 TMI       Davis-Besse (1977) 1975   Browns Ferry     Worst precursor Fire => loss of U1 ECCS 1986  Chernobyl Leningrad (1975) 1978   Rancho Seco     Next worst precursor 2011  Fukushima Blayais (1999)                              Human error (maintenance) => loss of NNI, LOFW 2002   Davis-Besse     Most recent significant precursor Multiple human/organizational faults
59 Empirical Experience Accidents Year Plant(s)
                                                            => RPV head corrosion
Precursor?
                                    *Per Accident Sequence Precursor (ASP) program 59
1979 TMI Davis-Besse (1977) 1986 Chernobyl Leningrad (1975) 2011 Fukushima Blayais (1999)
Some Significant* U.S. Precursors Year Plant(s)
Notes 1975 Browns Ferry Worst precursor Fire => loss of U1 ECCS 1978 Rancho Seco Next worst precursor Human error (maintenance) => loss of NNI, LOFW 2002 Davis-Besse Most recent significant precursor Multiple human/organizational faults  
=> RPV head corrosion
*Per Accident Sequence Precursor (ASP) program


Some Other Interesting International Events Year Plant(s)                                                           Scenario Type                 Notes 1957 Windscale 1 (UK)                                                   Fire                           Graphite fire in core, release to environment.
60 Some Other Interesting International Events Year Plant(s)
Power cable fire, loss of main feedwater, pressurizer safety 1975 Greifswald 1 (East Germany)                                        Fire valves fail to re-seat.
Scenario Type Notes 1957 Windscale 1 (UK)
Partial loss of offsite power (LOOP) and subsequent loss of 1977 Gundremmingen A (East Germany)                                    LOOP/LOCA cooling accident (LOCA) with internal flooding.
Fire Graphite fire in core, release to environment.
Turbine Building fire spreads into Main Control Room, collapses 1978 Beloyarsk 2 (Soviet Union)                                        Fire Turbine Building roof.
1975 Greifswald 1 (East Germany)
1981 Hinkley Point A-1, A-2 (UK)                                       External Flood; LOOP (weather) Severe weather LOOP and loss of ultimate heat sink (LOUHS).
Fire Power cable fire, loss of main feedwater, pressurizer safety valves fail to re-seat.
1982 Armenia 1 (Soviet Union)                                           Fire                           Fire-induced station blackout (SBO).
1977 Gundremmingen A (East Germany)
1989 Vandellos 1 (Spain)                                               Fire                           Fire-induced internal flood.
LOOP/LOCA Partial loss of offsite power (LOOP) and subsequent loss of cooling accident (LOCA) with internal flooding.
1991 Chernobyl 2 (Soviet Union)                                         Fire                           Fire-induced Turbine Building roof collapse.
1978 Beloyarsk 2 (Soviet Union)
1993 Narora 1 (India)                                                   Fire                           Fire-induced SBO.
Fire Turbine Building fire spreads into Main Control Room, collapses Turbine Building roof.
1993 Onagawa 1 (Japan)                                                 Reactivity Excursion           Seismically-induced reactivity excursion.
1981 Hinkley Point A-1, A-2 (UK)
1999 Blayais 1, 2 (France)                                             External Flood                 Severe weather LOOP and partial LOUHS.
External Flood; LOOP (weather)
2001 Maanshan 1 (Taiwan)                                               LOOP (Weather); Fire (HEAF)   Severe weather LOOP and subsequent SBO.
Severe weather LOOP and loss of ultimate heat sink (LOUHS).
Pickering 4-8; Darlington 1, 2, and 4; Bruce 3, 4, and 6 (Canada);
1982 Armenia 1 (Soviet Union)
2003 Fermi 2 , Fitzpatrick, Ginna, Indian Point 2 and 3, Nine Mile     LOOP (weather)                Northeast Blackout.
Fire Fire-induced station blackout (SBO).
Point 1 and 2, Oyster Creek, Perry (U.S.)
1989 Vandellos 1 (Spain)
2004 Madras 2 (India)                                                   External Flood                 Tsunami-induced LOUHS.
Fire Fire-induced internal flood.
2009 Cruas 2-4 (France)                                                 External Flood                 LOUHS due to flood debris.
1991 Chernobyl 2 (Soviet Union)
Fukushima Dai-ichi 5-6, Fukushima Dai-ni 1-4, Onagawa 1-3,                                       Earthquake- and tsunami-induced incidents (in addition to 2011 Tokai Dai-ni, Higashidori 1-2 (Japan)                             External Flood accidents at Fukushima Dai-ichi 1-3).
Fire Fire-induced Turbine Building roof collapse.
60
1993 Narora 1 (India)
Fire Fire-induced SBO.
1993 Onagawa 1 (Japan)
Reactivity Excursion Seismically-induced reactivity excursion.
1999 Blayais 1, 2 (France)
External Flood Severe weather LOOP and partial LOUHS.
2001 Maanshan 1 (Taiwan)
LOOP (Weather); Fire (HEAF)
Severe weather LOOP and subsequent SBO.
2003 Pickering 4-8; Darlington 1, 2, and 4; Bruce 3, 4, and 6 (Canada);
Fermi 2, Fitzpatrick, Ginna, Indian Point 2 and 3, Nine Mile Point 1 and 2, Oyster Creek, Perry (U.S.)
LOOP (weather)
Northeast Blackout.
2004 Madras 2 (India)
External Flood Tsunami-induced LOUHS.
2009 Cruas 2-4 (France)
External Flood LOUHS due to flood debris.
2011 Fukushima Dai-ichi 5-6, Fukushima Dai-ni 1-4, Onagawa 1-3, Tokai Dai-ni, Higashidori 1-2 (Japan)
External Flood Earthquake-and tsunami-induced incidents (in addition to accidents at Fukushima Dai-ichi 1-3).


External Hazards Scenario-Based Classification:
61 External Hazards Scenario-Based Classification:
An Aid for Completeness?
An Aid for Completeness?}}
61}}

Latest revision as of 08:54, 14 December 2024

Uncertainty Analysis Technology
ML20080N774
Person / Time
Issue date: 03/23/2020
From: Nathan Siu
Office of Nuclear Regulatory Research
To:
N. Siu
References
Download: ML20080N774 (61)
v • d • e


Text

Technology for the Treatment of Uncertainties:

History, Status, Commentary and Challenges Nathan Siu Senior Technical Adviser for PRA Analysis U.S. Nuclear Regulatory Commission Expanded version of a presentation originally developed for CRIEPI/NRRC and OECD/NEA Workshop on the Proper Treatment of Uncertainties in Reactor Safety Assessment March, 2020

2 Foreword On December 19, 2019, the Nuclear Risk Research Center (NRRC) of the Japan Central Research Institute of Electric Power Industry (CRIEPI) and the Organization for Economic Cooperation (OECD) Nuclear Energy Agency (NEA) invited the author to participate in a workshop on the improvement and enhancement of risk-informed decision making (RIDM) processes in reactor safety assessment. The workshop, titled A Workshop on the Proper Treatment of Uncertainties in Reactor Safety Assessment, was to be held on May 26-27, 2020 in Tokyo, Japan. At the request of the workshop organizers, the authors talk was to be titled Technology for the Treatment of Uncertainties: History, Status, and Some Challenges. On March 12, due to travel restrictions arising from the covid-19 pandemic, the author was directed to withdraw from the workshop. The following slides are an expanded version of the talk the author was planning on presenting.

3 Outline

  • Framework for discussion

- Parameter Uncertainties

- Model Uncertainties

- Completeness Uncertainties

- Communication

  • Current state of practice
  • History
  • Commentary and challenges tech*nol*o*gy, n. the sum of techniques, skills, methods, and processes used in the production of goods or services or in the accomplishment of objectives, such as scientific investigation. [Wikipedia]

In this talk:

technology {methods, models, computational tools, guidance, data}

4 DISCUSSION FRAMEWORK What are we talking about?

5 Context for Treatment of Uncertainties: Risk-Informed Decisionmaking (RIDM)

P{XlC,H}

subjective proposition conditions knowledge Discussion Framework Adapted from NUREG-2150

6 Parameter, Model, and Completeness Uncertainty:

A Practical Categorization M (Model of the World):

Scope, structure i: Parameters

Universe Known Unknowns Unknown Unknowns Discussion Framework mod*el, n. a representation of reality created with a specific objective in mind.

A. Mosleh, N. Siu, C. Smidts, and C. Lui, Model Uncertainty: Its Characterization and Quantification, Center for Reliability Engineering, University of Maryland, College Park, MD, 1995. (Also NUREG/CP-0138, 1994)

PRA models for NPPs Typically an assemblage of sub-models with parameters Implicitly include issues considered but not explicitly quantified

7 Parameter, Model, and Completeness Uncertainty:

A Practical Categorization M (Model of the World):

Scope, structure i: Parameters

Universe Known Unknowns Unknown Unknowns Discussion Framework mod*el, n. a representation of reality created with a specific objective in mind.

A. Mosleh, N. Siu, C. Smidts, and C. Lui, Model Uncertainty: Its Characterization and Quantification, Center for Reliability Engineering, University of Maryland, College Park, MD, 1995. (Also NUREG/CP-0138, 1994)

PRA models for NPPs Distinctions are not necessarily crisp Regardless of allocation to categories, need to consider in characterization of uncertainties

8 Parameter Uncertainty: An Example

  • Parameter of interest: frequency of flooding ()
  • Prior state-of-knowledge: minimal
  • Evidence: 10 events over 1877-2017 (140 years)
  • Posterior state-of-knowledge:

Date Flood Height (ft) 3/19/1936 36.5 6/1/1889 34.8 10/16/1942 33.8 10/1/1896 33.0 11/6/1985 30.1 9/8/1996 29.8 1/21/1996 29.4 11/25/1877 29.2 4/27/1937 29.0 6/23/1972 27.7 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Probability Density

Flood Frequency (/yr) 05 = 0.040/yr 50 = 0.069/yr 95 = 0.11/yr mean = 0.071/yr prior posterior Discussion Framework return period = 12 yr 1880 1900 1920 1940 1960 1980 2000

9 Hurricane Andrew: 8/22/1992, 1200 UTC (about 2 days before FL landfall)

Plot adapted from University of Wisconsin-Milwaukee (https://web.uwm.edu/hurricane-models/models/archive/)

Model Uncertainty:

Hurricane Example Discussion Framework

10 https://en.wikipedia.org/wiki/Hurricane_Irma#/media/File:Irma,_Jose_and_Katia_2017-09-07.png Completeness Uncertainty:

Multiple Hurricane Example (A Known Unknown)

Irma Jose Katia Discussion Framework Turkey Point

11 Risk Communication (Internal)

Discussion Framework Other Considerations Current regulations Safety margins Defense-in-depth Monitoring Quantitative Qualitative Adapted from NUREG-2150

12 CURRENT STATE-OF-PRACTICE What do people do now?

13 State-of-Practice: Parameter Uncertainties

  • Treatment involves Estimation (including expert elicitation)

Propagation

  • Straightforward mathematics and mechanics
  • Some practical challenges Current State of Practice

14 State-of-Practice:

Model Uncertainties

  • Important to acknowledge and treat (in context of decision)
  • Multiple approaches

- Consensus model

- Sensitivity analysis

- Weighted alternatives (e.g., SSHAC)

- Output uncertainties Current State of Practice Hurricane Andrew 8/22/1992, 1200 UTC Adapted from University of Wisconsin-Milwaukee (https://web.uwm.edu/hurricane-models/models/archive/)

Adapted from V.M. Andersen, Seismic Probabilistic Risk Assessment Implementation Guide, EPRI 3002000709, Electric Power Research Institute, Palo Alto, CA, December 2013 M.H. Salley and A. Lindeman, Verification and Validation of Selected Fire Models for Nuclear Power Plant Applications, NUREG-1824 Supplement 1/EPRI 3002002182, November 2016.

15 State-of-Practice:

Completeness Uncertainties

  • Potential concerns

- Known gaps (missing scope)

  • Scenario categories
  • Contributors within categories

- Unknown gaps

- Heuristics/biases

  • Excessive amplification (fear of the dark)
  • Excessive discounting (out of sight, out of mind)
  • Treatment

- Analysis guidance

- Additional analysis/R&D

- Risk-informed decisionmaking NUREG-1855 Rev. 1 (2017)

Options:

Progressive analysis (screening, bounding, conservative, detailed)

Change scope of risk-informed application RG 1.174 Rev. 3 (2019)

Current State of Practice

16 State-of-Practice: Internal Risk Communication

  • Often implicit (focus on mean values)
  • Various graphic displays
  • Includes story as well as numbers Current State of Practice Documents and Presentations (Flatland)

Interactive Discussion (Storytelling)

Likelihood Class 5 (10-5/yr) 4 (10-4/yr) 3 (10-3/yr) 2 (10-2/yr) 1 (10-1/yr)

Severity Class A

Marginal Undesirable Undesirable Critical Critical B

Marginal Marginal Undesirable Undesirable Critical C

No Action Marginal Marginal Undesirable Undesirable D

No Action No Action Marginal Marginal Undesirable E

No Action No Action No Action Marginal Marginal

17 A BRIEF HISTORY How did we get here?

18 A Series of Challenges and Responses 1940 1950 1960 1970 1980 1990 2000 2010 2020 Hanford to WASH-1400 Early PRAs Expansion Across Industry Modern Applications History

19 TMI-2 From Hanford to WASH-1400 SGHWR analysis WASH-740 For more information: T.R. Wellock, A Figure of Merit: Quantifying the Probability of a Nuclear Reactor Accident, Technology and Culture, 58, No. 3, July 2017, pp. 678-721.

Credible Accident System reliability studies Recommend:

accident chain analysis Hanford AEC/NRC UKAEA Technical Challenges: 1) Quantifying accident probability

2) Means to communicate risk not in the generation of the ACRS members present Farmer Curve WASH-1400 Estimates:

OpE (pessimistic)

Decomposition (optimistic)

History Windscale 1950 1960 1970 1980 System reliability studies

20 WASH-1400 Uncertainties (Level 1)

WASH-1400: it is reasonable to believe that the core melt probability of about 5x10-5 per reactor-year predicted by this study should not be significantly larger and would almost certainly not exceed the value of 3x10-4 which has been estimated as the upper bound for core melt probability.

Risk Assessment Review Group (NUREG/CR-0400):

We are unable to define whether the overall probability of a core melt given in WASH-1400 is high or low, but we are certain that the error bands are understated. We cannot say by how much.

1.E-05 1.E-04 1.E-03 CDF (/ry)

WASH-1400 Uncertainties (Estimated*)

Surry Peach Bottom 5th 50th 95th mean

  • Based on data from Tables V 3-14 (PWR) and 3-16 (BWR) of Appendix V, assuming distributions are lognormal; median values are somewhat higher than reported in Section 7.3.1 of the Main Report.

History

21 TMI-2 Chernobyl Some Early Developments and PRAs Challenges: 1) Filling known gaps (completeness uncertainty)

2) Clarifying meaning: models and results Clinch River (LMFBR)

Limerick Millstone Seabrook (full scope)

Fleming

(-factor)

Zion (full scope)

TMI-1 (full scope)

Oconee (full scope) 1980 1985 1975 Apostolakis (subjective probability)

Forsmark Koeberg

(~WASH-1400)

Super Phénix (FBR DHR)

AIPA (HTGR)

USDOE NRC US Industry International Other Notable Kaplan/

Garrick (risk)

History EC/JRC Benchmarks (systems, CCF, HRA)

RSSMAP/IREP Sizewell

(+DI&C)

Indian Point (full scope)

Oyster Creek

(+seismic)

Biblis

(+aircraft)

NUREG/CR-2300

22 Sample Level 1 Results Display History

23 Sample Results - Sub-Model Uncertainty Effect History Effects of fire model (COMPBRN) uncertainty on fire growth time N. Siu, "Modeling Issues in Nuclear Plant Fire Risk Analysis," in EPRI Workshop on Fire Protection in Nuclear Power Plants, EPRI NP-6476, J.-P. Sursock, ed., August 1989, pp. 14-1 through 14-16.

24 Sample Results - Model Uncertainty (User Effect)

Damage State Frequency (/yr), Review Damage State Frequency (/yr), Original 10-10 10-8 10-6 10-4 10-10 10-8 10-6 10-4 Early core melt, containment cooling Early core melt, no containment cooling Steam generator tube rupture Containment bypass Direct containment failure Late core melt, containment cooling Late core melt, no containment cooling 1.E-11 1.E-10 1.E-09 1.E-08 1.E-07 1.E-06 1.E-05 1.E-04 1.E-03 1.E-11 1.E-10 1.E-09 1.E-08 1.E-07 1.E-06 1.E-05 1.E-04 1.E-03 Original Review Internal Events 1.E-11 1.E-10 1.E-09 1.E-08 1.E-07 1.E-06 1.E-05 1.E-04 1.E-03 1.E-11 1.E-10 1.E-09 1.E-08 1.E-07 1.E-06 1.E-05 1.E-04 1.E-03 Original Review External Events Data source: G.J. Kolb, et al., Review and Evaluation of the Indian Point Probabilistic Safety Study, NUREG/CR-2934, December 1982.

(ML091540534)

History

25 Chernobyl 9/11 Expansion Across Industry (US)

Technical challenges: 1) Characterizing the fleet (variability)

2) Developing confidence for mainstreaming RIDM 1985 1990 2000 1995 GL 88-20 GL 88-20 Supplement 4 NUREG-1560 NUREG-1742 NUREG-1150 (final)

NUREG-1150 (draft)

Severe Accident Policy Statement Safety Goal Policy Statement PRA Policy Statement ASP Plant Class Models 1982 SPAR Models History NRC US Industry IPEs IPEEEs

26 NUREG-1150 Estimated* Uncertainties (Level 1)

Model Uncertainty Model Uncertainty

  • Notes: totals shown in this 1)

NUREG-1150 does not aggregate the hazard-specific results. The totals shown are rough estimates assuming that the NUREG-1150 distributions are lognormal.

2)

The WASH-1400 distributions are based on data from Tables V 3-14 (PWR) and 3-16 (BWR) of Appendix V, assuming that the distributions are lognormal. The median values are somewhat higher than reported in Section 7.3.1 of the Main Report History

27 IPE/IPEEE - Variability Across Fleet 0

10 20 30 40 Number BWR PWR CDF (/ry) 1x10-6 3x10-6 1x10-5 3x10-5 1x10-4 3x10-4 1x10-3 Internal Events + Internal Floods 0

10 20 30 40 Number BWR PWR CDF (/ry) 1x10-6 3x10-6 1x10-5 3x10-5 1x10-4 3x10-4 1x10-3 Total History

28 9/11 The Modern Era (US)

Technical challenges: 1) RIDM issues (e.g., realism, heterogeneity, aggregation)

2) Post-Fukushima issues (e.g., external hazards)
3) New/advanced reactors (e.g., conduct of operations)

NUREG-1855 History Fukushima RG 1.174 ASME PRA Standard 10 CFR 50.48(c)

(Fire Protection)

Risk-Informed ROP NFPA 805 NUREG-2150 NTTF Request for Information (Reevaluations) 2000 2010 2020 2005 2015 NRC US Industry SECY-98-144 Risk-Informed License Amendment Requests (LARs)

SAMAs (Life Extension)

SPAR Models NFPA 805 LARs (Fire Protection)

29 Variability in Recent Results (Level 1)

History 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35

-6.0

-5.5

-5.0

-4.5

-4.0

-3.5

-3.0 1E-6 1E-5 1E-4 1E-3 CDF (per reactor year)

Fraction of Plants Highest Reported:

1.3x10-4 Lowest Reported:

3.5x10-6 Population Mean:

4.7x10-5

30 Variability in Results - Comparison with IPE/IPEEE History 0.00 0.10 0.20 0.30 0.40 0.50 1

2 3

4 5

6 7

8 9

10 NFPA 805 IPE/IPEEE 0.01 0.1 1

10 100 1000 Fire CDF/Internal Events CDF Fraction of PRAs 0.00001 0.0001 0.001 1.00E-05 1.00E-04 1.00E-03 Total CDF (IPE + IPEEE)

Total CDF (Recent LARs) 1E-5 1E-4 1E-3 1E-5 1E-4 1E-3

31 COMMENTARY AND CHALLENGES Where might we do better and how?

32 An Important Note

  • Challenges regarding the treatment of uncertainty in PRA and RIDM exist for non-probabilistic approaches as well; the PRA/RIDM approach acknowledges these challenges explicitly.
  • The following slides are not a critique of the overall PRA/RIDM philosophy - they should be viewed in the framework of continuous improvement.

Commentary and Challenges

33 A Changing World

  • Evolving situation*

- market forces

- new nuclear technologies

- new analytical methods and data

- new professionals

  • Increased reliance on risk models, characterization of uncertainties
  • See Applying the Principles of Good Regulation as a Risk-Informed Regulator, October 15, 2019 (ADAMS ML19260E683)

Commentary and Challenges

34 Reminder: Parameter Uncertainties and Mean Values Commentary and Challenges: Parameter Uncertainties Mean = 7.6 x 10-5 /yr 95th = 2.6 x 10-4 /yr 50th (Median) = 3.9 x 10-5 /yr probability density function frequency (/yr)

Mean 0

Mathematically defined Affected by tail Does not correspond to a specific percentile

35 Parameter Uncertainties: Challenges

  • Quantification generally required, diverse views on value added
  • Technical challenges:

- Effect of data pre-processing

  • Selection
  • Interpretation

- Effect of analysis shortcuts

  • Standard prior distributions
  • Simplified expert elicitation
  • Independence assumption

- Ensuring correspondence with actual state-of-knowledge

  • Basic events (micro)
  • Overall results (macro)

Commentary and Challenges: Parameter Uncertainties 1.E-09 1.E-08 1.E-07 1.E-06 1.E-05 1.E-04 1.E-03 Probability Density Function (Normalized)

Failure Rate (/hr)

Service Water Normally Running Standby 2015 Industry-wide estimates from: https://nrcoe.inl.gov/resultsdb/AvgPerf/

Service Water Pumps: 2 failures in 16,292,670 hours0.00775 days <br />0.186 hours <br />0.00111 weeks <br />2.54935e-4 months <br /> Normally Running Pumps: 225 failures in 59,582,350 hours0.00405 days <br />0.0972 hours <br />5.787037e-4 weeks <br />1.33175e-4 months <br /> Standby Pumps (1st hour operation): 48 failures in 437,647 hours0.00749 days <br />0.18 hours <br />0.00107 weeks <br />2.461835e-4 months <br />

36 Model Uncertainties - Commentary

  • Model uncertainties can be large; importance depends on decision
  • Some practical approaches (e.g., consensus models, deterministic screening) can understate uncertainties
  • Subjective probability framework =>

- Need to include user effect

- Raises question regarding fundamental meaning of weighted average approaches

Hurricane Irma: 9/8/2017, 0000 UTC (about 2 days before FL landfall)

Outer prediction is closest to actual course

37 Model Uncertainty User Effects: HRA Example 1 Same method, different teams Same team, different methods All teams, all methods NRI, CREAM NRI, DT+ASEP NRC, SPAR-H INL, SPAR-H A Bye, et al., International HRA Empirical Study, NUREG/IA-0216, August 2011.

Commentary and Challenges: Model Uncertainties

38 Model Uncertainty User Effects: HRA Example 2 HFE 2A HFE 1C HFE 1A HFE 3A HFE 1B Decreasing difficulty Human Error Probability (HEP) 1.0E+0 1.0E-1 1.0E-2 1.0E-3 1.0E-4 1.0E-5 ASEP Team 1 ASEP Team 2 SPAR-H Team 1 SPAR-H Team 2 CBDT & HCR/ORE Team 1 CBDT & HCR/ORE Team 2 CBDT & HCR/ORE Team 3 ATHEANA Team 1 ATHEANA Team 2 Empirical 95th Percentile Empirical 5th Percentile Adapted from NUREG-2156 Commentary and Challenges: Model Uncertainties

39 Challenges: Quantification of Model Output Uncertainty

  • Bayesian methods

- Framework consistent with overall PRA

- Early approaches used in past PRAs

- Can address practical issues (e.g., non-homogeneous data)*

  • Challenges include

- Uncertainties in unmeasured parameters

- Sub-model limits of applicability

- Representativeness of computed results Time (s)

Experiment (K)

DRM (K) 180 400 450 360 465 510 720 530 560 840 550 565

  • See E. Droguett and Ali Mosleh, Bayesian methodology for model uncertainty using model performance data, Risk Analysis, 28, No. 5, 1457-1476, 2008.

Commentary and Challenges: Model Uncertainties Temperature (K)

Percentile Assume Homogeneous Data Assume Non-Homogeneous Data 1st 415.2 372.8 5th 437.5 400.7 50th 457.1 470.5 95th 479.7 559.4 99th 509.1 608.7 Data Output Uncertainty

40 Completeness Uncertainty

  • Sources

- Known gaps (missing scope)

- Unknown gaps

  • Concerns

- Excessive amplification (Fear of the dark)

- Excessive discounting (availability heuristic:

Out of sight, out of mind)

It would cease to be a danger if we could define it.

Sherlock Holmes (The Adventure of the Copper Beeches)

Commentary and Challenges: Completeness Uncertainties B. Fischhoff, P. Slovic, S. Lichtenstein, Fault trees: Sensitivity of estimated failure probabilities to problem representation, Journal of Experimental Psychology: Human Perception and Performance, 4(2), May 1978, 330-344.

Car Wont Start Battery Charge Insufficient Starting System Defective Ignition System Defective Mischievous Acts Of Vandalism All Other Problems Fuel System Defective Other Engine Problems

41 Known Gaps (Known Unknowns)

  • Broad scenario categories
  • Contributors within categories
  • Technology = {methods, models, tools, data}

Rationale Common Example(s)

Out of scope security/sabotage, operation outside approved limits Low significance (pre-analysis judgment) external floods (many plants pre-Fukushima)

Appropriate PRA technology* unavailable management and organizational factors PRA not appropriate software, security Category Example(s)

External hazards multiple coincident or sequential hazards Human reliability errors of commission, non-proceduralized recovery Passive systems thermal-hydraulic reliability Commentary and Challenges: Completeness Uncertainties

42 Unknown Unknowns: You Say Tomto Model Known Unknowns Unknown Unknowns

  • Explicit or implicit?
  • Extent of coverage?
  • Known by whom?
  • Known when?
  • Time from idea to theory to PRA implementation?

Viewpoint Precise classification is important only if it affects:

  • Understanding
  • Communication
  • Decision making Commentary and Challenges: Completeness Uncertainties

43 Unknown Unknowns: A Demonstrated Problem?

Model Known Unknowns Unknown Unknowns Then (a surprise?)

Now (treated in current PRAs?)

Browns Ferry fire (1975) - a long-recognized hazard; not in draft WASH-1400 but routinely treated now Chernobyl (1986) - precursor at Leningrad (1975); non-routine test during shutdown in any LPSD analyses?

TMI (1979) - precursors include Davis-Besse (1977); operator EOCs not in models; current recognition and some explorations Blayais flood (1999) - external floods often screened at time; current recognition, multi-hazard under development Maanshan HEAF/SBO (2001) - HEAF phenomenon known, in any PRAs at time? Now included as an initiator; smoke effect?

Davis-Besse RPV corrosion (2002) - RPV failure analyses focused on crack propagation; M&O failure not in PRAs Fukushima Daiichi (2011) - precursors: Blayais (1999), Indian Ocean (2004), hazard under review at time; PRA models under development Commentary and Challenges: Completeness Uncertainties

44 Illuminating Uncertainties: From Lampposts to Search Beacons Wheres the goat???

Commentary and Challenges: Completeness Uncertainties

45 What Can We (PRA R&D) Do?

  • Continue to develop technology to address known gaps

- Risk-informed prioritization

- Fully engage appropriate disciplines

- Take advantage of general computational and methodological developments

  • Facilitate re-emphasis on searching

- Demonstrate efficiency and effectiveness with current tools (e.g., MLD, HBFT) vs.

checklist/screening

- Develop improved tools (including OpE mining)

Event (NUREG/CR-4839), 1992 Aircraft impact Avalanche Coastal erosion Drought External flooding Extreme winds and tornadoes Fire Fog Forest fire Frost Hail High tide, high lake level, or high river stage

Commentary and Challenges: Completeness Uncertainties

46 Sources of Breakdowns: Risk Communication Between Risk Managers and Public*

  • Differences in perception of information

- Relevance

- Consistency with prior beliefs

  • Lack of understanding of underlying science
  • Conflicting agendas
  • Failure to listen
  • Trust Commentary and Challenges: Internal Risk Communication
  • J.L. Marble, N. Siu, and K. Coyne, Risk communication within a risk-informed regulatory decision-making environment, International Conference on Probabilistic Safety and Assessment (PSAM 11/ESREL 2012), Helsinki, Finland, June 25-29, 2012. (ADAMS ML120480139)

47 Risk Information: Inherently Complex Hyperdimensional

- Scenarios

- Likelihood

- Multiple consequence measures Heterogeneous

- Qualitative and quantitative

- Multiple technical disciplines Dynamic

- System changes (e.g., different operational modes, effects of decisions)

- Changing information (learning, adding/discounting data)

- New applications (and contexts)

Uncertain

- Sparse or non-existent data

- Outside range of personal experience Will somebody find me a one-handed scientist?!

- Senator Edmund Muskie (Concorde hearings, 1976)

I. Flatow, Truth, Deception, and the Myth of the One-Handed Scientist, October 18, 2012. Available from:

https://thehumanist.com/magazine/november-december-2012/features/truth-deception-and-the-myth-of-the-one-handed-scientist Commentary and Challenges: Internal Risk Communication

48 and the World is changing

  • Experiences, knowledge
  • Information content and delivery preferences
  • Comfort with analytics, risk, probability

- P.S. Dull, 1978 P.S. Dull, A Battle History of the Imperial Japanese Navy (1941-1945), Naval Institute Press, Annapolis, MD, 1978

49 Addressing Complexity (and Escaping Flatland)

  • Tufte model: use rich displays and reports, encourage user to explore

- Promotes active involvement of decision maker

- Increases general trust?

  • A graduated technical approach to assist?

Interface Interaction Mode Hyperlinked dashboards, reports Manual Video AI assist Visual immersion Multisensory immersion Time Commentary and Challenges: Internal Risk Communication Target audience(s)

- Heterogeneous

- Changing

- Constrained resources Schema

- No standards:

currently an art

- Solutions being developed intuitively; no scientific testing Continuing Challenges

50 Graphic adapted from https://www.flickr.com/photos/83823904@N00/64156219/

(permission CC-BY-2.0)

From Static to Interactive Dashboard to Sci-Fi?

M. Korsnick, Risk Informing the Commercial Nuclear Enterprise, Promise of a Discipline: Reliability and Risk in Theory and in Practice, University of Maryland, April 2, 2014.

Commentary and Challenges: Internal Risk Communication

51 Closing Remarks

  • RIDM, enabled by PRA, provides a practical approach to safety-related decisionmaking under uncertainty
  • Appropriate application of RIDM requires appropriate characterization and communication of uncertainties, supported by technology
  • Moving forward: bold exploration or avoidance?

Jason/(Momotar) or Pandora/

(Urashima Tar)?

Many calculations bring success; few calculations bring failure. No calculations at all spell disaster!

- Sun-Tzu (The Art of War)

52 Acknowledgments The author gratefully acknowledges helpful suggestions by G. Apostolakis, A. Mosleh, and M. Cheok on presentation structure, approach, and content, and technical information provided by M. Kazarians and J. Nakoski.

53 ADDITIONAL SLIDES

54 Reasonable Assurance of Adequate Protection 1940 1950 1960 1970 Atomic Energy Act (AEA) protect health and minimize danger AEA (Amended) adequate protection to the health and safety of the public AEC Chairman recognize every possible event assure that the probability of a mishap is satisfactorily low AEC Staff Credible Accident MIT Proposal for reactor risk study UKAEA Staff Farmer Curve TRG Report 1949(R)

SGHWR analysis WASH-740 Atoms for Peace Conf.

UKAEA call for comprehensive safety assessment NRC Letter reasonable assurance of adequate protection USAEC/USNRC UKAEA

55 Parameter Uncertainties:

Some Historical Results Industry results from: Garrick, B.J., Lessons learned from 21 nuclear plant probabilistic risk assessments, Nuclear Technology, 84, No. 3, 319-339(1989).

56 Uncertainty Reduction - Perspective Depends on Scaling

57 Early Views on Completeness W. F. Libby (Acting Chairman, AEC) - March 14, 1956 response to Senator Hickenlooper: it is incumbent upon the new industry and the Government to make every effort to recognize every possible event or series of events which could result in the release of unsafe amounts of radioactive material to the surroundings and to take all steps necessary to reduce to a reasonable minimum the probability that such events will occur in a manner causing serious overexposure to the public. [Emphasis added]

  • L. Silverman (Chairman, ACRS) - October 22, 1960 letter to AEC Chairman John A. McCone: We believe that a searching analysis which is necessary at this stage [reactor siting approval] should be done independently by the owner of the reactor [Emphases added]

58 ACRS Concerns with WASH-1400 Methodology*

Topic Signature Events[1]

Post-WASH-1400 Accident initiator quantification (Presumably external events)

Fukushima Extensive treatment: fires, earthquakes Inconsistent treatment: floods Atypical reactors Fermi 1 [2]

Multiple PRAs for non-LWRs Design errors

[3]

Many design and operational improvements identified by PRAs; database includes events involving design problems Operator error quantification TMI-2 Multiple methods emphasizing importance of context; still an active area of development Consequence modeling Chernobyl, Fukushima Continuing, evolutionary improvements (MACCS)

Data Many Improved hardware database; fits and starts with HRA; extreme natural hazards a continuing challenge

  • ACRS letter to Congressman Udall re: adequacy for estimating likelihood of low probability/high consequence events (Dec. 16, 1976)

Table Notes:

1.

Events whose key characteristics (for the given topic) might not have been captured by a WASH-1400 vintage analysis.

2.

Fermi 1 had limited fuel melting. However, without an analysis, it isnt clear if a WASH-1400 vintage analysis would have captured this scenario.

3.

Design weaknesses have played a role in multiple events. More detailed review is needed to determine if: a) these are errors, and b) if they would have been missed by a WASH-1400 vintage analysis.

59 Empirical Experience Accidents Year Plant(s)

Precursor?

1979 TMI Davis-Besse (1977) 1986 Chernobyl Leningrad (1975) 2011 Fukushima Blayais (1999)

Some Significant* U.S. Precursors Year Plant(s)

Notes 1975 Browns Ferry Worst precursor Fire => loss of U1 ECCS 1978 Rancho Seco Next worst precursor Human error (maintenance) => loss of NNI, LOFW 2002 Davis-Besse Most recent significant precursor Multiple human/organizational faults

=> RPV head corrosion

  • Per Accident Sequence Precursor (ASP) program

60 Some Other Interesting International Events Year Plant(s)

Scenario Type Notes 1957 Windscale 1 (UK)

Fire Graphite fire in core, release to environment.

1975 Greifswald 1 (East Germany)

Fire Power cable fire, loss of main feedwater, pressurizer safety valves fail to re-seat.

1977 Gundremmingen A (East Germany)

LOOP/LOCA Partial loss of offsite power (LOOP) and subsequent loss of cooling accident (LOCA) with internal flooding.

1978 Beloyarsk 2 (Soviet Union)

Fire Turbine Building fire spreads into Main Control Room, collapses Turbine Building roof.

1981 Hinkley Point A-1, A-2 (UK)

External Flood; LOOP (weather)

Severe weather LOOP and loss of ultimate heat sink (LOUHS).

1982 Armenia 1 (Soviet Union)

Fire Fire-induced station blackout (SBO).

1989 Vandellos 1 (Spain)

Fire Fire-induced internal flood.

1991 Chernobyl 2 (Soviet Union)

Fire Fire-induced Turbine Building roof collapse.

1993 Narora 1 (India)

Fire Fire-induced SBO.

1993 Onagawa 1 (Japan)

Reactivity Excursion Seismically-induced reactivity excursion.

1999 Blayais 1, 2 (France)

External Flood Severe weather LOOP and partial LOUHS.

2001 Maanshan 1 (Taiwan)

LOOP (Weather); Fire (HEAF)

Severe weather LOOP and subsequent SBO.

2003 Pickering 4-8; Darlington 1, 2, and 4; Bruce 3, 4, and 6 (Canada);

Fermi 2, Fitzpatrick, Ginna, Indian Point 2 and 3, Nine Mile Point 1 and 2, Oyster Creek, Perry (U.S.)

LOOP (weather)

Northeast Blackout.

2004 Madras 2 (India)

External Flood Tsunami-induced LOUHS.

2009 Cruas 2-4 (France)

External Flood LOUHS due to flood debris.

2011 Fukushima Dai-ichi 5-6, Fukushima Dai-ni 1-4, Onagawa 1-3, Tokai Dai-ni, Higashidori 1-2 (Japan)

External Flood Earthquake-and tsunami-induced incidents (in addition to accidents at Fukushima Dai-ichi 1-3).

61 External Hazards Scenario-Based Classification:

An Aid for Completeness?

Retrieved from "https://kanterella.com/w/index.php?title=ML20080N774&oldid=4379723"