Technical Uncertainty in Quantitative Policy Analysis — A Sulfur Air Pollution Example

Expert judgments expressed as subjective probability distributions provide an appropriate means of incorporating technical uncertainty in some quantitative policy studies. Judgments and distributions obtained from several experts allow one to explore the extent to which the conclusions reached in such a study depend on which expert one talks to. For the case of sulfur air pollution from coal-fired power plants, estimates of sulfur mass balance as a function of plume flight time are shown to vary little across the range of opinions of leading atmospheric scientists while estimates of possible health impacts are shown to vary widely across the range of opinions of leading scientists in air pollution health effects.     

Uncertain Benefits Estimates for Reductions in Fine Particle Concentrations

The Environmental Protection Agency's (EPA's) estimates of the benefits of improved air quality, especially from reduced mortality associated with reductions in fine particle concentrations, constitute the largest category of benefits from all federal regulation over the last decade. EPA develops such estimates, however, using an approach little changed since a 2002 report by the National Research Council (NRC), which was critical of EPA's methods and recommended a more comprehensive uncertainty analysis incorporating probability distributions for major sources of uncertainty. Consistent with the NRC's 2002 recommendations, we explore alternative assumptions and probability distributions for the major variables used to calculate the value of mortality benefits. For metropolitan Philadelphia, we show that uncertainty in air quality improvements and in baseline mortality have only modest effects on the distribution of estimated benefits. We analyze the effects of alternative assumptions regarding the value of reducing mortality risk, whether the toxicity is above or below the average for fine particles, and whether there is a threshold in the concentration‐response relationship, and show these assumptions all have large effects on the distribution of benefits.     

Using Uncertainty Analysis to Improve Consistency in Regulatory Assessments of Criteria Pollutant Standards

Regulatory impact analyses (RIAs), required for new major federal regulations, are often criticized for not incorporating epistemic uncertainties into their quantitative estimates of benefits and costs. “Integrated uncertainty analysis,” which relies on subjective judgments about epistemic uncertainty to quantitatively combine epistemic and statistical uncertainties, is often prescribed. This article identifies an additional source for subjective judgment regarding a key epistemic uncertainty in RIAs for National Ambient Air Quality Standards (NAAQS)—the regulator's degree of confidence in continuation of the relationship between pollutant concentration and health effects at varying concentration levels. An illustrative example is provided based on the 2013 decision on the NAAQS for fine particulate matter (PM_2.5). It shows how the regulator's justification for setting that NAAQS was structured around the regulator's subjective confidence in the continuation of health risks at different concentration levels, and it illustrates how such expressions of uncertainty might be directly incorporated into the risk reduction calculations used in the rule's RIA. The resulting confidence-weighted quantitative risk estimates are found to be substantially different from those in the RIA for that rule. This approach for accounting for an important source of subjective uncertainty also offers the advantage of establishing consistency between the scientific assumptions underlying RIA risk and benefit estimates and the science-based judgments developed when deciding on the relevant standards for important air pollutants such as PM_2.5.     

Maximizing Health Benefits and Minimizing Inequality: Incorporating Local‐Scale Data in the Design and Evaluation of Air Quality Policies

The U.S. Environmental Protection Agency undertook a case study in the Detroit metropolitan area to test the viability of a new multipollutant risk‐based (MP/RB) approach to air quality management, informed by spatially resolved air quality, population, and baseline health data. The case study demonstrated that the MP/RB approach approximately doubled the human health benefits achieved by the traditional approach while increasing cost less than 20%—moving closer to the objective of Executive Order 12866 to maximize net benefits. Less well understood is how the distribution of health benefits from the MP/RB and traditional strategies affect the existing inequalities in air‐pollution‐related risks in Detroit. In this article, we identify Detroit populations that may be both most susceptible to air pollution health impacts (based on local‐scale baseline health data) and most vulnerable to air pollution (based on fine‐scale PM_2.5 air quality modeling and socioeconomic characteristics). Using these susceptible/vulnerable subpopulation profiles, we assess the relative impacts of each control strategy on risk inequality, applying the Atkinson Index (AI) to quantify health risk inequality at baseline and with either risk management approach. We find that the MP/RB approach delivers greater air quality improvements among these subpopulations while also generating substantial benefits among lower‐risk populations. Applying the AI, we confirm that the MP/RB strategy yields less PM_2.5 mortality and asthma hospitalization risk inequality than the traditional approach. We demonstrate the value of this approach to policymakers as they develop cost‐effective air quality management plans that maximize risk reduction while minimizing health inequality.     

The Aggregation of Expert Judgment: Do Good Things Come to Those Who Weight?

Good policy making should be based on available scientific knowledge. Sometimes this knowledge is well established through research, but often scientists must simply express their judgment, and this is particularly so in risk scenarios that are characterized by high levels of uncertainty. Usually in such cases, the opinions of several experts will be sought in order to pool knowledge and reduce error, raising the question of whether individual expert judgments should be given different weights. We argue—against the commonly advocated “classical method”—that no significant benefits are likely to accrue from unequal weighting in mathematical aggregation. Our argument hinges on the difficulty of constructing reliable and valid measures of substantive expertise upon which to base weights. Practical problems associated with attempts to evaluate experts are also addressed. While our discussion focuses on one specific weighting scheme that is currently gaining in popularity for expert knowledge elicitation, our general thesis applies to externally imposed unequal weighting schemes more generally.     

Elicitation of Quantitative Data from a Heterogeneous Expert Panel: Formal Process and Application in Animal Health

This paper presents a protocol for a formal expert judgment process using a heterogeneous expert panel aimed at the quantification of continuous variables. The emphasis is on the process's requirements related to the nature of expertise within the panel, in particular the heterogeneity of both substantive and normative expertise. The process provides the opportunity for interaction among the experts so that they fully understand and agree upon the problem at hand, including qualitative aspects relevant to the variables of interest, prior to the actual quantification task. Individual experts' assessments on the variables of interest, cast in the form of subjective probability density functions, are elicited with a minimal demand for normative expertise. The individual experts' assessments are aggregated into a single probability density function per variable, thereby weighting the experts according to their expertise. Elicitation techniques proposed include the Delphi technique for the qualitative assessment task and the ELI method for the actual quantitative assessment task. Appropriately, the Classical model was used to weight the experts' assessments in order to construct a single distribution per variable. Applying this model, the experts' quality typically was based on their performance on seed variables. An application of the proposed protocol in the broad and multidisciplinary field of animal health is presented. Results of this expert judgment process showed that the proposed protocol in combination with the proposed elicitation and analysis techniques resulted in valid data on the (continuous) variables of interest. In conclusion, the proposed protocol for a formal expert judgment process aimed at the elicitation of quantitative data from a heterogeneous expert panel provided satisfactory results. Hence, this protocol might be useful for expert judgment studies in other broad and/or multidisciplinary fields of interest.     

Quantifying experts' uncertainty about the future cost of exotic diseases

Since the foot-and-mouth disease outbreak of 2001 in the United Kingdom, there has been debate about the sharing, between government and industry, both the costs of livestock disease outbreaks and responsibility for the decisions that give rise to them. As part of a consultation into the formation of a new body to manage livestock diseases, government veterinarians and economists produced estimates of the average annual costs for a number of exotic infectious diseases. In this article, we demonstrate how the government experts were helped to quantify their uncertainties about the cost estimates using formal expert elicitation techniques. This has enabled the decisionmakers to have a greater appreciation of government experts' uncertainty in this policy area.     

Representation,Propagation, and Decision Issues in Risk Analysis Under Incomplete Probabilistic Information

This article tries to clarify the potential role to be played by uncertainty theories such as imprecise probabilities, random sets, and possibility theory in the risk analysis process. Instead of opposing an objective bounding analysis, where only statistically founded probability distributions are taken into account, to the full‐fledged probabilistic approach, exploiting expert subjective judgment, we advocate the idea that both analyses are useful and should be articulated with one another. Moreover, the idea that risk analysis under incomplete information is purely objective is misconceived. The use of uncertainty theories cannot be reduced to a choice between probability distributions and intervals. Indeed, they offer representation tools that are more expressive than each of the latter approaches and can capture expert judgments while being faithful to their limited precision. Consequences of this thesis are examined for uncertainty elicitation, propagation, and at the decision‐making step.     

Can Property Values Capture Changes in Environmental Health Risks? Evidence from a Stated Preference Study in Italy and the United Kingdom

Hedonic models are a common nonmarket valuation technique, but, in practice, results can be affected by omitted variables and whether homebuyers respond to the assumed environmental measure. We undertake an alternative stated preference approach that circumvents these issues. We examine how homeowners in the United Kingdom and Italy value mortality risk reductions by asking them to choose among hypothetical variants of their home that differ in terms of mortality risks from air pollution and price. We find that Italian homeowners hold a value of a statistical life (VSL) of €6.4 million, but U.K. homeowners hold a much lower VSL (€2.1 million). This may be because respondents in the United Kingdom do not perceive air pollution where they live to be as threatening, and actually live in cities with relatively low air pollution. Italian homeowners value a reduction in the risk of dying from cancer more than from other causes, but U.K. respondents do not hold such a premium. Lastly, respondents who face higher baseline risks, due to greater air pollution where they live, hold a higher VSL, particularly in the United Kingdom. In both countries, the VSL is twice as large among individuals who perceive air pollution where they live as high.     

Quantifying the Effects of Expert Selection and Elicitation Design on Experts’ Confidence in Their Judgments About Future Energy Technologies

Expert elicitations are now frequently used to characterize uncertain future technology outcomes. However, their usefulness is limited, in part because: estimates across studies are not easily comparable; choices in survey design and expert selection may bias results; and overconfidence is a persistent problem. We provide quantitative evidence of how these choices affect experts’ estimates. We standardize data from 16 elicitations, involving 169 experts, on the 2030 costs of five energy technologies: nuclear, biofuels, bioelectricity, solar, and carbon capture. We estimate determinants of experts’ confidence using survey design, expert characteristics, and public R&D investment levels on which the elicited values are conditional. Our central finding is that when experts respond to elicitations in person (vs. online or mail) they ascribe lower confidence (larger uncertainty) to their estimates, but more optimistic assessments of best‐case (10th percentile) outcomes. The effects of expert affiliation and country of residence vary by technology, but in general: academics and public‐sector experts express lower confidence than private‐sector experts; and E.U. experts are more confident than U.S. experts. Finally, extending previous technology‐specific work, higher R&D spending increases experts’ uncertainty rather than resolves it. We discuss ways in which these findings should be seriously considered in interpreting the results of existing elicitations and in designing new ones.     

Integrated Uncertainty Analysis for Ambient Pollutant Health Risk Assessment: A Case Study of Ozone Mortality Risk

The U.S. Environmental Protection Agency (EPA) uses health risk assessment to help inform its decisions in setting national ambient air quality standards (NAAQS). EPA's standard approach is to make epidemiologically‐based risk estimates based on a single statistical model selected from the scientific literature, called the “core” model. The uncertainty presented for “core” risk estimates reflects only the statistical uncertainty associated with that one model's concentration‐response function parameter estimate(s). However, epidemiologically‐based risk estimates are also subject to “model uncertainty,” which is a lack of knowledge about which of many plausible model specifications and data sets best reflects the true relationship between health and ambient pollutant concentrations. In 2002, a National Academies of Sciences (NAS) committee recommended that model uncertainty be integrated into EPA's standard risk analysis approach. This article discusses how model uncertainty can be taken into account with an integrated uncertainty analysis (IUA) of health risk estimates. It provides an illustrative numerical example based on risk of premature death from respiratory mortality due to long‐term exposures to ambient ozone, which is a health risk considered in the 2015 ozone NAAQS decision. This example demonstrates that use of IUA to quantitatively incorporate key model uncertainties into risk estimates produces a substantially altered understanding of the potential public health gain of a NAAQS policy decision, and that IUA can also produce more helpful insights to guide that decision, such as evidence of decreasing incremental health gains from progressive tightening of a NAAQS.     

Risk Analysis for Autonomous Underwater Vehicle Operations in Extreme Environments

Autonomous underwater vehicles (AUVs) are used increasingly to explore hazardous marine environments. Risk assessment for such complex systems is based on subjective judgment and expert knowledge as much as on hard statistics. Here, we describe the use of a risk management process tailored to AUV operations, the implementation of which requires the elicitation of expert judgment. We conducted a formal judgment elicitation process where eight world experts in AUV design and operation were asked to assign a probability of AUV loss given the emergence of each fault or incident from the vehicle's life history of 63 faults and incidents. After discussing methods of aggregation and analysis, we show how the aggregated risk estimates obtained from the expert judgments were used to create a risk model. To estimate AUV survival with mission distance, we adopted a statistical survival function based on the nonparametric Kaplan‐Meier estimator. We present theoretical formulations for the estimator, its variance, and confidence limits. We also present a numerical example where the approach is applied to estimate the probability that the Autosub3 AUV would survive a set of missions under Pine Island Glacier, Antarctica in January–March 2009.     

Combining Experts’ Judgments: Comparison of Algorithmic Methods Using Synthetic Data

Expert judgment (or expert elicitation) is a formal process for eliciting judgments from subject‐matter experts about the value of a decision‐relevant quantity. Judgments in the form of subjective probability distributions are obtained from several experts, raising the question how best to combine information from multiple experts. A number of algorithmic approaches have been proposed, of which the most commonly employed is the equal‐weight combination (the average of the experts’ distributions). We evaluate the properties of five combination methods (equal‐weight, best‐expert, performance, frequentist, and copula) using simulated expert‐judgment data for which we know the process generating the experts’ distributions. We examine cases in which two well‐calibrated experts are of equal or unequal quality and their judgments are independent, positively or negatively dependent. In this setting, the copula, frequentist, and best‐expert approaches perform better and the equal‐weight combination method performs worse than the alternative approaches.     

Enhancing the Characterization of Epistemic Uncertainties in PM2.5 Risk Analyses

The Environmental Benefits Mapping and Analysis Program (BenMAP) is a software tool developed by the U.S. Environmental Protection Agency (EPA) that is widely used inside and outside of EPA to produce quantitative estimates of public health risks from fine particulate matter (PM_2.5). This article discusses the purpose and appropriate role of a risk analysis tool to support risk management deliberations, and evaluates the functions of BenMAP in this context. It highlights the importance in quantitative risk analyses of characterization of epistemic uncertainty, or outright lack of knowledge, about the true risk relationships being quantified. This article describes and quantitatively illustrates sensitivities of PM_2.5 risk estimates to several key forms of epistemic uncertainty that pervade those calculations: the risk coefficient, shape of the risk function, and the relative toxicity of individual PM_2.5 constituents. It also summarizes findings from a review of U.S.‐based epidemiological evidence regarding the PM_2.5 risk coefficient for mortality from long‐term exposure. That review shows that the set of risk coefficients embedded in BenMAP substantially understates the range in the literature. We conclude that BenMAP would more usefully fulfill its role as a risk analysis support tool if its functions were extended to better enable and prompt its users to characterize the epistemic uncertainties in their risk calculations. This requires expanded automatic sensitivity analysis functions and more recognition of the full range of uncertainty in risk coefficients.     

Combining Experts' Risk Judgments on Technology Performance of Phytoremediation: Self-Confidence Ratings, Averaging Procedures, and Formative Consensus Building

Expert panels and averaging procedures are common means for coping with the uncertainty of effects of technology application in complex environments. We investigate the connection between confidence and the validity of expert judgment. Moreover, a formative consensus building procedure (FCB) is introduced that generates probability statements on the performance of technologies, and we compare different algorithms for the statistical aggregation of individual judgments. The case study refers to an expert panel of 10 environmental scientists assessing the performance of a soil cleanup technology that uses the capability of certain plants to accumulate heavy metals from the soil in the plant body (phytoremediation). The panel members first provided individual statements on the effectiveness of a phytoremediation. Such statements can support policymakers, answering the questions concerning the expected performance of the new technology in contaminated areas. The present study reviews (1) the steps of the FCB, (2) the constraints of technology application (contaminants, soil structure, etc.), (3) the measurement of expert knowledge, (4) the statistical averaging and the discursive agreement procedures, and (5) the boundaries of application for the FCB method. The quantitative statement oriented part of FCB generates terms such as: "The probability that the concentration of soil contamination will be reduced by at least 50% is 0.8." The data suggest that taking the median of the individual expert estimates provides the most accurate aggregated estimate. The discursive agreement procedure of FCB appears suitable for deriving politically relevant singular statements rather than for obtaining comprehensive information about uncertainties as represented by probability distributions.     

Reassessing the human health benefits from cleaner air

Recent proposals to further reduce permitted levels of air pollution emissions are supported by high projected values of resulting public health benefits. For example, the Environmental Protection Agency recently estimated that the 1990 Clean Air Act Amendment (CAAA) will produce human health benefits in 2020, from reduced mortality rates, valued at nearly $2 trillion per year, compared to compliance costs of $65 billion ($0.065 trillion). However, while compliance costs can be measured, health benefits are unproved: they depend on a series of uncertain assumptions. Among these are that additional life expectancy gained by a beneficiary (with median age of about 80 years) should be valued at about $80,000 per month; that there is a 100% probability that a positive, linear, no-threshold, causal relation exists between PM(2.5) concentration and mortality risk; and that progress in medicine and disease prevention will not greatly diminish this relationship. We present an alternative uncertainty analysis that assigns a positive probability of error to each assumption. This discrete uncertainty analysis suggests (with probability >90% under plausible alternative assumptions) that the costs of CAAA exceed its benefits. Thus, instead of suggesting to policymakers that CAAA benefits are almost certainly far larger than its costs, we believe that accuracy requires acknowledging that the costs purchase a relatively uncertain, possibly much smaller, benefit. The difference between these contrasting conclusions is driven by different approaches to uncertainty analysis, that is, excluding or including discrete uncertainties about the main assumptions required for nonzero health benefits to exist at all.     

Valuing mortality risk reductions from environmental, transport, and health policies: a global meta-analysis of stated preference studies

We conduct, to our knowledge, the first global meta-analysis (MA) of stated preference (SP) surveys of mortality risk valuation. The surveys ask adults their willingness to pay (WTP) for small reductions in mortality risks, deriving estimates of the sample mean value of statistical life (VSL) for environmental, health, and transport policies. We explain the variation in VSL estimates by differences in the characteristics of the SP methodologies applied, the population affected, and the characteristics of the mortality risks valued, including the magnitude of the risk change. The mean (median) VSL in our full data set of VSL sample means was found to be around $7.4 million (2.4 million) (2005 U.S. dollars). The most important variables explaining the variation in VSL are gross domestic product (GDP) per capita and the magnitude of the risk change valued. According to theory, however, VSL should be independent of the risk change. We discuss and test a range of quality screening criteria in order to investigate the effect of limiting the MA to high-quality studies. When limiting the MA to studies that find statistically significant differences in WTP using external or internal scope tests (without requiring strict proportionality), we find that mean VSL from studies that pass both tests tend to be less sensitive to the magnitude of the risk change. Mean VSL also tends to decrease when stricter screening criteria are applied. For many of our screened models, we find a VSL income elasticity of 0.7-0.9, which is reduced to 0.3-0.4 for some subsets of the data that satisfy scope tests or use the same high-quality survey.     

Regulating Under Uncertainty: Newsboy for Exposure Limits

Setting action levels or limits for health protection is complicated by uncertainty in the dose-response relation across a range of hazards and exposures. To address this issue, we consider the classic newsboy problem. The principles used to manage uncertainty for that case are applied to two stylized exposure examples, one for high dose and high dose rate radiation and the other for ammonia. Both incorporate expert judgment on uncertainty quantification in the dose-response relationship. The mathematical technique of probabilistic inversion also plays a key role. We propose a coupled approach, whereby scientists quantify the dose-response uncertainty using techniques such as structured expert judgment with performance weights and probabilistic inversion, and stakeholders quantify associated loss rates.     

Capturing and Integrating Knowledge for Managing Risks in Tunnel Works

Risk‐related knowledge gained from past construction projects is regarded as potentially extremely useful in risk management. This article describes a proposed approach to capture and integrate risk‐related knowledge to support decision making in construction projects. To ameliorate the problem related to the scarcity of risks information often encountered in construction projects, Bayesian Belief Networks are used and expert judgment is elicited to augment available information. Particularly, the article provides an overview of judgment‐based biases that can appear in the elicitation of judgments for constructing Bayesian Networks and the provisos that can be made in this respect to minimize these types of bias. The proposed approach is successfully applied to develop six models for top risks in tunnel works. More than 30 tunneling experts in the Netherlands and Germany were involved in the investigation to provide information on identifying relevant scenarios than can lead to failure events associated with tunneling risks. The article has provided an illustration of the applicability of the developed approach for the case of “face instability in soft soils using slurry shields.”     

Effects of Acknowledging Uncertainty about Earthquake Risk Estimates on San Francisco Bay Area Residents’ Beliefs,Attitudes, and Intentions

We test here the risk communication proposition that explicit expert acknowledgment of uncertainty in risk estimates can enhance trust and other reactions. We manipulated such a scientific uncertainty message, accompanied by probabilities (20%, 70%, implicit [“will occur”] 100%) and time periods (10 or 30 years) in major (≥magnitude 8) earthquake risk estimates to test potential effects on residents potentially affected by seismic activity on the San Andreas fault in the San Francisco Bay Area (n = 750). The uncertainty acknowledgment increased belief that these specific experts were more honest and open, and led to statistically (but not substantively) significant increases in trust in seismic experts generally only for the 20% probability (vs. certainty) and shorter versus longer time period. The acknowledgment did not change judged risk, preparedness intentions, or mitigation policy support. Probability effects independent of the explicit admission of expert uncertainty were also insignificant except for judged risk, which rose or fell slightly depending upon the measure of judged risk used. Overall, both qualitative expressions of uncertainty and quantitative probabilities had limited effects on public reaction. These results imply that both theoretical arguments for positive effects, and practitioners’ potential concerns for negative effects, of uncertainty expression may have been overblown. There may be good reasons to still acknowledge experts’ uncertainties, but those merit separate justification and their own empirical tests.