Probability Elicitation Under Severe Time Pressure: A Rank‐Based Method

Probability elicitation protocols are used to assess and incorporate subjective probabilities in risk and decision analysis. While most of these protocols use methods that have focused on the precision of the elicited probabilities, the speed of the elicitation process has often been neglected. However, speed is also important, particularly when experts need to examine a large number of events on a recurrent basis. Furthermore, most existing elicitation methods are numerical in nature, but there are various reasons why an expert would refuse to give such precise ratio‐scale estimates, even if highly numerate. This may occur, for instance, when there is lack of sufficient hard evidence, when assessing very uncertain events (such as emergent threats), or when dealing with politicized topics (such as terrorism or disease outbreaks). In this article, we adopt an ordinal ranking approach from multicriteria decision analysis to provide a fast and nonnumerical probability elicitation process. Probabilities are subsequently approximated from the ranking by an algorithm based on the principle of maximum entropy, a rule compatible with the ordinal information provided by the expert. The method can elicit probabilities for a wide range of different event types, including new ways of eliciting probabilities for stochastically independent events and low‐probability events. We use a Monte Carlo simulation to test the accuracy of the approximated probabilities and try the method in practice, applying it to a real‐world risk analysis recently conducted for DEFRA (the U.K. Department for the Environment, Farming and Rural Affairs): the prioritization of animal health threats.     

Tailoring value elicitation to decision makers' numeracy and fluency: Expressing value judgments in numbers or words

In organizational settings, options evaluation requires managers to express value judgments on multiple criteria. This research investigates the influence of decision makers' numeracy (ability to use appropriate numerical principles) and fluency (ability to express oneself in words) on their subjective experience of value elicitation as supported by two different techniques: direct rating and MACBETH. The former asks for value judgments to be expressed numerically, the latter non-numerically. The results of our experiment indicate that the two techniques are not psychologically equivalent: decision makers with higher numeracy express values more easily when assisted by the numerical technique whereas decision makers with higher fluency find value elicitation easier with the non-numerical technique. These findings highlight the importance of tailoring value elicitation to decision makers' numeracy and fluency. Implications for decision scientists and analysts are discussed.     

An Integrated Approach to Oversight Assessment for Emerging Technologies

Analysis of oversight systems is often conducted from a single disciplinary perspective and by using a limited set of criteria for evaluation. In this article, we develop an approach that blends risk analysis, social science, public administration, legal, public policy, and ethical perspectives to develop a broad set of criteria for assessing oversight systems. Multiple methods, including historical analysis, expert elicitation, and behavioral consensus, were employed to develop multidisciplinary criteria for evaluating oversight of emerging technologies. Sixty‐six initial criteria were identified from extensive literature reviews and input from our Working Group. Criteria were placed in four categories reflecting the development, attributes, evolution, and outcomes of oversight systems. Expert elicitation, consensus methods, and multidisciplinary review of the literature were used to refine a condensed, operative set of criteria. Twenty‐eight criteria resulted spanning four categories: seven development criteria, 15 attribute criteria, five outcome criteria, and one evolution criterion. These criteria illuminate how oversight systems develop, operate, change, and affect society. We term our approach “integrated oversight assessment” and propose its use as a tool for analyzing relationships among features, outcomes, and tradeoffs of oversight systems. Comparisons among historical case studies of oversight using a consistent set of criteria should result in defensible and evidence‐supported lessons to guide the development of oversight systems for emerging technologies, such as nanotechnology.     

A Characterization for the Spherical Scoring Rule

Strictly proper scoring rules have been studied widely in statistical decision theory and recently in experimental economics because of their ability to encourage assessors to honestly provide their true subjective probabilities. In this article, we study the spherical scoring rule by analytically examining some of its properties and providing some new geometric interpretations for this rule. Moreover, we state a theorem which provides an axiomatic characterization for the spherical scoring rule. The objective of this analysis is to provide a better understanding of one of the most commonly available scoring rules, which could aid decision makers in the selection of an appropriate tool for evaluating and assessing probabilistic forecasts.      

Structured approach to the elicitation of expert beliefs for a Bayesian‐designed clinical trial: a case study

To quantify uncertainty in a formal manner, statisticians play a vital role in identifying a prior distribution for a Bayesian‐designed clinical trial. However, when expert beliefs are to be used to form the prior, the literature is sparse on how feasible and how reliable it is to elicit beliefs from experts. For late‐stage clinical trials, high importance is placed on reliability; however, feasibility may be equally important in early‐stage trials. This article describes a case study to assess how feasible it is to conduct an elicitation session in a structured manner and to form a probability distribution that would be used in a hypothetical early‐stage trial. The case study revealed that by using a structured approach to planning, training and conduct, it is feasible to elicit expert beliefs and form a probability distribution in a timely manner. We argue that by further increasing the published accounts of elicitation of expert beliefs in drug development, there will be increased confidence in the feasibility of conducting elicitation sessions. Furthermore, this will lead to wider dissemination of the pertinent issues on how to quantify uncertainty to both practicing statisticians and others involved with designing trials in a Bayesian manner. Copyright © 2013 John Wiley & Sons, Ltd.     

LinkIT: A Ludic Elicitation Game for Eliciting Risk Perceptions

The mental models approach, a leading strategy to develop risk communications, involves a time‐ and labor‐intensive interview process and a lengthy questionnaire to elicit group‐level risk perceptions. We propose that a similarity ratings approach for structural knowledge elicitation can be adopted to assist the risk mental models approach. The LinkIT game, inspired by games with a purpose (GWAP) technology, is a ludic elicitation tool designed to elicit group understanding of the relations between risk factors in a more enjoyable and productive manner when compared to traditional approaches. That is, consistent with the idea of ludic elicitation, LinkIT was designed to make the elicitation process fun and enjoyable in the hopes of increasing participation and data quality in risk studies. Like the mental models approach, the group mental model obtained via the LinkIT game can hence be generated and represented in a form of influence diagrams. In order to examine the external validity of LinkIT, we conducted a study to compare its performance with respect to a more conventional questionnaire‐driven approach. Data analysis results conclude that the two group mental models elicited from the two approaches are similar to an extent. Yet, LinkIT was more productive and enjoyable than the questionnaire. However, participants commented that the current game has some usability concerns. This presentation summarizes the design and evaluation of the LinkIT game and suggests areas for future work.     

Quantifying experts' uncertainty about the future cost of exotic diseases

Since the foot-and-mouth disease outbreak of 2001 in the United Kingdom, there has been debate about the sharing, between government and industry, both the costs of livestock disease outbreaks and responsibility for the decisions that give rise to them. As part of a consultation into the formation of a new body to manage livestock diseases, government veterinarians and economists produced estimates of the average annual costs for a number of exotic infectious diseases. In this article, we demonstrate how the government experts were helped to quantify their uncertainties about the cost estimates using formal expert elicitation techniques. This has enabled the decisionmakers to have a greater appreciation of government experts' uncertainty in this policy area.     

刍议英语启发式教学方法与技巧

本文从对启发式教学的认识入手,阐释了英语教学中应遵循的一些基本原则,以及可采用多种方法与技巧来启发学生积极思维,从而促进学生智慧的真正发展。     

On describing multivariate skewed distributions: A directional approach

Most multivariate measures of skewness in the literature measure the overall skewness of a distribution. These measures were designed for testing the hypothesis of distributional symmetry; their relevance for describing skewed distributions is less obvious. In this article, the authors consider the problem of characterizing the skewness of multivariate distributions. They define directional skewness as the skewness along a direction and analyze two parametric classes of skewed distributions using measures based on directional skewness. The analysis brings further insight into the classes, allowing for a more informed selection of classes of distributions for particular applications. The authors use the concept of directional skewness twice in the context of Bayesian linear regression under skewed error: first in the elicitation of a prior on the parameters of the error distribution, and then in the analysis of the skewness of the posterior distribution of the regression residuals.     

The impossibility of experimental elicitation of subjective probabilities

We show that if decision makers may have stakes in certain events then the experimental elicitation of their subjective probabilities of these events is impossible.We thank Eyal Sulganik for helpful comments on an earlier draft of this paper.