Evaluating Probabilistic Forecasts with Bayesian Signal Detection Models

We propose the use of signal detection theory (SDT) to evaluate the performance of both probabilistic forecasting systems and individual forecasters. The main advantage of SDT is that it provides a principled way to distinguish the response from system diagnosticity, which is defined as the ability to distinguish events that occur from those that do not. There are two challenges in applying SDT to probabilistic forecasts. First, the SDT model must handle judged probabilities rather than the conventional binary decisions. Second, the model must be able to operate in the presence of sparse data generated within the context of human forecasting systems. Our approach is to specify a model of how individual forecasts are generated from underlying representations and use Bayesian inference to estimate the underlying latent parameters. Given our estimate of the underlying representations, features of the classic SDT model, such as the receiver operating characteristic (ROC) curve and the area under the ROC curve (AUC), follow immediately. We show how our approach allows ROC curves and AUCs to be applied to individuals within a group of forecasters, estimated as a function of time, and extended to measure differences in forecastability across different domains. Among the advantages of this method is that it depends only on the ordinal properties of the probabilistic forecasts. We conclude with a brief discussion of how this approach might facilitate decision making.     

Keeping Surveys Valid,Reliable, and Useful: A Tutorial

This tutorial focuses on how to produce reliable and generalizable data from random‐digit‐dialing (RDD) landline and cell phone surveys. The article notes that RDD response rates have declined and explores the impact of this pronounced decline. The tutorial addresses order, response mode, and many other biases, sample size, cooperation and response rates, weighting, and hybrid designs‐all using examples from risk analysis to illustrate the key points. The article ends with a brief review of the advantages and disadvantages of major Internet and paper surveys tools, and how these can be molded and sometimes combined in repeated, longitudinal, and other designs to answer questions about risk preferences and perceptions.     

A simple two-stage design for quantitative responses with application to a study in diabetic neuropathic pain

Two-stage designs offer substantial advantages for early phase II studies. The interim analysis following the first stage allows the study to be stopped for futility, or more positively, it might lead to early progression to the trials needed for late phase II and phase III. If the study is to continue to its second stage, then there is an opportunity for a revision of the total sample size. Two-stage designs have been implemented widely in oncology studies in which there is a single treatment arm and patient responses are binary. In this paper the case of two-arm comparative studies in which responses are quantitative is considered. This setting is common in therapeutic areas other than oncology. It will be assumed that observations are normally distributed, but that there is some doubt concerning their standard deviation, motivating the need for sample size review. The work reported has been motivated by a study in diabetic neuropathic pain, and the development of the design for that trial is described in detail.     

Consultants' forum: should post hoc sample size calculations be done?

Pre‐study sample size calculations for clinical trial research protocols are now mandatory. When an investigator is designing a study to compare the outcomes of an intervention, an essential step is the calculation of sample sizes that will allow a reasonable chance (power) of detecting a pre‐determined difference (effect size) in the outcome variable, at a given level of statistical significance. Frequently studies will recruit fewer patients than the initial pre‐study sample size calculation suggested. Investigators are faced with the fact that their study may be inadequately powered to detect the pre‐specified treatment effect and the statistical analysis of the collected outcome data may or may not report a statistically significant result. If the data produces a “non‐statistically significant result” then investigators are frequently tempted to ask the question “Given the actual final study size, what is the power of the study, now, to detect a treatment effect or difference?” The aim of this article is to debate whether or not it is desirable to answer this question and to undertake a power calculation, after the data have been collected and analysed. Copyright © 2008 John Wiley & Sons, Ltd.     

Back to basics: explaining sample size in outcome trials,are statisticians doing a thorough job?

Time to event outcome trials in clinical research are typically large, expensive and high‐profile affairs. Such trials are commonplace in oncology and cardiovascular therapeutic areas but are also seen in other areas such as respiratory in indications like chronic obstructive pulmonary disease. Their progress is closely monitored and results are often eagerly awaited. Once available, the top line result is often big news, at least within the therapeutic area in which it was conducted, and the data are subsequently fully scrutinized in a series of high‐profile publications. In such circumstances, the statistician has a vital role to play in the design, conduct, analysis and reporting of the trial. In particular, in drug development it is incumbent on the statistician to ensure at the outset that the sizing of the trial is fully appreciated by their medical, and other non‐statistical, drug development team colleagues and that the risk of delivering a statistically significant but clinically unpersuasive result is minimized. The statistician also has a key role in advising the team when, early in the life of an outcomes trial, a lower than anticipated event rate appears to be emerging. This paper highlights some of the important features relating to outcome trial sample sizing and makes a number of simple recommendations aimed at ensuring a better, common understanding of the interplay between sample size and power and the final result required to provide a statistically positive and clinically persuasive outcome. Copyright © 2009 John Wiley & Sons, Ltd.     

Notes on authors

Social network analysis (SNA), a method which can be used to explore networks in various contexts, has received increasing attention. Drawing on the development of European smoke-free policy, this paper explores how a mixed-method approach to SNA can be utilised to investigate a complex policy network. Textual data from public documents, consultation submissions and websites were extracted, converted and analysed using plagiarism detection software and quantitative network analysis and qualitative data from public documents and 35 interviews were thematically analysed. While the quantitative analysis enabled understanding of the network’s structure and components, the qualitative analysis provided in-depth information about specific actors’ positions, relationships and interactions. The paper establishes that SNA is suited to empirically testing and analysing networks in EU policy-making. It contributes to methodological debates about the antagonism between qualitative and quantitative approaches and demonstrates that qualitative and quantitative network analysis can offer a powerful tool for policy analysis.     

A Class of Sampling Two Units with Probability Proportional to Size

A class of sampling two units without replacement with inclusion probability proportional to size is proposed in this article. Many different well known probability proportional to size sampling designs are special cases from this class. The first and second inclusion probabilities of this class satisfy important properties and provide a non-negative variance estimator of the Horvitz and Thompson estimator for the population total. Suitable choice for the first and second inclusion probabilities from this class can be used to reduce the variance estimator of the Horvitz and Thompson estimator. Comparisons between different proportional to size sampling designs through real data and artificial examples are given. Examples show that the minimum variance of the Horvitz and Thompson estimator obtained from the proposed design is not attainable for the most cases at any of the well known designs.     

应用油藏规模序列法预测东营凹陷剩余资源量

随着油气勘探程度的不断提高，未发现的剩余油气资源越来越少。如何准确预测高勘探程度即成熟探区的剩余油气资源，成为石油投资者和油气资源评价工作者关注的焦点。详细探讨了油藏规模序列法预测油气资源的原理、方法、操作流程及存在的问题，并运用该方法对东营凹陷剩余可探明储量以油气成藏体系为评价单元进行了预测。研究表明，东营凹陷剩余可探明储量约为17.03×10⁸t，仍具有较高的勘探潜力，并且主要集中在东营中央背斜带和东营凹陷北坡两个油气成藏体系内。实践证明，由于油藏规模序列法合理回避了油气成因机理问题，减少了资源评价工作中人为因素的影响，使资源评价结果更为客观。     

A comparison of test statistics for the recovery of rapid growth‐based enumeration tests

This paper considers five test statistics for comparing the recovery of a rapid growth‐based enumeration test with respect to the compendial microbiological method using a specific nonserial dilution experiment. The finite sample distributions of these test statistics are unknown, because they are functions of correlated count data. A simulation study is conducted to investigate the type I and type II error rates. For a balanced experimental design, the likelihood ratio test and the main effects analysis of variance (ANOVA) test for microbiological methods demonstrated nominal values for the type I error rate and provided the highest power compared with a test on weighted averages and two other ANOVA tests. The likelihood ratio test is preferred because it can also be used for unbalanced designs. It is demonstrated that an increase in power can only be achieved by an increase in the spiked number of organisms used in the experiment. The power is surprisingly not affected by the number of dilutions or the number of test samples. A real case study is provided to illustrate the theory. Copyright © 2013 John Wiley & Sons, Ltd.