Evaluating Probabilistic Forecasts with Bayesian Signal Detection Models

We propose the use of signal detection theory (SDT) to evaluate the performance of both probabilistic forecasting systems and individual forecasters. The main advantage of SDT is that it provides a principled way to distinguish the response from system diagnosticity, which is defined as the ability to distinguish events that occur from those that do not. There are two challenges in applying SDT to probabilistic forecasts. First, the SDT model must handle judged probabilities rather than the conventional binary decisions. Second, the model must be able to operate in the presence of sparse data generated within the context of human forecasting systems. Our approach is to specify a model of how individual forecasts are generated from underlying representations and use Bayesian inference to estimate the underlying latent parameters. Given our estimate of the underlying representations, features of the classic SDT model, such as the receiver operating characteristic (ROC) curve and the area under the ROC curve (AUC), follow immediately. We show how our approach allows ROC curves and AUCs to be applied to individuals within a group of forecasters, estimated as a function of time, and extended to measure differences in forecastability across different domains. Among the advantages of this method is that it depends only on the ordinal properties of the probabilistic forecasts. We conclude with a brief discussion of how this approach might facilitate decision making.     

Biased estimation of a variance

Summary Let , whereX _i are i.i.d. random variables with a finite variance σ² and is the usual estimate of the mean ofX _i. We consider the problem of finding optimal α with respect to the minimization of the expected value of |S ²(σ)−σ²|^k for variousk and with respect to Pitman's nearness criterion. For the Gaussian case analytical results are obtained and for some non-Gaussian cases we present Monte Carlo results regarding Pitman's criteron. This research was supported by Science Fund of Serbia, grant number 04M03, through Mathematical Institute, Belgrade.     

Comparing the Quality of Data Obtained by Minimally Balanced and Fully Balanced Attitude Questions

When measuring attitudes with questions that offer dichotomous,mutually exclusive response options, researchers can ask "fullybalanced" questions (which fully state both competing pointsof view) or "minimally balanced" questions (which fully stateone viewpoint and only briefly acknowledge the second viewpoint).The two studies reported here investigated whether the greaterefficiency of the latter approach brought with it reductionsin the quality of the data obtained. Two experiments embeddedin national sample surveys showed that minimally balanced andfully balanced attitude questions yielded similar distributionsof responses and that responses to the two question forms wereequivalent in terms of concurrent validity. These studies suggestthat greater efficiency can be achieved via minimal balancingat no cost in terms of data quality.     

How Too Little Can Give You a Little Too Much: Determining the Number of Household Phone Lines in RDD Surveys

Accurately assessing the number of household phone lines inrandom-digit dialed (RDD) surveys for use in weighting has becomemore complicated over time. This study evaluates phone lineweighting by asking an in-depth battery of seven questions designedto measure the number of available land lines in the home moreprecisely. The results show that this weighting correction isnot so simple, and that inadequately parsing the type and useof household phone numbers produces an inaccurate weight. Failingexplicitly to remove irrelevant phone numbers, especially cellphones, produces a larger-than-appropriate weighting adjustment.Potentially more important on a practical level is that, whendone correctly, and even when "overdone," phone line weightinghad little impact across a diverse set of opinion and demographicvariables.