Probabilistic evaluation of handwriting evidence: likelihood ratio for authorship

Summary.  The evaluation of handwritten characters that are selected from an anonymous letter and written material from a suspect is an open problem in forensic science. The individualization of handwriting is largely dependent on examiners who evaluate the characteristics in a qualitative and subjective way. Precise individual characterization of the shape of handwritten characters is possible through Fourier analysis: each handwritten character can be described through a set of variables such as the surface and harmonics as demonstrated by Marquis and co-workers in 2005. The assessment of the value of the evidence is performed through the derivation of a likelihood ratio for multivariate data. The methodology allows the forensic scientist to take into account the correlation between variables, and the non-constant variability within sources (i.e. individuals). Numerical procedures are implemented to handle the complexity and to compute the marginal likelihood under competing propositions.     

Frequentist-Bayesian Monte Carlo testing

Abstract

Conventional methods for statistical hypothesis testing has historically been categorized as frequentist or Bayesian. But, a third option based on a reconciling hybrid frequentist-Bayesian framework is quickly emerging. Although prominent, there are applications where the exact hybrid test is not computable. For such cases, the present paper introduces a straightforward Monte Carlo procedure for performing frequentist-Bayesian testing.     

Reporting Bayes factors or probabilities to decision makers of unknown loss functions

When competing interests seek to influence a decision maker, a scientist must report a posterior probability or a Bayes factor among those consistent with the evidence. The disinterested scientist seeks to report the value that is least controversial in the sense that it is best protected from being discredited by one of the competing interests. If the loss function of the decision maker is not known but can be assumed to satisfy two invariance conditions, then the least controversial value is a weighted generalized mean of the upper and lower bounds of the interval.     

Evidential inference for diffusion-type processes

This article analyses diffusion-type processes from a new point-of-view. Consider two statistical hypotheses on a diffusion process. We do not use a classical test to reject or accept one hypothesis using the Neyman–Pearson procedure and do not involve Bayesian approach. As an alternative, we propose using a likelihood paradigm to characterizing the statistical evidence in support of these hypotheses. The method is based on evidential inference introduced and described by Royall [Royall R. Statistical evidence: a likelihood paradigm. London: Chapman and Hall; 1997]. In this paper, we extend the theory of Royall to the case when data are observations from a diffusion-type process instead of iid observations. The empirical distribution of likelihood ratio is used to formulate the probability of strong, misleading and weak evidences. Since the strength of evidence can be affected by the sampling characteristics, we present a simulation study that demonstrates these effects. Also we try to control misleading evidence and reduce them by adjusting these characteristics. As an illustration, we apply the method to the Microsoft stock prices.     

The evaluation of DNA evidence in pedigrees requiring population inference

Summary. The evaluation of nuclear DNA evidence for identification purposes is performed here taking account of the uncertainty about population parameters. Graphical models are used to detail the hypotheses being debated in a trial with the aim of obtaining a directed acyclic graph. Graphs also clarify the set of evidence that contributes to population inferences and they also describe the conditional independence structure of DNA evidence. Numerical illustrations are provided by re-examining three case-studies taken from the literature. Our calculations of the weight of evidence differ from those given by the authors of case-studies in that they reveal more conservative values.     

Evaluation of trace evidence in the form of multivariate data

Summary.  The evaluation of measurements on characteristics of trace evidence found at a crime scene and on a suspect is an important part of forensic science. Five methods of assessment for the value of the evidence for multivariate data are described. Two are based on significance tests and three on the evaluation of likelihood ratios. The likelihood ratio which compares the probability of the measurements on the evidence assuming a common source for the crime scene and suspect evidence with the probability of the measurements on the evidence assuming different sources for the crime scene and suspect evidence is a well-documented measure of the value of the evidence. One of the likelihood ratio approaches transforms the data to a univariate projection based on the first principal component. The other two versions of the likelihood ratio for multivariate data account for correlation among the variables and for two levels of variation: that between sources and that within sources. One version assumes that between-source variability is modelled by a multivariate normal distribution; the other version models the variability with a multivariate kernel density estimate. Results are compared from the analysis of measurements on the elemental composition of glass.     

Interpreting statistical evidence by using imperfect models: robust adjusted likelihood functions

Summary. The strength of statistical evidence is measured by the likelihood ratio. Two key performance properties of this measure are the probability of observing strong misleading evidence and the probability of observing weak evidence. For the likelihood function associated with a parametric statistical model, these probabilities have a simple large sample structure when the model is correct. Here we examine how that structure changes when the model fails. This leads to criteria for determining whether a given likelihood function is robust (continuing to perform satisfactorily when the model fails), and to a simple technique for adjusting both likelihoods and profile likelihoods to make them robust. We prove that the expected information in the robust adjusted likelihood cannot exceed the expected information in the likelihood function from a true model. We note that the robust adjusted likelihood is asymptotically fully efficient when the working model is correct, and we show that in some important examples this efficiency is retained even when the working model fails. In such cases the Bayes posterior probability distribution based on the adjusted likelihood is robust, remaining correct asymptotically even when the model for the observable random variable does not include the true distribution. Finally we note a link to standard frequentist methodology—in large samples the adjusted likelihood functions provide robust likelihood-based confidence intervals.     

On the correspondence between frequentist and Bayesian tests

Modern theory for statistical hypothesis testing can broadly be classified as Bayesian or frequentist. Unfortunately, one can reach divergent conclusions if Bayesian and frequentist approaches are applied in parallel to analyze the same data set. This is a serious impasse since there is a lack of consensus on when to use one approach in detriment of the other. However, this conflict can be resolved. The present paper shows the existence of a perfect equivalence between Bayesian and frequentist methods for testing. Hence, Bayesian and frequentist decision rules can always be calibrated, in both directions, in order to present concordant results.     

P-Value Precision and Reproducibility

P-values are useful statistical measures of evidence against a null hypothesis. In contrast to other statistical estimates, however, their sample-to-sample variability is usually not considered or estimated, and therefore not fully appreciated. Via a systematic study of log-scale p-value standard errors, bootstrap prediction bounds, and reproducibility probabilities for future replicate p-values, we show that p-values exhibit surprisingly large variability in typical data situations. In addition to providing context to discussions about the failure of statistical results to replicate, our findings shed light on the relative value of exact p-values vis-a-vis approximate p-values, and indicate that the use of *, **, and *** to denote levels 0.05, 0.01, and 0.001 of statistical significance in subject-matter journals is about the right level of precision for reporting p-values when judged by widely accepted rules for rounding statistical estimates.     

How Often Likelihood Ratios are Misleading in Sequential Trials

How often would investigators be misled if they took advantage of the likelihood principle and used likelihood ratios—which need not be adjusted for multiple looks at the data—to frequently examine accumulating data? The answer, perhaps surprisingly, is not often. As expected, the probability of observing misleading evidence does increase with each additional examination. However, the amount by which this probability increases converges to zero as the sample size grows. As a result, the probability of observing misleading evidence remains bounded—and therefore controllable—even with an infinite number of looks at the data. Here we use boundary crossing results to detail how often misleading likelihood ratios arise in sequential designs. We find that the probability of observing a misleading likelihood ratio is often much less than its universal bound. Additionally, we find that in the presence of fixed-dimensional nuisance parameters, profile likelihoods are to be preferred over estimated likelihoods which result from replacing the nuisance parameters by their global maximum likelihood estimates.     

A bayesian test for a two-way contingency table using independence priors

One method of testing for independence in a two-way table is based on the Bayes factor, the ratio of the likelihoods under the independence hypothesis H and the alternative hypothesis H. The main difficulty in this approach is the specification of prior distributions on the composite hypotheses H and H. A new Bayesian test statistic is constructed by using a prior distribution on H that is concentrated about the “independence surface” H. Approximations are proposed which simplify the computation of the test statistic. The values of the Bayes factor are compared with values of statistics proposed by Gunel and Dickey (1974), Good and Crook (1987), and Spiegelhalter and Smith (1982) for a number of two-way tables. This investigation suggests a strong relationship between the new statistic and the p-value.     

Mixture Models in View of Evidential Analysis

In some practical inferential situations, it is needed to mix some finite sort of distributions to fit an adequate model for multi-modal observations. In this article, using evidential analysis, we determine the sample size for supporting hypotheses about the mixture proportion and homogeneity. An Expectation-Maximization algorithm is used to evaluate the probability of strong misleading evidence based on modified likelihood ratio as a measure of support.     

Axiomatic Development of Profile Likelihoods as the Strength of Evidence for Composite Hypotheses

One important type of question in statistical inference is how to interpret data as evidence. The law of likelihood provides a satisfactory answer in interpreting data as evidence for simple hypotheses, but remains silent for composite hypotheses. This article examines how the law of likelihood can be extended to composite hypotheses within the scope of the likelihood principle. From a system of axioms, we conclude that the strength of evidence for the composite hypotheses should be represented by an interval between lower and upper profiles likelihoods. This article is intended to reveal the connection between profile likelihoods and the law of likelihood under the likelihood principle rather than argue in favor of the use of profile likelihoods in addressing general questions of statistical inference. The interpretation of the result is also discussed.     

Weighing the Evidence for Hypotheses with Small Samples of Right-censored Exponential Data

The p-value evidence for an alternative to a null hypothesis regarding the mean lifetime can be unreliable if based on asymptotic approximations when there is only a small sample of right-censored exponential data. However, a guarded weight of evidence for the alternative can always be obtained without approximation, no matter how small the sample, and has some other advantages over p-values. Weights of evidence are defined as estimators of 0 when the null hypothesis is true and 1 when the alternative is true, and they are judged on the basis of the ensuing risks, where risk is mean squared error of estimation. The evidence is guarded in that a preassigned bound is placed on the risk under the hypothesis. Practical suggestions are given for choosing the bound and for interpreting the magnitude of the weight of evidence. Acceptability profiles are obtained by inversion of a family of guarded weights of evidence for two-sided alternatives to point hypotheses, just as confidence intervals are obtained from tests; these profiles are arguably more informative than confidence intervals, and are easily determined for any level and any sample size, however small. They can help understand the effects of different amounts of censoring. They are found for several small size data sets, including a sample of size 12 for post-operative cancer patients. Both singly Type I and Type II censored examples are included. An examination of the risk functions of these guarded weights of evidence suggests that if the censoring time is of the same magnitude as the mean lifetime, or larger, then the risks in using a guarded weight of evidence based on a likelihood ratio are not much larger than they would be if the parameter were known.     

Model fusion and multiple testing in the likelihood paradigm: shrinkage and evidence supporting a point null hypothesis

ABSTRACT

According to the general law of likelihood, the strength of statistical evidence for a hypothesis as opposed to its alternative is the ratio of their likelihoods, each maximized over the parameter of interest. Consider the problem of assessing the weight of evidence for each of several hypotheses. Under a realistic model with a free parameter for each alternative hypothesis, this leads to weighing evidence without any shrinkage toward a presumption of the truth of each null hypothesis. That lack of shrinkage can lead to many false positives in settings with large numbers of hypotheses. A related problem is that point hypotheses cannot have more support than their alternatives. Both problems may be solved by fusing the realistic model with a model of a more restricted parameter space for use with the general law of likelihood. Applying the proposed framework of model fusion to data sets from genomics and education yields intuitively reasonable weights of evidence.     

Bayes factors based on test statistics

Summary.  Traditionally, the use of Bayes factors has required the specification of proper prior distributions on model parameters that are implicit to both null and alternative hypotheses. I describe an approach to defining Bayes factors based on modelling test statistics. Because the distributions of test statistics do not depend on unknown model parameters, this approach eliminates much of the subjectivity that is normally associated with the definition of Bayes factors. For standard test statistics, including the χ ²-, F -, t - and z -statistics, the values of Bayes factors that result from this approach have simple, closed form expressions.     

Analysis of incomplete data under non-random mechanisms: bayesian inference

Inferences are made concerning population proportions when data are not missing at random.Both one sample and two sample situations are considered with examples in clinical trials.The one samplesituation involves the existence of response related incomplete data in a study conducted to make inferences involving the proportion. The two sample problem involves comparing two treatments in clinical trials when there exists dropouts due to both the treatment and the response to the treatment.Bayes procedures are used in estimating parameters of interest and testing hypotheses of interest in these two situations. An ad-hoc approach to the classical inference is presented for each ofthe two situations and compared with the Bayesian approach discussed. To illustrate the theory developed, data from clinical trials of severe head trauma patients at the Medical College of Virginia Head Injury Center from 1984 to 1987 is considered     

Stable local computation with conditional Gaussian distributions

This article describes a propagation scheme for Bayesian networks with conditional Gaussian distributions that does not have the numerical weaknesses of the scheme derived in Lauritzen (Journal of the American Statistical Association 87: 1098–1108, 1992).The propagation architecture is that of Lauritzen and Spiegelhalter (Journal of the Royal Statistical Society, Series B 50: 157– 224, 1988).In addition to the means and variances provided by the previous algorithm, the new propagation scheme yields full local marginal distributions. The new scheme also handles linear deterministic relationships between continuous variables in the network specification.The computations involved in the new propagation scheme are simpler than those in the previous scheme and the method has been implemented in the most recent version of the HUGIN software.     

Teaching Bayes’ Theorem: Strength of Evidence as Predictive Accuracy

Although teaching Bayes’ theorem is popular, the standard approach—targeting posterior distributions of parameters—may be improved. We advocate teaching Bayes’ theorem in a ratio form where the posterior beliefs relative to the prior beliefs equals the conditional probability of data relative to the marginal probability of data. This form leads to an interpretation that the strength of evidence is relative predictive accuracy. With this approach, students are encouraged to view Bayes’ theorem as an updating mechanism, to obtain a deeper appreciation of the role of the prior and of marginal data, and to view estimation and model comparison from a unified perspective.