A re-evaluation of random-effects meta-analysis

Summary.  Meta-analysis in the presence of unexplained heterogeneity is frequently undertaken by using a random-effects model, in which the effects underlying different studies are assumed to be drawn from a normal distribution. Here we discuss the justification and interpretation of such models, by addressing in turn the aims of estimation, prediction and hypothesis testing. A particular issue that we consider is the distinction between inference on the mean of the random-effects distribution and inference on the whole distribution. We suggest that random-effects meta-analyses as currently conducted often fail to provide the key results, and we investigate the extent to which distribution-free, classical and Bayesian approaches can provide satisfactory methods. We conclude that the Bayesian approach has the advantage of naturally allowing for full uncertainty, especially for prediction. However, it is not without problems, including computational intensity and sensitivity to a priori judgements. We propose a simple prediction interval for classical meta-analysis and offer extensions to standard practice of Bayesian meta-analysis, making use of an example of studies of 'set shifting' ability in people with eating disorders.     

On wasps and club dinners

Francis Ysidro Edgeworth could be termed a Victorian gentleman polymath: born on the family's estate in Edgeworthstown, Ireland, and a trained classicist and barrister, he was capable of both working as a Lecturer in Greek and deriving in 1883 (using what we now call a Bayesian argument) "what may the earliest appearance in any form of the Student's t distribution"¹. His unusual second name arose from his Catalan refugee mother, whom his father is reputed to have met on the steps of the British Museum.     

Bayesian graphical modelling: a case-study in monitoring health outcomes

Bayesian graphical modelling represents the synthesis of several recent developments in applied complex modelling. After describing a moderately challenging real example, we show how graphical models and Markov chain Monte Carlo methods naturally provide a direct path between model specification and the computational means of making inferences on that model. These ideas are illustrated with a range of modelling issues related to our example. An appendix discusses the BUGS software.     

Commissioned analysis of surgical performance using routine data: lessons from the Bristol inquiry

The public inquiry into paediatric cardiac surgery at the Bristol Royal Infirmary commissioned the authors to design and conduct analyses of routine data sources to compare surgical outcomes between centres. Such analyses are necessarily complex in this context but were further hampered by the inherent inconsistencies and mediocre quality of the various sources of data. Three levels of analysis of increasing sophistication were carried out. The reasonable consistency of the results arising from different sources of data, together with a number of sensitivity analyses, led us to conclude that there had been excess mortality in Bristol in open heart operations on children under 1 year of age. We consider criticisms of our analysis and discuss the role of statisticians in this inquiry and their contribution to the final report of the inquiry. The potential statistical role in future programmes for monitoring clinical performance is highlighted.     

Shipman's statistical legacy

Harold Shipman, who in January committed suicide in prison, has become notorious the world over as one of the most prolific serial killers of all time. His case has also seriously dented public confidence in doctors. David Spiegelhalter and Nicky Best explain how industrial quality control techniques could be adapted to signal when death rates among a doctor's patients are surprisingly high, and the tricky issues that would arise in implementing such a monitoring system.     

Surgical audit: statistical lessons from Nightingale and Codman

There is a long history of interest in examining and comparing surgical outcomes. The 'epidemiological' approach was initiated by Florence Nightingale in her suggestion for uniform surgical statistics, and she clearly predicted the problems that are associated with collecting, analysing and interpreting such data. Unfortunately those responsible for implementing and reporting her scheme appeared not to have shared her insight. The contrasting 'clinical' approach was championed by Ernest Codman in his search for full and honest appraisals of surgical errors. Once again, despite initial enthusiasm, others had great difficulty in following his example, although we discuss a recent instance of a reflective analysis of an individual surgeon's performance. We conclude by suggesting that a synthesis between these approaches is appropriate, but we follow others in warning of the inevitable extra-statistical difficulties that will arise.     

London murders: a predictable pattern?

Four stabbings to death in a single day. Ninety murders in 7 months. Shocking figures—or are they? Knife crime makes the headlines almost daily but are Londoners really at increased risk of being murdered? David Spiegelhalter and Arthur Barnett investigate—and find a predictable pattern of murder.     

Strategies for inference robustness in focused modelling

Advances in computation mean that it is now possible to fit a wide range of complex models to data, but there remains the problem of selecting a model on which to base reported inferences. Following an early suggestion of Box & Tiao, it seems reasonable to seek 'inference robustness' in reported models, so that alternative assumptions that are reasonably well supported would not lead to substantially different conclusions. We propose a four-stage modelling strategy in which we iteratively assess and elaborate an initial model, measure the support for each of the resulting family of models, assess the influence of adopting alternative models on the conclusions of primary interest, and identify whether an approximate model can be reported. The influence-support plot is then introduced as a tool to aid model comparison. The strategy is semi-formal, in that it could be embedded in a decision-theoretic framework but requires substantive input for any specific application. The one restriction of the strategy is that the quantity of interest, or 'focus', must retain its interpretation across all candidate models. It is, therefore, applicable to analyses whose goal is prediction, or where a set of common model parameters are of interest and candidate models make alternative distributional assumptions. The ideas are illustrated by two examples. Technical issues include the calibration of the Kullback-Leibler divergence between marginal distributions, and the use of alternative measures of support for the range of models fitted.