Meta-analysis using multilevel models with an application to the study of class size effects

Meta-analysis is formulated as a special case of a multilevel (hierarchical data) model in which the highest level is that of the study and the lowest level that of an observation on an individual respondent. Studies can be combined within a single model where the responses occur at different levels of the data hierarchy and efficient estimates are obtained. An example is given from studies of class sizes and achievement in schools, where study data are available at the aggregate level in terms of overall mean values for classes of different sizes, and also at the student level.     

Sensitivity analysis of the unconfoundedness assumption with an application to an evaluation of college choice effects on earnings

We evaluate the effects of college choice on earnings using Swedish register databases. This case study is used to motivate the introduction of a novel procedure to analyse the sensitivity of such an observational study to the assumption made that there are no unobserved confounders – variables affecting both college choice and earnings. This assumption is not testable without further information, and should be considered an approximation of reality. To perform a sensitivity analysis, we measure the departure from the unconfoundedness assumption with the correlation between college choice and earnings when conditioning on observed covariates. The use of a correlation as a measure of dependence allows us to propose a standardised procedure by advocating the use of a fixed value for the correlation, typically 1% or 5%, when checking the sensitivity of an evaluation study. A correlation coefficient is, moreover, intuitive to most empirical scientists, which makes the results of our sensitivity analysis easier to communicate than those of previously proposed methods. In our evaluation of the effects of college choice on earnings, the significantly positive effect obtained could not be questioned by a sensitivity analysis allowing for unobserved confounders inducing at most 5% correlation between college choice and earnings.     

A causal proportional hazards estimator under homogeneous or heterogeneous selection in an IV setting

Lifetime Data Analysis - In this paper we present a framework to do estimation in a structural Cox model when there may be unobserved confounding. The model is phrased in terms of a selection bias...     

Bayesian modelling of effects of prenatal alcohol exposure on child cognition based on data from multiple cohorts

High levels of prenatal alcohol exposure (PAE) result in significant cognitive deficits in children, but the exact nature of the dose-response relationship is less well understood. To investigate this relationship, data were assembled from six longitudinal birth cohort studies examining the effects of PAE on cognitive outcomes from early school age through adolescence. Structural equation models (SEMs) are a natural approach to consider, because of the way they conceptualise multiple observed outcomes as relating to an underlying latent variable of interest, which can then be modelled as a function of exposure and other predictors of interest. However, conventional SEMs could not be fitted in this context because slightly different outcome measures were used in the six studies. In this paper we propose a multi-group Bayesian SEM that maps the unobserved cognition variable to a broad range of observed outcomes. The relation between these variables and PAE is then examined while controlling for potential confounders via propensity score adjustment. By examining different possible dose-response functions, the proposed framework is used to investigate whether there is a threshold PAE level that results in minimal cognitive deficit.     

Multilevel modelling of the number of property crimes: household and area effects

Summary.  This study examines household and area effects on the incidence of total property crimes and burglaries and thefts. It uses data from the 2000 British Crime Survey and the 1991 UK census small area statistics. Results are obtained from estimated random-effects multilevel models, with an assumed negative binomial distribution of the dependent variable. Both household and area characteristics, as well as selected interactions, explain a significant portion of the variation in property crimes. There are also a large number of significant between-area random variances and covariances of household characteristics. The estimated fixed and random effects may assist in advancing victimization theory. The methods have potential for developing a better understanding of factors that give rise to crime and so assist in framing crime prevention policy.     

Variance partitioning in multilevel logistic models that exhibit overdispersion

Summary.  A common application of multilevel models is to apportion the variance in the response according to the different levels of the data. Whereas partitioning variances is straightforward in models with a continuous response variable with a normal error distribution at each level, the extension of this partitioning to models with binary responses or to proportions or counts is less obvious. We describe methodology due to Goldstein and co-workers for apportioning variance that is attributable to higher levels in multilevel binomial logistic models. This partitioning they referred to as the variance partition coefficient. We consider extending the variance partition coefficient concept to data sets when the response is a proportion and where the binomial assumption may not be appropriate owing to overdispersion in the response variable. Using the literacy data from the 1991 Indian census we estimate simple and complex variance partition coefficients at multiple levels of geography in models with significant overdispersion and thereby establish the relative importance of different geographic levels that influence educational disparities in India.     

A Discrete-Time Multilevel Mixture Model for Event History Data with Long-Term Survivors,with an Application to an Analysis of Contraceptive Sterilization in Bangladesh

Event history models typically assume that the entire population is at risk of experiencing the event of interest throughout the observation period. However, there will often be individuals, referred to as long-term survivors, who may be considered a priori to have a zero hazard throughout the study period. In this paper, a discrete-time mixture model is proposed in which the probability of long-term survivorship and the timing of event occurrence are modelled jointly. Another feature of event history data that often needs to be considered is that they may come from a population with a hierarchical structure. For example, individuals may be nested within geographical regions and individuals in the same region may have similar risks of experiencing the event of interest due to unobserved regional characteristics. Thus, the discrete-time mixture model is extended to allow for clustering in the likelihood and timing of an event within regions. The model is further extended to allow for unobserved individual heterogeneity in the hazard of event occurrence. The proposed model is applied in an analysis of contraceptive sterilization in Bangladesh. The results show that a woman's religion and education level affect her probability of choosing sterilization, but not when she gets sterilized. There is also evidence of community-level variation in sterilization timing, but not in the probability of sterilization.     

Decomposition analysis as a framework for understanding heterogeneity of treatment effects in non-randomized health care studies

This paper uses the decomposition framework from the economics literature to examine the statistical structure of treatment effects estimated with observational data compared to those estimated from randomized studies. It begins with the estimation of treatment effects using a dummy variable in regression models and then presents the decomposition method from economics which estimates separate regression models for the comparison groups and recovers the treatment effect using bootstrapping methods. This method shows that the overall treatment effect is a weighted average of structural relationships of patient features with outcomes within each treatment arm and differences in the distributions of these features across the arms. In large randomized trials, it is assumed that the distribution of features across arms is very similar. Importantly, randomization not only balances observed features but also unobserved. Applying high dimensional balancing methods such as propensity score matching to the observational data causes the distributional terms of the decomposition model to be eliminated but unobserved features may still not be balanced in the observational data. Finally, a correction for non-random selection into the treatment groups is introduced via a switching regime model. Theoretically, the treatment effect estimates obtained from this model should be the same as those from a randomized trial. However, there are significant challenges in identifying instrumental variables that are necessary for estimating such models. At a minimum, decomposition models are useful tools for understanding the relationship between treatment effects estimated from observational versus randomized data.     

Variation in caesarean delivery rates across hospitals: a Bayesian semi-parametric approach

This article presents a Bayesian semi-parametric approach for modeling the occurrence of cesarean sections using a sample of women delivering in 20 hospitals of Sardinia (Italy). A multilevel logistic regression has been fitted on the data using a Dirichlet process prior for modeling the random-effects distribution of the unobserved factors at the hospital level. Using the estimated random effects at the hospital level, a partition of the hospitals in terms of similar medical practice has been obtained that identifies different profiles of hospitals in terms of caesarean section risks. The limited number of clusters may be useful for suggesting policy implications that help to reduce the heterogeneity of caesarean delivery risks.     

High wage workers and low wage firms: negative assortative matching or limited mobility bias?

Summary.  In the empirical literature on assortative matching using linked employer–employee data, unobserved worker quality appears to be negatively correlated with unobserved firm quality. We show that this can be caused by standard estimation error. We develop formulae that show that the estimated correlation is biased downwards if there is true positive assortative matching and when any conditioning covariates are uncorrelated with the firm and worker fixed effects. We show that this bias is bigger the fewer movers there are in the data, which is 'limited mobility bias'. This result applies to any two-way (or higher) error components model that is estimated by fixed effects methods. We apply these bias corrections to a large German linked employer–employee data set. We find that, although the biases can be considerable, they are not sufficiently large to remove the negative correlation entirely.     

On selection with restriction

In this paper, we .consider the problem of prediction of an unobserved variable and then selecting a group of individuals which are superior with respect to this variable. It is desired at the same time that for this group, mean values of other unobserved variables or some other function of some variable are above certain prespecified levels. We provide the methods of computing the decision rules for these problems. These problems may be termed as the problems of restricted prediction.     

Interpretation of dispersion effects in a robust design context

The purpose of this paper is to discuss the interpretation of dispersion effects in un-replicated fractional factorials from a robust design perspective. We propose an interpretation of dispersion effects as manifested interactions between control factors and unobserved and uncontrolled factors, an interpretation shown to be useful in achieving robust designs. Further, we show the consequences this interpretation has on the identification of dispersion effects.     

Frailty modeling for clustered survival data: an application to birth interval in Bangladesh

The present work demonstrates an application of random effects model for analyzing birth intervals that are clustered into geographical regions. Observations from the same cluster are assumed to be correlated because usually they share certain unobserved characteristics between them. Ignoring the correlations among the observations may lead to incorrect standard errors of the estimates of parameters of interest. Beside making the comparisons between Cox's proportional hazards model and random effects model for analyzing geographically clustered time-to-event data, important demographic and socioeconomic factors that may affect the length of birth intervals of Bangladeshi women are also reported in this paper.     

The complexity of school and neighbourhood effects and movements of pupils on school differences in models of educational achievement

Summary.  Traditional studies of school differences in educational achievement use multilevel modelling techniques to take into account the nesting of pupils within schools. However, educational data are known to have more complex non-hierarchical structures. The potential importance of such structures is apparent when considering the effect of pupil mobility during secondary schooling on educational achievement. Movements of pupils between schools suggest that we should model pupils as belonging to the series of schools that are attended and not just their final school. Since these school moves are strongly linked to residential moves, it is important to explore additionally whether achievement is also affected by the history of neighbourhoods that are lived in. Using the national pupil database, this paper combines multiple membership and cross-classified multilevel models to explore simultaneously the relationships between secondary school, primary school, neighbourhood and educational achievement. The results show a negative relationship between pupil mobility and achievement, the strength of which depends greatly on the nature and timing of these moves. Accounting for pupil mobility also reveals that schools and neighbourhoods are more important than shown by previous analysis. A strong primary school effect appears to last long after a child has left that phase of schooling. The additional effect of neighbourhoods, in contrast, is small. Crucially, the rank order of school effects across all types of pupil is sensitive to whether we account for the complexity of the multilevel data structure.     

Evaluation of multiple-imputation procedures for three-mode component models

Three-mode analysis is a generalization of principal component analysis to three-mode data. While two-mode data consist of cases that are measured on several variables, three-mode data consist of cases that are measured on several variables at several occasions. As any other statistical technique, the results of three-mode analysis may be influenced by missing data. Three-mode software packages generally use the expectation–maximization (EM) algorithm for dealing with missing data. However, there are situations in which the EM algorithm is expected to break down. Alternatively, multiple imputation may be used for dealing with missing data. In this study we investigated the influence of eight different multiple-imputation methods on the results of three-mode analysis, more specifically, a Tucker2 analysis, and compared the results with those of the EM algorithm. Results of the simulations show that multilevel imputation with the mode with the most levels nested within cases and the mode with the least levels represented as variables gives the best results for a Tucker2 analysis. Thus, this may be a good alternative for the EM algorithm in handling missing data in a Tucker2 analysis.     

Analysing state dependences in emotional experiences by dynamic count data models

Summary. The paper presents a multilevel framework for the analysis of multivariate count data that are observed over several time periods for a random sample of individuals. The approach proposed facilitates studying observed and unobserved sources of dependences among the event categories in the presence of possibly higher order autoregressive effects. In an investigation of the relationships between pleasant and unpleasant emotional experiences and the personality traits neuroticism and extraversion over time, we find that the two personality factors are related to both the mean rates of the emotional experiences and their carry-over effects. Respondents with high neuroticism scores not only reported more unpleasant than pleasant emotional experiences but also exhibited higher carry-over effects for unpleasant than for pleasant emotions. In contrast, respondents with high extraversion scores reported fewer anxiety and more euphoria emotions than respondents with low extraversion scores with weaker carry-over effects for both pleasant and unpleasant emotions.     

Modelling method effects as individual causal effects

Summary.  Method effects often occur when different methods are used for measuring the same construct. We present a new approach for modelling this kind of phenomenon, consisting of a definition of method effects and a first model, the method effect model , that can be used for data analysis. This model may be applied to multitrait–multimethod data or to longitudinal data where the same construct is measured with at least two methods at all occasions. In this new approach, the definition of the method effects is based on the theory of individual causal effects by Neyman and Rubin. Method effects are accordingly conceptualized as the individual effects of applying measurement method j instead of k . They are modelled as latent difference scores in structural equation models. A reference method needs to be chosen against which all other methods are compared. The model fit is invariant to the choice of the reference method. The model allows the estimation of the average of the individual method effects, their variance, their correlation with the traits (and other latent variables) and the correlation of different method effects among each other. Furthermore, since the definition of the method effects is in line with the theory of causality, the method effects may (under certain conditions) be interpreted as causal effects of the method. The method effect model is compared with traditional multitrait–multimethod models. An example illustrates the application of the model to longitudinal data analysing the effect of negatively (such as 'feel bad') as compared with positively formulated items (such as 'feel good') measuring mood states.     

Impact evaluation of job training programmes: selection bias in multilevel models

This paper focuses on the evaluation of a job training programme composed of several different courses. The aim is to evaluate the impact of the programme for the participants with respect to non-participants, paying attention to possible differences in the effectiveness between the courses. The analysis is based on discrete data with a hierarchical structure. Multilevel modelling is the natural choice in this setting, but the results may be severely affected by selection bias. We propose a two-step procedure, which suits both the hierarchical structure and the observational nature of data. The method selects the appropriate control group, using standard results of the propensity score methodology. A suitable multilevel model is formulated, and the dependence of the results on the amount of non-random sample selection is analysed within a likelihood-based framework. As a result, rankings for comparative performances are obtained, adjusted for the amount of plausible selection bias. The procedure is illustrated with reference to a data set about a job training programme organized in Italy in the late 1990s.     

Outliers in multilevel data

This paper offers the data analyst a range of practical procedures for dealing with outliers in multilevel data. It first develops several techniques for data exploration for outliers and outlier analysis and then applies these to the detailed analysis of outliers in two large scale multilevel data sets from educational contexts. The techniques include the use of deviance reduction, measures based on residuals, leverage values, hierarchical cluster analysis and a measure called DFITS. Outlier analysis is more complex in a multilevel data set than in, say, a univariate sample or a set of regression data, where the concept of an outlying value is straightforward. In the multilevel situation one has to consider, for example, at what level or levels a particular response is outlying, and in respect of which explanatory variables; furthermore, the treatment of a particular response at one level may affect its status or the status of other units at other levels in the model.     

Multilevel models for longitudinal variables prognostic for survival

The issue of modelling the joint distribution of survival time and of prognostic variables measured periodically has recently become of interest in the AIDS literature but is of relevance in other applications. The focus of this paper is on clinical trials where follow-up measurements of potentially prognostic variables are often collected but not routinely used. These measurements can be used to study the biological evolution of the disease of interest; in particular the effect of an active treatment can be examined by comparing the time profiles of patients in the active and placebo group. It is proposed to use multilevel regression analysis to model the individual repeated observations as function of time and, possibly, treatment. To address the problem of informative drop-out—which may arise if deaths (or any other censoring events) are related to the unobserved values of the prognostic variables—we analyse sequentially overlapping portions of the follow-up information. An example arising from a randomized clinical trial for the treatment of primary biliary cirrhosis is examined in detail.