Estimating Variance Components and Random Effects Using the Box-Cox Transformation in the Linear Mixed Model

The linear mixed model assumes normality of its two sources of randomness: the random effects and the residual error. Recent research demonstrated that a simple transformation of the response targets normality of both sources simultaneously. However, estimating the transformation can lead to biased estimates of the variance components. Here, we provide guidance regarding this potential bias and propose a correction for it when such bias is substantial. This correction allows for accurate estimation of the random effects when using a transformation to achieve normality. The utility of this approach is demonstrated in a study of sleep-wake behavior in preterm infants.