Modelling spatial intensity for replicated inhomogeneous point patterns in brain imaging

Summary.  Pharmacological experiments in brain microscopy study patterns of cellular activ- ation in response to psychotropic drugs for connected neuroanatomic regions. A typical ex- perimental design produces replicated point patterns having highly complex spatial variability. Modelling this variability hierarchically can enhance the inference for comparing treatments. We propose a semiparametric formulation that combines the robustness of a nonparametric kernel method with the efficiency of likelihood-based parameter estimation. In the convenient framework of a generalized linear mixed model, we decompose pattern variation by kriging the intensities of a hierarchically heterogeneous spatial point process. This approximation entails discretizing the inhomogeneous Poisson likelihood by Voronoi tiling of augmented point patterns. The resulting intensity-weighted log-linear model accommodates spatial smoothing through a reduced rank penalized linear spline. To correct for anatomic distortion between subjects, we interpolate point locations via an isomorphic mapping so that smoothing occurs relative to common neuroanatomical atlas co-ordinates. We propose a criterion for choosing the degree and spatial locale of smoothing based on truncating the ordered set of smoothing covariates to minimize residual extra-dispersion. Additional spatial covariates, experimental design factors, hierarchical random effects and intensity functions are readily accommodated in the linear predictor, enabling comprehensive analyses of the salient properties underlying replicated point patterns. We illustrate our method through application to data from a novel study of drug effects on neuronal activation patterns in the brain of rats.