Generalized monotonic functional mixed models with application to modelling normal tissue complications

Summary.  Normal tissue complications are a common side effect of radiation therapy. They are the consequence of the dose of radiation that is received by the normal tissue surrounding the site of the tumour. Within a specified organ each voxel receives a certain dose of radiation, leading to a distribution of doses over the organ. It is often not known what aspect of the dose distribution drives the presence and severity of the complications. A summary measure of the dose distribution can be obtained by integrating a weighting function of dose ( w ( d )) over the density of dose. For biological reasons the weight function should be monotonic. We propose a generalized monotonic functional mixed model to study the dose effect on a clinical outcome by estimating this weight function non-parametrically by using splines and subject to the monotonicity constraint, while allowing for overdispersion and correlation of multiple obervations within the same subject. We illustrate our method with data from a head and neck cancer study in which the irradiation of the parotid gland results in loss of saliva flow.