Logistic Regression Models with Unspecified Low Dose–Response Relationships and Experimental Designs for Hormesis Studies

Hormesis refers to a nonmonotonic (biphasic) dose–response relationship in toxicology, environmental science, and related fields. In the presence of hormesis, a low dose of a toxic agent may have a lower risk than the risk at the control dose, and the risk may increase at high doses. When the sample size is small due to practical, logistic, and ethical considerations, a parametric model may provide an efficient approach to hypothesis testing at the cost of adopting a strong assumption, which is not guaranteed to be true. In this article, we first consider alternative parameterizations based on the traditional three‐parameter logistic regression. The new parameterizations attempt to provide robustness to model misspecification by allowing an unspecified dose–response relationship between the control dose and the first nonzero experimental dose. We then consider experimental designs including the uniform design (the same sample size per dose group) and the $c$‐optimal design (minimizing the standard error of an estimator for a parameter of interest). Our simulation studies showed that (1) the $c$‐optimal design under the traditional three‐parameter logistic regression does not help reducing an inflated Type I error rate due to model misspecification, (2) it is helpful under the new parameterization with three parameters (Type I error rate is close to a fixed significance level), and (3) the new parameterization with four parameters and the $c$‐optimal design does not reduce statistical power much while preserving the Type I error rate at a fixed significance level.