Modeling with Mixtures of Linear Regressions

Consider data (x ₁,y ₁),...,(x _n,y _n), where each x _i may be vector valued, and the distribution of y _i given x _i is a mixture of linear regressions. This provides a generalization of mixture models which do not include covariates in the mixture formulation. This mixture of linear regressions formulation has appeared in the computer science literature under the name Hierarchical Mixtures of Experts model.This model has been considered from both frequentist and Bayesian viewpoints. We focus on the Bayesian formulation. Previously, estimation of the mixture of linear regression model has been done through straightforward Gibbs sampling with latent variables. This paper contributes to this field in three major areas. First, we provide a theoretical underpinning to the Bayesian implementation by demonstrating consistency of the posterior distribution. This demonstration is done by extending results in Barron, Schervish and Wasserman (Annals of Statistics 27: 536–561, 1999) on bracketing entropy to the regression setting. Second, we demonstrate through examples that straightforward Gibbs sampling may fail to effectively explore the posterior distribution and provide alternative algorithms that are more accurate. Third, we demonstrate the usefulness of the mixture of linear regressions framework in Bayesian robust regression. The methods described in the paper are applied to two examples.     

Construction of uniform designs for mixture experiments with complex constraints

Abstract

Mixture experiments have attracted increasingly attention due to their great practical value in production and living, while uniform designs over irregular experimental regions have become a hot topic in the area of experimental designs in the past two decades. Noting that the experimental region of a mixture experiment with q components under some constraints is in fact a (q ? 1)-dimensional geometry, this article proposes a new method for searching nearly uniform designs for mixture experiments with any complex constraints. Two examples with some tables and figures are given to illustrate this method.     

Admissibility intervals for linear correlation coefficients

The correlation coefficient is widely used to quantify the degree of association between two quantitative variables. By resorting to the geometric representation of the linear correlation coefficient, it is possible to calculate the upper and lower bounds of the correlation coefficient between two variables x ₁,x ₂ when the correlation coefficients with a third variable x ₃ are available. Implications in observational studies, where x ₃ could be a proxy of a target variable x ₂, whose direct measurement is too expensive or impractical, are discussed.     

Alternative modeling techniques for the quantal response data in mixture experiments

Mixture experiments are commonly encountered in many fields including chemical, pharmaceutical and consumer product industries. Due to their wide applications, mixture experiments, a special study of response surface methodology, have been given greater attention in both model building and determination of designs compared with other experimental studies. In this paper, some new approaches are suggested on model building and selection for the analysis of the data in mixture experiments by using a special generalized linear models, logistic regression model, proposed by Chen et al. [7]. Generally, the special mixture models, which do not have a constant term, are highly affected by collinearity in modeling the mixture experiments. For this reason, in order to alleviate the undesired effects of collinearity in the analysis of mixture experiments with logistic regression, a new mixture model is defined with an alternative ratio variable. The deviance analysis table is given for standard mixture polynomial models defined by transformations and special mixture models used as linear predictors. The effects of components on the response in the restricted experimental region are given by using an alternative representation of Cox's direction approach. In addition, odds ratio and the confidence intervals of odds ratio are identified according to the chosen reference and control groups. To compare the suggested models, some model selection criteria, graphical odds ratio and the confidence intervals of the odds ratio are used. The advantage of the suggested approaches is illustrated on tumor incidence data set.     

The Design And Analysis of a Mixture Experiment on Glass Durability

This paper discusses the statistical design and analysis of a non–standard constrained mixture experiment on glass durability (the components sum to 1, and there are non–trivial lower or upper bounds on components or combinations of them). It notes some of the difficulties which arose during the study, and shows how the aims and methods of the experiment changed during its execution.     

A note comparing component-slope,Scheffé and Cox parameterizations of the linear mixture experiment model

A mixture experiment involves combining two or more components in various proportions and collecting data on one or more responses. A linear mixture model may adequately represent the relationship between a response and mixture component proportions and be useful in screening the mixture components. The Scheffé and Cox parameterizations of the linear mixture model are commonly used for analyzing mixture experiment data. With the Scheffé parameterization, the fitted coefficient for a component is the predicted response at that pure component (i.e. single-component mixture). With the Cox parameterization, the fitted coefficient for a mixture component is the predicted difference in response at that pure component and at a pre-specified reference composition. This article presents a new component-slope parameterization, in which the fitted coefficient for a mixture component is the predicted slope of the linear response surface along the direction determined by that pure component and at a pre-specified reference composition. The component-slope, Scheffé, and Cox parameterizations of the linear mixture model are compared and their advantages and disadvantages are discussed.     

Estimating the proportions of two populations in a mixture using linear maps

We address the problem of estimating the proportions of two statistical populations in a given mixture on the basis of an unlabeled sample of n–dimensional observations on the mixture. Assuming that the expected values of observations on the two populations are known, we show that almost any linear map from Rⁿ to R¹ yields an unbiased consistent estimate of the proportion of one population in a very easy way. We then find that linear map for which the resulting proportion estimate has minimum variance among all estimates so obtained. After deriving a simple expression for the minimum-variance estimate, we discuss practical aspects of obtaining this and related estimates.     

Perfect octagon quadrangle systems with upper C4-systems

An octagon quadrangle is the graph consisting of an 8-cycle (x₁, x₂,…, x₈) with two additional chords: the edges {x₁, x₄} and {x₅, x₈}. An octagon quadrangle system of order v and index ρ [OQS] is a pair (X,H), where X is a finite set of v vertices and H is a collection of edge disjoint octagon quadrangles (called blocks) which partition the edge set of ρK_v defined on X. An octagon quadrangle systemΣ=(X,H) of order v and index λ is said to be upper C₄-perfect if the collection of all of the upper4-cycles contained in the octagon quadrangles form a μ-fold 4-cycle system of order v; it is said to be upper strongly perfect, if the collection of all of the upper4-cycles contained in the octagon quadrangles form a μ-fold 4-cycle system of order v and also the collection of all of the outside8-cycles contained in the octagon quadrangles form a ?-fold 8-cycle system of order v. In this paper, the authors determine the spectrum for these systems.     

THE q-BETA-GEOMETRIC DISTRIBUTION AS A MODEL FOR FECUNDABILITY

The number of sterile couples in a retrospective study of the number of cycles to conception is necessarily zero; this is not so for a prospective study. The paper puts forward a modification of Weinberg and Gladen's beta geometric model for cycles to conception that is suitable for both types of investigation. The probability that a couple achieves conception at the xth cycle, but not earlier, is assumed to take the form R_x = (1 ? ρ)/(1 ? m ^x?1 ρ/u), instead of μ/(1 ? θ + θx). The set of parameter restraints (0 < m < 1, 0< ρ < 1, 1 < u) is appropriate for retrospective data, whilst the alternative set of restraints (1 < m, 1 < ρ, 0 < u < 1) is appropriate for prospective data. The decrease in R_x over time can be interpreted not only as a time effect, but also as a heterogeneity effect by replacing Weinberg and Gladen's beta mixture of geometric distributions by a q-beta mixture.     

Semiparametric location mixtures with distinct components

We consider a two-component location mixture model with symmetric components, one of which is assumed to be known, the other is unknown. We show identifiability under assumptions on the tails of the characteristic function for the true underlying mixture, and also construct asymptotically normal estimates. The model is an extension of the contamination model in Bordes et al. [Semiparametric estimation of a two-component mixture model when a component is known, Scand. J. Statist. 33 (2006), pp. 733–752], and also related to a location mixture of one symmetric density as in Bordes et al. [Semiparametric estimation of a two component mixture model, Ann. Statist. 34 (2006), pp. 1204–1232]. We show by simulation that estimating the additional location parameter leads to a slight loss of efficiency as compared with the contamination model.     

On the construction of hypercubic design

Hypercubic design was introduced by Shah (1958) and Kusumoto(1965) in which the t=v^m treatments are represented by n-plets (x₁, x₂, ..., x_m),where 1≤ x₁, x₂, ..., x_m ≤v, and two treatments are said to be i-th associates if they differ in exactly i components. This paper deals with the construction of some hypercubic designs and gives their application to confounding plans for symmetrical factorial experiments. The proposed methods prove to be quite flexible in terms of choice of possible block sizes and are easy to use.     

D-Optimal Axial Designs for Quadratic and Cubic Additive Mixture Models

The paper discusses D-optimal axial designs for the additive quadratic and cubic mixture models σ_1≤i≤q(β_ix_i + β_iix²_i) and σ_1≤i≤q(β_ix_i + β_iix²_i + β_iiix³_i), where x_i≥ 0, x₁ + . . . + x_q = 1. For the quadratic model, a saturated symmetric axial design is used, in which support points are of the form (x₁, . . . , x_q) = [1 ? (q?1)δ_i, δ_i, . . . , δ_i], where i = 1, 2 and 0 ≤δ₂ <_{δ1 ≤} 1/(q ?1). It is proved that when 3 ≤q≤ 6, the above design is D-optimal if δ₂ = 0 and δ₁ = 1/(q?1), and when q≥ 7 it is D-optimal if δ₂ = 0 and δ₁ = [5q?1 ? (9q²?10q + 1)^1/2]/(4q²). Similar results exist for the cubic model, with support points of the form (x₁, . . . , x_q) = [1 ? (q?1)δ_i, δ_i, . . . , δ_i], where i = 1, 2, 3 and 0 = δ₃ <_δ2 < δ_{1 ≤}1/(q?1). The saturated D-optimal axial design and D-optimal design for the quadratic model are compared in terms of their efficiency and uniformity.     

Statistical Inference and Applications of Mixture of Varying Coefficient Models

In this paper, we consider a new mixture of varying coefficient models, in which each mixture component follows a varying coefficient model and the mixing proportions and dispersion parameters are also allowed to be unknown smooth functions. We systematically study the identifiability, estimation and inference for the new mixture model. The proposed new mixture model is rather general, encompassing many mixture models as its special cases such as mixtures of linear regression models, mixtures of generalized linear models, mixtures of partially linear models and mixtures of generalized additive models, some of which are new mixture models by themselves and have not been investigated before. The new mixture of varying coefficient model is shown to be identifiable under mild conditions. We develop a local likelihood procedure and a modified expectation–maximization algorithm for the estimation of the unknown non‐parametric functions. Asymptotic normality is established for the proposed estimator. A generalized likelihood ratio test is further developed for testing whether some of the unknown functions are constants. We derive the asymptotic distribution of the proposed generalized likelihood ratio test statistics and prove that the Wilks phenomenon holds. The proposed methodology is illustrated by Monte Carlo simulations and an analysis of a CO₂‐GDP data set.     

Random search algorithm for optimal mixture experimental design

It is well known that it is difficult to obtain an accurate optimal design for a mixture experimental design with complex constraints. In this article, we construct a random search algorithm which can be used to find the optimal design for mixture model with complex constraints. First, we generate an initial set by the Monte-Carlo method, and then run the random search algorithm to get the optimal set of points. After that, we explain the effectiveness of this method by using two examples.     

Mixture designs for constrained components in orthogonal blocks

It is often the case in mixture experiments that some of the ingredients, such as additives or flavourings, are included with proportions constrained to lie in a restricted interval, while the majority of the mixture is made up of a particular ingredient used as a filler. The experimental region in such cases is restricted to a parallelepiped in or near one corner of the full simplex region. In this paper, orthogonally blocked designs with two experimental blends on each edge of the constrained region are considered for mixture experiments with three and four ingredients. The optimal symmetric orthogonally blocked designs within this class are determined and it is shown that even better designs are obtained for the asymmetric situation, in which some experimental blends are taken at the vertices of the experimental region. Some examples are given to show how these ideas may be extended to identify good designs in three and four blocks. Finally, an example is included to illustrate how to overcome the problems of collinearity that sometimes occur when fitting quadratic models to experimental data from mixture experiments in which some of the ingredient proportions are restricted to small values.     

Symmetry in mixture experiments

Two types of symmetry can arise when the proportions of mixture components are constrained by upper and lower bounds. These two types of symmetry are shown to be useful for blocking first-order designs, as well as for finding the centroid of the experimental region. Orthogonal blocking of first-order mixture designs provides a method of including process variables in the mixture experiment, with the mixture terms orthogonal to the process factors. Symmetric regions are used to develop spherical and rotatable response surface designs for mixtures. The central composite design and designs based on the icosahedron and the dodecahedron are given for four-component mixtures. The uniform shell designs are three-level designs when applied to mixture experiments.     

A graphical evaluation of logistic ridge estimator in mixture experiments

In comparison to other experimental studies, multicollinearity appears frequently in mixture experiments, a special study area of response surface methodology, due to the constraints on the components composing the mixture. In the analysis of mixture experiments by using a special generalized linear model, logistic regression model, multicollinearity causes precision problems in the maximum-likelihood logistic regression estimate. Therefore, effects due to multicollinearity can be reduced to a certain extent by using alternative approaches. One of these approaches is to use biased estimators for the estimation of the coefficients. In this paper, we suggest the use of logistic ridge regression (RR) estimator in the cases where there is multicollinearity during the analysis of mixture experiments using logistic regression. Also, for the selection of the biasing parameter, we use fraction of design space plots for evaluating the effect of the logistic RR estimator with respect to the scaled mean squared error of prediction. The suggested graphical approaches are illustrated on the tumor incidence data set.     

Optimal designs for experiments with mixtures: a survey

This is a survey article on known results about analytic solutions and numerical solutions of optimal designs for various regression models for experiments with mixtures. The regression models include polynomial models, models containing homogeneous functions, models containing inverse terms and ratios, log contrast models, models with quantitative variables, and mod els containing the amount of mixture, Optimality criteria considered include D-, A-, E-,φ_p- and I_λ-Optimalities. Uniform design and uniform optimal design for mixture components, and efficiencies of the {q,2} simplex-controid design are briefly discussed.     

Learning a multivariate Gaussian mixture model with the reversible jump MCMC algorithm

This paper is a contribution to the methodology of fully Bayesian inference in a multivariate Gaussian mixture model using the reversible jump Markov chain Monte Carlo algorithm. To follow the constraints of preserving the first two moments before and after the split or combine moves, we concentrate on a simplified multivariate Gaussian mixture model, in which the covariance matrices of all components share a common eigenvector matrix. We then propose an approach to the construction of the reversible jump Markov chain Monte Carlo algorithm for this model. Experimental results on several data sets demonstrate the efficacy of our algorithm.     

Robustness of the Multiple Correlation Coefficient When Sampling From a Mixture of Two Multivariate Normal Populations

The density of the multiple correlation coefficient is derived by direct integration when the sample covariance matrix has a linear non-central distribution. Using the density, we deduce the null and non-null distribution of the multiple correlation coefficient when sampling from a mixture of two multivariate normal populations with the same covariance matrix. We also compute actual significance levels of the test of the hypothesis H_o : ρ_1·2…p = 0 versus H_a:ρ_1·2…p > 0, given the mixture model.