A new search procedure of steepest ascent in response surface exploration

The method of steepest ascent direction has been widely accepted for process optimization in the applications of response surface methodology (RSM). The procedure of steepest ascent direction is performed on experiments run along the gradient of a fitted linear model. Therefore, the RSM practitioner needs to decide a suitable stopping rule such that the optimum point estimate in the search direction can be determined. However, the details of how to deflect and then halt a search in the steepest ascent direction are not thoroughly described in the literature. In common practice, it is convenient to use the simple stopping rules after one to three response deteriorations in a row after a series of fitted linear models used for exploration. In the literature, there are two formal stopping rules proposed, that is, Myers and Khuri's [A new procedure for steepest ascent, Comm. Statist. Theory Methods A 8(14) (1979), pp. 1359–1376] stopping rule and del Castillo's [Stopping rules for steepest ascent in experimental optimization, Comm. Statist. Simul. Comput. 26(4) (1997), pp. 1599–1615] stopping rule. This paper develops a new procedure for determining how to adjust and then when to stop a steepest ascent search in response surface exploration. This proposal wishes to provide the RSM practitioner with a clear-cut and easy-to-implement procedure that can attain the optimum mean response more accurately than the existing procedures. Through the study of simulation optimization, it shows that the average optimum point and response returned by using the new search procedure are considerably improved when compared with two existing stopping rules. The number of experimental trials required for convergence is greatly reduced as well.     

A new procedure for steepest ascent

The procedure of steepest ascent consists of performing a sequence of sets of trials. Each set of trials is obtained as a result of proceeding sequentially along the path of maximum increase in response. Until now there has been no formal stopping rule, When response values are subject to random error, the decision to stop can be premature due to a “false” drop in the observed response.

A new stopping rule procedure for steepest ascent is intro-duced that takes into account the random error variation in response values. The new procedure protects against taking too many observations when the true mean response is decreasing, it also protects against stopping. prematurely when the true mean response is increasing, A numerical example is given which illus-trates the method.     

On Response Surface Methodology and its Computer-Assisted Teaching

Since the teaching of response surface methodology involving the steepest ascent (descent) method requires a fair amount of instructor and student time even to complete one analysis, the routine aspects of the method were computerized. Flowcharts that contain the logic of first- and second-order experimentation to reach optimum conditions were also developed.     

A note on the optimal number of centre runs in a second phase design of response surface methods

In searching for optimum conditions, the response surface methods comprise two phases. In the first phase, the method of the steepest ascent with a 2 k-p design is used in searching for a region of improved response. The curvature of the response surface is checked in the second phase. For testing the evidence of curvature, a reasonable design is a 2 k-p fractional factorial design augmented by centre runs. Using c-optimality criterion, the optimal number of centre runs is investigated. Incorporating c-efficiencies for the curvature test with D-efficiencies and G-efficiencies of CCDs for the quadratic response surfaces and then, adopting the Mini-Max principle, i.e. maximizing the worst efficiency, we propose robust centre runs with respect to the three optimality criteria to be chosen.     

Extensions to the modified Gram–Schmidt strategy and its application in the steepest ascent method

The orthogonalization of undesigned experiments is introduced to increase statistical precision of the estimated regression coefficients. The goals are to minimize the covariance and the bias of the least squares estimator for estimating the path of the steepest ascent (SA) that leads the users toward the neighbour of the optimum response. An orthogonal design is established to decrease the inverse determinant of X′X and the angle between the true and the estimated SA paths. For orthogonalization of an undesigned matrix, our proposed solution is constructed on the modified Gram–Schmidt strategy relevant to the process of Gaussian elimination. The proposed solution offers an orthogonal basis, in full working accuracy, for the space spanned by the columns of the original matrix.     

Generalized Linear Factor Models: A New Local EM Estimation Algorithm

In this article, a general approach to latent variable models based on an underlying generalized linear model (GLM) with factor analysis observation process is introduced. We call these models Generalized Linear Factor Models (GLFM). The observations are produced from a general model framework that involves observed and latent variables that are assumed to be distributed in the exponential family. More specifically, we concentrate on situations where the observed variables are both discretely measured (e.g., binomial, Poisson) and continuously distributed (e.g., gamma). The common latent factors are assumed to be independent with a standard multivariate normal distribution. Practical details of training such models with a new local expectation-maximization (EM) algorithm, which can be considered as a generalized EM-type algorithm, are also discussed. In conjunction with an approximated version of the Fisher score algorithm (FSA), we show how to calculate maximum likelihood estimates of the model parameters, and to yield inferences about the unobservable path of the common factors. The methodology is illustrated by an extensive Monte Carlo simulation study and the results show promising performance.     

Genetic Algorithm in the Wavelet Domain for Large p Small n Regression

Many areas of statistical modeling are plagued by the “curse of dimensionality,” in which there are more variables than observations. This is especially true when developing functional regression models where the independent dataset is some type of spectral decomposition, such as data from near-infrared spectroscopy. While we could develop a very complex model by simply taking enough samples (such that n > p), this could prove impossible or prohibitively expensive. In addition, a regression model developed like this could turn out to be highly inefficient, as spectral data usually exhibit high multicollinearity. In this article, we propose a two-part algorithm for selecting an effective and efficient functional regression model. Our algorithm begins by evaluating a subset of discrete wavelet transformations, allowing for variation in both wavelet and filter number. Next, we perform an intermediate processing step to remove variables with low correlation to the response data. Finally, we use the genetic algorithm to perform a stochastic search through the subset regression model space, driven by an information-theoretic objective function. We allow our algorithm to develop the regression model for each response variable independently, so as to optimally model each variable. We demonstrate our method on the familiar biscuit dough dataset, which has been used in a similar context by several researchers. Our results demonstrate both the flexibility and the power of our algorithm. For each response variable, a different subset model is selected, and different wavelet transformations are used. The models developed by our algorithm show an improvement, as measured by lower mean error, over results in the published literature.     

Forecasting multivariate longitudinal binary data with marginal and marginally specified models

Forecasting with longitudinal data has been rarely studied. Most of the available studies are for continuous response and all of them are for univariate response. In this study, we consider forecasting multivariate longitudinal binary data. Five different models including simple ones, univariate and multivariate marginal models, and complex ones, marginally specified models, are studied to forecast such data. Model forecasting abilities are illustrated via a real-life data set and a simulation study. The simulation study includes a model independent data generation to provide a fair environment for model competitions. Independent variables are forecast as well as the dependent ones to mimic the real-life cases best. Several accuracy measures are considered to compare model forecasting abilities. Results show that complex models yield better forecasts.     

A Comparison of Three-Level Orthogonal Arrays in the Presence of Different Correlation Structures in Observations

When the experimenter suspects that there might be a quadratic relation between the response variable and the explanatory parameters, a design with at least three points must be employed to establish and explore this relation (second-order design). Orthogonal arrays (OAs) with three levels are often used as second-order response surface designs. Generally, we assume that the data are independent observations; however, there are many situations where this assumption may not be sustainable. In this paper, we want to compare three-level OAs with 18, 27, and 36 runs under the presence of three specific forms of correlation in observations. The aim is to derive the best designs that can be efficiently used for response surface modeling.     

Analyzing Designed Experiments with Multiple Responses

This paper is an overview of a unified framework for analyzing designed experiments with univariate or multivariate responses. Both categorical and continuous design variables are considered. To handle unbalanced data, we introduce the so-called Type II* sums of squares. This means that the results are independent of the scale chosen for continuous design variables. Furthermore, it does not matter whether two-level variables are coded as categorical or continuous. Overall testing of all responses is done by 50-50 MANOVA, which handles several highly correlated responses. Univariate p-values for each response are adjusted by using rotation testing. To illustrate multivariate effects, mean values and mean predictions are illustrated in a principal component score plot or directly as curves. For the unbalanced cases, we introduce a new variant of adjusted means, which are independent to the coding of two-level variables. The methodology is exemplified by case studies from cheese and fish pudding production.     

Path analysis and determining the distribution of indirect effects via simulation

The difference between a path analysis and the other multivariate analyses is that the path analysis has the ability to compute the indirect effects apart from the direct effects. The aim of this study is to investigate the distribution of indirect effects that is one of the components of path analysis via generated data. To realize this, a simulation study has been conducted with four different sample sizes, three different numbers of explanatory variables and with three different correlation matrices. A replication of 1000 has been applied for every single combination. According to the results obtained, it is found that irrespective of the sample size path coefficients tend to be stable. Moreover, path coefficients are not affected by correlation types either. Since the replication number is 1000, which is fairly large, the indirect effects from the path models have been treated as normal and their confidence intervals have been presented as well. It is also found that the path analysis should not be used with three explanatory variables. We think that this study would help scientists who are working in both natural and social sciences to determine sample size and different number of variables in the path analysis.     

Projection designs for mixture experiments in orthogonal blocks

Mixture experiments are often carried out in the presence of process variables, such as days of the week or different machines in a manufacturing process, or different ovens in bread and cake making. In such experiments it is particularly useful to be able to arrange the design in orthogonal blocks, so that the model in tue mixture vanauies may ue iitteu inucpenuentiy or tne UIOCK enects mtrouuceu to take account of the changes in the process variables. It is possible in some situations that some of the ingredients in the mixture, such as additives or flavourings, are present in soian quantities, pernaps as iuw a.s 5% ur even !%, resulting in the design space being restricted to only part of the mixture simplex. Hau and Box (1990) discussed the construction of experimental designs for situations where constraints are placed on the design variables. They considered projecting standard response surface designs, including factorial designs and central composite designs, into the restricted design space, and showed that the desirable property of block orthogonality is preserved by the projections considered. Here we present a number of examples of projection designs and illustrate their use when some of the ingredients are restricted to small values, such that the design space is restricted to a sub-region within the usual simplex in the mixture variables.     

A monte carlo study of robust and least squares response surface methods

Response surface methodology is useful for exploring a response over a region of factor space and in searching for extrema. Its generality, makes it applicable to a variety of areas. Classical response surface methodology for a continuous response variable is generally based on least squares fitting. The sensitivity of least squares to outlying observations carries over to the surface procedures. To overcome this sensitivity, we propose response surface methodology based on robust procedures for continuous response variables. This robust methodology is analogous to the methodology based on least squares, while being much less sensitive to outlying observations. The results of a Monte Carlo study comparing it and classical surface methodologies for normal and contaminated normal errors are presented. The results show that as the proportion of contamination increases, the robust methodology correctly identifies a higher proportion of extrema than the least squares methods and that the robust estimates of extrema tend to be closer to the true extrema than the least squares methods.     

Moments of generalized quadratic forms and distributions of certain multivariate test statistics

Summary Moments and distributions of quadratic forms or quadratic expressions in normal variables are available in literature. Such quadratic expressions are shown to be equivalent to a linear function of independent central or noncentral chi-square variables. Some results on linear functions of generalized quadratic forms are also available in literature. Here we consider an arbitrary linear function of matrix-variate gamma variables. Moments of the determinant of such a linear function are evaluated when the matrix-variate gammas are independently distributed. By using these results, arbitrary non-null moments as well as the non-null distribution of the likelihood ratio criterion for testing the hypothesis of equality of covariance matrices in independent multivariate normal populations are derived. As a related result, the distribution of a linear function of independent matrix-variate gamma random variables, which includes linear functions of independent Wishart matrices, is also obtained. Some properties of generalized special functions of several matrix arguments are used in deriving these results.     

A weighted linear quantile regression

In this article, we introduce a new weighted quantile regression method. Traditionally, the estimation of the parameters involved in quantile regression is obtained by minimizing a loss function based on absolute distances with weights independent of explanatory variables. Specifically, we study a new estimation method using a weighted loss function with the weights associated with explanatory variables so that the performance of the resulting estimation can be improved. In full generality, we derive the asymptotic distribution of the weighted quantile regression estimators for any uniformly bounded positive weight function independent of the response. Two practical weighting schemes are proposed, each for a certain type of data. Monte Carlo simulations are carried out for comparing our proposed methods with the classical approaches. We also demonstrate the proposed methods using two real-life data sets from the literature. Both our simulation study and the results from these examples show that our proposed method outperforms the classical approaches when the relative efficiency is measured by the mean-squared errors of the estimators.     

Testing the equality of two independent regression models

In some situations, for example in agriculture, biology, hydrology, and psychology, researchers wish to determine whether the relationship between response variable and predictor variables differs in two populations. In other words, we are interested in comparing two regression models for two independent datasets. In this work, we will use the parametric and nonparametric methods to establish hypothesis testing for the equality of two independent regression models. Then the simulation study is provided to investigate the performance of the proposed method.     

On Multi-Treatment Adaptive Allocation Design for Dichotomous Response

The use of ridit, as a probability score, is a very common practice to compare discrete random variables in discrete data analysis. In the present work we formulate ridit reliability functionals for some comparison of K independent binary random variables. We use such functionals to provide a generalized response-adaptive design (GRAD) on K(≥ +2) treatment-arms for dichotomous response variables. We exhibit some properties of the proposed design and compare it with some of the existing competitors by computing its various performance measures. We also provide a discussion towards a possible modification of the GRAD in the presence of covariates.     

The Representation of Hypergeometric Random Variables using Independent Bernoulli Random Variables

In this paper, we show that a hypergeometric random variable can be represented as a sum of independent Bernoulli random variables that are, except in degenerate cases, not identically distributed. In the proof, we use the factorial moment generating function. An asymptotic result on the probabilities of the Bernoulli random variables in the sum is also presented. Numerical examples are used to illustrate the results.     

Ridge Analysis 25 Years Later

The response surface technique called ridge analysis was originally introduced by Hoerl (1959) more than 25 years ago. Despite tremendous advantages over more conventional response surface procedures when more than two independent variables are present, ridge analysis has received little attention in the statistical literature since then, although numerous applications have appeared in engineering journals. This situation may be partially due to the fact that this procedure led to the discovery of ridge regression, which has completely overshadowed ridge analysis in the literature since. This discussion will briefly review the mathematics of ridge analysis, its literature, practical advantages, and relationship to ridge regression.     

Nonstationary regression models with a lagged dependent variable

This paper studies regression models with a lagged dependent variable when both the dependent and independent variables are nonstationary, and the regression model is misspecified in some dimension. In particular, we discuss the limiting properties of leastsquares estimates of the parameters in such regression models, and the limiting distributions of their test statistics. We show that the estimate of the lagged dependent variable tends to unity asymptotically independent of its true value, while the estimates of the independent variables tend to zero. The limiting distributions of their test statistics are shown to diverge with sample size.