Bayesian Network Structure Learning with Permutation Tests

In literature there are several studies on the performance of Bayesian network structure learning algorithms. The focus of these studies is almost always the heuristics the learning algorithms are based on, i.e., the maximization algorithms (in score-based algorithms) or the techniques for learning the dependencies of each variable (in constraint-based algorithms). In this article, we investigate how the use of permutation tests instead of parametric ones affects the performance of Bayesian network structure learning from discrete data. Shrinkage tests are also covered to provide a broad overview of the techniques developed in current literature.     

Multivariate Approach for Comparative Evaluations of Customer Satisfaction with Application to Transport Services

In problems related to evaluations of products or services (e.g. in customer satisfaction analysis) the main difficulties concern the synthesis of the information, which is necessary for the presence of several evaluators and many response variables (aspects under evaluation). In this article, the problem of determining and comparing the satisfaction of different groups of customers, in the presence of multivariate response variables and using the results of pairwise comparisons is addressed. Within the framework of group ranking methods and multi criteria decision making theory, a new approach, based on nonparametric techniques, for evaluating group satisfaction in a multivariate framework is proposed and the concept of Multivariate Relative Satisfaction is defined. An application to the evaluation of public transport services, like the railway service and the urban bus service, by students of the University of Ferrara (Italy) is also discussed.     

A permutation alternative and other test procedures for spatially correlated data in one-way anova

Research in many disciplines involves data with spatially correlated observations. Spatial dependence violates the independent errors assumption required for techniques such as the standard one-way analysis of variance for a completely randomized design. The testing methodology within the correlated errors approach has not been investigated within a spatial context. For one-way fixed effects analysis of variance, a permutation test and tests associated with the correlated errors approach are investigated through simulation. No single test was superior with respect to both power and size but the standard Wald F test and a simple adjustment to it performed well overall.     

Permutation tests of scale using deviances

Robust tests for comparing scale parameters, based on deviances—absolute deviations from the median—are examined. Higgins (2004) proposed a permutation test for comparing two treatments based on the ratio of deviances, but the performance of this procedure has not been investigated. A simulation study examines the performance of Higgins’ test relative to other tests of scale utilizing deviances that have been shown in the literature to have good properties. An extension of Higgins’ procedure to three or more treatments is proposed, and a second simulation study compares its performance to other omnibus tests for comparing scale. While no procedure emerged as a preferred choice in every scenario, Higgins’ tests are found to perform well overall with respect to Type I error rate and power.     

A comparison of recent nonparametric methods for testing effects in two-by-two factorial designs

The two-way two-levels crossed factorial design is a commonly used design by practitioners at the exploratory phase of industrial experiments. The F-test in the usual linear model for analysis of variance (ANOVA) is a key instrument to assess the impact of each factor and of their interactions on the response variable. However, if assumptions such as normal distribution and homoscedasticity of errors are violated, the conventional wisdom is to resort to nonparametric tests. Nonparametric methods, rank-based as well as permutation, have been a subject of recent investigations to make them effective in testing the hypotheses of interest and to improve their performance in small sample situations. In this study, we assess the performances of some nonparametric methods and, more importantly, we compare their powers. Specifically, we examine three permutation methods (Constrained Synchronized Permutations, Unconstrained Synchronized Permutations and Wald-Type Permutation Test), a rank-based method (Aligned Rank Transform) and a parametric method (ANOVA-Type Test). In the simulations, we generate datasets with different configurations of distribution of errors, variance, factor's effect and number of replicates. The objective is to elicit practical advice and guides to practitioners regarding the sensitivity of the tests in the various configurations, the conditions under which some tests cannot be used, the tradeoff between power and type I error, and the bias of the power on one main factor analysis due to the presence of effect of the other factor. A dataset from an industrial engineering experiment for thermoformed packaging production is used to illustrate the application of the various methods of analysis, taking into account the power of the test suggested by the objective of the experiment.     

A permutation approach for testing heterogeneity in two-sample categorical variables

In many sciences researchers often meet the problem of establishing if the distribution of a categorical variable is more concentrated, or less heterogeneous, in population P ₁ than in population P ₂. An approximate nonparametric solution to this problem is discussed within the permutation context. Such a solution has similarities to that of testing for stochastic dominance, that is, of testing under order restrictions, for ordered categorical variables. Main properties of given solution and a Monte Carlo simulation in order to evaluate its degree of approximation and its power behaviour are examined. Two application examples are also discussed.     

Saddlepoint p-values for group of linear rank tests

Many nonparametric tests in one sample problem, matched pairs, and competingrisks under censoring have the same underlying permutation distribution. This article proposes a saddlepoint approximation to the exact p-values of these tests instead of the asymptotic approximations. The performance of the saddlepoint approximation is assessed by using simulation studies that show the superiority of the saddlepoint methods over the asymptotic approximations in several settings. The use of the saddlepoint to approximate the p-values of class of two sample tests under complete randomized design is also discussed.     

On the use of the scholz-sievers method for dlstribution-free inference about the difference in slopes of two regression lines

A simple adaptation of a distribution-free method due to Scholz (1978) and Sievers (1978) for inference in a single regression setting is proposed for inference about the difference in slopes of two regression lines. We assume that the data are obtained from a designed experiment with common regression constants. A comparison of the proposed method to its competitors-one due to Hollander and the other due to Rao and Gore-indicates superiority of the new method.     

Optimal estimation of a finite population mean under generalized random permutation models

Employing certain generalized random permutation models and a general class of linear estimators of a finite population mean, it is shown that many of the conventional estimators are “optimal” in the sense of minimum average mean square error. Simple proofs are provided by using a well-known theorem on UMV estimation. The results also cover certain simple response error situations.     

Comparison of permutation methods for the partial correlation and partial mantel tests

This study compares empirical type I error and power of different permutation techniques that can be used for partial correlation analysis involving three data vectors and for partial Mantel tests. The partial Mantel test is a form of first-order partial correlation analysis involving three distance matrices which is widely used in such fields as population genetics, ecology, anthropology, psychometry and sociology. The methods compared are the following: (1) permute the objects in one of the vectors (or matrices); (2) permute the residuals of a null model; (3) correlate residualized vector 1 (or matrix A) to residualized vector 2 (or matrix B); permute one of the residualized vectors (or matrices); (4) permute the residuals of a full model. In the partial correlation study, the results were compared to those of the parametric t-test which provides a reference under normality. Simulations were carried out to measure the type I error and power of these permutatio methods, using normal and non-normal data, without and with an outlier. There were 10 000 simulations for each situation (100 000 when n = 5); 999 permutations were produced per test where permutations were used. The recommended testing procedures are the following:(a) In partial correlation analysis, most methods can be used most of the time. The parametric t-test should not be used with highly skewed data. Permutation of the raw data should be avoided only when highly skewed data are combined with outliers in the covariable. Methods implying permutation of residuals, which are known to only have asymptotically exact significance levels, should not be used when highly skewed data are combined with small sample size. (b) In partial Mantel tests, method 2 can always be used, except when highly skewed data are combined with small sample size. (c) With small sample sizes, one should carefully examine the data before partial correlation or partial Mantel analysis. For highly skewed data, permutation of the raw data has correct type I error in the absence of outliers. When highly skewed data are combined with outliers in the covariable vector or matrix, it is still recommended to use the permutation of raw data. (d) Method 3 should never be used.