Towards Program Evaluation for Practitioner Learning: Human Service Practitioners' Perceptions of Evaluation

Abstract

While human service practice improvement is acknowledged as an important outcome of program evaluation, much evaluation is narrowly focused on accountability requirements. This type of evaluation often has limited use and relevance to human service practitioners. In exploring forms of evaluation that do have practice relevance, this article presents the experiences and perceptions of human service practitioners subject to evaluations that set out to foster learning. Across the cases participants expressed concerns about data quality and credibility, the relevance of evaluation data to the program context, the invasiveness of the evaluation, management of the process (particularly limited consultation with practitioners), and the lack of effective dissemination. These cases illustrate the challenges for evaluation to be useful to practice, and to foster meaningful improvements to services.     

Cases for the nugget in modeling computer experiments

Most surrogate models for computer experiments are interpolators, and the most common interpolator is a Gaussian process (GP) that deliberately omits a small-scale (measurement) error term called the nugget. The explanation is that computer experiments are, by definition, “deterministic”, and so there is no measurement error. We think this is too narrow a focus for a computer experiment and a statistically inefficient way to model them. We show that estimating a (non-zero) nugget can lead to surrogate models with better statistical properties, such as predictive accuracy and coverage, in a variety of common situations.     

Components of change in the number of female family heads ages 15 to 44, an update and reanalysis: United States, 1940 to 1983

We review the literature dealing with the various components of change in the number of female family heads and conclude that P. Cutright's (1974, Journal of Marriage and the Family 36, 714–721) four components of change in the number of ever-married female family heads can be usefully applied to a reanalysis of decade-by-decade changes in this quantity among white and nonwhite women ages 15–44, from 1940 through 1980. However, we substantially revise and update Cutright's (1974) original data. Our findings include the fact that while the relative importance of the various components of increases in ever-married female family headship vary from decade to decade, overall trends are fairly similar for white and nonwhite women. Moreover, the rapid rise between 1970 and 1980 in the number of white and nonwhite female family heads among the never-married (single) continued through the 1980 to 1983 period. A decade ago, Cutright (1974) predicted that a change in the number of female family heads from 1970 on would reflect little more than changes in the numbers of ever- and never-married women ages 15–44. We discuss this inaccurate prediction and conclude that wishful thinking is a poor guide in forecasting the future.     

Robustness and power of parametric, nonparametric, robustified and adaptive tests—The multi-sample location problem

This paper deals with a survey of different types of tests, parametric, nonparametric, robustified and adaptive ones, and with an application to the two-sided c-sample location problem. Some concepts of robustness are discussed, such as breakdown point, influence function, gross-error sensitivity and especially α- and β-robustness. A robustness study on level α in the case of heteroscedasticity and nonnormal distributions is carried out via Monte Carlo methods and also a power comparison of all the tests considered. It turns out that robustified versions of the F-test and Welch-test where the original observations are replaced by its ranks behave well over a broad class of distributions, symmetric ones with different tail weight and asymmetric ones, but, on the whole, an adaptive test is to prefer.     

An approximation to the distribution of the sample variance

A simple three-moment approximation is introduced for the distribution of the sample variance. Comparisons are given with other approximations discussed by Tan and Wong (1977) and with an approximation developed very recently by Mudholkar and Trivedi (1981).     

Maximum Test versus Adaptive Tests for the Two-Sample Location Problem

For the non-parametric two-sample location problem, adaptive tests based on a selector statistic are compared with a maximum and a sum test, respectively. When the class of all continuous distributions is not restricted, the sum test is not a robust test, i.e. it does not have a relatively high power across the different possible distributions. However, according to our simulation results, the adaptive tests as well as the maximum test are robust. For a small sample size, the maximum test is preferable, whereas for a large sample size the comparison between the adaptive tests and the maximum test does not show a clear winner. Consequently, one may argue in favour of the maximum test since it is a useful test for all sample sizes. Furthermore, it does not need a selector and the specification of which test is to be performed for which values of the selector. When the family of possible distributions is restricted, the maximin efficiency robust test may be a further robust alternative. However, for the family of t distributions this test is not as powerful as the corresponding maximum test.     

Adaptive bootstrap tests and its competitors in the c-sample scale problem

This paper deals with a study of different types of tests for the two-sided c-sample scale problem. We consider the classical parametric test of Bartlett [M.S. Bartlett, Properties of sufficiency and statistical tests, Proc. R. Stat. Soc. Ser. A. 160 (1937), pp. 268–282] several nonparametric tests, especially the test of Fligner and Killeen [M.A. Fligner and T.J. Killeen, Distribution-free two-sample tests for scale, J. Amer. Statist. Assoc. 71 (1976), pp. 210–213], the test of Levene [H. Levene, Robust tests for equality of variances, in Contribution to Probability and Statistics, I. Olkin, ed., Stanford University Press, Palo Alto, 1960, pp. 278–292] and a robust version of it introduced by Brown and Forsythe [M.B. Brown and A.B. Forsythe, Robust tests for the equality of variances, J. Amer. Statist. Assoc. 69 (1974), pp. 364–367] as well as two adaptive tests proposed by Büning [H. Büning, Adaptive tests for the c-sample location problem – the case of two-sided alternatives, Comm. Statist.Theory Methods. 25 (1996), pp. 1569–1582] and Büning [H. Büning, An adaptive test for the two sample scale problem, Nr. 2003/10, Diskussionsbeiträge des Fachbereich Wirtschaftswissenschaft der Freien Universität Berlin, Volkswirtschaftliche Reihe, 2003]. which are based on the principle of Hogg [R.V. Hogg, Adaptive robust procedures. A partial review and some suggestions for future applications and theory, J. Amer. Statist. Assoc. 69 (1974), pp. 909–927]. For all the tests we use Bootstrap sampling strategies, too. We compare via Monte Carlo Methods all the tests by investigating level α and power β of the tests for distributions with different strength of tailweight and skewness and for various sample sizes. It turns out that the test of Fligner and Killeen in combination with the bootstrap is the best one among all tests considered.