Performance measurement: Some conceptual issues

Measures of how well a system is operating are clearly of interest to a wide range of users, from organizational researchers to system managers. The current literatures in such areas as “performance measurement, ” “organizational effectiveness, ” and the like show little consistency in their definitions of terms or in their methods for generating measures. The present paper attempts a clarification of these issues in the form of a conceptual minimalist position which requires only three basic definitions, and leaves, as far as possible, all remaining issues open to empirical investigation.We first review the literature on “organizational effectiveness” contrasting the organizational goals and systems paradigms, and note the lack of either theoretical or empirical convergence between the two. An examination of the nature of effectiveness statements suggests that this failure of convergence flows mainly from the different criterion sets generated by the two paradigms — and, importantly, that one should not expect convergence on a single measure or set of measures which uniquely define how well a system is performing. One's view of how well a given system is performing is a function of where one stands (either theoretically or in relationship to the system), and pursuit of the one true set of performance measures is a futile exercise. Instead, we propose to redirect attention to the identification of the various individuals and groups (“constituencies”) with an interest in system performance, and to the investigation of those items of system relevant information (their “performance measure sets”) which do, in fact, change their evaluations of how well the system is performing. This perspective will, we hope, redirect effort from futile theoretical debate to empirical investigations of what measures are used, by whom, and to what effect, in specific settings.     

On summary measures of binarized dominance data

Data on outcomes of dominance contests in groups of animals are often binarized (the original matrix of counts transformed into a matrix of ones and zeros indicating, for each pair, which animal won more contests). Various measures attempt to summarize the structure of the binarized data. We discuss uncertainty in such summary measures, and suggest ways to account for this uncertainty. We consider the summary measure both for an observed finite sample and for the unobserved underlying probability structure. We illustrate the general ideas with one particular summary measure, Appleby's K, and one data set, Appleby's red deer data.     

A new approach to measure socioeconomic inequality in health

This paper introduces a new approach to rank and measure socioeconomic inequality in health. A novel feature of the approach is the use of an income-health matrix that relates socioeconomic class with health status; each row of the matrix corresponds to a socioeconomic class and contains the respective probability distribution of health. By invoking a monotone assumption on the income-health matrix, which is a direct translation of the well-known hypothesis of social gradients in health outcomes, we derive a set of welfare-dominance and inequality-dominance conditions for ranking health distributions. Unlike other existing health inequality measures, our conditions require no cardinal specification of ordinal heath data and, hence, are robust. We then apply the dominance conditions to compare socioeconomic health inequality in the US and Canada using the newly released JCUSH data.     

Bridging: Locating critical connectors in a network

This paper proposes several measures for bridging in networks derived from Granovetter's (1973) insight that links which reduce distances in a network are important structural bridges. Bridging is calculated by systematically deleting links and calculating the resultant changes in network cohesion (measured as the inverse average path length). The average change for each node's links provides an individual level measure of bridging. We also present a normalized version which controls for network size and a network-level bridging index. Bridging properties are demonstrated on hypothetical networks, empirical networks, and a set of 100 randomly generated networks to show how the bridging measure correlates with existing network measures such as degree, personal network density, constraint, closeness centrality, betweenness centrality, and vitality. Bridging and the accompanying methodology provide a family of new network measures useful for studying network structure, network dynamics, and network effects on substantive behavioral phenomenon.     

Models and procedures for evaluating government provided leisure services

This paper briefly describes an experimental project in which government attempted to set up a viable management information and feedback system for evaluating accountability in services delivery. Conceptual models for agency goals and services delivery were designed and measures of program performance were developed in the provision of leisure and recreational services. Two citizen surveys are described which contributed data for service provider feedback. A discussion of perspectives on the definition of equity is presented.     

Harnessing the reliability of outcome measures

A method is presented for engineering the necessary levels of measurement reliability for evaluating ongoing programs. Two studies of levels of client functioning at a community mental health center (CMHC), in which an outcome measure, the Global Assessment Scale (GAS), did not perform as expected, drew attention to the need for better control of outcome measure reliabilities. Drawing from generalizability theory, a study was conducted of three sources of GAS score variance — clients, raters, and training in the use of the scale. Several estimates of reliability (ERs) were developed, depending on the manner in which the GAS ratings were, or would be, obtained in the CMHC. The differences among these ERs clarified why the GAS had lower reliability when used in our setting. Finally, two hypothetical examples are described to illustrate the utility of applying generalizability theory to achieve higher reliabilities for outcome measures.     

Unemployment duration and the measurement of unemployment

We propose a framework in which the welfare bases of various unemployment measures available in the literature can be analyzed and a decomposition of these measures can be made into aggregate unemployment and the concentration of its distribution. A set of axioms for an unemployment measure is then introduced, which relate to the sensitivity of an unemployment measure to the inequality in the distribution of the unemployment. An unemployment measure is derived on the basis of these axioms that captures the distributional considerations in the measurement of unemployment.      

Network analysis in evaluation: some words of caution

Network analysis, a methodology derived from general systems' theory, can be utilized as a community mental health administrative-evaluation procedure. Evaluation parameters derived from analysis of patient data as they “flow” through a network of agency services, provide measures of systemic functioning. These parameters include “longest paths” and various ratio relationships as evaluation measures of service delivery. The limitations of network analysis are examined by means of conceptual analyses, and phenomena that emerged from research experience. The necessity for both quantitative and qualitative data to ensure a meaningful evaluation of mental health services is explained. Conclusions about the value of the network analysis approach are considered.     

Applicability of multiple matrix sampling to estimating effectiveness of educational programs

When evaluating the effectiveness of an educational program, a sampling strategy known as “multiple matrix sampling” is particularly effective and efficient when the goal of the evaluation is estimating group (as opposed to individual) performance. Although this technique involves sampling both program participants and items and, subsequently, different participants being assessed over different sets of items, the results may be used to estimate those which would have been obtained had all participants been administered all items. The technique is here described in some detail, with advantages and disadvantages discussed, and examples of its application are given.     

Ranking inequality: Applications of multivariate subset selection

Inequality measures are often presented in the form of a rank ordering to highlight their relative magnitudes. However, a rank ordering may produce misleading inference, because the inequality measures themselves are statistical estimators with different standard errors, and because a rank ordering necessarily implies multiple comparisons across all measures. Within this setting, if differences between several inequality measures are simultaneously and statistically insignificant, the interpretation of the ranking is changed. This study uses a multivariate subset selection procedure to make simultaneous distinctions across inequality measures at a pre-specified confidence level. Three applications of this procedure are explored using country-level data from the Luxembourg Income Study. The findings show that simultaneous precision plays an important role in relative inequality comparisons and should not be ignored. An erratum to this article can be found at     

Treatment participation and outcome in a program for problem drinker-drivers

Relationships were examined between 13 predictor variables and two criteria of improvement in a program for problem drinker-drivers. Four measures of social background (income, educations, age, marital status), four measures of psychopathology based on the Minnesota Multiphasic Personality Inventory (MMPI), one measure of general alcoholism, and four indicators of treatment involvement served as predictor variables. Two factorially derived criteria served as outcome indicators, one reflecting general improvement, the other reflecting improved social adjustment, but a failure to reduce drinking. This second factor was interpreted as indicating resistance. Analysis included partial correlation and multiple regression, from which explained variance was completely partitioned into sources unique to each predictor, and sources shared by combinations of predictors. Results suggested that with other factors controlled, treatment involvement was linked both to improvement and to resistance. Similar conclusions were suggested for social background, but slightly different ones for psychopathology and alcoholism. Psychopathology was uniquely related to improvement; alcoholism, to resistance.     

A multitrait-multimethod investigation of interpersonal distance

Prior research has investigated the validity of interpersonal distance measures with respect to only one trait — comfort. This study expanded the previous research by measuring interpersonal distance along the trait dimensions of comfort, control and privacy. In addition, measures were taken along three different measurement techniques: stop distance, silhouette placement, and a paper and pencil measure (CID). The results were interpreted in terms of a multitrait-multimethod matrix. Results indicated that these measures were reliable and valid and the traits did have convergent validity. However, all of these measures were uncorrelated with a disguised distance measure of interpersonal distance.     

Power indices expressed in terms of minimal winning coalitions

A voting situation is given by a set of voters and the rules of legislation that determine minimal requirements for a group of voters to pass a motion. A priori measures of voting power, such as the Shapley–Shubik index and the Banzhaf value, show the influence of the individual players in a voting situation and are calculated by looking at marginal contributions in a simple game consisting of winning and losing coalitions derived from the legislative rules. We introduce a new way to calculate these measures directly from the set of minimal winning coalitions and derive explicit formulae for the Shapley–Shubik and Banzhaf values. This new approach logically appealing as it writes measures as functions of the rules of the legislation. For certain classes of games that arise naturally in applications the logical shortcut drastically simplifies the numerical calculations to obtain the indices. The technique generalises directly to all semivalues.     

The client satisfaction questionnaire. Psychometric properties and correlations with service utilization and psychotherapy outcome

An 18-item version of the Client Satisfaction Questionnaire (CSQ-18) was included in an experimental study of the effects of pretherapy orientation on psychotherapy outcome. The psychometric properties of the CSQ-18 in this study were compared with earlier findings. In addition, the correlations of the CSQ-18 with service utilization and psychotherapy outcome measures were examined. Results indicated that the CSQ-18 had high internal consistency (coefficient alpha = .91) and was substantially correlated with remainer-terminator status (rs = .61) and with number of therapy sessions attended in one month (r = .54). The CSQ-18 was also correlated with change in client-reported symptoms (r = -.35), indicating that greater satisfaction was associated with greater symptom reduction. Results also demonstrated that a subset of items from the scale (the CSQ-8) performed as well as the CSQ-18 and often better. The excellent performance of the CSQ-8, coupled with its brevity, suggests that it may be especially useful as a brief global measure of client satisfaction.     

Marital status and psychological well-being: An individual-environment congruence perspective

The role of the individual's relationship to his environment has achieved central prominence in social science theory, but has not yet been adequately represented and measured through numerical methods. A variety of multivariate regression techniques have been proposed to incorporate individual and environmental level measures in a single regression equation. This article presents a special case of a contextual analysis model in which two sets of predictor variables are statistically treated to create a third set of predictor variables called congruence measures. Each of the three sets of predictor variables are then run separately in multiple regression equations against a fourth set of individual level outcome variables to determine the relative gain in predictive power of the congruence measures over the individual and environmental level variables when they are run alone. Martital status variables are used to illustrate and examine the model.     

Toward Comprehensive Health Status Measures*

While health, itself, is the central concept in the sociology of health, little agreement exists on what constitutes an appropriate definition of health. This paper develops two composite health status measures (one continuous summary measure, and one set of eight discrete health state measures) based on the World Health Organization's conceptualization of the physical, social, and psychological dimensions of health. Data from a 1978 regional survey demonstrates the continuous summary measure's reliability (alpha = .70), and validity (factor analytic support for the hypothesized dimensions). The results also indicate that although either of the comprehensive measures significantly increase the goodness of fit of structural models of health service utilization, the set of discrete health state measures increases more the goodness of fit, bringing the explained variance of physician visits up from 16 percent to 30 percent. The magnitude of the increments clearly demonstrates the considerable utility of the comprehensive health status measurement approach.     

Entropy Measures of Inequality*

Although sociologists have long been interested in the analysis of inequality, there have been relatively few systematic attempts to measure it. Most measurement has utilized popular existing measures such as the Gini coefficient, which unfortunately is relatively difficult to compute and interpret. This article presents a new measure of inequality (B) that is based upon the concept of entropy. The B measure is used for categorical data and complements Theil's (T) entropy measure for continuous data. The B measure is illustrated for both income and wealth data and successfully meets several criteria for evaluating measures of inequality. These include the criteria of scale invariance, sensitivity to transfers, and adequate upper bounds. A further advantage of sociological entropy measures of inequality is that they facilitate interdisciplinary comparison of work on inequality between sociology and other disciplines (such as economics) which use entropy measures of inequality.