Conceptual and methodological issues in evaluating emergent programs

This paper identifies and discusses conceptual and methodological issues related to five attributes of emergent programs that frequently apply across the spectrum of human services: undefined client populations, inadequate causal evidence relating inputs to outcomes, shifting objectives, identification of criteria for standardizing treatments, and temporal constraints on program development. A case study of the American Red Cross nutrition program is used to illuminate the evaluation problems associated with these issues to mitigate them. Strategies found to be useful in the evaluation of emergent programs include the examination of the relationship among observed client characteristics and needs, external reviews of objectives, identification of client preconditions that support or inhibit performance, examination of normal treatment variation, and a multi-stage process.     

Determinants of Neighborhood Satisfaction: A Metropolitan-Level Analysis*

Students of neighborhood satisfaction have traditionally regarded the individual as the appropriate unit of analysis. In this present paper we break with tradition proposing that satisfaction can also be viewed as a property of populations. Data from the 1974–76 U.S. Annual Housing Surveys are employed to test three alternative explanations of variation in aggregate levels of satisfaction across 60 metropolitan areas Consistent with the urban-scale hypothesis, a negative relationship is found between metropolitan population size and the percentage of residents who rate their neighborhoods “excellent.” Further analysis reveals that variables representing the compositional and quality-of-life perspectives affect satisfaction in the expected directions as well: the more residents who are incentive- or resource-deficient and the more residents who perceive local conditions as problems, the lower the level of neighborhood satisfaction tends to be. In terms of relative explanatory power the quality-of-life variables appear dominant, although the persistent effects of the urban-scale measure may prove of greatest theoretical interest.     

Experimental and contextual models of program evaluation

Evaluation research is one of the most rapidly evolving fields of applied behavioral science. As demand for program assessment has increased, the number of alternative evaluation approaches has also grown. As a result, everyday practitioners have often lacked sufficient guidelines for the choice of appropriate evaluation strategies.The present paper articulates an underlying epistemological distinction between (a) experimental evaluation models which simplify program realities in generalizable analyses of discrete causes and effects, and (b) contextual evaluation models which holistically examine particular program operations. These two evaluation approaches are directed at different purposes and are applicable to different program settings. A topology of program characteristics (breadth of goals, scope of treatment, specificity of results, and clarity of theory) is developed and linked to the appropriateness of experimental and contextual evaluation.     

Developing a public health policy-research nexus: An evaluation of Nurse Practitioner models in aged care

A frustration often expressed by researchers and policy-makers in public health is an apparent mismatch between respective priorities and expectations for research. Academics bemoan an oversimplification of their work, a reticence for independent critique and the constant pressure to pursue evaluation funding. Meanwhile, policy-makers look for research reports written in plain language with clear application, which are attuned to current policy settings and produced quickly. In a context where there are calls in western nations for evidence based policy with stronger links to academic research, such a mismatch can present significant challenges to policy program evaluation. The purpose of this paper is to present one attempt to overcome these challenges. Specifically, the paper describes the development of a conceptual framework for a large-scale, multifaceted evaluation of an Australian Government health initiative to expand Nurse Practitioner models of practice in aged care service delivery. In doing so, the paper provides a brief review of key points for the facilitation of a strong research-policy nexus in public health evaluations, as well as describes how this particular evaluation embodies these key points. As such, the paper presents an evaluation approach which may be adopted and adapted by others undertaking public health policy program evaluations.     

The embeddedness of organizational performance: Multiple Membership Multiple Classification Models for the analysis of multilevel networks

We present a Multiple Membership Multiple Classification (MMMC) model for analysing variation in the performance of organizational sub-units embedded in a multilevel network. The model postulates that the performance of organizational sub-units varies across network levels defined in terms of: (i) direct relations between organizational sub-units; (ii) relations between organizations containing the sub-units, and (iii) cross-level relations between sub-units and organizations. We demonstrate the empirical merits of the model in an analysis of inter-hospital patient mobility within a regional community of health care organizations. In the empirical case study we develop, organizational sub-units are departments of emergency medicine (EDs) located within hospitals (organizations). Networks within and across levels are delineated in terms of patient transfer relations between EDs (lower-level, emergency transfers), hospitals (higher-level, elective transfers), and between EDs and hospitals (cross-level, non-emergency transfers). Our main analytical objective is to examine the association of these interdependent and partially nested levels of action with variation in waiting time among EDs – one of the most commonly adopted and accepted measures of ED performance. We find evidence that variation in ED waiting time is associated with various components of the multilevel network in which the EDs are embedded. Before allowing for various characteristics of EDs and the hospitals in which they are located, we find, for the null models, that most of the network variation is at the hospital level. After adding these characteristics to the model, we find that hospital capacity and ED uncertainty are significantly associated with ED waiting time. We also find that the overall variation in ED waiting time is reduced to less than a half of its estimated value from the null models, and that a greater share of the residual network variation for these models is at the ED level and cross level, rather than the hospital level. This suggests that the covariates explain some of the network variation, and shift the relative share of residual variation away from hospital networks. We discuss further extensions to the model for more general analyses of multilevel network dependencies in variables of interest for the lower level nodes of these social structures.     

Motor variability--an important issue in occupational life

Several recent reviews have reported that 'repetitive movements' is a risk factor for occupational musculoskeletal disorders (MSD) in the neck, shoulder and arm regions. More variation in biomechanical exposure is often suggested as an effective intervention in such settings. While increasing variation using extrinsic methods like job rotation may not always be possible in an industrial context, the intrinsic variability of the motor system may offer an alternative opportunity to increase variation. Motor variability (MV) refers to the natural variation in postures, movements and muscle activity observed to different extents in all tasks. The current review explores the state of the art in MV research from motor control, sports and occupational biomechanics literature to answer whether MV is important to consider in an occupational context, and if yes, whether this literature stimulates further studies to test if MV can be manipulated as a deliberate intervention for increasing biomechanical variation without jeopardizing production.     

Evolutionary Evaluation: Implications for evaluators,researchers, practitioners,funders and the evidence-based program mandate

Evolutionary theory, developmental systems theory, and evolutionary epistemology provide deep theoretical foundations for understanding programs, their development over time, and the role of evaluation. This paper relates core concepts from these powerful bodies of theory to program evaluation. Evolutionary Evaluation is operationalized in terms of program and evaluation evolutionary phases, which are in turn aligned with multiple types of validity. The model of Evolutionary Evaluation incorporates Chen's conceptualization of bottom-up versus top-down program development. The resulting framework has important implications for many program management and evaluation issues. The paper illustrates how an Evolutionary Evaluation perspective can illuminate important controversies in evaluation using the example of the appropriate role of randomized controlled trials that encourages a rethinking of “evidence-based programs”. From an Evolutionary Evaluation perspective, prevailing interpretations of rigor and mandates for evidence-based programs pose significant challenges to program evolution. This perspective also illuminates the consequences of misalignment between program and evaluation phases; the importance of supporting both researcher-derived and practitioner-derived programs; and the need for variation and evolutionary phase diversity within portfolios of programs.     

DOES OPTION THEORY HOLD FOR MAJOR LEAGUE BASEBALL CONTRACTS?

We test whether option theory indicated by the Black–Scholes formula is consistent with contract negotiations between teams and players in Major League Baseball (MLB). Teams use contract options in order to obtain the right to buy a player's services for an extra year when negotiating multiple year contracts. These options are similar to European call options, and should reduce the risk of buying volatile assets (players with high variation in performance). In this paper, we model the decision to buy an option (include an option year in the contract) and whether to exercise the option (pay the option salary in the option year versus buying out the option). The results indicate that the behavior of MLB teams is consistent with standard option theory both in terms of buying and exercising options. The sample is the complete population of free agent hitters who signed new contracts between the years 2003 and 2011. We were unable to find any published literature that has addressed this issue. (JEL ZO, H3)     

Reflections on Accuracy

The difference between test accuracy and predictive accuracy is presented and defined. The failure to distinguish between these two types of measures is shown to have led to a misguided debate over the interpretation of prevalence estimates. The distinction between test accuracy defined as sensitivity and specificity, and predictive accuracy defined as positive and negative predictive value is shown to reflect the choice of the denominator used to calculate true positive, false positive, false negative, and true negative rates. It is further shown that any instrument will tend to overestimate prevalence in low base rate populations and underestimate it in those populations where prevalence is high. The implications of these observations are then discussed in terms of the need to define diagnostic thresholds that have clinical and policy relevance.     

"Show me your impact": evaluating transitional justice in contested spaces

This paper discusses some of the most significant challenges and opportunities for evaluating the effects of programs in support of transitional justice - the field that addresses how post-conflict or post authoritarian societies deal with legacies of wide spread human rights violations. The discussion is empirically grounded in a case study that assesses the efforts of the International Development Research Centre (IDRC) and one of its Guatemalan partners to evaluate the effects of a museum exposition that is attempting to recast historic memory and challenge racist attitudes in post-conflict Guatemala. The paper argues that despite the increasing trend to fund transitional justice programs, many international aid donors are stuck in traditional and arguably orthodox paradigms of program evaluation. This is having a negative effect not only upon the administration of aid but also upon how transitional justice research is perceived and valued by local populations. The case study experience indicates that there is no perfect evaluation model or approach for evaluating transitional justice programming - only choices to be made by commissioners of evaluation, evaluators, and those being evaluated. These are profoundly influenced by the extreme politics and moral values that define transitional justice settings as contested spaces in which calls to remember the tragic past must be balanced with aspirations to re-build a hopeful future.     

The Validation of Screening Tests: Meet the New Screen same as the Old Screen?

The focus of this report is to examine the process of validation of new screening tests designed to detect the problem gambler in research and practice settings. A hierarchical or phases of evaluation model is presented as a conceptual framework to describe the basic features of the validation process and its implications for application and interpretation of test results. The report describes a number of threats to validity in the form of sources of unintended bias that when unrecognized may lead to incorrect interpretations of study results and the drawing of incorrect conclusions about the usefulness of the new screening tests. Examples drawn from the gambling literature on problem gambling are used to illustrate some of the more important concepts including spectrum bias and clinical variation in test accuracy. The concept of zones of severity and the bias inherent in selecting criterion thresholds are reviewed. A definition of reference or study gold standard is provided. The use of 2-stage designs to establish validity by efficiently using reference standards to determine indices of accuracy and prevalence is recommended.     

Coming Out Issues of Gay and Lesbian Mental Health Professionals in Voluntary and Involuntary Settings

Abstract

Mental health professionals are increasingly called upon to provide consultation and treatment in settings different from the traditional outpatient and hospital inpatient settings. This paper provides an overview of the unique problems posed when considering self-disclosure by gay or lesbian mental health professionals working with institutionalized populations. A review of literature concerning self-disclosure to patients in outpatient settings is provided and applied to institutionalized settings. Benefits of, as well as cautionary notes regarding, self-disclosure of homosexuality by mental health staff are reviewed.     

Relationship and touch in public settings

Gender differences in touch in U.S. populations have been well demonstrated. The age of participants and the setting in which touch occurs have been shown to affect the gender differences. Some investigators have concluded that a gender asymmetry exists with men touching women more than women touch men. A number of studies have shown that men and women interpret touch differently. Past research has provided little information about the effect of the relationship between a couple and the meaning of their touch. In the present study touch initiation among couples was observed in a variety of public settings and then the couples were asked to identify their relationship. It was found that men were more likely to initiate touch during courtship and women were more likely to initiate touch after marriage. A sex difference in reproductive strategies was suggested as one explanation for the phenomenon.     

Unravelling Cultural Constructions in Social Work Education: Journeying Toward Cultural Competence

Cultural understanding is often so deeply internalised it is difficult to know when and how values, beliefs and traditions were learned and even whether the ingrained messages carried from the past remain relevant in the present. Finding creative and culturally inclusive ways for students to unravel the origins of their own cultural constructions evolved into an assignment for a family course in social work education at the University of Otago, New Zealand. The student group were distance taught, mature students, most of whom were working in social service settings. Students self‐selected into groups and were asked to research, analyse and present their understanding of the influence of a culturally specific and influential myth, legend or fairytale. A key requirement was that this occurred within the context of working with families. Students were required to examine how the messages from well known ‘stories’ are perpetuated and how these then influence social work practice. This method of assessment was monitored and evaluated in terms of the students' experiences along with the learning outcomes for the course. It is the rationale, the process, and evaluation of this assignment that will be discussed in this paper.     

It's Not What You Do,But Who You Are: Informal Social Control,Social Status,and Normative Seriousness in Organizations

This paper analyzes the impact of managerial social status on the normative evaluation of managerial acts in organizational contexts. We test several propositions on the relationship between social status and normative evaluation derived from Donald Black's theoretical framework on social control. The research design consists of a factorial survey of 200 managers. Each respondent evaluated the seriousness of a normatively questionable managerial act. In each vignette, the perpetrator's social status was systematically manipulated in either a high or a low condition. The results generally support the argument that the higher a manager's social status the less vulnerable that individual is to unfavorable normative evaluations, holding constant the act. The paper closes with discussion of our findings in light of social structural and rational choice perspectives on informal social control in organizations. Additionally, we discuss methodological issues related to experimental research on informal social control in organizations, the consistency of our findings with those from previous studies of social control across diverse settings, potential theoretical applications and extensions of Black's framework in organizational contexts, and practical implications for the implementation of corporate codes of conduct and corporate dispute resolution systems.     

Attitudes to variation in spoken Faroese

This paper presents results from a language attitude test in the Faroe Islands, a small country with considerable dialectal variation and no standard spoken language. The test used is a replica of the verbal guise test method that has been used through the SLICE research network, which includes a verbal guise test to elicit reactions to five regional varieties of Faroese and a label ranking task to elicit statements on language variation. There were 203 Faroese 15‐year‐old students tested. The results from the label ranking task indicate a common notion of a national Faroese dialect hierarchy that favors that of the capital. However, the reaction patterns in the verbal guise test reflect no such hierarchy. The Faroese results thus diverge from the evaluation patterns found in a number of comparable European verbal guise tests, yet are similar to the patterns found in one European test locality: Western Norway.     

Variations in the conceptualization and measurement of rurality: Conflicting findings on the elderly widowed

This paper argues that the nature of any observed distinctions between the rural and urban aged are a function of whether rurality is conceptualized in ecological or socio-cultural terms. Four measures of rurality are contrasted. Two measures, based primarily on ecological criteria, define rurality in terms of (1) current residence, and (2) duration of community residence. Two additional measures, based primarily on socio-cultural criteria, define rurality in terms of (3) place of residence at age 16, and (4) a self-definition as rural. While different kinds of rural populations are identified by each measure, a self-definition of rurality seems to capture proportionally more of those who are also rural on the other criteria. However, this is not the case when urbanity is defined using the same criteria. The practice of classifying elderly populations as urban or rural based on current residence alone identifies ‘urban’ individuals who are in fact largely urban on both ecological and socio-cultural criteria; a significant proportion of those identified as rural by ecological criteria do not, however, fit the socio-cultural definition. The implications of these findings are discussed in reference to a sample of 151 elderly widowed men and women. Very different patterns of variability between rural and urban widowed elderly emerge, depending on which definition of rurality is used.     

Performance measurement: Some conceptual issues

Measures of how well a system is operating are clearly of interest to a wide range of users, from organizational researchers to system managers. The current literatures in such areas as “performance measurement, ” “organizational effectiveness, ” and the like show little consistency in their definitions of terms or in their methods for generating measures. The present paper attempts a clarification of these issues in the form of a conceptual minimalist position which requires only three basic definitions, and leaves, as far as possible, all remaining issues open to empirical investigation.We first review the literature on “organizational effectiveness” contrasting the organizational goals and systems paradigms, and note the lack of either theoretical or empirical convergence between the two. An examination of the nature of effectiveness statements suggests that this failure of convergence flows mainly from the different criterion sets generated by the two paradigms — and, importantly, that one should not expect convergence on a single measure or set of measures which uniquely define how well a system is performing. One's view of how well a given system is performing is a function of where one stands (either theoretically or in relationship to the system), and pursuit of the one true set of performance measures is a futile exercise. Instead, we propose to redirect attention to the identification of the various individuals and groups (“constituencies”) with an interest in system performance, and to the investigation of those items of system relevant information (their “performance measure sets”) which do, in fact, change their evaluations of how well the system is performing. This perspective will, we hope, redirect effort from futile theoretical debate to empirical investigations of what measures are used, by whom, and to what effect, in specific settings.     

Situation and ethnic identity

This paper illustrates situational variation as regards ethnic identification and develops some ideas about identity formation and the expression of ethnic identity in multiple ethnic contexts. The author discusses both social (ethnic) identity and subjective social identity (self-identifications). The author shows how expressions of ethnic origin and identification vary with changes in a number of hypothetical situational settings, explained in terms of a combination of the present and the past. The material consists of interview data from 110 Turks and 114 Yugoslavs aged 16-24, selected randomly, in Stockholm, Sweden. The 1st interview question asked of the Turks and Yugoslavs is how they would describe where they came from in several hypothetical situations. Answers vary more by place asked than by person asking. A tendency exists to relate oneself positively to the person asking, by stressing commonalities. The 2nd question was "To whom do you think you most strongly belong?" Answers show that these immigrants primarily relate themselves to their own ethnic groups and to Swedes, while other immigrant groups in Sweden are less important. The final question was "How would you feel if someone here in Sweden told you that you seem to be just like other Swedes?" The most usual answer is that it does not matter. Both Turks and Yugoslavs would appreciate the statements more from a Swede than a compatriot, and the Turks would become more angry than the Yugoslavs if a compatriot said it. Turks have preserved more ethnic and traditional lifestyles, lack close ties to the Swedes, and experience greater social distance to Swedes. Migrants often develop complex multiple identities, and situational variation, at least in part, can be understood as a competence for living with and making use of this complexity. The article concludes with a typology that illustrates how the interplay between the past and the present creates more or less complex patterns of identification, so that a person cannot express more than parts of his or her identifications in a given concrete situation.     

Latent Class Marginal Models for Cross-Classifications of Counts

The standard latent class model is a finite mixture of indirectly observed multinomial distributions, each of which is assumed to exhibit statistical independence. Latent class analysis has been applied in a wide variety of research contexts, including studies of mobility, educational attainment, agreement, and diagnostic accuracy, and as measurement error models in social research. One of the attractive features of the latent class model in these settings is that the parameters defining the individual multinomials are readily interpretable marginal probabilities, conditional on the unobserved latent variable(s), that are often of substantive interest. There are, however, settings where the local-independence axiom is not supported, and hence it is useful to consider some form of local dependence. In this paper we consider a family of models defined in terms of finite mixtures of multinomial models where the multinomials are parameterized in terms of a set of models for the univariate marginal distributions and for marginal associations. Local dependence is introduced through the models for marginal associations, and the standard latent class model obtains as a special case. Three examples are analyzed with the models to illustrate their utility in analyzing complex cross-classifications.