Evaluating Social Service Agencies: Fuzzy Pictures of Organizational Effectiveness

This article presents a case study of how members of three funding organizations evaluated the same two agencies in Canada. The research on which the article is based sheds light on the organizational effectiveness construct, on the ways in which the evaluators use it to reach conclusions on agency effectiveness, and the relation between these conclusions and funders' decisions on agency funding. The authors describe a framework for understanding evaluation processes, describe three funders in terms of this framework, set out predominant patterns in evaluation processes the funders used, and show the effects of these patterns from the agency's perspective. They then discuss the implications of the findings for agency managers and how the findings relate to theories of organization.     

Satisfaction with home-based employment

Using data from nine state regional research project on at-home income generation, the relationships of three satisfaction variables to demographic and work situation variables of 899 household managers in households with home-based employment are investigated. The satisfaction variables include quality of life, family income, and control over everyday life. The majority of households are satisfied with their quality of life and control over life although only moderately satisfied with income. One variable is related to the three satisfaction variables, the wage earner's control over the amount of work done in a day.This paper reports results from the Cooperative Regional Research Project, NE-167, entitled, At-Home Income Generation: Impact on Management, Productivity and Stability in Rural and Urban Families, partially supported by Cooperative States Research Service, U.S. Department of Agriculture, and the Experiment Stations at the University of Hawaii, Iowa State University, Lincoln University (Missouri), Michigan State University, Cornell University (New York), The Ohio State University, The Pennsylvania State University, Utah State University, and the University of Vermont. This article was accepted in 1992 under the editorship of Charles B. Hennon.Her research interests include evaluation of teaching/learning, program evaluation, and entrepreneurship. She received her Ph.D. from Cornell University.     

An examination of methods in the evaluation of volunteer programs.

Although many volunteer programs have been called "successful" and the use of paraprofessionals urged, few program evaluations with a benefit-cost analysis have been reported. Several major studies of volunteer utilization are reviewed, and particular needs with reference to program evaluation are discussed. Three alternative evaluation methodologies are proposed. The importance of utilizing a comprehensive evaluation incorporating all three methodologies is discussed. An ongoing volunteer program in a community mental health center is used as an illustration.     

Nudity and Framing: Classifying Art,Pornography, Information,and Ambiguity

Drawing on 45 interviews, this paper addresses how people interpret and make sense of nude images within popular culture. I suggest that the nude image resides comfortably in three familiar and bounded frames: art, pornography, and information. These frames clearly provide the reader with cues that aid his/her interpretation of the nude as well as his/her evaluation of the nude. A fourth frame is under construction this is the commodified frame used in selling products and magazines. Older adults refuse to address the acceptability of this frame; younger adults believe it to be a bounded category close to the other three. Age distinguishes between the kinds and amounts of cultural work that people do in deciding the acceptability of the frames that contain nude images.     

Priority setting for evaluation: Developing a strategic evaluation portfolio

Resources for evaluation are frequently scarce and best use should be made of them to deliver against the typical purposes of an evaluation function to (i) enhance accountability and (ii) promote operational improvement and learning. This paper presents a method for analyzing and prioritizing potential evaluations to improve the selection of a portfolio of activities that give the greatest pay-off. The method establishes the relative priority of ‘evaluation opportunities’ against criteria that relate to the usual primary purposes of an evaluation function. The method was developed in the context of a multilateral organization but is of general utility to the wider evaluation community and, with suitable adaptation can be applied to help ensure that scarce evaluation resources are used to their best advantage.A range of benefits are expected to accrue to an organization from adopting a more thorough, analytical priority setting process. These include:

• •Enhancing the relevance of evaluations to the wider organizational agenda
• •Providing structured justification for allocating funds to evaluation activities
• •More transparent, predictable decision-making
• •Enabling realignment of evaluation priorities with changing needs and circumstances
• •Improved credibility with stakeholders
• •Providing a clear direction and sense of purpose to evaluation staff
• •Documenting a clear framework for the development of an organization's evaluation portfolio.

The paper describes a priority setting method, including the key criteria that are used to assess ‘evaluation opportunities’, and presents different analyses of an evaluation portfolio. Examples from a practical application of the approach to the preparation of an evaluation work plan in a multilateral environmental agency are given.     

Are cost-inclusive evaluations worth the effort?

Relative costs are as important as relative effectiveness when choosing between program alternatives or among a set of programs competing for scarce funds. Nevertheless, the number of cost-inclusive evaluations remains comparatively small. This article presents the results of three first-time cost-inclusive evaluations each performed by an experienced evaluator. Each evaluator performed a different type of cost-inclusive evaluation using different tobacco control programs as examples: "standard" cost-effectiveness analysis, threshold or break-even analysis, and a simulation model. Results are presented in terms of the challenges faced, and informational and insight benefits gained, as well as in terms of program cost-effectiveness. All three evaluators agreed that the benefits from performing cost-inclusive evaluations are well worth time and effort involved. They also found that this type of evaluation provides abundant information that can be used to improve program effectiveness and cost-effectiveness.     

Using macro variables in program evaluation

This paper discusses the importance of using macro-level variables for program analysis. It is argued that three types of macro variables--cultural, economic, and demographic--provide a context for evaluation, and offer alternative explanations for the success or failure of a program. The problems of causality and availability of data are discussed. Macro variables are seen to be especially useful in "meta evaluation," and in integrating the results of specific program evaluations into general conclusions.     

The other side of the story. Student narratives on the California Drug, Alcohol, and Tobacco Education Programs

Within the context of a large-scale, comprehensive evaluation of the California Drug Alcohol Tobacco Education (DATE) program, this study sought to extend knowledge of student perceptions of prevention education using a naturalistic approach. The constant comparative method was used to analyze 40 focus group interviews of risk and thriving groups conducted in 11 high, middle, and elementary school districts. This article presents three assertions generated solely from 490 "narrative stories" found in the data set. "At-risk" and "thriving" students at all three levels of schooling (a) use "story" to make sense of prevention education, and (b) distinguish use from abuse. High school students of both groups (c) believe that hearing only one side of the substance use/abuse story and strict expulsion policies further alienate students most in need of help. Implications for the use of story as an assessment tool are discussed, as are implications for substance use prevention policy.     

Improved Self-exclusion Program: Preliminary Results

The gambling industry has offered self-exclusion programs for quite a long time. Such measures are designed to limit access to gaming opportunities and provide problem gamblers with the help they need to cease or limit their gambling behaviour. However, few studies have empirically evaluated these programs. This study has three objectives: (1) to observe the participation in an improved self-exclusion program that includes an initial voluntary evaluation, phone support, and a mandatory meeting, (2) to evaluate satisfaction and usefulness of this service as perceived by self-excluders, (3) to measure the preliminary impact of this improved program. One hundred sixteen self-excluders completed a questionnaire about their satisfaction and their perception of the usefulness during the mandatory meeting. Among those participants, 39 attended an initial meeting. Comparisons between data collected at the initial meeting and data taken at the final meeting were made for those 39 participants. Data showed that gamblers chose the improved self-exclusion program 75% of the time; 25% preferred to sign a regular self-exclusion contract. Among those who chose the improved service, 40% wanted an initial voluntary evaluation and 37% of these individuals actually attended that meeting. Seventy percent of gamblers came to the mandatory meeting, which was a required condition to end their self-exclusion. The majority of participants were satisfied with the improved self-exclusion service and perceived it as useful. Major improvements were observed between the final and the initial evaluation on time and money spent, consequences of gambling, DSM-IV score, and psychological distress. The applicability of an improved self-exclusion program is discussed and, as shown in our study, the inclusion of a final mandatory meeting might not be so repulsive for self-excluders. Future research directives are also proposed.

Collaborative evaluation within a framework of stakeholder-oriented evaluation approaches

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation planning and implementation. Unlike "distanced" evaluation approaches, which reject stakeholder participation as evaluation team members, Collaborative Evaluation assumes that active, on-going engagement between evaluators and program staff, result in stronger evaluation designs, enhanced data collection and analysis, and results that stakeholder understand and use. Among similar "participant-oriented" evaluation approaches (Fitzpatrick, Sanders, & Worthen, 2011), Collaborative Evaluation distinguishes itself in that it uses a sliding scale for levels of collaboration. This means that different program evaluations will experience different levels of collaborative activity. The sliding scale is applied as the evaluator considers each program's evaluation needs, readiness, and resources. While Collaborative Evaluation is a term widely used in evaluation, its meaning varies considerably. Often used interchangeably with participatory and/or empowerment evaluation, the terms can be used to mean different things, which can be confusing. The articles use a comparative Collaborative Evaluation Framework to highlight how from a theoretical perspective, Collaborative Evaluation distinguishes itself from the other participatory evaluation approaches.     

Facilitating Fatherhood: A Longitudinal Examination of Father Involvement Among Young Minority Fathers

The effects of father absence on children have been well documented in research and range from increased risk of poverty, to increased risk of incarceration (Anderson et al. Family Relations 51(2):148–155, 2002). This study presents a longitudinal evaluation of young father’s involvement with their children conducted within the scope of a teen parenting program in Arlington County, Virginia. The respondents in the study are young, mainly Hispanic fathers who come from lower socio-economic groups. The theoretical foundation for the parenting program was derived from Prochaska’s Transtheoretical Model of Change (1979). Three scales of fatherhood involvement were assessed at four time periods, including two years post-program. Results from the evaluation indicated that all three fatherhood scales increased steadily and significantly from pre-test to two years post-program. Results support the notion that fatherhood involvement curricula have the potential to positively influence young fathers and keep them involved with their children. Suggestions for further research and discussion of public policy implications are included in the study.

A participatory evaluation framework in the establishment and implementation of transdisciplinary collaborative centers for health disparities research

ObjectiveWe describe the formulation and implementation of a participatory evaluation plan for three Transdisciplinary Collaborative Centers for Health Disparities Research funded by the National Institute of Minority Health and Health Disparities.MethodsAlthough different in scope of work, all three centers share a common goal of establishing sustainable centers in health disparities science in three priority areas – social determinants of health, men’s health research, and health policy research.ResultsThe logic model guides the process, impact, and outcome evaluation. Emphasis is placed on process evaluation in order to establish a “blue print” that can guide other efforts as well as assure that activities are being implemented as planned.ConclusionWe have learned three major lessons in this process: (1) Significant engagement, participation, and commitment of all involved is critical for the evaluation process; (2) Having a “roadmap” (logic model) and “directions” (evaluation worksheets) are instrumental in getting members from different backgrounds to follow the same path; and (3) Participation of the evaluator in the leadership and core meetings facilitates continuous feedback.     

Evaluation approaches in mental health: Implications of the new federalism

Since the introduction of federal mental health legislation in 1963, there has been a changing emphasis on evaluation and accountability. With direct federal funding of community mental health services, accountability demands were met through expectations for local agency evaluation activities which were overseen by federal authorities. The advent of the New Federalism and the shift to block grant funding of mental health services to state mental health authorities have shifted responsibility for evaluation to the states and local programs. This paper reviews federal mental health statutes to trace the extent and locus of required evaluation activities and discusses two approaches to carrying out program evaluation: "top-down" where the evaluation topic, method, and data collection are mandated by an administering or funding body; and "bottom-up" approaches where the subject, method of study, and data to be collected are developed in response to a felt need at the local agency level. A case study of each approach as used at the state level in mental health is examined. Based on the literature and the case studies, conclusions are presented on the pro's and con's of each method in meeting accountability demands and the barriers which must be overcome for either method to be successful.     

The evaluation of the PACA regional public health plan: reconciling the managerial, scientific and democratic finalities

For more than a decade now, evaluation has developed considerably in France, thanks in particular to the Société Fran?aise de l'évaluation, whose charter sets out a number of principles designed to guide the work of evaluators. This article examines how the evaluation process surrounding a regional public health plan (referred to as PRSP)--itself being a new instrument for regional planning in France--accords with one of these principles, which specifies that evaluation must be framed according to "a three-fold logic involving public management, democracy and scientific debate." Our analysis shows that while this evaluation was driven primarily by managerial concerns (i.e., assessing the capacity of the plan to structure health policy in a region), it also provided an Opportunity for restoring dialogue with a range of actors by opening up a space of cooperation and consultation. In addition, in order to ensure the legitimacy of the evaluation's conclusions, the knowledge produced by the evaluators had to rest on an irreproachable methodology. This example illustrates how evaluation, in the French tradition, is a process that strives to reconcile the viewpoints and expectations of managers, scientists and the general public; it is also a process that brings out lines of tension and areas of complementariness between these three logics.     

Measuring societal effects of transdisciplinary research projects: design and application of an evaluation method

Most Transdisciplinary Research (TdR) projects combine scientific research with the building of decision making capacity for the involved stakeholders. These projects usually deal with complex, societally relevant, real-world problems. This paper focuses on TdR projects, which integrate the knowledge of researchers and stakeholders in a collaborative transdisciplinary process through structured methods of mutual learning. Previous research on the evaluation of TdR has insufficiently explored the intended effects of transdisciplinary processes on the real world (societal effects). We developed an evaluation framework for assessing the societal effects of transdisciplinary processes. Outputs (measured as procedural and product-related involvement of the stakeholders), impacts (intermediate effects connecting outputs and outcomes) and outcomes (enhanced decision making capacity) are distinguished as three types of societal effects. Our model links outputs and outcomes of transdisciplinary processes via the impacts using a mediating variables approach. We applied this model in an ex post evaluation of a transdisciplinary process. 84 out of 188 agents participated in a survey. The results show significant mediation effects of the two impacts "network building" and "transformation knowledge". These results indicate an influence of a transdisciplinary process on the decision making capacity of stakeholders, especially through social network building and the generation of knowledge relevant for action.     

The South Oaks Gambling Screen Revised for Adolescents (SOGS-RA): Further Psychometric Findings from a Community Sample

The broad expansion of gambling across North America during the last two decades has generated concern about the extent of gambling and problem gambling in youth, and the need to more accurately monitor it. The South Oaks Gambling Screen-Revised for Adolescents (SOGS-RA) is a promising instrument for screening problem gambling (Winters, Stinchfield, & Fulkerson, 1993) that requires more evaluation. Accordingly, further psychometric analysis of the instrument was conducted as part of a community survey of gambling in a sample of 1,000 male and female youth, aged 12 to 17 years. The analyses extended previous focus by including females, young adolescents, and an evaluation of youth classified as at-risk. Consistent with preliminary findings obtained during scale development, the distribution of item endorsement revealed trends of over-endorsement for some items (e.g., gambled more than intended, felt bad about the amount bet), and under-endorsement for others (e.g., criticized or told you had a gambling problem). These results suggest consideration of some form of weighting procedure, item deletion or re-wording. A factor analysis of the SOGS-RA items suggested a two-factor solution, with one factor interpreted as Control over Gambling and the other Gambling Consequences. It is proposed that the two factors may represent early versus more severe levels of gambling problems, respectively. The results highlight the need for further psychometric evaluation and refinement of instruments used to identify gambling problems in young people.     

Promoting ergonomics in Algeria: activities of "the research and training laboratory" in the University of Oran

The growing need in Algeria to develop ergonomics knowledge and practice in industry was behind the initiative to develop a training and research project within the ergonomics laboratory at Oran University. Since 2005 the laboratory team is running an academic option master in work design and ergonomics. The evaluation of the academic master in 2010 revealed the acute need of the local industry for professional competences in ergonomic and work psychology. A professional training master program in "ergonomics & work psychology" was then developed in partnership with local industry, five European Universities and six Universities from three Maghreb countries. Research projects were initiated around the two training programs, in conjunction with a number of ergonomics dissemination and promotion activities. Preliminary results of the project are presented and discussed in relation to the local context, and in the light of similar cases in Industrially Developing Countries.     

Environmental assessment and evaluation research: examples from mental health and substance abuse programs.

Evaluation researchers have been broadening the traditional client input--"black box" treatment--client outcome paradigm that has guided many evaluations of mental health, substance abuse treatment, and other intervention programs. The points of expansion are in the areas of treatment implementation and treatment processes, as well as "extratreatment" influences on treatment selection, duration, and, especially, outcome. This review illustrates the application of environmental assessment procedures--particularly social climate measures--in four aspects of evaluation research suggested by the more comprehensive model: (a) evaluating treatment implementation; (b) exploring treatment processes; (c) identifying extratreatment influences on client posttreatment functioning; and (d) operationalizing outcome variables. Conceptual and methodological issues raised by these applications are discussed, and the benefits to be derived from an expanded model of evaluation research--especially the greater potential for program improvement--are considered.     

Engaging New Social Movements

Organizations involved in the development ofinnovative social programs are coming under increasingpressure to establish structures through which theusers, targets, or beneficiaries of program activities can be involved in program planning andimplementation. The dynamic that this sets up betweenthe program and its wider environment, particularly inareas in which new social movements areoperating, can be quite challenging. This article seeks toexamine these dynamics, particularly in the context ofprogram evaluation. It is argued that an engagedevaluation approach (drawing on an action research orientation) has particular strengths in sucha situation, although the limitations and challenges ofsuch an approach are also outlined. It may, for example,require careful management in terms of the dynamics of the evaluation team. The discussionis illustrated by examples drawn from the evaluation ofa European program for disabled people in which theauthor as member of a research team from The Tavistock Institute was involved.     

Problems in the evaluation of persons who request sex-change surgery

Not all individuals who seek an evaluation for sex-change surgery are true, primary transsexuals. Evaluations need to be thorough and evaluators need to commit themselves to working with a difficult, demanding client population. Problems are related to the state of imperfection in nosology, diagnosis, and scientific knowledge. There is a great need for controlled research, long-term follow-up and consistency in procedural protocols.