Collaborative evaluation within a framework of stakeholder-oriented evaluation approaches

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation planning and implementation. Unlike "distanced" evaluation approaches, which reject stakeholder participation as evaluation team members, Collaborative Evaluation assumes that active, on-going engagement between evaluators and program staff, result in stronger evaluation designs, enhanced data collection and analysis, and results that stakeholder understand and use. Among similar "participant-oriented" evaluation approaches (Fitzpatrick, Sanders, & Worthen, 2011), Collaborative Evaluation distinguishes itself in that it uses a sliding scale for levels of collaboration. This means that different program evaluations will experience different levels of collaborative activity. The sliding scale is applied as the evaluator considers each program's evaluation needs, readiness, and resources. While Collaborative Evaluation is a term widely used in evaluation, its meaning varies considerably. Often used interchangeably with participatory and/or empowerment evaluation, the terms can be used to mean different things, which can be confusing. The articles use a comparative Collaborative Evaluation Framework to highlight how from a theoretical perspective, Collaborative Evaluation distinguishes itself from the other participatory evaluation approaches.     

The evaluation and monitoring of a Medicaid second surgical opinion program.

The evaluation of newly implemented and controversial programs often present real problems for the evaluator. When a program is accompanied by a legislatively mandated evaluation, the issues become even more acute. The evaluator, wanting to impact on policy, must strike a balance between the adherence to principles of scientific inquiry, yet be responsive to the needs of policymakers. This paper describes the evaluation and follow-up monitoring of a recently established second opinion program for Medicaid recipients. The program was to terminate in 17 months unless extended by the Wisconsin Legislature. The evaluation found, among other things, marked reductions in the use of surgical services after implementation of the second opinion program. Based in large part on the evaluation, the program was extended with a provision for continued monitoring. The paper discusses the factors that contributed to the evaluation having a major impact on the decision to extend the program.     

Dissemination and implementation science in program evaluation: A telemental health clinical consultation case example

Increased attention has been placed on evaluating the extent to which clinical programs that support the behavioral health needs of youth have effective processes and result in improved patient outcomes. Several theoretical frameworks from dissemination and implementation (D&I) science have been put forth to guide the evaluation of behavioral health program implemented in the context of real-world settings. Although a strong rationale for the integration of D&I science in program evaluation exists, few examples exist available to guide the evaluator in integrating D&I science in the planning and execution of evaluation activities.This paper seeks to inform program evaluation efforts by outlining two D&I frameworks and describing their integration in program evaluation design. Specifically, this paper seeks to support evaluation efforts by illustrating the use of these frameworks via a case example of a telemental health consultation program in pediatric primary care designed to improve access to behavioral health care for children and adolescents in rural settings. Lessons learned from this effort, as well as recommendations regarding the future evaluation of programs using D&I science to support behavioral health care in community-based settings are discussed.     

Lessons learned using a values-engaged approach to attend to culture,diversity, and equity in a STEM program evaluation

Evaluation must attend meaningfully and respectfully to issues of culture, race, diversity, power, and equity. This attention is especially critical within the evaluation of science, technology, engineering, and mathematics (STEM) educational programming, which has an explicit agenda of broadening participation. The purpose of this article is to report lessons learned from the implementation of a values-engaged, educative (Greene et al., 2006) evaluation within a multi-year STEM education program setting. This meta-evaluation employed a case study design using data from evaluator weekly systematic reflections, review of evaluation and program artifacts, stakeholder interviews, and peer review and assessment. The main findings from this study are (a) explicit attention to culture, diversity, and equity was initially challenged by organizational culture and under-developed evaluator–stakeholder professional relationship and (b) evidence of successful engagement of culture, diversity, and equity emerged in formal evaluation criteria and documents, and informal dialogue and discussion with stakeholders. The paper concludes with lessons learned and implications for practice.     

Changing stakeholder needs and changing evaluator roles: the Central Valley Partnership of the James Irvine foundation

This case describes the changing roles of an evaluator as the program he was evaluating changed and developed, and as the needs of his client and primary intended users changed over time. The focus in this case is on the relationship between the external evaluator and the staff of a philanthropic foundation as a major initiative unfolded. The focus of the evaluation was learning and improvement, and as the evaluator developed recommendations for organizational capacity-building, he was commissioned by the foundation to implement his own recommendations, a form of developmental evaluation. The case explores the dynamics and challenges of such an arrangement and relationship. Those involved have agreed to have their real names used because this is a highly visible initiative and attempting to disguise the foundation and key players might undermine, in this instance, the authenticity of the case. The initiative that is the focus of the case is the Central Valley Partnership of The James Irvine Foundation headquartered in San Francisco.     

The operation was a success but the patient died: A critique of “the implementation and evaluation of a problem-solving training program for adolescents”

This critique examines George S. Tellado's account of his evaluation of a problem-solving program for adolescents in a public junior high school in Pennsylvania. The program was apparently successful (i.e., it was shown by the evaluation to improve students' problem-solving capacities), and the evaluation was, for the most part, technically sound, but the program did not survive. This critique suggests that by examining the problem-solving program as an organizational change intervention, several problems can be uncovered regarding the program's design, implementation, evaluation, and institutionalization. In particular, it appears that the school system's staff did not have sufficient ownership of the program to understand it or to wish to carry it on after the program's originator and evaluator were no longer present. This article addresses these problems in detail and recommends techniques from the field of organizational change consultation to increase the likelihood that such programs will live on into the future.     

The relocation of nursing home residents: An illustration of the advantages gained by planning a new program and designing an implementation evaluation together

This paper describes a collaboration between an evaluation team and the management and staff responsible for relocating nursing home residents to a new, state-of-the-art facility. Both the relocation plan and the implementation evaluation of this plan are detailed in order to illustrate the link between program planning and evaluation. I argue that combining program planning and implementation evaluation activities is of value to a wide range of organizations contemplating the development and implementation of a new program. Then I draw upon the research utilization literature to discuss how the collaborative relationship between the evaluation team and management contributed to the success of both program planning and program evaluation activities.     

Legislative history and evaluation

Legislative histories can be done with those evaluative studies which are based on programs which have a history in the legislature. ESEA Title I, Title IV, and the Title I Technical Assistance Centers are examples of programs which have their direct source in federal legislation. The evaluator could ask whether the intent (aims and objectives) of a program is the same as the intent of the legislation or could ask whether the outcomes of a program were the same as those intended by the legislature. The purpose of this paper is to introduce the reader to the doing of legislative history in an evaluation study. The paper introduces the reader to five major sources for legislative history. The paper then discusses seven acceptable sources of information. Following this is a five-step guide to doing a legislative history.     

An evaluator of the future

For the evaluator who is part of the management of a human service organization, future expectations will be more demanding than at present. Operating staff, managers, and funders will already be familiar with the methods and the usefulness of capable program evaluation. Improved information tools will help the evaluator meet this challenge, by keeping the organization visible and manageable. A more fundamental change will be an improved “language of accountability”. Program standards, accreditation, and review now focus on structure and process characteristics. What is emerging is a growing consensus on definitions of client problems and program objectives in relation to these problems. This consensus spurs psychometric research on measures of problems and of the attainment of program objectives. Dependable, standard measures and meaningful comparative data will make the evaluation of program effectiveness possible.     

Are cost-inclusive evaluations worth the effort?

Relative costs are as important as relative effectiveness when choosing between program alternatives or among a set of programs competing for scarce funds. Nevertheless, the number of cost-inclusive evaluations remains comparatively small. This article presents the results of three first-time cost-inclusive evaluations each performed by an experienced evaluator. Each evaluator performed a different type of cost-inclusive evaluation using different tobacco control programs as examples: "standard" cost-effectiveness analysis, threshold or break-even analysis, and a simulation model. Results are presented in terms of the challenges faced, and informational and insight benefits gained, as well as in terms of program cost-effectiveness. All three evaluators agreed that the benefits from performing cost-inclusive evaluations are well worth time and effort involved. They also found that this type of evaluation provides abundant information that can be used to improve program effectiveness and cost-effectiveness.     

Environmental assessment and evaluation research: examples from mental health and substance abuse programs.

Evaluation researchers have been broadening the traditional client input--"black box" treatment--client outcome paradigm that has guided many evaluations of mental health, substance abuse treatment, and other intervention programs. The points of expansion are in the areas of treatment implementation and treatment processes, as well as "extratreatment" influences on treatment selection, duration, and, especially, outcome. This review illustrates the application of environmental assessment procedures--particularly social climate measures--in four aspects of evaluation research suggested by the more comprehensive model: (a) evaluating treatment implementation; (b) exploring treatment processes; (c) identifying extratreatment influences on client posttreatment functioning; and (d) operationalizing outcome variables. Conceptual and methodological issues raised by these applications are discussed, and the benefits to be derived from an expanded model of evaluation research--especially the greater potential for program improvement--are considered.     

Staff and client perspectives on the Journey Mapping online evaluation tool in a drug court program

The purpose of this study was to assess staff and client perspectives on the Internet-based Journey Mapping program evaluation tool. A drug court program was chosen for a case study research design. Six staff and 10 clients participated in interviews and observations, and also responded to a questionnaire. A staff survey provided additional data. The study revealed evidence that (1) Journey Mapping was user-friendly, saved time, promoted positive thinking, facilitated communication, increased learning, and created client voice; (2) Journey Mapping, as a program evaluation tool, supported both administrative and treatment functions of the drug court; and (3) Journey Mapping's positive orientation supplied momentum for clients’ treatment progress. Journey Mapping's distinctive technology offered a viable option for including narrative methods along with quantitative methods in program evaluation.     

The power of the context map: Designing realistic outcome evaluation strategies and other unanticipated benefits

Developing a feasible evaluation plan is challenging when multiple activities, often sponsored by multiple agencies, work together toward a common goal. Often, resources are limited and not every agency's interest can be represented in the final evaluation plan. The article illustrates how the Antecedent Target Measurement (ATM) approach to logic modeling was adapted to meet this challenge. The key adaptation is the context map generated in the first step of the ATM approach. The context map makes visually explicit many of the underlying conditions contributing to a problem as possible. The article also shares how a prioritization matrix can assist the evaluator in filtering through the context map to prioritize the outcomes to be included in the final evaluation plan as well as creating realistic outcomes. This transparent prioritization process can be especially helpful in managing evaluation expectations of multiple agencies with competing interests. Additional strategic planning benefits of the context map include pinpointing redundancies caused by overlapping collaborative efforts, identifying gaps in coverage, and assisting the coordination of multiple stakeholders.     

Clarifying the role of belief-motive explanations in multi-stakeholder realist evaluation

In realist evaluation, where researchers aim to make program theories explicit, they can encounter competing explanations as to how programs work. Managing explanatory tensions from different sources of evidence in multi-stakeholder projects can challenge external evaluators, especially when access to pertinent data, like client records, is mediated by program stakeholders. In this article, we consider two central questions: how can program stakeholder motives shape a realist evaluation project; and how might realist evaluators respond to stakeholders’ belief-motive explanations, including those about program effectiveness, based on factors such as supererogatory commitment or trying together in good faith? Drawing on our realist evaluation of a service reform initiative involving multiple agencies, we describe stakeholder motives at key phases, highlighting a need for tactics and skills that help to manage explanatory tensions. In conclusion, the relevance of stakeholders’ belief-motive explanations (‘we believe the program works’) in realist evaluation is clarified and discussed.     

Design of a collaborative monitoring and evaluation system for a community-based nutrition project in rural Bangladesh

Poor diet and undernutrition are common among children living in Bangladesh. To promote appropriate complementary feeding of young children, an economic development (ED) program involving income-generating asset transfer was implemented alongside a social and behavior change (SBC) program. This paper introduces a collaborative monitoring and evaluation (M&E) system in which diverse collaborators (“research group”, “implementation team”, and “coordinators”) facilitate M&E data acquisition by leveraging their comparative advantages. The implementation team built a monitoring system to track the ED (n = 2960) and SBC participants (n=∼10,000) over 12 months. Based on the baseline design and the monitoring records, the collaborators planned an impact evaluation introducing a quasi-experimental design using two cross-sectional surveys and a prospective cohort survey of child feeding and nutritional status. Using various data sources generated from the M&E system, the collaborators will also reveal the program impact pathway through which each intervention component is delivered, received, and utilized alongside the context-specific facilitators and barriers, including the programs’ uptake. The collaborative M&E system enables the sharing of program goals, strengthens collaborators' commitment to the program, and extends the understanding of the program's progress and evaluation activities.     

Ethical dilemmas in evaluating programs with family court related clients

Two ethical dilemmas are discussed; breaching client confidentiality in order to intervene in the client's illegal behavior, and defining the role of evaluator as expert witness. The evaluation context concerns a combined program development and evaluation project with clients involved in family court proceedings for child abuse or neglect. Different perspectives, approaches, and experiences in resolving each dilemma are reviewed. The conclusion argues that approaches to ethical dilemmas in evaluation projects must go beyond techniques to disguise data, to include an assessment of the values and value commitments connected to the different roles an evaluator must perform.     

Expanding evaluator competency research: Exploring competencies for program evaluation using the context of non-formal education

The overlap of competencies between general program evaluation and specific contexts or content will always be reality because evaluators may need unique competencies to answer evaluation questions for particular contexts or content areas. Limited research exists that explores the essential competencies required by professionals who use evaluation as one part of their job portfolio, which leaves unanswered questions regarding the applicability of current evaluator competency models in such settings. We used a modified three-round Delphi technique to identify evaluator competencies for non-formal educators in Cooperative Extension (CE). Our panelists identified 36 competencies in the non-formal educational programming context for CE educators that they considered important to be included in evaluation capacity building efforts. We categorized our 36 identified competencies from the Delphi study into the five competency domains proposed by the American Evaluation Association. Our findings provide information to help guide professional development among non-formal educators related to program evaluation.     

Register data in the evaluation and program planning of addiction treatment programs: Using Sweden as an example

Data from large-scale registers is often underutilized when evaluating addiction treatment programs. Since many programs collect register data regarding clients and interventions, there is a potential to make greater use of such records for program evaluation. The purpose of this article is to discuss the value of using large-scale registers in the evaluation and program planning of addiction treatment systems and programs. Sweden is used as an example of a country where register data is both available and is starting to be used in national evaluation and program planning efforts.The article focuses on possibilities, limitations and practicalities when using large-scale register data to conduct evaluations and program planning of addiction treatment programs. Main conclusions are that using register data for evaluation provides large amounts of data at low cost, limitations associated to the use of register data may be handled statistically, register data can answer important questions in planning of addiction treatment programs, and more accurate measures are needed to account for the diversity of client populations.