An organizational model to distinguish between and integrate research and evaluation activities in a theory based evaluation

Theory-based evaluation (TBE) is an evaluation method that shows how a program will work under certain conditions and has been supported as a viable, evidence-based option in cases where randomized trials or high-quality quasi-experiments are not feasible. Despite the model's widely accepted theoretical appeal there are few examples of its well-implemented use, probably due to time and money limitations necessary for planning and a confusion over the definitions between research and evaluation functions and roles. In this paper, we describe the development of a theory-based evaluation design in a Math and Science Partnership (MSP) research project funded by the U.S. National Science Foundation (NSF). Through this work we developed an organizational model distinguishing between and integrating evaluation and research functions, explicating personnel roles and responsibilities, and highlighting connections between research and evaluation work. Although the research and evaluation components operated on independent budgeting, staffing, and implementation activities, we were able to combine datasets across activities to allow us to assess the integrity of the program theory, not just the hypothesized connections within it. This model has since been used for proposal development and has been invaluable as it creates a research and evaluation plan that is seamless from the beginning.     

Planning and implementation of a participatory evaluation strategy: A viable approach in the evaluation of community-based participatory programs addressing cancer disparities

Community-based participatory research (CBPR) has been posited as a promising methodology to address health concerns at the community level, including cancer disparities. However, the major criticism to this approach is the lack of scientific grounded evaluation methods to assess development and implementation of this type of research. This paper describes the process of development and implementation of a participatory evaluation framework within a CBPR program to reduce breast, cervical, and colorectal cancer disparities between African Americans and whites in Alabama and Mississippi as well as lessons learned. The participatory process involved community partners and academicians in a fluid process to identify common ground activities and outcomes. The logic model, a lay friendly approach, was used as the template and clearly outlined the steps to be taken in the evaluation process without sacrificing the rigorousness of the evaluation process. We have learned three major lessons in this process: (1) the importance of constant and open dialogue among partners; (2) flexibility to make changes in the evaluation plan and implementation; and (3) importance of evaluators playing the role of facilitators between the community and academicians. Despite the challenges, we offer a viable approach to evaluation of CBPR programs focusing on cancer disparities.     

Conflict of interest in the evaluation and dissemination of "model" school-based drug and violence prevention programs

Conflict of interest refers to a set of conditions in which professional judgment concerning the validity of research might be influenced by a secondary competing interest. The competing interest that has received most attention in the literature addressing the prevalence and effects of such conflicts on the practice of empirical research has been that of financial relationships between investigators and research sponsors. The potential for conflicts of interest to arise in the evaluation of drug prevention programs was raised by Moskowitz in this journal in 1993, but to date there has been no attempt made to estimate the scope of this problem. The present study addressed this issue using a sample of “model” school-based drug and violence prevention interventions by first, identifying the types or relationships that exist between program developers and program distributors, and, second, by assessing how many of the evaluations of these programs published in peer-reviewed journals had been conducted by the developers of the programs compared to independent evaluation teams. The data presented indicate that there are relatively few published evaluations that do not involve program developers and that there are few instances in which there is complete separation between the program developer and program distributor. Using the open systems model of the Institute of Medicine Committee on Research Integrity as a framework, it is argued that the culture and norms of the program developer and those of the program evaluator are fundamentally distinct and therefore failure to separate these roles produces high potential for conflict of interest to arise.     

Open systems evaluation and the logic model: Program planning and evaluation tools

This paper describes two case management projects and efforts to evaluate outcomes. Lessons learned from the first effort suggest that a tool referred to as the logic model and a particular approach to evaluation called the “open systems” model might be useful to planners and program evaluators. The logic model provides a means of presenting conditions a program is intended to address, activities that constitute a program, short term outcomes resulting from program activities and long term impacts. The open systems perspective emphasizes evaluation as a tool to be used to achieve program objectives as opposed to establishing cause and effect relationships. This paper also describes how the logic model and open systems evaluation were used to facilitate the development of an evaluation plan for a project designed to assist homeless, substance abusing, pregnant women.     

The relocation of nursing home residents: An illustration of the advantages gained by planning a new program and designing an implementation evaluation together

This paper describes a collaboration between an evaluation team and the management and staff responsible for relocating nursing home residents to a new, state-of-the-art facility. Both the relocation plan and the implementation evaluation of this plan are detailed in order to illustrate the link between program planning and evaluation. I argue that combining program planning and implementation evaluation activities is of value to a wide range of organizations contemplating the development and implementation of a new program. Then I draw upon the research utilization literature to discuss how the collaborative relationship between the evaluation team and management contributed to the success of both program planning and program evaluation activities.     

Using logic model and visualization to conduct portfolio evaluation

Portfolio evaluation is the evaluation of multiple projects with a common purpose. While logic models have been used in many ways to support evaluation, and data visualization has been used widely to present and communicate evaluation findings, adopting logic models for portfolio evaluation and using data visualization to share findings simultaneously is surprisingly limited in the literature. With the data from a sample portfolio of 209 projects which aims to improve the system of early care and education (ECE), this study illustrated how to use logic model and data visualization techniques to conduct a portfolio evaluation by answering two evaluation questions: “To what extent are the elements of a logic model (strategies, sub-strategies, activities, outcomes, and impacts) reflected in the sample portfolio?” and “Which dominant paths through the logic model were illuminated by the data visualization technique?” For the first question, the visualization technique illuminated several dominant strategies, sub-strategies, activities, and outcomes. For the second question, our visualization techniques made it convenient to identify critical paths through the logic model. Implications for both program evaluation and program planning were discussed.     

Enhancing program review for evaluation and management

Program review has not received the attention it warrants as a program evaluation tool despite its wide use for evaluation and management purposes. The use of the program review will probably endure on the strength of its face validity and irrespective of other developments in the field of program evaluation. Evaluators should realize this and, accordingly, attempt to improve its effectiveness. This paper presents one organization's approach to achieving this objective through the explication of development principles, implementation guidelines and review items. This paper also discusses benefits that can be expected from a systematic development of this tool and presents various research directions and potentials in this area.     

A comparison of linear and systems thinking approaches for program evaluation illustrated using the Indiana Interdisciplinary GK-12

Logic models are based on linear relationships between program resources, activities, and outcomes, and have been used widely to support both program development and evaluation. While useful in describing some programs, the linear nature of the logic model makes it difficult to capture the complex relationships within larger, multifaceted programs. Causal loop diagrams based on a systems thinking approach can better capture a multidimensional, layered program model while providing a more complete understanding of the relationship between program elements, which enables evaluators to examine influences and dependencies between and within program components. Few studies describe how to conceptualize and apply systems models for educational program evaluation. The goal of this paper is to use our NSF-funded, Interdisciplinary GK-12 project: Bringing Authentic Problem Solving in STEM to Rural Middle Schools to illustrate a systems thinking approach to model a complex educational program to aid in evaluation. GK-12 pairs eight teachers with eight STEM doctoral fellows per program year to implement curricula in middle schools. We demonstrate how systems thinking provides added value by modeling the participant groups, instruments, outcomes, and other factors in ways that enhance the interpretation of quantitative and qualitative data. Limitations of the model include added complexity. Implications include better understanding of interactions and outcomes and analyses reflecting interacting or conflicting variables.     

Implementation and process evaluation of a school-based drug abuse prevention program: Project Towards No Drug Abuse

The present study provides an implementation, process, and immediate outcomes evaluation of the classroom component of Project Towards No Drug Abuse (TND). This project involves development and evaluation of a school-based drug abuse prevention curriculum for continuation high school youth, who are at relatively high risk for drug abuse. Three randomized conditions were evaluated: a standard care, classroom only, and classroom plus school-as-community. The latter condition was an enhanced school-based condition which involved outside-of-classroom meetings and activities. Implementation was high in both program conditions even though this was a higher risk context. Process evaluation data were favorable and did not vary between the two program conditions. Immediate outcomes data (knowledge) was higher in the two program conditions than in the standard care condition. Regarding the classroom program, addition of extra-classroom activities does not appear to alter the quality of delivery of the program.     

Evaluation of a cross-cultural training program for Pakistani educators: Lessons learned and implications for program planning

In this paper, we share the results of a summative evaluation of PEILI, a US-based adult professional development/training program for secondary school Pakistani teachers. The evaluation was guided by the theories of cultural competence (American Psychological Association, 2003, Bamberger, 1999, Wadsworth, 2001) and established frameworks for the evaluation of professional development/training and instructional design (Bennett, 1975, Guskey, 2002, King, 2014, Kirkpatrick, 1967). The explicit and implicit stakeholder assumptions about the connections between program resources, activities, outputs, and outcomes are described. Participant knowledge and skills were measured via scores on a pre/posttest of professional knowledge, and a standards-based performance assessment rubric. In addition to measuring short-term program outcomes, we also sought to incorporate theory-driven thinking into the evaluation design. Hence, we examined participant self-efficacy and access to social capital, two evidenced-based determinants or “levers” that theoretically explain the transformative space between an intervention and its outcomes (Chen, 2012). Data about program determinants were collected and analyzed through a pre/posttest of self-efficacy and social network analysis. Key evaluation findings include participant acquisition of new instructional skills, increased self-efficacy, and the formation of a nascent professional support network. Lessons learned and implications for the design and evaluation of cross-cultural teacher professional development programs are discussed.     

The unique character of involvement in multi-site evaluation settings

As the number of large federal programs increases, so, too, does the need for a more complete understanding of how to conduct evaluations of such complex programs. The research literature has documented the benefits of stakeholder participation in smaller-scale program evaluations. However, given the scope and diversity of projects in multi-site program evaluations, traditional notions of participatory evaluation do not apply. The purpose of this research is to determine the ways in which stakeholders are involved in large-scale, multi-site STEM evaluations. This article describes the findings from a survey of 313 program leaders and evaluators and from follow-up interviews with 12 of these individuals.Findings from this study indicate that attendance at meetings and conferences, planning discussions within the project related to use of the program evaluation, and participation in data collection should be added to the list of activities that foster feelings of evaluation involvement among stakeholders. In addition, perceptions of involvement may vary according to breadth or depth of evaluation activities, but not always both. Overall, this study suggests that despite the contextual challenges of large, multi-site evaluations, it is feasible to build feelings of involvement among stakeholders.     

Leverage and evaluation effectiveness

Weaknesses in evaluations often can be traced to structural limitations in the positions of evaluation researchers. Conventional human relations techniques often are an insufficient basis for securing strong support for evaluation research. Strategies for increasing evaluation research leverage are reviewed. Alignment of evaluation research with regulatory bodies with authority to suspend public program expenditures is advocated. Several likely obstacles in the development of the regulatory evaluation model are anticipated and addressed.     

Agent-based modeling as a tool for program design and evaluation

Recently, systems thinking and systems science approaches have gained popularity in the field of evaluation; however, there has been relatively little exploration of how evaluators could use quantitative tools to assist in the implementation of systems approaches therein. The purpose of this paper is to explore potential uses of one such quantitative tool, agent-based modeling, in evaluation practice. To this end, we define agent-based modeling and offer potential uses for it in typical evaluation activities, including: engaging stakeholders, selecting an intervention, modeling program theory, setting performance targets, and interpreting evaluation results. We provide demonstrative examples from published agent-based modeling efforts both inside and outside the field of evaluation for each of the evaluative activities discussed. We further describe potential pitfalls of this tool and offer cautions for evaluators who may chose to implement it in their practice. Finally, the article concludes with a discussion of the future of agent-based modeling in evaluation practice and a call for more formal exploration of this tool as well as other approaches to simulation modeling in the field.     

Theory and method in treatment evaluation

We outline a conceptual model of evaluation research that has both guided and flowed from our research on the process and outcome of treatment for alcohol abuse and for depression. The conceptual model encompasses three basic processes: the process of treatment selection, the process of treatment, and the process underlying the onset and course of the disorder or problem being treated. A theory guiding a program evaluation can focus on one or more of these processes, and findings from evaluations can contribute to the development of better theories of these processes. We discuss how theories can guide both experimental and naturalistic evaluations, as well as aid in the estimation of treatment effects with either type of design. We then consider the role of theory and method in exploring patient-treatment matching. Finally, we discuss the implications of theory-guidance for the utility of evaluations—particularly the utility for evaluation researchers.     

Updating the FORECAST formative evaluation approach and some implications for ameliorating theory failure,implementation failure,and evaluation failure

Historically, there has been considerable variability in how formative evaluation has been conceptualized and practiced. FORmative Evaluation Consultation And Systems Technique (FORECAST) is a formative evaluation approach that develops a set of models and processes that can be used across settings and times, while allowing for local adaptations and innovations. FORECAST integrates specific models and tools to improve limitations in program theory, implementation, and evaluation. In the period since its initial use in a federally funded community prevention project in the early 1990s, evaluators have incorporated important formative evaluation innovations into FORECAST, including the integration of feedback loops and proximal outcome evaluation. In addition, FORECAST has been applied in a randomized community research trial. In this article, we describe updates to FORECAST and the implications of FORECAST for ameliorating failures in program theory, implementation, and evaluation.     

Experimental and contextual models of program evaluation

Evaluation research is one of the most rapidly evolving fields of applied behavioral science. As demand for program assessment has increased, the number of alternative evaluation approaches has also grown. As a result, everyday practitioners have often lacked sufficient guidelines for the choice of appropriate evaluation strategies.The present paper articulates an underlying epistemological distinction between (a) experimental evaluation models which simplify program realities in generalizable analyses of discrete causes and effects, and (b) contextual evaluation models which holistically examine particular program operations. These two evaluation approaches are directed at different purposes and are applicable to different program settings. A topology of program characteristics (breadth of goals, scope of treatment, specificity of results, and clarity of theory) is developed and linked to the appropriateness of experimental and contextual evaluation.     

Building evaluation capacity: definitional and practical implications from an Australian case study

Internationally, evaluation capacity-building activities have mushroomed as demands have increased for government-funded programs to demonstrate that they are effective and efficient. Despite this, there is a lack of clarity in the way in which evaluation capacity-building is defined and conceptualized. The current paper presents a case study of a national evaluation capacity building exercise that we are in the midst of conducting in Australia, and discusses the findings in relation to definitional, conceptual and practical issues. Specifically, we describe an evaluation capacity building exercise involving over 100 mental health projects, detailing the methods that we employ, some of the challenges that we have faced, and the benefits we feel we are achieving. Our key message is that definitions of evaluation capacity-building should not only make reference to equipping organizations to routinely conduct evaluations, but should also stress the varied uses to which evaluation findings can be put. In addition, such definitions should acknowledge some of the valuable by-products of evaluation capacity building activities, such as the development of shared understandings of the program or project being evaluated.     

Evaluation plan of the 6for6 research skills program for rural and remote physicians

Overwhelming issues and barriers often prevent rural and remote physicians (RRPs) from pursuing the many socially accountable research questions they encounter on a daily basis. Although research training programs can empower RRPs to rise to these challenges, there is a lack of evidence on how they should be developed and refined. At Memorial University, a faculty development program (FDP) called 6for6 has been helping RRPs surmount their research quagmires and engage in scholarship since 2014. After an initial three-year (2014-17) pilot, we prepared a detailed plan to evaluate the 6for6 research FDP for RRPs and inform future years of delivery. Using a modified Delphi method and participatory action model a group of program team members, stakeholders and evaluation experts developed an evaluation plan including a logic model and an evaluation matrix addressing five key themes. To our knowledge, this is the first evaluation plan for a research-focused FDP targeting RRPs. While this plan was developed specifically for the 6for6 FDP, our approach to its development may be useful to any institution interested in evaluating an FDP with limited resources.