Open systems evaluation and the logic model: Program planning and evaluation tools

This paper describes two case management projects and efforts to evaluate outcomes. Lessons learned from the first effort suggest that a tool referred to as the logic model and a particular approach to evaluation called the “open systems” model might be useful to planners and program evaluators. The logic model provides a means of presenting conditions a program is intended to address, activities that constitute a program, short term outcomes resulting from program activities and long term impacts. The open systems perspective emphasizes evaluation as a tool to be used to achieve program objectives as opposed to establishing cause and effect relationships. This paper also describes how the logic model and open systems evaluation were used to facilitate the development of an evaluation plan for a project designed to assist homeless, substance abusing, pregnant women.     

State Government Funding of Research on Science and Technology: A Report on Michigan

Abstract

The question of this paper is: what factors explain the decisions of state governments to fund research on science and technology? The theoretical problem underlying the research question is patronage in the classical sense, that is, the support by one person or group for the activities of another person or group. Three models of explanation are proposed: “budget incrementalism,” a “rational” model, and a power model. The models are evaluated with information from a case study of a state government program in Michigan. The rational model best explains the state's selection of some proposals out of all the proposals submitted by each state university for funding. The power model best explains the overall allocation of program funds among Michigan's public universities. The budget incrementalism model helps explain the considerable stability from year to year in funding both for the entire program and for each university's share.     

Context matters: Using mixed methods timelines to provide an accessible and integrated visual for complex program evaluation data

The need for conducting evaluations which reflect of the influence of context on complex programs is increasingly recognized in the field of evaluation. Better data visualization techniques for connecting context with program evaluation data are needed. We share our experience developing a mixed methods timeline to visualize complexity and context with evaluation data. Mixed methods timelines provide a meaningful way to show change over time in both a visually stimulating and accessible format for evaluation audiences. This paper provides an innovative example of using mixed methods timelines to integrate evaluation data with key program activities and milestones, while also showing internal and external contextual influences in one cohesive visual. We present methods and best practices for collecting contextual data and for incorporating a variety of data sources into such a visual. We discuss several strategies to collect and organize context related data including: qualitative interviews, program materials, narrative reports, and member checking with stakeholders and staff. Gathering multiple perspectives is essential to better capture the multi-layered elements of program activities and context.     

Evaluating health services research capacity building programs: Implications for health services and human resource development

Evaluation of research capacity building programs presents challenges, including defining “research capacity building” and measuring it. The present study evaluates infrastructure capacity-building strategies and barriers and facilitators of two federally funded health-related research capacity building programs within the United States. We developed a conceptual framework and abstracted 223 data elements from existing project files. We administered questionnaires to 17 principal investigators and conducted 2-h interviews of 15 principal investigators. Finally, we conducted one-and-a-half-day site visits with five projects. We aggregated data and analysis results across the two programs. Impacts and strategies were also analyzed as to whether host institutions had little or no existing research infrastructure (“seed projects”) or had well-developed research infrastructure (“fertilizer projects”). Projects were associated with increases in publication and presentation rates by Principal Investigators and staff, increases in research funding, changes in organizational research culture, and positive impacts on the community. Project characteristics and activities associated with each of these outcomes were identified. It was also noted that the correlations between these outcomes were small and non-significant.     

Using a systems orientation and foundational theory to enhance theory-driven human service program evaluations

This paper offers a framework for using a systems orientation and “foundational theory” to enhance theory-driven evaluations and logic models. The framework guides the process of identifying and explaining operative relationships and perspectives within human service program systems. Self-Determination Theory exemplifies how a foundational theory can be used to support the framework in a wide range of program evaluations. Two examples illustrate how applications of the framework have improved the evaluators’ abilities to observe and explain program effect. In both exemplars improvements involved addressing and organizing into a single logic model heretofore seemingly disparate evaluation issues regarding valuing (by whose values); the role of organizational and program context; and evaluation anxiety and utilization.     

A comparison of linear and systems thinking approaches for program evaluation illustrated using the Indiana Interdisciplinary GK-12

Logic models are based on linear relationships between program resources, activities, and outcomes, and have been used widely to support both program development and evaluation. While useful in describing some programs, the linear nature of the logic model makes it difficult to capture the complex relationships within larger, multifaceted programs. Causal loop diagrams based on a systems thinking approach can better capture a multidimensional, layered program model while providing a more complete understanding of the relationship between program elements, which enables evaluators to examine influences and dependencies between and within program components. Few studies describe how to conceptualize and apply systems models for educational program evaluation. The goal of this paper is to use our NSF-funded, Interdisciplinary GK-12 project: Bringing Authentic Problem Solving in STEM to Rural Middle Schools to illustrate a systems thinking approach to model a complex educational program to aid in evaluation. GK-12 pairs eight teachers with eight STEM doctoral fellows per program year to implement curricula in middle schools. We demonstrate how systems thinking provides added value by modeling the participant groups, instruments, outcomes, and other factors in ways that enhance the interpretation of quantitative and qualitative data. Limitations of the model include added complexity. Implications include better understanding of interactions and outcomes and analyses reflecting interacting or conflicting variables.     

Design of a collaborative monitoring and evaluation system for a community-based nutrition project in rural Bangladesh

Poor diet and undernutrition are common among children living in Bangladesh. To promote appropriate complementary feeding of young children, an economic development (ED) program involving income-generating asset transfer was implemented alongside a social and behavior change (SBC) program. This paper introduces a collaborative monitoring and evaluation (M&E) system in which diverse collaborators (“research group”, “implementation team”, and “coordinators”) facilitate M&E data acquisition by leveraging their comparative advantages. The implementation team built a monitoring system to track the ED (n = 2960) and SBC participants (n=∼10,000) over 12 months. Based on the baseline design and the monitoring records, the collaborators planned an impact evaluation introducing a quasi-experimental design using two cross-sectional surveys and a prospective cohort survey of child feeding and nutritional status. Using various data sources generated from the M&E system, the collaborators will also reveal the program impact pathway through which each intervention component is delivered, received, and utilized alongside the context-specific facilitators and barriers, including the programs’ uptake. The collaborative M&E system enables the sharing of program goals, strengthens collaborators' commitment to the program, and extends the understanding of the program's progress and evaluation activities.     

Striking a Balance between Program-Specific and Portfolio-Level Evaluation: Lessons Learned from a Multi-Site Evaluation on the Texas-Mexico Border

Funders are increasingly making strategic investments across multiple grantees, aiming for their portfolio to improve targeted outcomes in a specific issue area. To this end, funders might use multi-site evaluation (MSE) approaches to examine the impact of their collective investments. However, it is important to recognize that each program—and its own program evaluation—must be tailored to its setting, population, and local context to best meet the needs of its target population. Therefore, multi-site evaluations need to account for this complexity. This paper describes the Sí Texas project, a large initiative of eight grantees implementing different integrated behavioral health models to improve physical and mental health outcomes along the Texas-Mexico border. With over 4,200 MSE study participants, the evaluation for Sí Texas used a partnership-centered approach to both enhance the evidence base and build local organizational capacity. This paper describes this approach, the process of tailoring evaluation practices to the grantees’ context, and the challenge of balancing consistency at the grantee-level for the portfolio multi-site evaluation. Successes, challenges, and lessons learned related to study design, data collection, grantee partnership, and capacity building are discussed.     

From plan to action: Adapting evaluation to serve the developmental needs of a newly-funded multidisciplinary research center

The Southeastern Coastal Center for Agricultural Health and Safety (SCCAHS) is one of many newly-funded federal research centers, housing five multidisciplinary research projects and seven pilot projects, and serving a multi-state region. In the early stages of such a complex project, with multiple teams separated by geography and disciplines, the evaluation program has been integral in connecting internal and external stakeholders at the center and project levels. We used a developmental evaluation (DE) framework to respond to the complex political environment surrounding agricultural health and safety in the southeast; to engage external stakeholders in guiding the center’s research and outreach trajectories; to support center research teams in a co-creation process to develop logic models and tailored indicators; and to provide timely and feedback within the center to address communications gaps identified by the evaluation program. By using DE principles to shape monitoring and evaluation approaches, our evaluation program has adapted to the dynamic circumstances presented as our center’s progress has been translated from a plan in a grant proposal to implementation.     

What Works at Work? Toward an Integrative Model Examining Workplace Campaign Strategies

Many US employees are regularly asked to give charitable donations through work. The techniques used to solicit workplace donations vary. Drawing on a nationally representative survey, the study used a sample of donor responses to examine the effectiveness of several widely used campaign strategies: donor choice, company matching, public recognition, and solicitation support. The theoretical framework built on workplace research by Barman (2007) and established charitable giving mechanisms (Bekkers and Wiepking 2011a, 2011b). The research question was, “Do workplace campaign strategies lead employees to participate and to make (larger) donations in the workplace?” The positive outcomes of the strategies, aside from donor choice, were limited, suggesting that tried‐and‐true workplace fundraising strategies warrant additional scrutiny. The findings are meaningful to campaign managers seeking to identify approaches that generate workplace giving. For researchers, the results confirm growing attention to the importance of purpose‐based giving in comparison with community‐based giving.     

Devil is in the details: Using logic models to investigate program process

Theory-based logic models are commonly developed as part of requirements for grant funding. As a tool to communicate complex social programs, theory based logic models are an effective visual communication. However, after initial development, theory based logic models are often abandoned and remain in their initial form despite changes in the program process. This paper examines the potential benefits of committing time and resources to revising the initial theory driven logic model and developing detailed logic models that describe key activities to accurately reflect the program and assist in effective program management. The authors use a funded special education teacher preparation program to exemplify the utility of drill down logic models. The paper concludes with lessons learned from the iterative revision process and suggests how the process can lead to more flexible and calibrated program management.     

Semi-structured interview protocol for constructing logic models

This paper details a semi-structured interview protocol that evaluators can use to develop a logic model of a program's services and outcomes. The protocol presents a series of questions, which evaluators can ask of specific program informants, that are designed to: (1) identify key informants basic background and contextual information, (2) generate logic model elements, (3) model program inputs, activities, outputs, and outcomes, (4) build a rational theory, (5) develop a program theory, (6) prioritize logic model elements, and (7) build a graphical or tabular logic model. The paper will also provide an example of how this approach was used to develop a logic model for a youth mentoring program. It is our hope and belief that with this interview protocol, novice evaluators will be able to generate comprehensive logic models like seasoned professional evaluators.     

Dealing with context in logic model development: Reflections from a realist evaluation of a community health worker programme in Nigeria

Logic models (LMs) have been used in programme evaluation for over four decades. Current debate questions the ability of logic modelling techniques to incorporate contextual factors into logic models. We share experience of developing a logic model within an ongoing realist evaluation which assesses the extent to which, and under what circumstances a community health workers (CHW) programme promotes access to maternity services in Nigeria. The article contributes to logic modelling debate by: i) reflecting on how other scholars captured context during LM development in theory-driven evaluations; and ii) explaining how we explored context during logic model development for realist evaluation of the CHW programme in Nigeria. Data collection methods that informed our logic model development included documents review, email discussions and teleconferences with programme stakeholders and a technical workshop with researchers to clarify programme goals and untangle relationships among programme elements. One of the most important findings is that, rather than being an end in itself, logic model development is an essential step for identifying initial hypotheses for tentative relevant contexts, mechanisms and outcomes (CMOs) and CMO configurations of how programmes produce change. The logic model also informed development of a methodology handbook that is guiding verification and consolidation of underlying programme theories.     

Failing better: The stochastic art of evaluating community-led environmental action programs

This article provides insights into the evaluation of a government-funded action for climate change program. The UK-based program aimed to reduce CO2 emissions and encourage behavioral change through community-led environmental projects. It, thus, employed six community development facilitators, with expertise in environmental issues. These facilitators supported and learnt from 18 community groups over an 18-month period. The paper explores the narratives of the six professional facilitators. These facilitators discuss their experiences of supporting community groups. They also explain their contribution to the wider evaluation of the community-led projects. This paper reflects on the facilitator experience of the program’s outcome-led evaluation process. In turn, it also explores how the groups they supported experienced the process. The facilitator’s narratives reveal that often community-group objectives did not align with predefined outcomes established through theory of change or logic model methodologies, which had been devised in attempt to align to program funder aims. Assisting community action emerges in this inquiry as a stochastic art that requires funder and facilitator willingness to experiment and openness to the possibilities of learning from failure. Drawing on in-depth accounts, the article illustrates that a reflexive, interpretive evaluation approach can enhance learning opportunities and provides funders with more trustworthy representations of community-led initiatives. Yet, it also addresses why such an approach remains marginal within policy circles.     

Democracy in Arts Development 1

Too often in the arts, projects are undertaken without clear‐cut aims and ways of assessing whether these aims are accomplished. With regard to “Art for Social Change” projects, it is not difficult to identify activities that might be labelled “dishonest”, “ill‐conceived” or even “unethical”. It is argued here that it is possible, and necessary, to set clear aims, and also to devise effective assessment strategies. It is essential for artists to embrace evaluation in order to help them improve and better understand their work.

     

Evaluation of a cross-cultural training program for Pakistani educators: Lessons learned and implications for program planning

In this paper, we share the results of a summative evaluation of PEILI, a US-based adult professional development/training program for secondary school Pakistani teachers. The evaluation was guided by the theories of cultural competence (American Psychological Association, 2003, Bamberger, 1999, Wadsworth, 2001) and established frameworks for the evaluation of professional development/training and instructional design (Bennett, 1975, Guskey, 2002, King, 2014, Kirkpatrick, 1967). The explicit and implicit stakeholder assumptions about the connections between program resources, activities, outputs, and outcomes are described. Participant knowledge and skills were measured via scores on a pre/posttest of professional knowledge, and a standards-based performance assessment rubric. In addition to measuring short-term program outcomes, we also sought to incorporate theory-driven thinking into the evaluation design. Hence, we examined participant self-efficacy and access to social capital, two evidenced-based determinants or “levers” that theoretically explain the transformative space between an intervention and its outcomes (Chen, 2012). Data about program determinants were collected and analyzed through a pre/posttest of self-efficacy and social network analysis. Key evaluation findings include participant acquisition of new instructional skills, increased self-efficacy, and the formation of a nascent professional support network. Lessons learned and implications for the design and evaluation of cross-cultural teacher professional development programs are discussed.     

Using client exit interviews to illuminate outcomes in program logic models: a case example

This paper describes how client exit interviews can assist human service administrators and workers to better understand the outcomes their programs are designed to accomplish. Specifically, the qualitative component of a demonstration family literacy program evaluation is used to illustrate how client input can be used to fine-tune the outcomes component of a program's logic model. An analysis of semi-structured exit interviews with 35 clients, who were randomly selected from all 89 served in the first year of the program, resulted in revision to the program's original logic model, creating explicit ‘testable’ pathways to achieving intended outcomes.     

The development of evaluation as a concession

The organization of evaluation activities takes place within the already existing programs and power arrangements of complex organizations. In this context, evaluation is likely to develop along the lines of a licensed concession operating within a particular program area and as an expression of a concession made by complex organizations to the ideal of rational decision-making. In addition, the formation of professional career paths in evaluation may lead to career immobility. The “evaluation as a profession” issue itself could be resolved not by evaluation becoming a profession but by evaluation becoming a specialty of the professional manager.     

Development of a logic model for a service learning,dog walking course for college students

Despite overwhelming evidence of the importance of physical activity, only 20% of US adults meet physical activity recommendations. A subsector of the population vulnerable to inactivity is college students. There is a disconnect between evidence-based physical activity programming and dissemination into college communities, which may be overcome through logic model development and evaluation. The purpose of this paper is to discuss logic model development for a service-learning, dog walking program for undergraduates. Participants were undergraduates (18–25 years). Process evaluations and observational data were used in a content analysis to determine major themes to create the logic model. A set of resources were needed to complete program activities, which led to short- and long-term outcomes. The overarching goal of the program is to improve the overall health status of college students and companion animals. This logic model may have significant public health impact as it can serve as a framework for universities wanting to develop, implement, and evaluate a service-learning, dog walking program for students.