首页 | 本学科首页   官方微博 | 高级检索  
相似文献
 共查询到20条相似文献,搜索用时 203 毫秒
1.
The need for conducting evaluations which reflect of the influence of context on complex programs is increasingly recognized in the field of evaluation. Better data visualization techniques for connecting context with program evaluation data are needed. We share our experience developing a mixed methods timeline to visualize complexity and context with evaluation data. Mixed methods timelines provide a meaningful way to show change over time in both a visually stimulating and accessible format for evaluation audiences. This paper provides an innovative example of using mixed methods timelines to integrate evaluation data with key program activities and milestones, while also showing internal and external contextual influences in one cohesive visual. We present methods and best practices for collecting contextual data and for incorporating a variety of data sources into such a visual. We discuss several strategies to collect and organize context related data including: qualitative interviews, program materials, narrative reports, and member checking with stakeholders and staff. Gathering multiple perspectives is essential to better capture the multi-layered elements of program activities and context.  相似文献   

2.
This paper explores avenues for navigating evaluation design challenges posed by complex social programs (CSPs) and their environments when conducting studies that call for generalizable, causal inferences on the intervention’s effectiveness. A definition is provided of a CSP drawing on examples from different fields, and an evaluation case is analyzed in depth to derive seven (7) major sources of complexity that typify CSPs, threatening assumptions of textbook-recommended experimental designs for performing impact evaluations. Theoretically-supported, alternative methodological strategies are discussed to navigate assumptions and counter the design challenges posed by the complex configurations and ecology of CSPs. Specific recommendations include: sequential refinement of the evaluation design through systems thinking, systems-informed logic modeling; and use of extended term, mixed methods (ETMM) approaches with exploratory and confirmatory phases of the evaluation. In the proposed approach, logic models are refined through direct induction and interactions with stakeholders. To better guide assumption evaluation, question-framing, and selection of appropriate methodological strategies, a multiphase evaluation design is recommended.  相似文献   

3.
4.
The article presents a mixed-methods evaluation of regional libraries in Namibia, which incorporates three perspectives: the patron perspective (library users), the library perspective (library staff, management, and related officials), and the external perspective (including evaluators and monitoring data). Seven data collection methods were used: patron surveys, patron panel studies, focus group discussions, key informant and staff interviews, secondary data analysis, media analysis, and observations. The goal of the evaluation was to assess library performance for both formative and summative purposes by addressing evaluation questions on areas such as library services, use, and operations. Building upon the literature review of how mixed-methods approaches can contribute to library evaluation, the aim of this article is to show how a mixed-methods evaluation can be designed to examine multi-faceted library performance and to illustrate how the evaluation design allows information complementarity and can be utilized to present diverse viewpoints of the above three perspectives. The evaluation design, analysis process, and lessons learned from this study may be useful to evaluators engaged in evaluation of public services or programs (including public libraries) that examine multiple aspects of service performance and involve a variety of stakeholders.  相似文献   

5.
Latent growth modeling (LGM) has emerged as a flexible analytic technique for modeling change over time because it can describe developmental processes at both the inter- and intra-individual levels. The LGM method can also provide a means for testing the contribution of other variables in order to explain variability in growth trajectories. This paper didactically illustrates the use of LGM as an analytical tool in program evaluation. Specifically, a hypothetical evaluation of a high school drug prevention program was used to demonstrate: (a) how LGM can be used to assess the longitudinal impact of a prevention program by comparing treatment and control populations with respect to individual differences in initial status and in rate of change; and (b) how predictors of initial status (post-intervention) and growth selected on the basis of a particular program theory can be incorporated in the model to explain program impact. Some advantages and limitations of using LGM in program evaluation are highlighted.  相似文献   

6.
Multi-sectoral programs that involve stakeholders in agriculture, nutrition and health care are essential for responding to nutrition problems such as vitamin A deficiency among pregnant and lactating women and their infants in many poor areas of lower income countries. Yet planning such multi-sectoral programs and designing appropriate evaluations, to respond to different disciplinary cultures of evidence, remain a challenge. We describe the context, program development process, and evaluation design of the Mama SASHA project (Sweetpotato Action for Security and Health in Africa) which promoted production and consumption of a bio-fortified, orange-fleshed sweetpotato (OFSP). In planning the program we drew upon information from needs assessments, stakeholder consultations, and a first round of the implementation evaluation of a pilot project. The multi-disciplinary team worked with partner organizations to develop a program theory of change and an impact pathway which identified aspects of the program that would be monitored and established evaluation methods. Responding to the growing demand for greater rigour in impact evaluations, we carried out quasi-experimental allocation by health facility catchment area, repeat village surveys for assessment of change in intervention and control areas, and longitudinal tracking of individual mother-child pairs. Mid-course corrections in program implementation were informed by program monitoring, regular feedback from implementers and partners’ meetings. To assess economic efficiency and provide evidence for scaling we collected data on resources used and project expenses. Managing the multi-sectoral program and the mixed methods evaluation involved bargaining and trade-offs that were deemed essential to respond to the array of stakeholders, program funders and disciplines involved.  相似文献   

7.
The article addresses the relevance and need for complex mixed methods designs when developing standardized measurement instruments for research in strictly hierarchical institutions. Using the evaluation of courses for officer’s training in the army service of the German Bundeswehr, we demonstrate how a suitable questionnaire can be constructed by conducting an extensive qualitative preliminary study and using mixed methods for an iterative validation process. In addition, the specific nature of both the research field and evaluation studies are considered when addressing the problems of compliance with quality criteria of empirical social research in terms of field access and sample production.  相似文献   

8.
This paper discusses the development of a mixed methods approach to analyse research funding. Research policy has taken on an increasingly prominent role in the broader political scene, where research is seen as a critical factor in maintaining and improving growth, welfare and international competitiveness. This has motivated growing emphasis on the impacts of science funding, and how funding can best be designed to promote socio-economic progress. Meeting these demands for impact assessment involves a number of complex issues that are difficult to fully address in a single study or in the design of a single methodology. However, they point to some general principles that can be explored in methodological design. We draw on a recent evaluation of the impacts of research grant funding, discussing both key issues in developing a methodology for the analysis and subsequent results. The case of research grant funding, involving a complex mix of direct and intermediate effects that contribute to the overall impact of funding on research performance, illustrates the value of a mixed methods approach to provide a more robust and complete analysis of policy impacts. Reflections on the strengths and weaknesses of the methodology are used to examine refinements for future work.  相似文献   

9.
Evaluators are challenged to keep pace with the vast array of Veteran support programs operating in the United States, resulting in a situation in which many programs lack any evidence of impact. Due to this lack of evidence, there is no efficient way to suggest which programs are most effective in helping Veterans in need of support. One potential solution to this dilemma is to reconceptualize program evaluation, by moving away from evaluating programs individually to evaluating what is common across programs. The Common Components Analysis (CCA) is one such technique that aggregates findings from programs that have undergone rigorous evaluation at the level of program components (e.g., content, process, barrier reduction). Given that many Veteran programs lack outcome evidence from rigorous studies, an adaptation to CCA is needed. This report examines cross-sectional data from a pilot study using an adapted CCA across five domains of well-being (i.e., employment, education, legal/financial/housing, mental/physical health, and social/personal relationships). The purpose of this preliminary study is to determine the feasibility of eliciting program nominations and program components from Veterans via an online survey. When coupled with a longitudinal research design, this adaptation to CCA will allow for stronger causal claims about the expected impact of different program components within and across a variety of domains.  相似文献   

10.
Multi-site evaluations, particularly of federally funded service programs, pose a special set of challenges for program evaluation. Not only are there contextual differences related to project location, there are often relatively few programmatic requirements, which results in variations in program models, target populations and services. The Jail Diversion and Trauma Recovery–Priority to Veterans (JDTR) National Cross-Site Evaluation was tasked with conducting a multi-site evaluation of thirteen grantee programs that varied along multiple domains. This article describes the use of a mixed methods evaluation design to understand the jail diversion programs and client outcomes for veterans with trauma, mental health and/or substance use problems. We discuss the challenges encountered in evaluating diverse programs, the benefits of the evaluation in the face of these challenges, and offer lessons learned for other evaluators undertaking this type of evaluation.  相似文献   

11.
Evaluators often neglect to question whether a program has been appropriately designed for a new context prior to a feasibility study’s commencement. In this paper, we document the results of a case study that closely examined context when determining the feasibility of implementing a FoodShare outreach program in rural northern Wisconsin. Using community-based participatory mixed methods, we examine how stakeholder engagement led to both study refinement and a more comprehensive understanding of community food security based on contextual factors and systems thinking. Lessons learned when conducting feasibility studies from this perspective are provided to improve evaluation practice.  相似文献   

12.
Poor diet and undernutrition are common among children living in Bangladesh. To promote appropriate complementary feeding of young children, an economic development (ED) program involving income-generating asset transfer was implemented alongside a social and behavior change (SBC) program. This paper introduces a collaborative monitoring and evaluation (M&E) system in which diverse collaborators (“research group”, “implementation team”, and “coordinators”) facilitate M&E data acquisition by leveraging their comparative advantages. The implementation team built a monitoring system to track the ED (n = 2960) and SBC participants (n=∼10,000) over 12 months. Based on the baseline design and the monitoring records, the collaborators planned an impact evaluation introducing a quasi-experimental design using two cross-sectional surveys and a prospective cohort survey of child feeding and nutritional status. Using various data sources generated from the M&E system, the collaborators will also reveal the program impact pathway through which each intervention component is delivered, received, and utilized alongside the context-specific facilitators and barriers, including the programs’ uptake. The collaborative M&E system enables the sharing of program goals, strengthens collaborators' commitment to the program, and extends the understanding of the program's progress and evaluation activities.  相似文献   

13.
Evolutionary theory, developmental systems theory, and evolutionary epistemology provide deep theoretical foundations for understanding programs, their development over time, and the role of evaluation. This paper relates core concepts from these powerful bodies of theory to program evaluation. Evolutionary Evaluation is operationalized in terms of program and evaluation evolutionary phases, which are in turn aligned with multiple types of validity. The model of Evolutionary Evaluation incorporates Chen's conceptualization of bottom-up versus top-down program development. The resulting framework has important implications for many program management and evaluation issues. The paper illustrates how an Evolutionary Evaluation perspective can illuminate important controversies in evaluation using the example of the appropriate role of randomized controlled trials that encourages a rethinking of “evidence-based programs”. From an Evolutionary Evaluation perspective, prevailing interpretations of rigor and mandates for evidence-based programs pose significant challenges to program evolution. This perspective also illuminates the consequences of misalignment between program and evaluation phases; the importance of supporting both researcher-derived and practitioner-derived programs; and the need for variation and evolutionary phase diversity within portfolios of programs.  相似文献   

14.
This article examines recent progress at assigning monetary values to what are normally considered “hard to quantify” benefits of transportation projects. It focuses on three types of impacts – environmental quality, health and wider economic impacts – to examine how transportation project evaluation methods have evolved in recent years and how they compare to methods used for evaluation of non-transportation programs. Examples of recent practice are provided to show how transport agencies are continuing to refine performance measures to include broader impacts in project evaluation. A classification is provided to distinguish direct traveler effects from indirect effects on non-travelers, a step important to maximize coverage and minimize double-counting of impacts. For each type of impact, the paper discusses the range of variation in monetized values and shows that the variation is due less to imprecision in measurement than to fundamental issues about whether to use damage compensation, impact avoidance costs, stated preferences or behavioral valuation perspectives to define those values. Case studies as diverse as Australian roads, Wisconsin energy programs and Appalachian economic development programs are used to show how common methods are evolving among transport and non-transport agencies to improve impact measurement and its use in project evaluation.  相似文献   

15.
Monitoring and evaluation (M&E) of gender-based violence (GBV) programs is challenging in humanitarian settings. To address these challenges, we used SenseMaker® as a mixed methods M&E tool for GBV services and programs in Lebanon. Over a three-month period in 2018, a total of 198 self-interpreted stories were collected from women and girls accessing GBV programs from six service providers across five locations. The resultant mixed-methods analysis provided holistic and nuanced insights on how perceived benefits differed by type of GBV program, how motivations for accessing programs differed by location, and how feelings while accessing programs differed by participant nationality. SenseMaker reinforced the intersectionality between events leading up to the accessed services, the experiences of accessing the services, and subsequent outcomes as a result of having accessed the services, helping to contextualize the findings within the broader experiences of participating women and girls. Limited literacy and technology skills among participants proved to be a challenge and future work should investigate how technology might facilitate use of the tool among participants with lower literacy and technology skills in addition to exploring the feasibility and added value of SenseMaker as an M&E tool in acute humanitarian settings.  相似文献   

16.
In this paper, the author shares with readers his journey towards creating and evaluating a mixed‐methods research course in education using a course design framework to guide the discussion. He discusses important questions such as what goes into a mixed‐methods syllabus, what resources are available to instructors and students, what learning goals we might set for our students and how we might assess whether students achieve these goals in the context of teaching his own course for the first time. There is a need for dialogue among the ‘first generation’ of instructors of mixed‐methods research courses about how we teach these courses – readers are encouraged to continue this dialogue about teaching mixed‐methods research courses by following up on and extending the ideas in this paper.  相似文献   

17.
This paper examines the application of Complexity Theory constructs to a research-for-development program evaluation and presents an overview of the implications and promising approaches for evaluating complex programs. We discuss lessons learned from an evaluation completed for the International Development Research Centre’s Food, Environment and Health (FEH) program, which investigated the integration and outcomes of five strategic program priorities: partnerships, southern leadership, gender and equity, scale, and environmental sustainability. We present interpretations from a secondary, thematic content analysis that categorized evaluation findings across four complexity constructs: emergence, unpredictability, contradiction and self-organization. Viewing the evaluation through these constructs surfaced some important features of the FEH program to date, specifically its evolving approach, adaptiveness to emergent issues, non-linear outcomes, and self-organizing agents, which had several implications for the evaluative process. We conclude that the most appropriate evaluation designs for complex funding programs are participatory (to explore all stakeholders' influence), adaptive (to capture the unexpected) and assess external contexts. The application of complexity constructs may be useful for evaluators to gain a deeper understanding of how program contexts change in the face of complexity and why some evaluation methods work more effectively than others.  相似文献   

18.
19.
This article describes two stages of the Juvenile Justice Educational Enhancement Program's pre-, post-, and longitudinal evaluation research. Pilot studies were used to explore how to design statewide research of pre- and postassessment scores and community reintegration outcomes. Preliminary findings suggest that higher performing educational programs produce greater educational gains as measured by academic achievement tests, credits earned, and pupil progression rates. The findings also indicate that these programs have more students returning to school and lower recidivism rates. Building on the pilot studies, refinements were made to the research designs to enable more comprehensive statewide evaluation. Current research includes collection of pre- and postassessment scores from official sources on approximately 16,000 juvenile justice youths. In addition, a research design has been developed to examine program effectiveness by measuring community reintegration variables. Multiple data sources, including official and self-reported data on family, school, employment, and subsequent crime involvement, will be used in the longitudinal study.  相似文献   

20.
Although the Drug-Free Schools and Communities Act of 1986 and the 1989 Amendments to the Act require states to evaluate their drug-education programs, no guidelines for conducting these evaluations have been produced, and little has been reported on how the states are conducting such evaluations. In this article, the results of a telephone survey on current state-level efforts to evaluate school programs funded under the Act are reported. Some states report studies of the implementation of the program and some report drug- and alcohol-use surveys. Together, these two types of evaluation efforts form the foundation of an approach for conducting evaluations under the Act. Reasons are presented why experimental and quasi-experimental designs might be inappropriate and impractical for the evaluations, and an evaluation approach linking program implementation findings and drug- and alcohol-use survey results is suggested.  相似文献   

设为首页 | 免责声明 | 关于勤云 | 加入收藏

Copyright©北京勤云科技发展有限公司  京ICP备09084417号