Putting measures of individual well-being to use for ex-ante policy evaluation

Most studies using microsimulation techniques have considered the effect of potential reforms on the income distribution. However, it has become increasingly recognized, both at the academic and political level, that focusing purely on income provides a limited picture of social progress. We illustrate how ex-ante policy evaluation can be performed in terms of richer concepts of individual well-being, such as subjective life satisfaction and equivalent incomes. Our analysis makes use of EUROMOD, the EU-wide tax-benefit microsimulation model, along with 2013 EU-SILC data for Sweden, which for the first time provides information on life satisfaction. Our results show that the effect of potential reforms varies widely depending on the well-being concept used in the evaluation. We discuss the normative questions that are raised by this finding.     

Conceptual Framework for Marriage Education Programs for Stepfamily Couples with Considerations for Socioeconomic Context

Stepfamily couples face unique challenges that put them at higher risk for dissolution than non-stepfamily couples. Risks for stepfamily couples are magnified in the context of lower economic resources, making low-income stepfamily couples a key target population for intervention and for community education programs. Existing programs and services for couples often do not address the unique situations of stepfamilies. This article builds on previous reviews that have offered information on important research themes and assessment of available curricula for stepfamily couples and presents a more comprehensive conceptual model to guide program design and services for stepfamily couples inclusive of specific objectives, as well as conditions and antecedents affecting content and prioritized needs. The goal is to enhance efforts to strengthen more vulnerable couples in complex families.     

New Curriculum Policy Statement: The Problem of Professional Cohesion–3

This paper presents the Community Learning Center Project, a multicultural training program, as a successful model for the acquisition of culturally relevant social work practice competencies. Cultural pluralism is presented as the underlying framework for the project. The design and model of an evaluation study of the project is described and mechanisms for the implementation of such a culturally relevant effort are suggested.     

Ethical problems in program evaluation: Advice for trapped evaluators

This paper argues that ethical problems in program evaluation are increased when conflicting or incompatible evaluation models are applied concurrently. Three models are illustrated: an Amelioration Model, generating better information for a program's own decision makers; an Accountability Model, focusing on public data disclosure and citizen participation in evaluation; and an Advocacy Model, in which the evaluation is designed to advance the program's interests in the competition for resources. Ethical problems inherent in each model, and arising from combining models are presented, along with case examples and discussion. The authors suggest several directions that program evaluators can consider to reduce their vulnerability to ethical problems.     

Evaluating the impact of decision making during construction on transport project outcome

Decisions made during the project construction phase may bear considerable impacts on the success of transport projects and undermine the ex-ante project evaluation. An innovative and holistic approach has been taken to assess and address this issue by (a) examining the decision process and procedure during project construction, through a field survey, (b) assessing the impact of decisions made during construction on respective transport project and, finally, (c) developing a quality monitoring framework model which links decisions made during the project implementation (construction) phase with the ex-ante and ex-post project evaluations. The framework model is proposed as a guiding and support tool for decision makers.     

Implementing the ecological approach in tobacco control programs: results of a case study

The ‘new public health’ has stimulated calls to adopt an ecological approach for action aimed at improving population health. However, the implementation of ecological programs has often remained a challenge. The objective of this study was to identify factors facilitating the implementation of ecological tobacco control programming in two Public Health Departments in Québec (Canada) using Scheirer's model of program implementation as a conceptual framework and an exemplary case study design. Analysis revealed two distinct configurations of environmental, organizational and professional characteristics that were related to equally successful implementations of the framework. Key roles for factors such as provincial ministerial guidelines and financial resources, human resources, and relationships and alliances with other organizations, were highlighted. These results replicate findings from quantitative surveys and allow better understanding of the conditions required for planning ecological programming.     

Program analysis using the client oriented cost outcome system.

The client oriented cost outcome system has been under development in Pennsylvania community and hospital programs since 1972. The system builds upon the behavioral and decision data generated with or on behalf of consumers. Flexibility for local program system design is permitted if the procedures of consumer intake, review, and termination document each consumer's (a) problems, resources, and goals, (b) overall functioning level in their ordinary community, and (c) services intended and rendered as related to (a), above. While the system's primary application is in providing feedback for local program quality assurance and evaluation procedures, aggregation of data permits program planning and evaluation at county, state, and federal levels in terms of (a) client demographic or diagnostic characteristics, and (b) program service characteristics and objectives.     

Assessing local capacity for health intervention

Because of their location within the practice realm, participatory, community-based public health coalitions offer many lessons about implementing and sustaining local interventions. This paper presents a case study of capacity assessment at the local level. Capacity evaluation methods are presented, with emphasis on the theoretical framework used to guide the evaluation. The capacity evaluation framework presented herein was theoretically based and designed to generate practical information to facilitate the adoption of a locally tailored youth obesity prevention program, VERB Summer Scorecard (VSS). Using multiple methods, four aspects of community capacity were assessed, including community, knowledge and skills, resources, and power. Within each category, factors that facilitated or impeded program implementation were distinguished. The evaluation protocol was designed to generate information increasing community capacity to sustain a community-based obesity prevention program. Capacity tables were used as a program-planning tool and as a system for sharing implementation and sustainability requirements with potential adopters. This case study also explores how to use capacity assessment results to empower coalitions to serve as catalysts for development of local programs in other communities.     

A model-guided process evaluation: Office-based prescribing and pharmacy dispensing of methadone

This article presents an exemplar of a model-guided process evaluation that specifies the treatment model, assesses its implementation, monitors the fidelity of the model throughout the project, assesses model exposure and absorption, and helps understand the program's intermediate effects (proximal outcomes) as well as final effects (distal outcomes). The New Mexico study on office-based prescribing and community pharmacy dispensing of methadone is a research demonstration project that phases a small group of female methadone maintenance patients out of methadone clinics and into a program where they will obtain their scheduled doses of methadone at pharmacies that work in collaboration with physicians and a social worker.The patient's methadone treatment will in this way become part of their overall health care. Early detection of problems of implementation (e.g., the omission of program content or the delivery of inaccurate information) enables the researcher to make adjustments before the problems become unmanageable and the integrity of the original research design is compromised. A model-guided process evaluation can critically inform health services research demonstrations designed for enabling continuous, ongoing feedback and improvement of client-related services.     

Planning an integrated agriculture and health program and designing its evaluation: Experience from Western Kenya

Multi-sectoral programs that involve stakeholders in agriculture, nutrition and health care are essential for responding to nutrition problems such as vitamin A deficiency among pregnant and lactating women and their infants in many poor areas of lower income countries. Yet planning such multi-sectoral programs and designing appropriate evaluations, to respond to different disciplinary cultures of evidence, remain a challenge. We describe the context, program development process, and evaluation design of the Mama SASHA project (Sweetpotato Action for Security and Health in Africa) which promoted production and consumption of a bio-fortified, orange-fleshed sweetpotato (OFSP). In planning the program we drew upon information from needs assessments, stakeholder consultations, and a first round of the implementation evaluation of a pilot project. The multi-disciplinary team worked with partner organizations to develop a program theory of change and an impact pathway which identified aspects of the program that would be monitored and established evaluation methods. Responding to the growing demand for greater rigour in impact evaluations, we carried out quasi-experimental allocation by health facility catchment area, repeat village surveys for assessment of change in intervention and control areas, and longitudinal tracking of individual mother-child pairs. Mid-course corrections in program implementation were informed by program monitoring, regular feedback from implementers and partners’ meetings. To assess economic efficiency and provide evidence for scaling we collected data on resources used and project expenses. Managing the multi-sectoral program and the mixed methods evaluation involved bargaining and trade-offs that were deemed essential to respond to the array of stakeholders, program funders and disciplines involved.     

Designing useful evaluations: The stakeholder survey

The failure of program evaluations to provide policy decision-makers with useful information is a familiar complaint. This paper discusses an approach towards designing and implementing more useful evaluations — the stakeholder survey. Stakeholders are individuals, or groups, who have a direct interest in the program being evaluated. The central premise of the paper is that the information needs of stakeholders should be incorporated directly into the formulation of evaluation objectives, and the design of the evaluation to achieve those objectives. The paper presents a rationale for the stakeholder survey, describes its major components, and reports on the application of the approach in a large program evaluation. Key issues involved in applying the approach are also considered. The evidence from this study suggests that such surveys are not only feasible, but also of considerable value to the evaluation team.     

Evaluating utility conservation programs: The BPA plan

This paper summarizes a detailed plan for evaluation of the residential energy conservation programs offered by the Bonneville Power Administration (BPA). The major purpose of the evaluation is to determine the energy savings attributable to the programs and to identify differences between program participants and nonparticipants. The focus of this paper is on the sampling issues associated with reliable estimates of energy savings. The methodology attempts to correct for the self-selection bias likely to occur in conservation program evaluation by using intentional nonrandom sampling. In addition, the paper suggests a variety of analytical approaches to analyzing the data with respect to energy savings.The proposed design, using four different groups to estimate program effects, is a comprehensive endeavor. This is appropriate for BPA because of the substantial investment it is making in conservation. However, the design is flexible and allows for use offewer groups to reduce evaluation costs.     

The theory-driven approach to validity

Current approaches for dealing with validity tend to focus on one type of validity and play down the importance of others. For example, internal validity is treated by Campbell and Stanley as “sine qua non,” whereas, Cronbach judges external validity to be prime importance in research. However, these single validity oriented approaches may achieve one type of validity at the expense of other types of validity.As an alternative, we propose a theory-driven approach to validity. Our central argument is that a model or theory should be formulated in a program evaluation and the modeling process should include the identification of potential threats to validity in research. In this sense, the purpose of applying a technique such as randomization is to strengthen rather than to replace the model or theory. That is, program evaluation should be model or theory-oriented instead of research design oriented. Randomization can be used in any research if it is applicable, but we need rely less on randomization as a safeguard to internal validity if the threats in a research can be adequately identified and controlled.Since the structural relationships among the relevant variables and intervening processes are carefully modeled, the theory-driven approach avoids the pitfalls of black-box evaluation and provides better understanding of the causal mechanisms underlying the relationships between treatment and effects. Unlike randomized experiments which aim only at internal validity, a theory-driven approach provides a general framework to deal with internal, external, construct and statistical conclusion validity.     

Challenges related to the analytical process in realist evaluation and latest developments on the use of NVivo from a realist perspective

ABSTRACT

Realist evaluation (RE) is a research design increasingly used in program evaluation, that aims to explore and understand the influence of context and underlying mechanisms on intervention or program outcomes. Several methodological challenges, however, are associated with this approach. This article summarizes RE key principles and examines some documented challenges and solutions when analyzing RE data, including the development of Context-Mechanism-Outcome configurations. An analytic method using NVivo features is also presented. This method makes it possible to respond to certain analytic difficulties associated with RE by facilitating the identification of patterns and ensuring transparency in the analytical process.     

Evaluation of a cross-cultural training program for Pakistani educators: Lessons learned and implications for program planning

In this paper, we share the results of a summative evaluation of PEILI, a US-based adult professional development/training program for secondary school Pakistani teachers. The evaluation was guided by the theories of cultural competence (American Psychological Association, 2003, Bamberger, 1999, Wadsworth, 2001) and established frameworks for the evaluation of professional development/training and instructional design (Bennett, 1975, Guskey, 2002, King, 2014, Kirkpatrick, 1967). The explicit and implicit stakeholder assumptions about the connections between program resources, activities, outputs, and outcomes are described. Participant knowledge and skills were measured via scores on a pre/posttest of professional knowledge, and a standards-based performance assessment rubric. In addition to measuring short-term program outcomes, we also sought to incorporate theory-driven thinking into the evaluation design. Hence, we examined participant self-efficacy and access to social capital, two evidenced-based determinants or “levers” that theoretically explain the transformative space between an intervention and its outcomes (Chen, 2012). Data about program determinants were collected and analyzed through a pre/posttest of self-efficacy and social network analysis. Key evaluation findings include participant acquisition of new instructional skills, increased self-efficacy, and the formation of a nascent professional support network. Lessons learned and implications for the design and evaluation of cross-cultural teacher professional development programs are discussed.     

Explaining variation in the effects of welfare-to-work programs

Evaluations of government-funded training programs often combine results from similar operations in multiple sites. Findings inevitably vary. It is common to relate site-to-site variations in outcomes to variations in program design, participant characteristics, and the local environment. Frequently, such connections are constructed in a narrative synthesis of multisite results. This article uses findings from the evaluations of California's Greater Avenues for Independence (GAIN) program and the National Evaluation of Welfare-to-Work Strategies (NEWWS) to illustrate why it is important to question the legitimacy of such syntheses. The discussion is carried out using a simple multilevel evaluation model that incorporates models of both individual outcomes within sites and variation in program effects across sites. The results indicate that tempting generalizations about GAIN and NEWWS effects are statistically unjustified but that significant progress might be made in identifying the determinants of program effects in future demonstrations with some changes in evaluation strategy.     

Open systems evaluation and the logic model: Program planning and evaluation tools

This paper describes two case management projects and efforts to evaluate outcomes. Lessons learned from the first effort suggest that a tool referred to as the logic model and a particular approach to evaluation called the “open systems” model might be useful to planners and program evaluators. The logic model provides a means of presenting conditions a program is intended to address, activities that constitute a program, short term outcomes resulting from program activities and long term impacts. The open systems perspective emphasizes evaluation as a tool to be used to achieve program objectives as opposed to establishing cause and effect relationships. This paper also describes how the logic model and open systems evaluation were used to facilitate the development of an evaluation plan for a project designed to assist homeless, substance abusing, pregnant women.     

Educational guidance for functional visual diversity in Nicaragua. Design and evaluation of a teacher-training program

The article exposes the process of design and implementation of a training program for teachers that teach people with Visual Functional Diversity (VFD), taking into account their needs both in training and in the provision of resources in Nicaragua. To verify the effectiveness of the program, a quasi-experimental design was carried out with a non-equivalent pretest and posttest control group. The experimental group received training while the control group did not receive training. The evaluation of the results obtained after the application of the program was carried out through two instruments: an efficacy questionnaire and a satisfaction questionnaire. The results showed significant differences between the experimental and control group after the application of the program (Z = −4,383; p = 0.000) in favor of the experimental group. It is confirmed that the program is effective in training teachers on inclusive education for the visually impaired as the teachers who took part in the training program significantly increased their knowledge in that issue.     

An evaluation model for school psychological services

An evaluation model is presented which provides a structure and a process for the evaluation of school psychological services. The structure incorporates psychological service areas along required and desired service dimensions; information to evaluate a school psychological service program is gathered from school files and interviews with students, parents, educators, and psychological services personnel. The method involves a five-stage process which may lead toward program modification efforts and subsequent re-evaluation. Further, the need for evaluation models in school psychology and for psychological service evaluation is presented and discussed.