Prerequisites of EAP Evaluation:

This chapter's main contention is that EAP evaluators, coordinators and administrators need to spend more time in thoughtfully planning EAP evaluations as such activities directly influence the quality of any subsequent evaluation. The chapter outlines four prerequisites to be considered in this regard. These are: (1) a sincere commitment to evaluate the program, (2) answering the why, when, and how of the evaluation; (3) the goals and objectives of the EAP require specification; and (4) information should be in retrievable form. These prerequisites are discussed within the context of a cooperative planning process between the administration, the EAP coordinator and EAP evaluator(s), and are described as to how they may directly influence the EAP evaluation.     

Application of logic models in a large scientific research program

It is the purpose of this article to discuss the development and application of a logic model in the context of a large scientific research program within the Commonwealth Scientific and Industrial Research Organisation (CSIRO). CSIRO is Australia's national science agency and is a publicly funded part of Australia's innovation system. It conducts mission-driven scientific research focussed on delivering results with relevance and impact for Australia, where impact is defined and measured in economic, environmental and social terms at the national level. The Australian Government has recently signalled an increasing emphasis on performance assessment and evaluation, which in the CSIRO context implies an increasing emphasis on ensuring and demonstrating the impact of its research programs. CSIRO continues to develop and improve its approaches to impact planning and evaluation, including conducting a trial of a program logic approach in the CSIRO Preventative Health National Research Flagship. During the trial, improvements were observed in clarity of the research goals and path to impact, as well as in alignment of science and support function activities with national challenge goals. Further benefits were observed in terms of communication of the goals and expected impact of CSIRO's research programs both within CSIRO and externally. The key lesson learned was that significant value was achieved through the process itself, as well as the outcome. Recommendations based on the CSIRO trial may be of interest to managers of scientific research considering developing similar logic models for their research projects. The CSIRO experience has shown that there are significant benefits to be gained, especially if the project participants have a major role in the process of developing the logic model.     

The relocation of nursing home residents: An illustration of the advantages gained by planning a new program and designing an implementation evaluation together

This paper describes a collaboration between an evaluation team and the management and staff responsible for relocating nursing home residents to a new, state-of-the-art facility. Both the relocation plan and the implementation evaluation of this plan are detailed in order to illustrate the link between program planning and evaluation. I argue that combining program planning and implementation evaluation activities is of value to a wide range of organizations contemplating the development and implementation of a new program. Then I draw upon the research utilization literature to discuss how the collaborative relationship between the evaluation team and management contributed to the success of both program planning and program evaluation activities.     

Jane Addams and the Campaign of Theodore Roosevelt

Abstract

Built on the assumption that both rational and nonrational thought go into problem solving, but only rational approaches have been presented as appropriate technology for program development, this article offers an alterative way of designing social programs. We call this approach “emergent planning.” Our aim is to provide a conceptualization of planning processes that will be useful regardless of culture, mission or goals of the human service organization within which planning and service occur. Our hope is to deconstruct what constitutes rational thinking in program planning in order to critically consider alternative ways of thinking and planning within the context of multiculturalism and globalization. The goal is recognition of a subjugated model of planning so that those who do not plan rationally are not automatically assumed to lack competence to engage in “real” planning and effective problem solving.     

Interview. Maria Isabal Plata,executive director of Profamilia,talks to Caroline Sweetman about the work of the organisation in Colombia

In this interview, Maria Isabal Plata discusses the work of the nongovernmental organization Profamilia in Colombia. Since its founding in 1965, Profamilia has assumed direct and indirect responsibility for nearly 70% of family planning (FP) and reproductive health activities in the country. These activities are complemented by a legal service program, an evaluation and research program, and a documentation center. In Colombia, women gained equal rights under the law in 1974 and a constitutional prohibition on discrimination in 1991, but sex stereotypes still dictate responsibility for family chores. When women realize their economic and social rights, poverty and inequalities will diminish. Thus, it is crucial to safeguard reproductive and sexual rights and ban violence against women. The traditional concept of family forces a disregard for the rights of family members and allows societies to oppose the interests of women. The aftermath of the UN Fourth Women's Conference in Colombia will be to work towards achieving full citizenship for women and democratic societies. The experience women's groups had at the conference has created a situation in which grassroots organizations can consider beginning to work with development organizations to achieve some of these goals.     

Using practitioners' theories to document program results

A form of self-assessment, termed practitioner-centered evaluation, is proposed for human services agencies. In contrast to assessment methods that may require an outside evaluator, this qualitative technique focuses on the informal theories of behavioral change that underlie a program-as-implemented. It takes into account interacting cause-and-effect relationships and relies on the ability of staff members to document success and failure in the context of their own theories. As a vehicle for identifying program elements that are effective in achieving behavioral change and other goals, practitioner-centered evaluation can be the basis of franker, more productive conversations between funders and grantees and between staff members and their governing boards.     

A principles-focused evaluation of community mental health short-term case management

Principles-focused evaluations allow evaluators to appraise each principle that guides an organization or program. This study completed a principles-focused evaluation of a new community mental health intervention called Short Term Case Management (STCM) in Toronto, Canada. STCM is a time limited intervention for clients to address unmet needs and personalized goals over 3 months. Findings showcase that a principles-focused evaluation, assessing whether program principles are guiding, useful, inspiring, developmental and/or evaluable (GUIDE), is a practical formative evaluation approach. Specifically, offering an understanding of a novel intervention, including its key components of assessment and planning, support plan implementation and evaluation and care transitions. Findings also highlight that STCM may work best for those clients ready to participate in achieving their own goals. Future research should explore how best to apply the GUIDE framework to complex interventions, including multiple principles, to increase evaluation feasibility and focus.     

Dissemination and implementation science in program evaluation: A telemental health clinical consultation case example

Increased attention has been placed on evaluating the extent to which clinical programs that support the behavioral health needs of youth have effective processes and result in improved patient outcomes. Several theoretical frameworks from dissemination and implementation (D&I) science have been put forth to guide the evaluation of behavioral health program implemented in the context of real-world settings. Although a strong rationale for the integration of D&I science in program evaluation exists, few examples exist available to guide the evaluator in integrating D&I science in the planning and execution of evaluation activities.This paper seeks to inform program evaluation efforts by outlining two D&I frameworks and describing their integration in program evaluation design. Specifically, this paper seeks to support evaluation efforts by illustrating the use of these frameworks via a case example of a telemental health consultation program in pediatric primary care designed to improve access to behavioral health care for children and adolescents in rural settings. Lessons learned from this effort, as well as recommendations regarding the future evaluation of programs using D&I science to support behavioral health care in community-based settings are discussed.     

Engaging New Social Movements

Organizations involved in the development ofinnovative social programs are coming under increasingpressure to establish structures through which theusers, targets, or beneficiaries of program activities can be involved in program planning andimplementation. The dynamic that this sets up betweenthe program and its wider environment, particularly inareas in which new social movements areoperating, can be quite challenging. This article seeks toexamine these dynamics, particularly in the context ofprogram evaluation. It is argued that an engagedevaluation approach (drawing on an action research orientation) has particular strengths in sucha situation, although the limitations and challenges ofsuch an approach are also outlined. It may, for example,require careful management in terms of the dynamics of the evaluation team. The discussionis illustrated by examples drawn from the evaluation ofa European program for disabled people in which theauthor as member of a research team from The Tavistock Institute was involved.     

Goal-oriented evaluation for program planning: A case study

Utilizing a contextual model of evaluation, a goal-oriented method was applied to the Health Psychology program, a doctoral program in its early stages at the University of California, San Francisco. There were five stages involved in implementing this method: (1) clarification of the goals and objectives of the program, (2) prioritizing the objectives, (3) judging the attainment of the objectives, (4) organization of faculty/student input, and (5) feedback to the program management. All faculty members and students were invited to participate as selfevaluators in this evaluation effort. The results indicated that there were significant differences between the faculty group and the student group on their ranking and rating of the importance of specific educational and resource objectives. A one year follow-up was obtained by interviewing the director of the program to assess the impact of the project on program planning. The advantages and disadvantages of the approach were discussed in light of this attempt to analyze a new educational program.     

Planning for permanency planning in foster care

The perceived severity of the problem of “drift” in the foster care system has encouraged agencies nationwide to accept permanency planning, but perhaps with limited attention to the planning process. This article examines the phenomenon of permanency planning from a planning perspective by focusing on: (a) definition of the permanency planning problem, (b) understanding of issues relevant to permanency planning (caseworker, program and systemic), (c) appreciation of the operational context (political, economic and social factors), and (d) evaluation of available options and possible consequences in developing or improving a permanency planning program. Permanency planning is not a simple, ready-made program applicable to any situation; it involves coordinated efforts by workers, service systems, and society in general, and requires time, commitment and resources. Successful planning for permanency planning can permit more children to grow up in homes providing continuity and stability.     

Faith-Based Programs and the Role of Empirical Research

Abstract

After some decades of being subject to neglect or contempt, faith-based social care programs are receiving increased attention and resources enabling them to undertake a greater role in the national network of human services. Faith-based programs receiving public support can expect to be rightly scrutinized by the public, in terms of their ability to attain professed program goals. The tools of conventional empirically-oriented program evaluation research have tremendous potential to help demonstrate the effectiveness of faith-based programs, which will justify their receipt of ongoing support from public funds. Negative research findings can be properly scrutinized by the faith-based community of service providers to help make tough, data-based, decisions on funding priorities. Several examples are described, illustrating how various types of faith-based programs have profitably participated in program evaluation studies.     

An organizational model to distinguish between and integrate research and evaluation activities in a theory based evaluation

Theory-based evaluation (TBE) is an evaluation method that shows how a program will work under certain conditions and has been supported as a viable, evidence-based option in cases where randomized trials or high-quality quasi-experiments are not feasible. Despite the model's widely accepted theoretical appeal there are few examples of its well-implemented use, probably due to time and money limitations necessary for planning and a confusion over the definitions between research and evaluation functions and roles. In this paper, we describe the development of a theory-based evaluation design in a Math and Science Partnership (MSP) research project funded by the U.S. National Science Foundation (NSF). Through this work we developed an organizational model distinguishing between and integrating evaluation and research functions, explicating personnel roles and responsibilities, and highlighting connections between research and evaluation work. Although the research and evaluation components operated on independent budgeting, staffing, and implementation activities, we were able to combine datasets across activities to allow us to assess the integrity of the program theory, not just the hypothesized connections within it. This model has since been used for proposal development and has been invaluable as it creates a research and evaluation plan that is seamless from the beginning.     

An organizational model to distinguish between and integrate research and evaluation activities in a theory based evaluation

Theory-based evaluation (TBE) is an evaluation method that shows how a program will work under certain conditions and has been supported as a viable, evidence-based option in cases where randomized trials or high-quality quasi-experiments are not feasible. Despite the model's widely accepted theoretical appeal there are few examples of its well-implemented use, probably due to time and money limitations necessary for planning and a confusion over the definitions between research and evaluation functions and roles. In this paper, we describe the development of a theory-based evaluation design in a Math and Science Partnership (MSP) research project funded by the U.S. National Science Foundation (NSF). Through this work we developed an organizational model distinguishing between and integrating evaluation and research functions, explicating personnel roles and responsibilities, and highlighting connections between research and evaluation work. Although the research and evaluation components operated on independent budgeting, staffing, and implementation activities, we were able to combine datasets across activities to allow us to assess the integrity of the program theory, not just the hypothesized connections within it. This model has since been used for proposal development and has been invaluable as it creates a research and evaluation plan that is seamless from the beginning.     

Planning an integrated agriculture and health program and designing its evaluation: Experience from Western Kenya

Multi-sectoral programs that involve stakeholders in agriculture, nutrition and health care are essential for responding to nutrition problems such as vitamin A deficiency among pregnant and lactating women and their infants in many poor areas of lower income countries. Yet planning such multi-sectoral programs and designing appropriate evaluations, to respond to different disciplinary cultures of evidence, remain a challenge. We describe the context, program development process, and evaluation design of the Mama SASHA project (Sweetpotato Action for Security and Health in Africa) which promoted production and consumption of a bio-fortified, orange-fleshed sweetpotato (OFSP). In planning the program we drew upon information from needs assessments, stakeholder consultations, and a first round of the implementation evaluation of a pilot project. The multi-disciplinary team worked with partner organizations to develop a program theory of change and an impact pathway which identified aspects of the program that would be monitored and established evaluation methods. Responding to the growing demand for greater rigour in impact evaluations, we carried out quasi-experimental allocation by health facility catchment area, repeat village surveys for assessment of change in intervention and control areas, and longitudinal tracking of individual mother-child pairs. Mid-course corrections in program implementation were informed by program monitoring, regular feedback from implementers and partners’ meetings. To assess economic efficiency and provide evidence for scaling we collected data on resources used and project expenses. Managing the multi-sectoral program and the mixed methods evaluation involved bargaining and trade-offs that were deemed essential to respond to the array of stakeholders, program funders and disciplines involved.     

The power of the context map: Designing realistic outcome evaluation strategies and other unanticipated benefits

Developing a feasible evaluation plan is challenging when multiple activities, often sponsored by multiple agencies, work together toward a common goal. Often, resources are limited and not every agency's interest can be represented in the final evaluation plan. The article illustrates how the Antecedent Target Measurement (ATM) approach to logic modeling was adapted to meet this challenge. The key adaptation is the context map generated in the first step of the ATM approach. The context map makes visually explicit many of the underlying conditions contributing to a problem as possible. The article also shares how a prioritization matrix can assist the evaluator in filtering through the context map to prioritize the outcomes to be included in the final evaluation plan as well as creating realistic outcomes. This transparent prioritization process can be especially helpful in managing evaluation expectations of multiple agencies with competing interests. Additional strategic planning benefits of the context map include pinpointing redundancies caused by overlapping collaborative efforts, identifying gaps in coverage, and assisting the coordination of multiple stakeholders.     

Stakeholder management and program planning in German public theaters

Taking the example of a German public professional theater, this study is the first to analyze how stakeholder interests are managed in the process of program planning. We address the following questions: (1) What are the goals and interests of the most influential stakeholders? (2) How are these goals and interests reflected by theater management when deciding the season's program? (3) Which strategies does theater management apply to balance conflicting stakeholder goals and interests? Results of our qualitative study identify five decisions to be made with regard to program planning: selection of the season's theme, plays, stage directors, cast, and chronological order of productions. Four strategies of balancing conflicting stakeholder goals and interests are identified: setting priorities, combining stakeholder interests, focusing on neutral goals, and developing audiences.     

Program analysis using the client oriented cost outcome system.

The client oriented cost outcome system has been under development in Pennsylvania community and hospital programs since 1972. The system builds upon the behavioral and decision data generated with or on behalf of consumers. Flexibility for local program system design is permitted if the procedures of consumer intake, review, and termination document each consumer's (a) problems, resources, and goals, (b) overall functioning level in their ordinary community, and (c) services intended and rendered as related to (a), above. While the system's primary application is in providing feedback for local program quality assurance and evaluation procedures, aggregation of data permits program planning and evaluation at county, state, and federal levels in terms of (a) client demographic or diagnostic characteristics, and (b) program service characteristics and objectives.     

Using activity theory to evaluate and improve K-12 school and university partnerships

This article reports on an evaluation methodology development study for K-12 school and university partnerships. The method is based on Engeström's (1987). Learning by expanding: An activity-theoretical approach to developmental research. Helsinki: Orienta-Konsultit Oy activity systems analysis that allows researchers to examine qualitative datasets of complex human interactions. This study was designed for participants to evaluate partnership relations and activities. We investigated how the use of activity systems analysis in K-12 school and university partnership evaluation meetings affected participant communication processes. In this study, during a 1-day retreat K-12 school staff and university staff used a modified activity systems model to identify persisting institutional tensions in their program that often trigger miscommunications and strain their relations. During the discussion sessions, study participants collaboratively examined their partnership relations and identified strategies for overcoming difficulties. Additionally, in subsequent monthly partnership meetings during the school year, participants examined findings from the evaluation to design and implement improvement strategies. The results from this methodology development and implementation study provided researchers and partnership participants a new means to (a) evaluate partnership activities, (b) identify institutional barriers, (c) plan future activities, and (d) listen to and incorporate the ideas of less vocal staff members into the planning of future activities.