Evaluating unintended program outcomes through Ripple Effects Mapping (REM): Application of REM using grounded theory

Several evaluation models exist for investigating unintended outcomes, including goal-free and systems evaluation. Yet methods for collecting and analyzing data on unintended outcomes remain under-utilized. Ripple Effects Mapping (REM) is a promising qualitative evaluation method with a wide range of program planning and evaluation applications. In situations where program results are likely to occur over time within complex settings, this method is useful for uncovering both intended and unintended outcomes. REM applies an Appreciative Inquiry facilitation technique to engage stakeholders in visually mapping sequences of program outcomes. Although it has been used to evaluate community development and health promotion initiatives, further methodological guidance for applying REM is still needed. The purpose of this paper is to contribute to the methodological development of evaluating unintended outcomes and extend the foundations of REM by describing steps for integrating it with grounded theory.     

The use of metaevaluation to plan evaluations of conservation programs

The U.S. Department of Energy (DOE) is sponsoring and will be required to evaluate a wide range of conservation programs. The development of evaluation plans for such programs can often be improved by reviewing evaluations that have been done on similar programs. The Residential Conservation Service (RCS), sponsored by DOE, is discussed as an example of how conducting metaevaluations can improve evaluation planning. Many utilities have sponsored programs that are similar to the RCS. A review of the utility program evaluations suggested key methodological issues and data problems that should be considered in the development of an RCS evaluation plan. The implications of these metaevaluation findings for an RCS evaluation plan are discussed.     

A case study in defining developmental costs for quality assurance in mental health center programs

Issues related to the state-of-the-art of ambulatory mental health care review as well as to the resources necessary for planning and operating a quality assurance program may be hampering development of such programs in community mental health centers nationwide. Faced with the problem of responsible and prudent program management, it has become increasingly difficult for CMHC directors to implement innovative, but unproven, and often unpriced programs which have the potential to draw scarce professional resources away from direct service delivery. Pressure to implement CMHC program requirements that are not ordinarily considered revenue-producing in nature, for example, consultation and education services and quality assurance activities, may further create anxiety for mental health program directors, especially during periods when they are ostensibly confronted with the need to maximize CMHC revenues. Before mental health professionals and administrators of mental health programs can be expected to introduce quality assurance programs, they need to have some idea how much the system costs. The figures reported from three North Carolina mental health centers may offer some indication of what other mental health centers might expect to spend in developing a quality assurance program. Salary expenses for multidisciplinary staff participation in quality assurance activities are presented.     

Professionalization of evaluative research: Conflict as a sign of health

Morell and Flaherty (1978) present arguments about the current status of evaluative research as a profession. This paper contains a critique, using a process model of professional development, of the perspectives employed by Morell and Flaherty. Issues discussed are the nature and substance of conflicts in evaluative research, including conflicts over methodological perspectives, the role of the evaluator in program planning, and the appropriate uses and users of evaluative information. The paper concludes with a discussion of the needs within the profession of evaluative research if it is to remain viable and responsive to change.     

Post hoc procedures for planning and evaluation.

A model designed to isolate, in post hoc fashion, current program services is presented. Concepts from multidimensional scaling are united with program planning techniques to yield an approach which allows for the specification of the five basic dimensions of program (Clients, Problems, Services, Professionals, and Facilities). These five dimensions are also employed in priority setting procedures. Discussion of the application of the model in the field of program planning and evaluation is presented.     

Register data in the evaluation and program planning of addiction treatment programs: Using Sweden as an example

Data from large-scale registers is often underutilized when evaluating addiction treatment programs. Since many programs collect register data regarding clients and interventions, there is a potential to make greater use of such records for program evaluation. The purpose of this article is to discuss the value of using large-scale registers in the evaluation and program planning of addiction treatment systems and programs. Sweden is used as an example of a country where register data is both available and is starting to be used in national evaluation and program planning efforts.The article focuses on possibilities, limitations and practicalities when using large-scale register data to conduct evaluations and program planning of addiction treatment programs. Main conclusions are that using register data for evaluation provides large amounts of data at low cost, limitations associated to the use of register data may be handled statistically, register data can answer important questions in planning of addiction treatment programs, and more accurate measures are needed to account for the diversity of client populations.     

Utilizing grassroots workers in family planning programs in India: prospects and problems

In order to rapidly expand the network of delivery systems and speed up the process of acceptance of family planning messages and methods, a shift took place in the Indian family planning program from the bureaucratic "clinical" approach to the people oriented "extension" approach. As a result, there is an increasing emphasis on moving the family planning efforts closer to the grassroots level. A key methodological issue centers on the proper selection, cultural acceptability, and the effectiveness of the grassroots workers who are to be trained and through whom family planning motivational messages and methods are to be introduced. The Indian government, from time to time, has trained and utilized different groups of grassroots workers in its family planning promotional efforts. Anthropological field studies were conducted in two different regions in India to examine the potential and actual roles of two groups of grassroots workers--opinion leaders and traditional birth attendants--in the delivery of family planning services in the rural areas. These studies revealed that while the traditional birth attendants can be trained and utilized to a limited extent in promoting family planning efforts, especially to the eligible female clients, the role of the opinion leaders in such efforts is at best questionable. Based on these field studies, cultural and technical (including bureaucratic) problems in training and utilizing opinion leaders and traditional birth attendants are explored in detail. Modifications in the training program strategies are suggested to improve and expand the family planning delivery system in rural India.     

Is implementation fidelity associated with improved access to care in a School-based Child and Family Team model?

Effective child and family centered service planning is crucial to addressing vulnerable children's needs. However, there is limited evidence about what facets of these processes improve service use and outcomes. The current study used a Poisson random effects hazard model to test correlations between fidelity to NC's Child and Family Support Team model and time to service receipt, using case management data for 3396 children served by that program during the 2008–2009 school year. Students were more likely to receive recommended services more quickly when caregivers and the students attended planning meetings, when their plans included services for caregivers, and when child and family team leaders followed up after meetings to verify service receipt. Contrary to the Child and Family Support Team theory of change, match between student needs and the lead agency of the meeting was not associated with the odds of quicker service receipt, nor was attendance by natural supports. Findings from this study demonstrate the potential effectiveness of using case management systems to measure service planning process fidelity, as well as how results thereof can both inform process improvement and potential refinements to models’ theories of change.     

The relocation of nursing home residents: An illustration of the advantages gained by planning a new program and designing an implementation evaluation together

This paper describes a collaboration between an evaluation team and the management and staff responsible for relocating nursing home residents to a new, state-of-the-art facility. Both the relocation plan and the implementation evaluation of this plan are detailed in order to illustrate the link between program planning and evaluation. I argue that combining program planning and implementation evaluation activities is of value to a wide range of organizations contemplating the development and implementation of a new program. Then I draw upon the research utilization literature to discuss how the collaborative relationship between the evaluation team and management contributed to the success of both program planning and program evaluation activities.     

Challenges related to the analytical process in realist evaluation and latest developments on the use of NVivo from a realist perspective

ABSTRACT

Realist evaluation (RE) is a research design increasingly used in program evaluation, that aims to explore and understand the influence of context and underlying mechanisms on intervention or program outcomes. Several methodological challenges, however, are associated with this approach. This article summarizes RE key principles and examines some documented challenges and solutions when analyzing RE data, including the development of Context-Mechanism-Outcome configurations. An analytic method using NVivo features is also presented. This method makes it possible to respond to certain analytic difficulties associated with RE by facilitating the identification of patterns and ensuring transparency in the analytical process.     

The State of Our Art

Abstract

In this two-part paper, the author examines inter-generational program research and evaluation based on a framework derived from a collaborative UNESCO (2000)-sponsored review of the intergenerational program field. In Part One, conceptual foundations for intergenerational programming are considered, taking into account theories that focus on individuals and groups within interactive contexts, those that focus primarily on individual development, and conceptually based program evaluations. In Part Two, appearing in the next issue of this journal, effects of intergenerational program participation are described, with emphasis on program activities and various program contexts. Challenges and questions emerging from the literature are presented, identifying the need for a greater use of theory in research, more cross cultural research, expanded outcomes, and solutions to some of the methodological challenges in intergenerational program research and evaluation.     

The State of Our Art

Abstract

In the second part of this two-part paper, the author examines intergenerational program research and evaluation based on a framework derived from a collaborative UNESCO (2000)-sponsored review of the intergenerational program field. In Part One, which appeared in the previous issue, conceptual foundations for intergenerational programming were considered, taking into account theories that focus on individuals and groups within interactive contexts, those that focus primarily on individual development, and conceptually based program evaluations. In Part Two, effects of intergenerational program participation are described, with emphasis on program activities and various program contexts. Challenges and questions emerging from the literature are presented, identifying the need for a greater use of theory in research, more cross-cultural research, expanded outcomes, and solutions to some of the methodological challenges in intergenerational program research and evaluation.     

Community-based behavioral health interventions: Developing strong community partnerships

The popularity of community-based interventions has experienced a revival over the last two decades. The general theme behind this trend is that greater community involvement provides researchers and practitioners with culturally relevant information to make interventions successful. This involvement is usually solicited through the formation of community coalitions and advisory boards. Although the positive influence of community partnerships is well established, strategies for successfully developing and sustaining these partnerships are less clear. To address this gap, this paper presents evaluation findings from The Making Connections Initiative, a national effort emphasizing community-level prevention strategies for improving the mental health of men/boys. Sixteen grantees were tasked with developing a coalition of community organizations to provide input into the development and implementation of a local prevention initiative. An evaluation of the 16 grantees’ planning activities identified four themes related to developing and sustaining successful community partnerships. This article reports on the strategies that were found most successful across grantees. With the increasing popularity of community-based programs, strategies for successfully partnering with local organizations are particularly important. This study adds to the limited research on best practices for forming community partnerships and should serve as a guide for community-based evaluators and program planners.     

Staff Development as a School Climate Intervention to Support Transgender and Gender Nonconforming Students: An Integrated Research Partnership Model for School Counselors and Counselor Educators

ABSTRACT

This article posits that an integrative research partnership model can assist school counselors in addressing the marginalization of transgender students in schools through staff development. A methodological integration of action research and program evaluation through quantitative and qualitative data collection is explained, and a collaborative process is highlighted. A case example is provided to demonstrate the use of the design.     

The results framework--an innovative tool for program planning and evaluation.

This article constitutes a case study of the development and implementation of the "results framework," an innovative planning and evaluation tool that is rapidly becoming a standard requirement for United States Agency for International Development (USAID) projects. The framework is used in a USAID-funded regional initiative for HIV/AIDS prevention in Central America. This new program evaluation and monitoring tool provides many advantages over traditional evaluation approaches that use outside consultants to provide midterm and end-of-project evaluations. The results-framework process, which spans the life of the project, provides an opportunity for program staff, donors, partners, and evaluators to work as a team to collect and use rich, longitudinal data for project planning, implementation, and evaluation purposes.     

UPSTREAM! Together evaluation results from community efforts to prevent mental,emotional, and behavioral health problems

Mental, emotional, and behavioral (MEB) health problems are prevalent globally. Despite effective programs that can prevent MEB problems and promote mental health, there has not been widespread adoption. UPSTREAM! Together was a planning project in three Colorado communities. Communities partnered with academic and policy entities to 1) translate evidence about MEB problem prevention into locally-relevant messages and materials and 2) develop long-term plans for broad implementation of interventions to prevent high-priority MEB problems. Community members recognized the need to talk about MEB problems to prevent them. The UPSTREAM! communities localized messages designed to start conversations and sustain attention on preventing MEB problems. The communities understood that prevention takes sustained community attention and advocacy, knowing that important outcomes may be years away. Long-term implementation plans aimed to strengthen families and enhance social connections among youth. Despite community readiness and capacity to implement evidence-based programs, there were few funding opportunities, delaying program implementation and revealing gaps between funding policies and community readiness. This community-engaged experience suggests an achievable approach, acceptable to communities, and worthy of further development and testing. Policies that cultivate and support local expertise may help to increase wider community adoption of evidence-based programs that promote mental health among youth.     

Jane Addams and the Campaign of Theodore Roosevelt

Abstract

Built on the assumption that both rational and nonrational thought go into problem solving, but only rational approaches have been presented as appropriate technology for program development, this article offers an alterative way of designing social programs. We call this approach “emergent planning.” Our aim is to provide a conceptualization of planning processes that will be useful regardless of culture, mission or goals of the human service organization within which planning and service occur. Our hope is to deconstruct what constitutes rational thinking in program planning in order to critically consider alternative ways of thinking and planning within the context of multiculturalism and globalization. The goal is recognition of a subjugated model of planning so that those who do not plan rationally are not automatically assumed to lack competence to engage in “real” planning and effective problem solving.     

Evaluation readiness: improved evaluation planning using a data inventory framework

Factors intrinsic to many programs, such as ambiguously stated objectives, inadequately defined performance measures, and incomplete or unreliable databases, often conspire to limit the evaluability of these programs. Current evaluation planning approaches are somewhat constrained in their ability to overcome these obstacles and to achieve full preparedness for evaluation. In this paper, the concept of evaluation readiness is introduced as a complement to other evaluation planning approaches, most notably that of evaluability assessment. The basic products of evaluation readiness--the formal program definition and the data inventory framework--are described, along with a guide for assuring more timely and appropriate evaluation response capability to support the decision making needs of program managers. The utility of evaluation readiness for program planning, as well as for effective management, is also discussed.     

Applied behavioral intervention in rural school settings: Problems in research design,implementation, and evaluation

This article discusses a number of theoretical, practical, and methodological difficulties encountered in the design, implementation, and evaluation of treatment programs conducted in the natural environment. In order to illustrate how these difficulties might be overcome, a case example is provided. Contingency contracts were used to modify the assaultive and disruptive behaviors of a 6-year-old child attending a preschool program for emotionally and physically handicapped children. Treatment effectiveness was evaluated through the use of a multiple baseline design and maintenance of behavior change was assessed in periodic follow ups of the child for 1 year after his admittance to public school. The importance of, planning for the generalization of newly acquired behaviors to different environments is stressed and suggestions for facilitating this process are discussed.     

Taxonomy and environment—Aging interaction: a case for small town diversity

This paper demonstrates the utility of an environment-behavior conceptual scheme proposed by Lawton for guiding the development of taxonomies of social and environmental attributes of rural small towns. A twelve-dimension psychosocial and environmental taxonomy is presented based on data gathered from 989 older residents of 18 small Kansas towns of 2500 and less in population. Implications of taxonomic development for the further conceptualization and definition of rurality is discussed. The role which taxonomic development plays in environmental design, planning and program intervention is also presented.