The use of evaluation models for decision making: Application to the community anti-crime program

Evaluations are typically designed to provide useful information, including measures of program effectiveness. Although this information is intended to be useful, the general literature on evaluations suggests that instances in which evaluations are explicitly used for decision making are rare. This article discusses the use of decision-theoretic evaluation models, and their applicability in practice as well as in principle. A case study is presented from the evaluation of LEAA's Community Anti-Crime Program, in which a decision analysis based on a decision-theoretic evaluation model influenced a programmatic decision which arose during the conduct of the evaluation.     

The results framework--an innovative tool for program planning and evaluation.

This article constitutes a case study of the development and implementation of the "results framework," an innovative planning and evaluation tool that is rapidly becoming a standard requirement for United States Agency for International Development (USAID) projects. The framework is used in a USAID-funded regional initiative for HIV/AIDS prevention in Central America. This new program evaluation and monitoring tool provides many advantages over traditional evaluation approaches that use outside consultants to provide midterm and end-of-project evaluations. The results-framework process, which spans the life of the project, provides an opportunity for program staff, donors, partners, and evaluators to work as a team to collect and use rich, longitudinal data for project planning, implementation, and evaluation purposes.     

Blazing an evaluation pathway: Lessons learned from applying utilization-focused evaluation to a conservation education program

In 2005, state fish and wildlife agency wanted to examine how one of its conservation education programs was providing science-based understanding and outdoor experiences by evaluating students’ knowledge, skills, attitudes and intended behavioral outcomes related to fish, fishing and aquatic habitats in Montana. A key factor in this study was the acceptance by program stakeholders to conduct the evaluation using a utilization-focused evaluation approach to promote usability and accuracy of evaluation results. Using a quasi-experimental non-equivalent group design, more than 2000 students in participating classrooms throughout Montana received a pre-survey, post-survey and an extended post-survey; 114 teachers participated in an Internet survey and 16 program instructors took part in a structured open-ended telephone interview. The participatory approach and mixed methods enhanced abilities to interpret results of student surveys in particular. The user-focused approach was discovered to be personal and situational, allowed the facilitation of the evaluation process with consideration for increased application of evaluation findings and implementation of recommendations from beginning to end. Further development of evaluation pathways is needed to more effectively evaluate outcomes and implement practical and transferable measures to determine if environmental education activities produce desired participant outcomes.     

The promotion of participatory methods for group events in Zambia: a process evaluation report for unicef 1994–1997

Beginning in early 1994, UNICEF Zambia made a commitment to emphasize the use of participatory methods in conducting its business. Specifically, this has meant adopting and experimenting with a participatory methodology named VIPP (Visualization in Participatory Programmes) which had been used in a wide variety of group events, ranging from planning and review meetings to training and team building. Three years later, a process evaluation was commissioned to examine the question: How has this initiative been accepted and what effect is it having on UNICEF-supported programmes in Zambia?Employing a widely-utilized framework for evaluating training effectiveness, the evaluation team traced the history of VIPP implementation in Zambia through document reviews and interviews. In addition, the evaluators collected data from three different programme sites in Zambia where VIPP had been used. Several participatory evaluation techniques were used to conduct this evaluation, the first exercise of its kind for assessing VIPP application in Africa.Through this evaluation exercise, the evaluation team has gained knowledge about the strategic benefits and application of VIPP that will help with future planning for VIPP in Zambia and that will also inform other UNICEF offices and organizations about the experience of implementing a participatory methodology for programme and organisational purposes. © 1998 Elsevier Science Ltd. All rights reserved.     

Using conceptual models as a planning and evaluation tool in conservation

Conservation projects are dynamic interventions that occur in complex contexts involving intricate interactions of social, political, economic, cultural, and environmental factors. These factors are constantly changing over time and space as managers learn more about the context within which they work. This complex context poses challenges for planning and evaluating conservation project. In order for conservation managers and evaluation professionals to design good interventions and measure project success, they simultaneously need to embrace and deconstruct contextual complexity.In this article, we describe conceptual models—a tool that helps articulate and make explicit assumptions about a project's context and what a project team hopes to achieve. We provide real-world examples of conceptual models, discuss the relationship between conceptual models and other evaluation tools, and describe various ways that conceptual models serve as a key planning and evaluation tool. These include, for example, that they document assumptions about a project site and they provide a basis for analyzing theories of change.It is impractical to believe that we can completely eliminate detail or dynamic complexity in projects. Nevertheless, conceptual models can help reduce the effects of this complexity by helping us understand it.     

Using evaluation to improve program quality based on the BELL model

Building Educated Leaders for Life (BELL) is a national not-for-profit organization whose mission is to increase the educational achievements, self-esteem, and life opportunities of elementary school children living in low-income urban communities. BELL has been engaged in formal evaluation, internally and externally, for more than five years and has built internal evaluation capacity by investing in a specialized full-time evaluation team. As part of a continuous program improvement model of evaluation, BELL uses the data to refine program implementation and replicate successful elements of the services and operations. In this chapter, the authors highlight best practices from the field by outlining BELL's approach to using evaluation data for continuous program improvement. Key strategies include (1) carefully identifying intended users of the evaluation throughout the organization and among its external stakeholders, then working closely with intended users throughout the evaluation process, ensuring full engagement at every step of the process; (2) reporting findings in a readable, user-friendly format and timing the reporting so that it is aligned with programmatic decision making and planning cycles; and (3) making and supporting explicit recommendations for the next program cycle, where intended users have agreed to recommendations and ownership is assigned. BELL's successful use of data for improvement is evidenced by the consistently strong outcomes for the students it serves as well as increased efficiency and satisfaction related to service delivery that has supported the replication of BELL's programs nationally.     

Peer cluster theory and adolescent drug use: a reanalysis

Peer cluster theory hypothesizes that peer drug association has a direct effect on adolescent drug use. In turn, peer drug associations are influenced by familial factors (sanctions and strength) and individual variables (religious identification, school adjustment). Oetting and Beauvais evaluated peer cluster theory in a cross-sectional survey of 415 high school students [1]. We hypothesized, evaluated, and found support for an alternative model in which poorer school adjustment was specified as a consequence of drug use, peer drug associations, lack of family sanctions against drug use, low religious identification, and absence of family strength. This reanalysis illustrates that interpretations of structural equation modeling results from cross-sectional data are especially problematic.     

Collaborative evaluation within a framework of stakeholder-oriented evaluation approaches

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation planning and implementation. Unlike "distanced" evaluation approaches, which reject stakeholder participation as evaluation team members, Collaborative Evaluation assumes that active, on-going engagement between evaluators and program staff, result in stronger evaluation designs, enhanced data collection and analysis, and results that stakeholder understand and use. Among similar "participant-oriented" evaluation approaches (Fitzpatrick, Sanders, & Worthen, 2011), Collaborative Evaluation distinguishes itself in that it uses a sliding scale for levels of collaboration. This means that different program evaluations will experience different levels of collaborative activity. The sliding scale is applied as the evaluator considers each program's evaluation needs, readiness, and resources. While Collaborative Evaluation is a term widely used in evaluation, its meaning varies considerably. Often used interchangeably with participatory and/or empowerment evaluation, the terms can be used to mean different things, which can be confusing. The articles use a comparative Collaborative Evaluation Framework to highlight how from a theoretical perspective, Collaborative Evaluation distinguishes itself from the other participatory evaluation approaches.     

The strategic use of Twitter to manage personal public relations

This study examined the effect of personal public relations developed through Twitter on cognitive and attitudinal aspects, particularly focusing on the use of Twitter by CEOs. According to the responses of young consumers who participated in this study, their attitudes toward the use of Twitter by CEOs positively influenced their perception of transformational leadership (H1). Further, perceived transformational leadership was positively associated with attitudes toward CEOs using Twitter; this in turn positively influenced attitudes toward corporations (H2). The positive evaluation of the use of Twitter by CEOs directly influenced attitudes toward such CEOs (H3), whereas attitudes toward such use did not influence attitudes toward corporations. The results indicated that microblogging is beneficial for the development of effective personal public relations by corporate leaders.     

Personal value of alcohol use as a predictor of intention to decrease post-college alcohol use.

This study examines the utility of a new measure of personal value of alcohol use in predicting intentions to reduce post-college alcohol use. This measure is intended to distinguish college students who drink due to situational norms versus intrinsic commitment to alcohol use. As expected, alcohol consumption quantity/frequency and frequency of consuming five or more drinks per occasion were unrelated to intention to reduce alcohol use after college. When these alcohol consumption measures were controlled using partial correlation, the personal value of alcohol measure was significantly and negatively related to intention to reduce alcohol use. Conversely, when personal value of alcohol use was controlled in partial correlations, the relationship between alcohol consumption variables and intention to reduce alcohol use after college became positive, though only at marginally significant levels. Implications are discussed.     

Substance use among college athletes: a comparison based on sport/team affiliation

OBJECTIVE: Prior research shows that college athletes have higher rates of substance use, especially alcohol, than do college students who are not involved in athletics. To augment the literature, the author sought to determine which sports/teams are at the greatest risk for substance use. PARTICIPANTS: The author used data from the 1999 Harvard School of Public Health College Alcohol Study, a national survey of college and university students in the United States. METHODS: A series of chi-square and logistic regression models examined variation in substance use among college athletes on the basis of sport/team affiliation. RESULTS: Findings indicated that male hockey and female soccer athletes were the most likely to report substance use and that male basketball and cross-country/track athletes reported lower levels of substance use. CONCLUSION: There is variation in substance use on the basis of sport/team affiliation, and future researchers should examine why certain groups of athletes have higher rates of substance use.     

How was a national moving and handling people guideline intended to work? The underlying programme theory

In healthcare, moving and handling people (MHP) often cause musculoskeletal disorders. To prevent musculoskeletal disorders due to MHP, many national evidence-based guidelines have been developed. However, little is known about how these guidelines were intended to work, i.e. their ‘programme theory’, how implementation by intended users is influenced by contextual factors and mechanisms to produce outcomes. This paper identifies the programme theory of a national MHP guideline (MHPG) using thematic analysis of the MHPG document, three organisational planning documents, and interviews with MHPG developers. The analysis identified the intended users of the MHPG as health and safety managers and MHP coordinators. The programme theory comprised contextual factors, potentially hindering (e.g. budget constraints) or facilitating (e.g. changing demographics) implementation, being influenced by mechanisms mainly based on ethical (quality of care, evidence-based practices), and economic reasoning (reducing cost of MHP, return on investment) to reduce injuries caused by MHP – the intended outcome.     

Child welfare workers use of knowledge in their daily work

The debate continues on what constitutes core knowledge in professional work in child welfare and child protection. The aims of this article are to examine what different sources of knowledge social workers report to use in their daily work and to determine the importance of different knowledge sources in relation to their work experience, educational background and type of employment (caseworker/manager). Data were obtained from a survey completed by 390 social workers in municipal child welfare services in Western Norway. Social workers answered questions regarding their use of different knowledge sources in their daily work. The information they obtained in a particular case and their personal experiences were the most frequently used sources. Factor analysis revealed four subgroups of knowledge sources: The most frequently used were: (a) Colleagues and Supervision and (b) Personal Experience, followed by (c) Social Work Education, Courses and Municipal and State Documents, with (d) Journal Articles, Textbooks and External Sources being the least frequently used. The most important source for social workers with less work experience was Colleagues and Supervision, whereas the factor Journal Articles, Textbooks and External Sources was more often important for social workers with a master’s degree than for those without this level of education. Overall, social workers seem to rely mostly on personal experience and their colleagues as sources of knowledge in their work.     

Improving child welfare services with family team meetings: A mixed methods analysis of caseworkers' perceived challenges

Various forms of family team meetings have been increasingly employed in child welfare systems to empower families and provide effective individualized services through community partnerships. However, many family team meetings often fail to achieve their intended goals mainly due to ongoing challenges that team members face. Using data from a survey of caseworkers, this study examined dynamic processes of family team meetings to improve child welfare service outcomes, specifically focusing on caseworkers' perceived challenges. Also, responses to an open ended question were reviewed using content analysis to identify similar or different challenges encountered by diverse team members. Survey data were analyzed using path analysis and found that logistical barriers tended to decrease both family and stakeholder engagements, which in turn made child welfare services less accessible, less utilized, and less effective for children and families. Content analysis revealed that caseworkers perceived disagreement/conflict and knowledge deficits about child welfare as common challenges for all team members. This paper concludes with practice implications suggesting ways to minimize challenges identified and maximize the effectiveness of a family team meeting.     

Land use planning practice under the public land ownership policy in Tanzania

Land-use planning can be defined as a programme of state intervention in land use and environmental change to mediate conflicts of interests over how land should be used, developed, and coordinate individual activities which if left to proceed otherwise would lead to an environment for living that is characterised by negative externalities, inefficient use of land and services, inequity and unfair distribution of resources. The land-use programme is realised through a planning system, constituted by institutional arrangements, instruments and tools. Being one of the land policy tool by which the government intervenes in land-use change and development, land-use planning practice is, obviously, influenced by the content of the land policy as well as interests extant in a locality to include those of the planners and other land administrators. How does planning practice looks like under a public landownership policy? Through an empirical case of planning in practice, this paper provides an answer to this question.     

Prevalence and reasons for Juul use among college students

Abstract

Objective: Examine Juul use patterns, sociodemographic and personal factors associated with Juul use, and reasons for Juul initiation and current use, among college students. Participants: Convenience sample of 371 undergraduates at a large university in the southeast; recruited April 2018. Methods: Cross-sectional design using an online survey. Logistic regression identified the personal risk factors for current use. Results: Over 80% of participants recognized Juul; 36% reported ever use and 21% past 30-day use. Significant risk factors for current Juul use were: male, White/non-Hispanic, lower undergraduate, and current cigarette smoker. Current Juul users chose ease of use and lack of a bad smell as reasons for use. Ever Juul users most commonly endorsed curiosity and use by friends as reasons for trying Juul. Conclusions: Given the propensity for nicotine addiction among youth and young adults, rates of Juul use are alarming and warrant immediate intervention.     

Lapses,infidelities, and creative adaptations: Lessons from evaluation of a participatory market development approach in the Andes

Participatory approaches are frequently recommended for international development programs, but few have been evaluated. From 2007 to 2010 the Andean Change Alliance evaluated an agricultural research and development approach known as the “Participatory Market Chain Approach” (PMCA). Based on a study of four cases, this paper examines the fidelity of implementation, the factors that influenced implementation and results, and the PMCA change model. We identify three types of deviation from the intervention protocol (lapses, creative adaptations, and true infidelities) and five groups of variables that influenced PMCA implementation and results (attributes of the macro context, the market chain, the key actors, rules in use, and the capacity development strategy). There was insufficient information to test the validity of the PMCA change model, but results were greatest where the PMCA was implemented with highest fidelity. Our analysis suggests that the single most critical component of the PMCA is engagement of market agents – not just farmers – throughout the exercise. We present four lessons for planning and evaluating participatory approaches related to the use of action and change models, the importance of monitoring implementation fidelity, the limits of baseline survey data for outcome evaluation, and the importance of capacity development for implementers.     

Expectation changes and team characteristics in a participatory design process

A human factors specialist researched the expectations of a culturally and professionally diverse team throughout a year long participatory design process of a large processing facility. For a deeper understanding of high-level team expectations and characteristics, the specialist collected data and information through in-situ ethnography and traditional case study methods, personal interviews, and a questionnaire that included a likert scale rating for expectation levels. Results found that expectation levels rated extremely satisfied for individual team members and the overall team itself before and during the participatory process. In contrast, expectations for upper management from the team were satisfied before the participatory process, but changed to uncertain, to unsatisfied, to extremely unsatisfied during the process. Additionally, the participatory design team exhibited high-level team characteristics to include honesty, competence, commitment, communication, creativity, and clear expectations.     

To What Extent Did Ethnicity and Economic Issues Matter in the 2007 Disputed Kenyan Elections?

This article assesses the key factors that influenced the intention to vote in the 2007 presidential election in Kenya, using data from a survey and other external datasets. Ethnicity was found to be one of the main factors. However, access to private and public goods, poverty and differences in wealth across ethnic groups also mattered, depending on how the voter claimed to be self‐identified. Those who self‐identified in terms of their ethnicity were influenced the most by access to services such as schools, health clinics and police stations, evidence suggesting that voters use ethnicity as a proxy to gauge which candidates will give them greater access to services provided by the state.     

Strategies for engaging foster care youth in permanency planning family team conferences

The present study examines strategies conference facilitators used to engage foster care youth in decision-making in the context of permanency planning family team conferences. Data collection included observations of permanency planning family team conferences, followed by interviews with foster care youth and conference facilitators. Data analysis focused on gaining a deeper understanding of how facilitators incorporate youth into decision-making, with a specific focus on the strategies they employed. Four strategies were identified in the analysis: creating a safe space, encouraging the youth voice, re-balancing power, and establishing a personal connection. The study's policy and practice implications, limitations and areas of further research are presented.