A large-scale initiative to disseminate an evidence-based drug abuse prevention program in Italy: Lessons learned for practitioners and researchers

Across developed countries, experimentation with alcohol, tobacco, and other drugs often begins in the early adolescent years. Several evidence-based programs have been developed to prevent adolescent substance use. Many of the most rigorously tested and empirically supported prevention programs were initially developed and tested in the United States. Increasingly, these interventions are being adopted for use in Europe and throughout the world. This paper reports on a large-scale comprehensive initiative designed to select, adapt, implement, and sustain an evidence-based drug abuse prevention program in Italy. As part of a large-scale regionally funded collaboration in the Lombardy region of Italy, we report on processes through which a team of stakeholders selected, translated and culturally adapted, planned, implemented and evaluated the Life Skills Training (LST) school-based drug abuse prevention program, an evidence-based intervention developed in the United States. We discuss several challenges and lessons learned and implications for prevention practitioners and researchers attempting to undertake similar international dissemination projects. We review several published conceptual models designed to promote the replication and widespread dissemination of effective programs, and discuss their strengths and limitations in the context of planning and implementing a complex, large-scale real-world dissemination effort.     

“Compassion,pleasantry, and hope”: A process evaluation of a volunteer-based nonprofit

As funders continue to emphasize the importance of documented results, nonprofit organizations must work to complete program evaluations that are both valuable and feasible. The purpose of this paper is to document a practical process evaluation of a southeastern nonprofit, a local Meals on Wheels. Using a mixed methods approach, we sought to answer four evaluation questions: (1) What are the essential program components, as identified by key stakeholders; (2) To what extent are volunteers implementing the identified essential components as intended; (3) What is the level of volunteer satisfaction with the program; and (4) What suggestions do stakeholders have for improving the program? Our findings indicate that most aspects of the program were implemented as intended, but inconsistencies occurred when volunteers were unsure of their assigned duties. In addition, volunteers had high levels of satisfaction and specific suggestions for improvement. From these results, we developed a conceptual model of factors contributing to quality of implementation and volunteer satisfaction that may be generalizable to other volunteer-based nonprofits. Specifically, we identified three factors that helped to facilitate satisfaction and performance: leadership, social contact, and fulfillment. Finally, this process evaluation demonstrates the feasibility of developing and implementing evaluation tools in similar organizations.     

Agent-based modeling as a tool for program design and evaluation

Recently, systems thinking and systems science approaches have gained popularity in the field of evaluation; however, there has been relatively little exploration of how evaluators could use quantitative tools to assist in the implementation of systems approaches therein. The purpose of this paper is to explore potential uses of one such quantitative tool, agent-based modeling, in evaluation practice. To this end, we define agent-based modeling and offer potential uses for it in typical evaluation activities, including: engaging stakeholders, selecting an intervention, modeling program theory, setting performance targets, and interpreting evaluation results. We provide demonstrative examples from published agent-based modeling efforts both inside and outside the field of evaluation for each of the evaluative activities discussed. We further describe potential pitfalls of this tool and offer cautions for evaluators who may chose to implement it in their practice. Finally, the article concludes with a discussion of the future of agent-based modeling in evaluation practice and a call for more formal exploration of this tool as well as other approaches to simulation modeling in the field.     

A formative evaluation of the recovery public works programme in Blantyre City,Malawi

Public works programmes (PWPs) are popular social protection instruments in the context of chronic poverty but very little has been published in the way of implementation and outcomes of these programmes. This paper presents a formative process and outcome evaluation of the recovery PWP in Blantyre City, Malawi. The evaluation used longitudinal household survey data of PWP beneficiaries, programme records and interview responses from programme staff and community leaders. Largely, the process evaluation findings showed an agreement between actual and planned activities. The outcome evaluation found indications that the PWP community assets offered some potential benefits to the communities, and that PWP wages allowed the beneficiaries to purchase some food. This however, did not translate into more meals per day, nor did the earnings prevent the decline in household assets as expected. Given a plausible PWP theory and high implementation fidelity, the PWP wage rate or number of days was either just enough to smooth participant income, or insufficient altogether, to enable achievement of more distal outcomes.     

Who performs better under time pressure? Results from a field experiment

We investigate whether and how time pressure affects performance. We conducted a field experiment in which students from an Italian University are proposed to choose between two exam schemes: a standard scheme without time pressure and an alternative scheme consisting of two written intermediate tests, one of which to be taken under time pressure. Students deciding to sustain the alternative exam are randomly assigned to a “time pressure” and a “no time pressure” group. Students performing under time pressure at the first test perform in absence of time pressure at the second test and vice versa. We find that being exposed to time pressure exerts a negative and statistically significant impact on students’ performance. The effect is driven by a strong negative impact on females’ performance, while there is no statistically significant effect on males. Both the quantity and quality of females’ work is hampered by time pressure. Using data on students’ expectations, we also find that the effect produced by time pressure on performance was correctly perceived by students. Female students expect a lower grade when working under time pressure, while males do not. These findings contribute to explain why women tend to shy away from jobs and careers involving time pressure.