Evaluating school capacity to implement new programs

An eight-factor survey-based Bayesian model (Bridge-It) for assessing school capacity to implement health and education programs was tested in secondary analyses of data from 47 schools in the Texas Tobacco Prevention Initiative (TTPI). Bridge-It was used during the pre-implementation phase and again at mid-course of the TTPI 2 years later. Achieved implementation status was evaluated in follow-up almost 4 years after the start of the TTPI. The Bridge-It score aggregated across all eight of the capacity factors predicted both quality of adherence to the Guidelines for School Programs to Prevent Tobacco Use and Addiction and quantity of implementing activity. The school-based leadership factor was an independent predictor of quality of adherence whereas the facilitation processes factor predicted quantity of implementing activity. Integration of Bridge-It, or comparable multi-attribute tools, into the planning and evaluation of school-centered programs can increase understanding of factors that influence implementation and provide guidance for capacity building.     

The other side of the story. Student narratives on the California Drug, Alcohol, and Tobacco Education Programs

Within the context of a large-scale, comprehensive evaluation of the California Drug Alcohol Tobacco Education (DATE) program, this study sought to extend knowledge of student perceptions of prevention education using a naturalistic approach. The constant comparative method was used to analyze 40 focus group interviews of risk and thriving groups conducted in 11 high, middle, and elementary school districts. This article presents three assertions generated solely from 490 "narrative stories" found in the data set. "At-risk" and "thriving" students at all three levels of schooling (a) use "story" to make sense of prevention education, and (b) distinguish use from abuse. High school students of both groups (c) believe that hearing only one side of the substance use/abuse story and strict expulsion policies further alienate students most in need of help. Implications for the use of story as an assessment tool are discussed, as are implications for substance use prevention policy.     

Building evaluation capacity in Spain: a case study of rural development and empowerment in the European Union

The development of European Community administrative authority has greatly influenced the development of an evaluation culture among the southern and central member states of the European Union. The present case study from Spain provides an example of this diffusion through the use of an empowerment evaluation approach to build evaluation capacity within the context of rural development. The study focuses on the evaluation process over a 10-year period of three local development programs run by three local partnerships created in the rural areas of the Madrid region through the European LEADER initiative. Critical components of empowerment evaluation are discussed, as well as the impact, limitations, difficulties, and applicability of the approach to rural development in the European Union. Conclusions are presented concerning evaluation capacity building and the development of an evaluation culture.     

Building a community-based culture of evaluation

In this article we argue for a community-based approach as a means of promoting a culture of evaluation. We do this by linking two bodies of knowledge – the 70-year theoretical tradition of community-based research and the trans-discipline of program evaluation – that are seldom intersected within the evaluation capacity building literature. We use the three hallmarks of a community-based research approach (community-determined; equitable participation; action and change) as a conceptual lens to reflect on a case example of an evaluation capacity building program led by the Ontario Brian Institute. This program involved two community-based groups (Epilepsy Southwestern Ontarioand the South West Alzheimer Society Alliance) who were supported by evaluators from the Centre for Community Based Research to conduct their own internal evaluation. The article provides an overview of a community-based research approach and its link to evaluation. It then describes the featured evaluation capacity building initiative, including reflections by the participating organizations themselves. We end by discussing lessons learned and their implications for future evaluation capacity building. Our main argument is that organizations that strive towards a community-based approach to evaluation are well placed to build and sustain a culture of evaluation.     

Can a workbook work? Examining whether a practitioner evaluation toolkit can promote instrumental use

In large-scale, multi-site contexts, developing and disseminating practitioner-oriented evaluation toolkits are an increasingly common strategy for building evaluation capacity. Toolkits explain the evaluation process, present evaluation design choices, and offer step-by-step guidance to practitioners. To date, there has been limited research on whether such resources truly foster the successful design, implementation, and use of evaluation findings. In this paper, we describe a multi-site project in which we developed a practitioner evaluation toolkit and then studied the extent to which the toolkit and accompanying technical assistance was effective in promoting successful completion of local-level evaluations and fostering instrumental use of the findings (i.e., whether programs directly used their findings to improve practice, see Patton, 2008). Forensic nurse practitioners from six geographically dispersed service programs completed methodologically rigorous evaluations; furthermore, all six programs used the findings to create programmatic and community-level changes to improve local practice. Implications for evaluation capacity building are discussed.     

Experiments in evaluation capacity building: Enhancing brain disorders research impact in Ontario

This paper is the introductory paper on a forum on evaluation capacity building for enhancing impacts of research on brain disorders. It describes challenges and opportunities of building evaluation capacity among community-based organizations in Ontario involved in enhancing brain health and supporting people living with a brain disorder. Using an example of a capacity building program called the “Evaluation Support Program”, which is run by the Ontario Brain Institute, this forum discusses multiple themes including evaluation capacity building, evaluation culture and evaluation methodologies appropriate for evaluating complex community interventions. The goal of the Evaluation Support Program is to help community-based organizations build the capacity to demonstrate the value that they offer in order to improve, sustain, and spread their programs and activities. One of the features of this forum is that perspectives on the Evaluation Support Program are provided by multiple stakeholders, including the community-based organizations, evaluation team members involved in capacity building, thought leaders in the fields of evaluation capacity building and evaluation culture, and the funders.     

Measuring societal effects of transdisciplinary research projects: design and application of an evaluation method

Most Transdisciplinary Research (TdR) projects combine scientific research with the building of decision making capacity for the involved stakeholders. These projects usually deal with complex, societally relevant, real-world problems. This paper focuses on TdR projects, which integrate the knowledge of researchers and stakeholders in a collaborative transdisciplinary process through structured methods of mutual learning. Previous research on the evaluation of TdR has insufficiently explored the intended effects of transdisciplinary processes on the real world (societal effects). We developed an evaluation framework for assessing the societal effects of transdisciplinary processes. Outputs (measured as procedural and product-related involvement of the stakeholders), impacts (intermediate effects connecting outputs and outcomes) and outcomes (enhanced decision making capacity) are distinguished as three types of societal effects. Our model links outputs and outcomes of transdisciplinary processes via the impacts using a mediating variables approach. We applied this model in an ex post evaluation of a transdisciplinary process. 84 out of 188 agents participated in a survey. The results show significant mediation effects of the two impacts "network building" and "transformation knowledge". These results indicate an influence of a transdisciplinary process on the decision making capacity of stakeholders, especially through social network building and the generation of knowledge relevant for action.     

Evaluating the engagement of universities in capacity building for sustainable development in local communities

Universities have the potential to play a leading role in enabling communities to develop more sustainable ways of living and working however, sustainable communities may only emerge with facilitation, community learning and continual efforts to build their capacities. Elements of programme planning and evaluation on the one hand, and capacity building on the other, are needed. The latter entails approaches and processes that may contribute to community empowerment; universities may either lead such approaches, or be key partners in an endeavour to empower communities to address the challenges posed by the need for sustainable development. Although capacity building and the promotion of sustainable development locally, are on the agenda for universities who take seriously regional engagement, very little is published that illustrates or describes the various forms of activities that take place. Further, there is a paucity of studies that have evaluated the work performed by universities in building capacity for sustainable development at the local level. This paper is an attempt to address this need, and entails an empirical study based on a sample of universities in the United Kingdom, Germany, Portugal and Brazil. The paper examines the extent to which capacity building for sustainable development is being undertaken, suggests the forms that this might take and evaluates some of the benefits for local communities. The paper concludes by reinforcing that universities have a critical role to play in community development; that role has to prioritise the sustainability agenda.     

Engaging Parents in Parentline Plus’ Time to Talk Community Programme as Part of England’s Teenage Pregnancy Strategy: Lessons for Policy and Practice

Based on 65 interviews with professionals and parents conducted during 2007–2008, this 16‐month, mainly qualitative evaluation of Parentline Plus’ Time to Talk Community Programme (a preventative initiative within England’s teenage pregnancy strategy) found that a community development approach and an ethos of partnership with parents and professionals facilitated engagement. Respectful and realistic implementation involved skilled workers building relationships of trust. By consulting with other professionals and parents it was possible to establish local need and to build on existing provision. Initially, insufficient prior local consultation and negotiation created perceptions of an imposed initiative, led by outsiders, resulting in wary local parents and professionals. The engagement and evaluation processes are considered in relation to the short‐term nature of the initiative and potential conflicts between responding flexibly to the needs of parents and delivering targeted programmes.     

Mentoring Policy Research Organizations: Project Evaluation Results

The international community has identified a positive link between good governance and economic development. There is an increasing appreciation of the effective role that local policy research organizations (PROs) in transition and developing countries can play in providing evidence-based policy recommendations as the basis for sound legislation and in assessing the efficacy of existing programs. Capacity building programs for PROs are seen as a key component of growth strategies. The donor community has responded with support for such efforts, which take many forms. On the other hand, the evaluation of the effectiveness of these capacity building and mentoring programs has been inadequate. This article presents the evaluation of one program, focusing on two important capacity building program outcomes: improved research capacity and greater effectiveness in the policy arena. A reflexive design is used. The results suggest the program??s approach is promising.     

Honoring,Training, and Building a Statewide Network of Elder Activists: The California Senior Leaders Program (2002–2012)

This case study explores the first decade (2002–2012) of the California Senior Leaders Program (CSLP), including participants' creation of a formal advocacy group, the California Senior Leaders Alliance. Grounded in concerns with ageism and invisibility, the CSLP provides recognition and support for diverse California elders engaged in community building and social justice work. This study employs qualitative analysis of data from participant interviews, event evaluations, program documents, video footage, and participant observation. Findings show emotional, learning, and networking benefits for participants, intergenerational influences, collective capacity and coalition building, and contributions to policy. Program challenges are described, and future directions discussed.     

Striking a Balance between Program-Specific and Portfolio-Level Evaluation: Lessons Learned from a Multi-Site Evaluation on the Texas-Mexico Border

Funders are increasingly making strategic investments across multiple grantees, aiming for their portfolio to improve targeted outcomes in a specific issue area. To this end, funders might use multi-site evaluation (MSE) approaches to examine the impact of their collective investments. However, it is important to recognize that each program—and its own program evaluation—must be tailored to its setting, population, and local context to best meet the needs of its target population. Therefore, multi-site evaluations need to account for this complexity. This paper describes the Sí Texas project, a large initiative of eight grantees implementing different integrated behavioral health models to improve physical and mental health outcomes along the Texas-Mexico border. With over 4,200 MSE study participants, the evaluation for Sí Texas used a partnership-centered approach to both enhance the evidence base and build local organizational capacity. This paper describes this approach, the process of tailoring evaluation practices to the grantees’ context, and the challenge of balancing consistency at the grantee-level for the portfolio multi-site evaluation. Successes, challenges, and lessons learned related to study design, data collection, grantee partnership, and capacity building are discussed.     

Building evaluation capacity: definitional and practical implications from an Australian case study

Internationally, evaluation capacity-building activities have mushroomed as demands have increased for government-funded programs to demonstrate that they are effective and efficient. Despite this, there is a lack of clarity in the way in which evaluation capacity-building is defined and conceptualized. The current paper presents a case study of a national evaluation capacity building exercise that we are in the midst of conducting in Australia, and discusses the findings in relation to definitional, conceptual and practical issues. Specifically, we describe an evaluation capacity building exercise involving over 100 mental health projects, detailing the methods that we employ, some of the challenges that we have faced, and the benefits we feel we are achieving. Our key message is that definitions of evaluation capacity-building should not only make reference to equipping organizations to routinely conduct evaluations, but should also stress the varied uses to which evaluation findings can be put. In addition, such definitions should acknowledge some of the valuable by-products of evaluation capacity building activities, such as the development of shared understandings of the program or project being evaluated.     

Building capacity for urban management in Ghana: some critical considerations

Severe capacity gaps continue to constrain effective urban management in Ghana, despite numerous donor-supported training programmes. This paper examines the nature of these capacity gaps and analyses specifically the factors contributing to the low attraction and retention of local government staff: politicisation of the bureaucracy, excessive use of external experts, excessive state intervention in skill development and low remuneration of local government staff. Against this background, the paper reviews critically a recent capacity building initiative: the establishment of the (new) Institute of Local Government Studies in Accra. Suggestions to improve capacity building for urban management in Ghana are offered. These include the need to prepare a coherent strategy to guide and co-ordinate stakeholder inputs in capacity building; to place training in the context of institutional development; and encourage the development and use of local expertise in donor assisted projects; and, promote merit-based remuneration of local government staff.     

Evaluation capacity building in the nonformal education context: Challenges and strategies

Policymakers’ demand for increased accountability has compelled organizations to pay more attention to internal evaluation capacity building (ECB). The existing literature about ECB has focused on capacity building experiences and organizational research, with limited attention on challenges that internal evaluation specialists face in building organizational evaluative capacity. To address this knowledge gap, we conducted a Delphi study with evaluation specialists in the United States’ Cooperative Extension Service and developed a consensus on the most pervasive ECB challenges as well as the most useful strategies for overcoming ECB challenges. Challenges identified in this study include limited time and resources, limited understanding of the value of evaluation, evaluation considered as an afterthought, and limited support and buy-in from administrators. Alternatively, strategies found in the study include a shift in an organizational culture where evaluation is appreciated, buy-in and support from administration, clarifying the importance of quality than quantity of evaluations, and a strategic approach to ECB. The challenges identified in this study have persisted for decades, meaning administrators must understand the persistence of these issues and make an earnest investment (financial and human resource) to make noticeable progress. The Delphi approach can be used more often to prioritize ECB efforts.     

Reflections on experiential learning in evaluation capacity building with a community organization,Dancing With Parkinson’s

This paper discusses what was learned about evaluation capacity building with community organizations who deliver services to individuals with neurological disorders. Evaluation specialists engaged by the Ontario Brain Institute Evaluation Support Program were paired with community organizations, such as Dancing With Parkinson’s. Some of the learning included: relationship building is key for this model of capacity building; community organizations often have had negative experiences with evaluation and the idea that evaluations can be friendly tools in implementing meaningful programs is one key mechanism by which such an initiative can work; community organizations often need evaluation most to be able to demonstrate their value; a strength of this initiative was that the focus was not just on creating products but mostly on developing a learning process in which capacities would remain; evaluation tools and skills that organizations found useful were developing a theory of change and the concept of heterogeneous mechanisms (informed by a realist evaluation lens).     

Making urban sector lending work; lessons from a capacity building programme in Karnataka, India

This article presents experiences with human resource development and ‘capacity building programmes’ in urban sector lending by multilateral agencies. The article first reviews the history of ‘capacity building’ and provides a view on its meaning. The experiences with a capacity building programme, part of an infrastructure loan to the state of Karnataka in India, are subsequently elaborated as an illustration. The article seeks to identify the factors determining the success of capacity building programmes which are part of urban lending. It concludes first that Terms of Reference, drafted in a bottom–up and demand-driven fashion are a better guarantee for ownership, commitment and positive results. Secondly, capacity building activities need to be tailored to the ability of the towns to make changes in their operation. Finally, capacity building at local government level can only be effective if reciprocated by supportive measures of state governments. The paper also argues that Technical Assistance (TA) as a component of capacity building should be more strongly integrated in the loan component activities and continue throughout the project period.     

Registered domestic partnerships among gay men and lesbians: the role of economic factors

In this paper, we predict the demand for a marriage-like status—registered domestic partnership–among same-sex couples. Domestic partnership in the state of California now comes with almost all of the rights and responsibilities of marriage that a state can provide. We use the LGBT (Lesbian, Gay, Bisexual, and Transgender) Tobacco Use Survey conducted by the California Department of Health and the Field Research Corporation in 2003. From this telephone survey, we use a probability sample of 1,002 lesbian and gay individuals in California. Using multinomial probit models of partnership status (single, not cohabiting, cohabiting, or registered), we find limited evidence of economic motivations in the choice to register. Gay men’s likelihood of registration rises with income; lesbians’ probability of registration rises with age. Couples with longer duration are more likely to register, suggesting that registration and duration are complementary signals of commitment and possibly of the need for rights and benefits of registration.

Poverty Research in a Development Policy Context

This article provides an evaluation of the extent to which science is addressing worldwide poverty and hunger. It is based on the literature contained in the Thomson Reuters' Web of Science Citation Index (1980 to 2008) and focuses on the WOTRO Strategy Plan 2007–2010 of the Netherlands Foundation for the Advancement of Tropical Research, although similar policies from other countries might also be assessed. The data show that poverty/hunger research has grown steadily over time in many disciplines, most significantly in Environmental Sciences and Technology. Much of this research is hidden; hence the construction of an internationally recognised open‐access database is recommended so that scientists can easily identify critical gaps related to scientific capacity‐building.     

Narrative review of strategies by organizations for building evaluation capacity

Program evaluation is an important source of information to assist organizations to make “evidence-informed” decisions about program planning and development. The objectives of this study were to identify evaluated strategies used by organizations and program developers to build the program evaluation capacity of their workforce, and to describe success factors and lessons learned. Common elements for successful evaluation capacity building (ECB) include: a tailored strategy based on needs assessment, an organizational commitment to evaluation and ECB, experiential learning, training with a practical element, and some form of ongoing technical support within the workplace. ECB is a relatively new field of endeavor, and, while existing studies in ECB are characterized by lower levels of evidence, they suggest the most successful approaches to ECB are likely to be multifaceted. To build the level of evidence in this field, more rigorous study designs need to be implemented in the future.