Collective Impact: A Review of the Peer-reviewed Research

ABSTRACT

Collective Impact (CI) is a framework for collaborative social change that has gained considerable attention over the past seven years. In this article we review the peer-reviewed academic research on the CI framework to understand what is known about its use. Nineteen articles were located. We identified four themes in the research: A need to adapt CI to the specific context, its technical nature, the importance of relationships, and concerns about power and equity. The literature on the CI framework is in its infancy and as such tends to focus on capacity building or model development. Whilst CI promotes important discussion about collaboration and social change, issues of power, equity and grass-roots engagement require further attention.

IMPLICATIONS

• Peer reviewed academic research on the application of CI is at an early stage and further research is needed.

• Social workers should carefully consider critiques of CI before adopting this framework for social change.

     

Fostering CHANGE: Lessons from implementing a multi-country,multi-sector nutrition-sensitive agriculture project

Using agriculture to improve nutrition is an approach growing in popularity, with programs becoming increasingly complex and multisectoral. While there is an active line of research assessing the impacts of such programs, little has been written about the process of successfully implementing them. As such, this paper uses a multisectoral nutrition-sensitive agriculture program implemented in four African countries as a case study to address key challenges in and lessons learned from implementation. We highlight the overall flexibility of nutrition-sensitive agriculture but also the need to adapt certain aspects to the particular context, as well as the opportunities for cross-context learning (and the limits to this). Integrating rigorous evaluation into such complex programs and forging diverse cross-sectoral partnerships offer both rewards and challenges, upon which we reflect. Main lessons learned from the program include the importance of carefully sequencing interventions, retaining flexibility in implementation, allowing for considerable time for cross-sector integration and coordination, and considering community impacts when designing research.     

People,Pipelines, and Probabilities: Clarifying Significance and Uncertainty in Environmental Impact Assessments

Determinations of significance play a pivotal role in environmental impact assessments because they point decision makers to the predicted effects of an action most deserving of attention and further study. Impact predictions are always subject to uncertainty because they rely on estimates of future consequences. Yet uncertainty is often neglected or treated in a perfunctory manner as part of the characterization, evaluation, and communication of anticipated consequences and their significance. Proposals to construct fossil fuel pipelines in North America provide a highly visible example; casual treatment of how uncertainty affects significance determinations has resulted in poorly informed stakeholders, frustrated industry proponents, and inconsistent choices on the part of public decision makers. Using environmental assessments for recent pipeline proposals as examples, we highlight five ways in which uncertainty is often neglected when determining impact significance and suggest that a mix of known methods, new guidelines, and appropriate oversight could greatly improve current practices.     

Using theory of change as a framework for examining community context and philanthropic impact

As part of the process of examining their theory of change (TOC) and reflecting on grant making activity, one Midwestern foundation employed a multipronged strategy to assess 209 community-based mental health grants across seven years of funding. This article details the evaluation approach, which comprised these areas of the TOC: grantees’ use of evidence-based interventions, cultural competency, quality improvement, community collaboration, and use of integrated care. Inductive analyses identified grantees’ use of innovative practices, trauma-informed care, and use of validated instruments. In an iterative exercise spanning multiple years, the foundation and researchers found ways to gather information about community capacity and impact while simultaneously minimizing grantee data collection/reporting burden. Findings revealed that community agencies were improving in their uptake of evidence-based interventions and use of validated instruments. Community grantee use of trauma informed care also improved over the study period. Grantee desire to collaborate and coordinate services within the community was also strong. The research also revealed that in multiple domains (such as technology and non-profit operating costs) grantees needed support in building agency capacity. These findings shed light on which aspects of the foundation’s TOC were being addressed through funding, and which areas of the community needed additional support.     

Contribution analysis to analyze the effects of the health impact assessment at the local level: A case of urban revitalization

The health impact assessment (HIA) is a tool used to estimate the potential impact on health of non-health-related proposals prior to implementation. While it is increasingly used in Quebec, Canada, studies have not analyzed its medium-term impacts and potential long-term impacts. We conducted a contribution analysis using in-depth interviews with key stakeholders, as well as documents, observation and images related to HIA in order to analyze its impacts on the revitalization of road infrastructure, parks and green spaces, and residential housing. Our analysis not only reflects on the decision-making process through the adoption and implementation of HIA recommendations, but also on the link between actions implemented in the field and health outcomes.     

Developing a methodology to assess the impact of research grant funding: A mixed methods approach

This paper discusses the development of a mixed methods approach to analyse research funding. Research policy has taken on an increasingly prominent role in the broader political scene, where research is seen as a critical factor in maintaining and improving growth, welfare and international competitiveness. This has motivated growing emphasis on the impacts of science funding, and how funding can best be designed to promote socio-economic progress. Meeting these demands for impact assessment involves a number of complex issues that are difficult to fully address in a single study or in the design of a single methodology. However, they point to some general principles that can be explored in methodological design. We draw on a recent evaluation of the impacts of research grant funding, discussing both key issues in developing a methodology for the analysis and subsequent results. The case of research grant funding, involving a complex mix of direct and intermediate effects that contribute to the overall impact of funding on research performance, illustrates the value of a mixed methods approach to provide a more robust and complete analysis of policy impacts. Reflections on the strengths and weaknesses of the methodology are used to examine refinements for future work.     

Evaluation of the impact of intervention programmes on education organisations: Application to a Quality Management System

This paper analyses the importance of evaluating the various components of the programmes or actions carried out by education organisations. It highlights the need to assess the impact of the intervention on the organisation and consider how changes are consolidated over time in interaction with the context. We propose an impact evaluation model and as an example have chosen the implementation of Quality Management Systems in schools. The paper analyses the results obtained in 40 schools in three regions (Spanish Autonomous Communities) with varying levels of implementation. The results show overall impact on these education centres as the teachers and management teams of the centres perceive it. This impact is more evident in some of the dimensions considered in the study than in others. The results also confirm the differences between regional contexts.     

Are Overall Journal Rankings a Good Mapping for Article Quality in Specialty Fields?

Overall journal rankings, which are generated with sample articles in different research fields, are commonly used to measure the research productivity of academic economists. In this article, we investigate a growing concern in the profession that the use of the overall journal rankings to evaluate scholars’ relative research productivity may exhibit a downward bias toward researchers in some specialty fields if their respective field journals are under-ranked in the overall journals rankings. To address this concern, we constructed new journal rankings based on the intellectual influence of research in 8 specialty fields using a sample consisting of 26,401 articles published across 60 economics journals from 1998 to 2007. We made various comparisons between the newly constructed journal rankings in specialty fields and the traditional overall journal ranking. Our results show that the overall journal ranking provides a considerably good mapping for the article quality in specialty fields. Supplementary materials for this article are available online.     

Causal inferences on the effectiveness of complex social programs: Navigating assumptions,sources of complexity and evaluation design challenges

This paper explores avenues for navigating evaluation design challenges posed by complex social programs (CSPs) and their environments when conducting studies that call for generalizable, causal inferences on the intervention’s effectiveness. A definition is provided of a CSP drawing on examples from different fields, and an evaluation case is analyzed in depth to derive seven (7) major sources of complexity that typify CSPs, threatening assumptions of textbook-recommended experimental designs for performing impact evaluations. Theoretically-supported, alternative methodological strategies are discussed to navigate assumptions and counter the design challenges posed by the complex configurations and ecology of CSPs. Specific recommendations include: sequential refinement of the evaluation design through systems thinking, systems-informed logic modeling; and use of extended term, mixed methods (ETMM) approaches with exploratory and confirmatory phases of the evaluation. In the proposed approach, logic models are refined through direct induction and interactions with stakeholders. To better guide assumption evaluation, question-framing, and selection of appropriate methodological strategies, a multiphase evaluation design is recommended.