Narrative review of strategies by organizations for building evaluation capacity

Program evaluation is an important source of information to assist organizations to make “evidence-informed” decisions about program planning and development. The objectives of this study were to identify evaluated strategies used by organizations and program developers to build the program evaluation capacity of their workforce, and to describe success factors and lessons learned. Common elements for successful evaluation capacity building (ECB) include: a tailored strategy based on needs assessment, an organizational commitment to evaluation and ECB, experiential learning, training with a practical element, and some form of ongoing technical support within the workplace. ECB is a relatively new field of endeavor, and, while existing studies in ECB are characterized by lower levels of evidence, they suggest the most successful approaches to ECB are likely to be multifaceted. To build the level of evidence in this field, more rigorous study designs need to be implemented in the future.     

Conducting and utilizing evaluation for multiple accountabilities: A study of nonprofit evaluation capacities

This article combines the evaluation capacity building and nonprofit accountability literature to examine how nonprofits use evaluation data to address stakeholder expectations. Our study investigates how staff competency, technological resources, learning climate, and strategic planning influence a nonprofit's ability to demonstrate upward, lateral, and downward accountabilities. Results indicate different combinations of evaluation capacities matter more for particular stakeholder groups. We argue a more integrative nonprofit accountability requires that managers and staff know how to utilize evaluation results for internal and external audiences. Nuances between specific evaluative capacities and their influences on multiple accountabilities suggest several implications for practice.     

Building evaluative culture in community services: Caring for evidence

An organization with a strong evaluative culture engages in self-reflection, evidence-based learning and experimentation. It sees evidence as essential for managing well, but building such a culture is challenging. Community service organizations seek to provide effective services for their clients. To build an evaluative culture, they need to acquire basic monitoring and evaluation capabilities, be provided with opportunities for using these capabilities and be adequately motivated to care about evidence as a means to improve services to their clients. Leadership along with a phased in approach are key in bringing about these behaviour changes.     

Organizational change in the U.S. Afghan diaspora: A response to homeland events or heightened government scrutiny?

This article examines factors behind a post–September 11, 2001, increase in the number of U.S.‐based Afghan nonprofits registering or filing with the Internal Revenue Service (IRS) for the first time. The nonprofit literature suggests that increased government scrutiny of charitable activities has caused nonprofits to change their behavior. However, empirical evidence behind these claims is lacking. Philanthropy researchers also discuss the response of ethnic diasporas in the United States to crises and other events in their homelands, though rarely from an organizational perspective. This study examines whether heightened government scrutiny, homeland events, or some other factor was most responsible for the increase in U.S.‐based Afghan nonprofits initiating IRS contact following September 11. Survey research finds that Afghan nonprofits, though arguably greater targets for IRS scrutiny due to their Middle East connections, were more motivated by events in their homeland than by heightened government scrutiny to register and file with the IRS.     

Pre- and post-planned evaluation: Which is preferable?

This paper compares the advantages and disadvantages of the “pre-planned” and “post-planned” approaches to evaluating program effectiveness. These evaluative approaches are compared along a number of dimensions which include: (a) Reliability of data and the cost of collecting it; (b) Internal validity; (c) External validity; (d) Evaluation obtrusiveness and threat; and (e) Program goal displacement and program direction. A model designed to help program managers decide when and under what conditions either of these two evaluative approaches should be employed is presented. One major theme throughout this discussion is that despite the growing interest in and use of pre-planned evaluation, the post-planned method has many advantages which often go unnoticed. This paper will help program administrators, planners and evaluators in selecting and implementing these two methods in light of their respective strengths and limitations.     

Evaluating the integration of strategic priorities within a complex research-for-development funding program

This paper examines the application of Complexity Theory constructs to a research-for-development program evaluation and presents an overview of the implications and promising approaches for evaluating complex programs. We discuss lessons learned from an evaluation completed for the International Development Research Centre’s Food, Environment and Health (FEH) program, which investigated the integration and outcomes of five strategic program priorities: partnerships, southern leadership, gender and equity, scale, and environmental sustainability. We present interpretations from a secondary, thematic content analysis that categorized evaluation findings across four complexity constructs: emergence, unpredictability, contradiction and self-organization. Viewing the evaluation through these constructs surfaced some important features of the FEH program to date, specifically its evolving approach, adaptiveness to emergent issues, non-linear outcomes, and self-organizing agents, which had several implications for the evaluative process. We conclude that the most appropriate evaluation designs for complex funding programs are participatory (to explore all stakeholders' influence), adaptive (to capture the unexpected) and assess external contexts. The application of complexity constructs may be useful for evaluators to gain a deeper understanding of how program contexts change in the face of complexity and why some evaluation methods work more effectively than others.     

Collaborative evaluation of a high school prevention curriculum: How methods of collaborative evaluation enhanced a randomized control trial to inform program improvement

Blending high-quality and rigorous research with pure evaluation practice can often be best accomplished through thoughtful collaboration. The evaluation of a high school drug prevention program (All Stars Senior) is an example of how perceived competing purposes and methodologies can coexist to investigate formative and summative outcome variables that can be used for program improvement. Throughout this project there were many examples of client learning from evaluator and evaluator learning from client. This article presents convincing evidence that collaborative evaluation can improve the design, implementation, and findings of the randomized control trial. Throughout this paper, we discuss many examples of good science, good evaluation, and other practical benefits of practicing collaborative evaluation. Ultimately, the authors created the term pre-formative evaluation to describe the period prior to data collection and before program implementation, when collaborative evaluation can inform program improvement.     

“Compassion,pleasantry, and hope”: A process evaluation of a volunteer-based nonprofit

As funders continue to emphasize the importance of documented results, nonprofit organizations must work to complete program evaluations that are both valuable and feasible. The purpose of this paper is to document a practical process evaluation of a southeastern nonprofit, a local Meals on Wheels. Using a mixed methods approach, we sought to answer four evaluation questions: (1) What are the essential program components, as identified by key stakeholders; (2) To what extent are volunteers implementing the identified essential components as intended; (3) What is the level of volunteer satisfaction with the program; and (4) What suggestions do stakeholders have for improving the program? Our findings indicate that most aspects of the program were implemented as intended, but inconsistencies occurred when volunteers were unsure of their assigned duties. In addition, volunteers had high levels of satisfaction and specific suggestions for improvement. From these results, we developed a conceptual model of factors contributing to quality of implementation and volunteer satisfaction that may be generalizable to other volunteer-based nonprofits. Specifically, we identified three factors that helped to facilitate satisfaction and performance: leadership, social contact, and fulfillment. Finally, this process evaluation demonstrates the feasibility of developing and implementing evaluation tools in similar organizations.     

A novel approach for evaluating programs designed to serve military veterans: Using an adapted common components analysis

Evaluators are challenged to keep pace with the vast array of Veteran support programs operating in the United States, resulting in a situation in which many programs lack any evidence of impact. Due to this lack of evidence, there is no efficient way to suggest which programs are most effective in helping Veterans in need of support. One potential solution to this dilemma is to reconceptualize program evaluation, by moving away from evaluating programs individually to evaluating what is common across programs. The Common Components Analysis (CCA) is one such technique that aggregates findings from programs that have undergone rigorous evaluation at the level of program components (e.g., content, process, barrier reduction). Given that many Veteran programs lack outcome evidence from rigorous studies, an adaptation to CCA is needed. This report examines cross-sectional data from a pilot study using an adapted CCA across five domains of well-being (i.e., employment, education, legal/financial/housing, mental/physical health, and social/personal relationships). The purpose of this preliminary study is to determine the feasibility of eliciting program nominations and program components from Veterans via an online survey. When coupled with a longitudinal research design, this adaptation to CCA will allow for stronger causal claims about the expected impact of different program components within and across a variety of domains.     

Puberty Education in a Global Context: Knowledge Gaps,Opportunities, and Implications for Policy

Youth programs and policies provide opportunities for institutions and societies to support healthy adolescent development. Puberty education programs are universally important, as they provide crucial knowledge and skills to help youth and their caregivers navigate the physical, emotional, and interpersonal changes of puberty with positive outcomes. However, few puberty programs have been rigorously evaluated, resulting in a lack of evidence‐based knowledge and practice in this area. This review examines the status of research on puberty education and related programs and draws on the broader intervention literature and recent research findings on adolescence to identify program features that might improve program effectiveness. Implications for policy are also discussed. The need for rigorous program evaluation is emphasized throughout.     

Organizational capacity building: addressing a research and practice gap

The purpose of this article is to address the gap between evaluation research, and the practice of capacity building with nonprofits. This study describes a 5-year capacity building initiative with grassroots organizations including a longitudinal evaluation of the implementation and outcomes achieved. Formative processes yielded many lessons that were used to improve the capacity building model of services. The results show that the majority of groups met a priori expectations for participation success. Organizational staff valued technology, consultants, and program funding the most. Increases were found in board membership and perceptions of visibility of the organization were enhanced. Executive directors reported greater awareness of needs and improved management knowledge. These small organizations fill many unmet needs and more capacity building evaluation studies are needed to understand the mechanisms that support their efforts and the impact on their sustainability.     

Using WWC sanctioned rigorous methods to develop comparison groups for evaluation

Evaluation of program impact in the field of education has been a controversial topic over the years. Although randomized control trials have great advantages in causal inference, they often raise ethical and economic concerns in practice. As an alternative, quasi-experimental designs may provide valid evidence of influence if they are well-designed. In this article, we presented an evaluation case of a district-wide early learning improvement program. To strike a balance between practicability and academic rigor, we developed comparison groups from multiple perspectives, and used a series of tests consistent with WWC 3.0 standards to reach the most valid comparisons. Implications for evaluation practice were discussed.     

Charity Watchdogs and the Limits of Information-Based Regulation

Transparency concerns and the concomitant accountability challenges have motivated policy and legal scholars to explore information-based regulatory approaches. We examine their usefulness in the context of the nonprofit sector which tends to show signs of governance failure. Although nonprofits are required by law to disclose information on fund use, nonprofit donors face difficulties in accessing and interpreting information about how nonprofits are deploying resources. Charity watchdogs make this information available to donors in a convenient format. In theory, this should allow donors to reward nonprofits that devote resources to service delivery and to punish those that are less careful about controlling overheads. To test the relationship between charity ratings and donations, we examine 90 nonprofits in the state of Washington for the period 2004–2007. Drawing on ratings data provided by Charity Navigator, we find that changes in charity ratings tend not to affect donor support to these nonprofits. We explore this statistical finding via interviews with 10 charities located in Washington State. Supporting the statistical results, we find that charities believe that donors tend not to systematically embed ratings in their donation decisions. Instead, they believe that donors assess nonprofits’ effectiveness and trustworthiness via other means such as familiarity, word-of-mouth, or the visibility of the nonprofit in their community. In sum, the policy challenge is to provide information which users desire such as organizational effectiveness as opposed to basic fund allocation in the case of non-profits. What matters for policy efficacy is not how much information is provided but of what type.     

Administrators as information brokers: A managerial perspective on naturalistic evaluation

The managerial perspective on justifying conclusions from naturalistic evaluation is based on how administrators function as brokers who exchange, highlight, and interpret information for others. Information brokering is particularly important within a cognitive perspective on organizations, and consists of process and outcome dimensions. Brokering processes include using evaluative information for conceptualizing, motivating action, and monitoring performance. Brokering outcomes refer to conceptual, instrumental, and symbolic results of information use. The combination of these dimensions results in a variety of substantive and methodological criteria that administrators might use to justify knowledge claims from program evaluation.     

Why so many “rigorous” evaluations fail to identify unintended consequences of development programs: How mixed methods can contribute

Many widely-used impact evaluation designs, including randomized control trials (RCTs) and quasi-experimental designs (QEDs), frequently fail to detect what are often quite serious unintended consequences of development programs. This seems surprising as experienced planners and evaluators are well aware that unintended consequences frequently occur. Most evaluation designs are intended to determine whether there is credible evidence (statistical, theory-based or narrative) that programs have achieved their intended objectives and the logic of many evaluation designs, even those that are considered the most “rigorous,” does not permit the identification of outcomes that were not specified in the program design. We take the example of RCTs as they are considered by many to be the most rigorous evaluation designs. We present a numbers of cases to illustrate how infusing RCTs with a mixed-methods approach (sometimes called an “RCT+” design) can strengthen the credibility of these designs and can also capture important unintended consequences. We provide a Mixed Methods Evaluation Framework that identifies 9 ways in which UCs can occur, and we apply this framework to two of the case studies.     

The Nonprofit Sector and State Governments: Public Policy Issues Facing Nonprofits in North Carolina and Other States

The challenges facing nonprofits in North Carolina and most states include devolution and ripple effects at the state level from federal budget cutbacks, government efforts to tax or further regulate nonprofits, scandals involving a few organizations or their leaders, and efforts to limit nonprofits' right to lobby and advocate for their causes. But the future also holds a number of opportunities for nonprofits, including the increased clout of nonprofits acting together as a sector; partnership initiatives with nonprofits by North Carolina's governor and chief executives in other states, legislative measures to increase charitable giving and reduce red tape, state-level summits following up on the president's push for increased volunteerism, and a reservoir of public trust in nonprofits. North Carolina's experience may predict the future dangers and opportunities in other states.     

Social Service Research and Religion

Abstract

Development of accountability standards to demonstrate cause and effect relationships are gaining rapid advancement in the field of social sciences. Many governmental agencies, foundations, and other funders have developed approaches that require organizations to utilize science-based programs and incorporate evaluative methods to show improved outcomes and cost benefits to society. This article will examine the need for increased accountability in developing effective interventions by faith-based organizations in the delivery of social service interventions. Recently, there has been a strong movement toward governmental funding for faith-based institutions to provide social services, although there has been inadequate scientific data to demonstrate that approaches implemented are effective in meeting needs or yielding favorable outcomes. Similarly, many faith-based organizations provide innovative and effective programs that could serve as model programs if there was appropriate empirical evidence. This article will discuss how rigorous evaluative approaches such as randomized clinical control trials can produce scientific data on program effectiveness. We will use a case example in the field of drug and alcohol treatment to illustrate these points.     

Evolutionary Evaluation: Implications for evaluators,researchers, practitioners,funders and the evidence-based program mandate

Evolutionary theory, developmental systems theory, and evolutionary epistemology provide deep theoretical foundations for understanding programs, their development over time, and the role of evaluation. This paper relates core concepts from these powerful bodies of theory to program evaluation. Evolutionary Evaluation is operationalized in terms of program and evaluation evolutionary phases, which are in turn aligned with multiple types of validity. The model of Evolutionary Evaluation incorporates Chen's conceptualization of bottom-up versus top-down program development. The resulting framework has important implications for many program management and evaluation issues. The paper illustrates how an Evolutionary Evaluation perspective can illuminate important controversies in evaluation using the example of the appropriate role of randomized controlled trials that encourages a rethinking of “evidence-based programs”. From an Evolutionary Evaluation perspective, prevailing interpretations of rigor and mandates for evidence-based programs pose significant challenges to program evolution. This perspective also illuminates the consequences of misalignment between program and evaluation phases; the importance of supporting both researcher-derived and practitioner-derived programs; and the need for variation and evolutionary phase diversity within portfolios of programs.     

The evolution of Rare Pride: Using evaluation to drive adaptive management in a biodiversity conservation organization

Rare Pride is a social marketing program that stimulates human behavior change in order to promote biodiversity conservation in critically threatened regions in developing countries. A series of formal evaluation studies, networking strategies, and evaluative inquiries have driven a 20-year process of adaptive management that has resulted in extensive programmatic changes within Pride. This paper describes the types of evaluation that Rare used to drive adaptive management and the changes it caused in Pride's theory-of-change and programmatic structure. We argue that (a) qualitative data gathered from partners and staff through structured interviews is most effective at identifying problems with current programs and procedures, (b) networking with other organizations is the most effective strategy for learning of new management strategies, and (c) quantitative data gathered through surveys is effective at measuring program impact and quality. Adaptive management has allowed Rare to increase its Pride program from implementing about two campaigns per year in 2001 to more than 40 per year in 2009 while improving program quality and maintaining program impact.     

Expanding the repertoire of evaluation tools so that evaluation recommendations can assist nonprofits to enhance strategic planning and design of program operations

Data are the lifeblood of decision-making and provide a critical component into the complex mosaic from which decisions emerge. Evaluators and decision-makers should, therefore, continuously seek to explore the use of tools that can produce more meaningful, insightful, and useful data so that decision-making can be enhanced and improved. Traditional economic appraisal methods offer much and are very useful and relevant. Standard financial accounting reporting for tax purposes also provide insightful information on the organization’s financial health and that of its individual programs. However, neither offer insight into cost behavior—an important consideration for making program operations more efficient and for long-term strategic planning, forecasting, and design of program operations. Before COVID-19, programs were already challenged with competition for donor dollars and the pursuit for financial sustainability. The COVID-19 environment has considerably exacerbated these challenges. In this current environment, it is more important than ever for nonprofits to optimize limited financial resources to do more societal good. This paper will illustrate how selected concepts from cost and management accounting can be used by both independent program evaluators to improve their recommendations, as well as program administrators in-charge of nonprofits to enhance decision-making.