Identifying and controlling for program-level differences in comparative cost analysis: lessons from the economic evaluation of the National Breast and Cervical Cancer Early Detection Program

Performing economic evaluations of established health care programs is essential to identify and control for underlying program-level variations and to make valid comparisons. At a time when the need for such evaluations is growing, health care professionals have limited information on the methodological challenges of performing these evaluations. In this study, we used the National Breast and Cervical Cancer Early Detection Program to illustrate these potential underlying variations. We performed site visits to four grantees and collected activity-based cost data from nine additional representative programs. We identified five specific types of cost factors that should be considered when evaluating and comparing health care programs: clinical services, service mix, in-kind contributions, indirect costs, and year-to-year expenditures of specific activities. A key lesson is that case studies and pilot testing should be performed before initiating cost analysis to identify underlying variation and to test appropriate methods to adequately control for these differences.     

Register data in the evaluation and program planning of addiction treatment programs: Using Sweden as an example

Data from large-scale registers is often underutilized when evaluating addiction treatment programs. Since many programs collect register data regarding clients and interventions, there is a potential to make greater use of such records for program evaluation. The purpose of this article is to discuss the value of using large-scale registers in the evaluation and program planning of addiction treatment systems and programs. Sweden is used as an example of a country where register data is both available and is starting to be used in national evaluation and program planning efforts.The article focuses on possibilities, limitations and practicalities when using large-scale register data to conduct evaluations and program planning of addiction treatment programs. Main conclusions are that using register data for evaluation provides large amounts of data at low cost, limitations associated to the use of register data may be handled statistically, register data can answer important questions in planning of addiction treatment programs, and more accurate measures are needed to account for the diversity of client populations.     

State “Technical Assistance Programs” for Nursing Home Quality Improvement: Variations and Potential Implications

To improve nursing home quality, many states have developed “technical assistance programs” that provide on-site consultation and training for nursing facility staff. We conducted a national survey on these state programs to collect data on program design, operations, financing, and perceived effectiveness. As of 2010, 17 states had developed such programs. Compared to existing state nursing home quality regulations, these programs represent a collaborative, rather than enforcement-oriented, approach to quality. However, existing programs vary substantially in key structural features such as staffing patterns, funding levels, and relationship with state survey and certification agencies. Perceived effectiveness by program officials on quality was high, although few states have performed formal evaluations. Perceived barriers to program effectiveness included lack of appropriate staff and funding, among others. In conclusion, state technical assistance programs for nursing homes vary in program design and perceived effectiveness. Future comparative evaluations are needed to inform evidence-based quality initiatives.     

The cost of implementing a jail diversion program for people with mental illness in San Antonio,Texas

Jail diversion programs for people with mental illness are designed to redirect offenders with mental illness into community treatment. Although much has been published about program models and their successes, little detail is available to policy makers and community stakeholders on the resources required to start and implement a jail diversion program and which agencies bear how much of the burden. The current study used data on a model jail diversion program in San Antonio, Texas, to address this research gap. Data on staff costs, client contacts, planning, and implementation were collected for three types of diversion: pre-booking police, post-booking bond, and post-booking docket. An activity-based costing algorithm was developed to which parameter values were applied. The start-up cost for the program was $556,638.69. Pre-booking diversion cost $370 per person; 90% of costs were incurred by community mental health agencies for short-term monitoring and screening (>80% of activities). Post-booking bond and docket diversion cost $238 and $205 per person, respectively; the majority of costs were incurred by the courts for court decisions. Developing a multiple-intercept jail diversion program requires significant up-front investment. The share of costs varies greatly depending on the type of diversion.     

Cost-utility analysis in educational needs assessment

Educational needs assessments (hereafter NAs) are inattentive to cost considerations and are frequently dominated by elite stakeholder groups. In this article I make a case for adopting a cost–utility approach, illustrating the argument with data generated in a NA of central library services in a Canadian school district. Using survey data from eight stakeholder groups, I found that (1) NAs based on the service preferences of a single stakeholder group can be misleading; (2) service preferences can be integrated into a single set of priorities, even when there are disagreements, by using the stakeholder group as the unit of analysis and assigning weights that privilege input from knowledgeable respondents; and (3) that the ranking of service operations produced by user preferences was not significantly correlated with the ranking produced by integrating preferences with costs. Cost–utility analysis would be more helpful if the utilities represented rigorously determined benefits of the services assessed, as well as stakeholder perceptions of the value of these benefits. Cost–utility analysis in NA will not reach its potential until cost considerations are routinely included in educational program evaluations.     

Self-Employment as a Social and Economic Development Intervention for Recipients of AFDC

Summary

This paper reports findings from program evaluations of two demonstrations designed to (1) test self-employment as a social and economic development strategy for recipients of AFDC, and (2) identify policy barriers to improving the economic well-being of this population. The authors collected data through interviews with program participants and program operators and used secondary data contained in operators' management information systems and state Department of Human Services electronic files. Program results showed that self-employment is a viable social and economic development strategy for self-selected welfare recipients who receive business training and assistance in locating operating capital. The paper discusses implications for social work practice, education, and research.     

Response Effects in the Electronic Survey

This report examines the electronic survey as a research tool.In an electronic survey, respondents use a text processing programto self-administer a computer-based questionnaire. As more peoplehave access to computers, electronic surveys may become widespread.The electronic survey can reduce processing costs because itautomates the transformation of raw data into computer-readableform. It can combine advantages of interviews (e.g., prompts,complex branching) with those of paper mail surveys (e.g., standardization,anonymity). An important issue is how the electronic surveyaffects the responses of people who use it. We conducted anexperimental sample survey on health attitudes, behaviors, andpersonal traits using two forms of administration: electronicand paper mail. Closed-end responses in the electronic surveywere less socially desirable and tended to be more extreme thanwere responses in the paper survey. Open-ended responses thatcould be edited by respondents were relatively long and disclosing.These findings are consistent with other research on computer-mediatedcommunication, raising general issues about using computersto collect self-report data.     

The effect of alternative staff time data collection methods on drug treatment service cost estimates

Although a limited number of service cost estimates exist, no study has evaluated how differences in the method used to collect the staff time allocation across treatment services contribute to differences in service cost estimates. Three alternative data collection methods for estimating service-level costs in methadone treatment programs were evaluated: key informants, staff surveys, and staff diaries. We analyzed data from 25 methadone clinics across the United States. Results indicate that for the three primary services offered at methadone clinics—individual counseling, group counseling, and methadone dosing—no statistically significant differences exist in the mean estimates of costs per session across programs. Of the other five services analyzed, we found no statistically significant differences in two of the mean costs per session and a small but statistically significant difference in another service. We found large and statistically significant differences in mean costs for two services, initial patient assessment and initial medical services. Although there is no gold standard available to judge which method is the best to use, we concluded that the key informant method yields more reliable cost estimates compared with the staff methods and is less burdensome to both the treatment programs and to researchers. Our findings suggest that the key informant method is the preferred method for costing substance abuse treatment services.     

Enhancing the safety of children in foster care and family support programs: automated critical incident reporting

The Adoption and Safe Families Act of 1997 has made child safety an explicit focus in child welfare. The authors describe an automated critical incident reporting program designed for use in foster care and family-support programs. The program, which is based in Lotus Notes and uses e-mail to route incident reports from direct service staff to supervisors and administrators, facilitates timely clinical oversight and risk management and ensures the security of clients' protected health information. The authors present data collected using the program to illustrate how it can be used to monitor abuse and neglect allegations in a foster care program. A survey of users found that the program saved time, was easy to use, and helped manage critical incident reports.     

State-level evaluations of school programs funded under the Drug-Free Schools and Communities Act.

Although the Drug-Free Schools and Communities Act of 1986 and the 1989 Amendments to the Act require states to evaluate their drug-education programs, no guidelines for conducting these evaluations have been produced, and little has been reported on how the states are conducting such evaluations. In this article, the results of a telephone survey on current state-level efforts to evaluate school programs funded under the Act are reported. Some states report studies of the implementation of the program and some report drug- and alcohol-use surveys. Together, these two types of evaluation efforts form the foundation of an approach for conducting evaluations under the Act. Reasons are presented why experimental and quasi-experimental designs might be inappropriate and impractical for the evaluations, and an evaluation approach linking program implementation findings and drug- and alcohol-use survey results is suggested.     

The effects of Project SUCCESS on student academic performance: a quasi-experimental study

Project SUCCESS (PS) is a substance use prevention program that targets indicated high school students. We used archival data to explore the program's effects on students' academic achievement and disciplinary problems. It is essential to demonstrate such effects, if prevention curricula are to survive in schools that face multiple competing demands for class time. Using a quasi-experimental design that compared students exposed to the program to those not exposed in five schools, we examined the effects of PS on grades, absenteeism, and disciplinary problems. Using longitudinal analyses that linked individual students' program exposure data to their academic performance, we found that PS had a positive impact on both grades and absenteeism for two years following exposure. We note that these findings are particularly fortuitous, as PS was not designed to affect these outcomes. We recommend that future evaluations of school-based prevention curricula routinely include academic performance.     

Using rapid city surveys to inform municipal social policy—an application in Cali, Colombia

Many developing countries assign local governments increasing responsibilities in fighting poverty. This requires local social policy to go beyond the execution of centrally designed and funded education and health programs. Hence, local governments and their partners have both an opportunity and a need to analyze key local bottlenecks for poverty reduction and social development. Drawing on an example from Cali, Colombia, this paper describes a tool for such policy formulation at the local level, a rapid city household survey. Although the survey uses pre-coded and closed ended questions, it is contextual in the sense that it is tailor-made to social and economic conditions in Cali. The survey places particular emphasis on collecting both key quantitative information, like household welfare and service access, as well as qualitative information, such as service evaluations and population priorities. Combining the quantitative and qualitative data allows, e.g., to map population budget priorities or service satisfaction levels by welfare group. With costs of the survey moderate and turnover time short, rapid city household surveys could provide an important tool for the development of local social policies.     

The unique character of involvement in multi-site evaluation settings

As the number of large federal programs increases, so, too, does the need for a more complete understanding of how to conduct evaluations of such complex programs. The research literature has documented the benefits of stakeholder participation in smaller-scale program evaluations. However, given the scope and diversity of projects in multi-site program evaluations, traditional notions of participatory evaluation do not apply. The purpose of this research is to determine the ways in which stakeholders are involved in large-scale, multi-site STEM evaluations. This article describes the findings from a survey of 313 program leaders and evaluators and from follow-up interviews with 12 of these individuals.Findings from this study indicate that attendance at meetings and conferences, planning discussions within the project related to use of the program evaluation, and participation in data collection should be added to the list of activities that foster feelings of evaluation involvement among stakeholders. In addition, perceptions of involvement may vary according to breadth or depth of evaluation activities, but not always both. Overall, this study suggests that despite the contextual challenges of large, multi-site evaluations, it is feasible to build feelings of involvement among stakeholders.     

Expanding the repertoire of evaluation tools so that evaluation recommendations can assist nonprofits to enhance strategic planning and design of program operations

Data are the lifeblood of decision-making and provide a critical component into the complex mosaic from which decisions emerge. Evaluators and decision-makers should, therefore, continuously seek to explore the use of tools that can produce more meaningful, insightful, and useful data so that decision-making can be enhanced and improved. Traditional economic appraisal methods offer much and are very useful and relevant. Standard financial accounting reporting for tax purposes also provide insightful information on the organization’s financial health and that of its individual programs. However, neither offer insight into cost behavior—an important consideration for making program operations more efficient and for long-term strategic planning, forecasting, and design of program operations. Before COVID-19, programs were already challenged with competition for donor dollars and the pursuit for financial sustainability. The COVID-19 environment has considerably exacerbated these challenges. In this current environment, it is more important than ever for nonprofits to optimize limited financial resources to do more societal good. This paper will illustrate how selected concepts from cost and management accounting can be used by both independent program evaluators to improve their recommendations, as well as program administrators in-charge of nonprofits to enhance decision-making.     

Use of social impact bonds to address social problems: Understanding contractual risks and transaction costs

Social impact bonds, a recent innovation in social finance, are designed to harness capital and knowledge from private nonprofit, for‐profit, and public entities to address pressing social problems. Although there is great policy interest in understanding how social impact bonds can be used to tackle social problems, the emergent nature of social impact bonds makes it hard to find relevant data and evidence. To overcome this challenge, we use single‐significant‐case sampling as our research design strategy. We conduct an in‐depth case study of the Social Innovation Financing Youth Recidivism Project in Massachusetts. Our case study is comprised of a qualitative analysis of the multiparty contract and multiyear quantitative benefit–cost analysis to understand transaction costs. We draw upon contract theory to develop an analytical framework for the case analysis and highlight the risks and safeguards for the various parties to the contract, and conduct a formal benefit–cost analysis to map out transaction costs. We conclude with a discussion of study implications and future research.     

Are cost-inclusive evaluations worth the effort?

Relative costs are as important as relative effectiveness when choosing between program alternatives or among a set of programs competing for scarce funds. Nevertheless, the number of cost-inclusive evaluations remains comparatively small. This article presents the results of three first-time cost-inclusive evaluations each performed by an experienced evaluator. Each evaluator performed a different type of cost-inclusive evaluation using different tobacco control programs as examples: "standard" cost-effectiveness analysis, threshold or break-even analysis, and a simulation model. Results are presented in terms of the challenges faced, and informational and insight benefits gained, as well as in terms of program cost-effectiveness. All three evaluators agreed that the benefits from performing cost-inclusive evaluations are well worth time and effort involved. They also found that this type of evaluation provides abundant information that can be used to improve program effectiveness and cost-effectiveness.     

Context matters: Using mixed methods timelines to provide an accessible and integrated visual for complex program evaluation data

The need for conducting evaluations which reflect of the influence of context on complex programs is increasingly recognized in the field of evaluation. Better data visualization techniques for connecting context with program evaluation data are needed. We share our experience developing a mixed methods timeline to visualize complexity and context with evaluation data. Mixed methods timelines provide a meaningful way to show change over time in both a visually stimulating and accessible format for evaluation audiences. This paper provides an innovative example of using mixed methods timelines to integrate evaluation data with key program activities and milestones, while also showing internal and external contextual influences in one cohesive visual. We present methods and best practices for collecting contextual data and for incorporating a variety of data sources into such a visual. We discuss several strategies to collect and organize context related data including: qualitative interviews, program materials, narrative reports, and member checking with stakeholders and staff. Gathering multiple perspectives is essential to better capture the multi-layered elements of program activities and context.     

DESIGN AND EVALUATION OF A MIGRATION EXPERIMENT IN THAILAND

In many less developed countries, planners are concerned about the concentration of population in one or two major cities. The authors implemented a small-scale experimental program in Northeast Thailand designed to alter patterns of outinigration from a rural area, specifically to redirect migration to urban centers within the same region rather than toward Bangkok. Using data from a baseline survey and a follow-up survey 12 months later, strong effects on mobility behavior, modest effects on information, and little or no effect on community evaluations and mobility plans are discerned. Implications for the attitude-behavior literature are discussed.