Are cost-inclusive evaluations worth the effort?

Relative costs are as important as relative effectiveness when choosing between program alternatives or among a set of programs competing for scarce funds. Nevertheless, the number of cost-inclusive evaluations remains comparatively small. This article presents the results of three first-time cost-inclusive evaluations each performed by an experienced evaluator. Each evaluator performed a different type of cost-inclusive evaluation using different tobacco control programs as examples: "standard" cost-effectiveness analysis, threshold or break-even analysis, and a simulation model. Results are presented in terms of the challenges faced, and informational and insight benefits gained, as well as in terms of program cost-effectiveness. All three evaluators agreed that the benefits from performing cost-inclusive evaluations are well worth time and effort involved. They also found that this type of evaluation provides abundant information that can be used to improve program effectiveness and cost-effectiveness.     

Should we provide oral health training for staff caring for people with intellectual disabilities in community based residential care? A cost-effectiveness analysis

Oral health training is often introduced into community-based residential settings to improve the oral health of people with intellectual disabilities (ID). There is a lack of appropriate evaluation of such programs, leading to difficulty in deciding how best to allocate scarce resources to achieve maximum effect. This article reports an economic analysis of one such oral health program, undertaken as part of a cluster randomized controlled trial. Firstly, we report a cost-effectiveness analysis of training care-staff compared to no training, using incremental cost-effectiveness ratios (ICERs). Effectiveness was measured as change in knowledge, reported behaviors, attitude and self-efficacy, using validated scales (K&BAS). Secondly, we costed training as it was scaled up to include all staff within the service provider in question. Data were collected in Dublin, Ireland in 2009. It cost between €7000 and €10,000 more to achieve modest improvement in K&BAS scores among a subsample of 162 care-staff, in comparison to doing nothing. Considering scaled up first round training, it cost between €58,000 and €64,000 to train the whole population of staff, from a combined dental and disability service perspective. Less than €15,000–€20,000 of this was additional to the cost of doing nothing (incremental cost). From a dental perspective, a further, second training cycle including all staff would cost between €561 and €3484 (capital costs) and €5815 (operating costs) on a two yearly basis. This study indicates that the program was a cost-effective means of improving self-reported measures and possibly oral health, relative to doing nothing. This was mainly due to low cost, rather than the large effect. In this instance, the use of cost effectiveness analysis has produced evidence, which may be more useful to decision makers than that arising from traditional methods of evaluation. There is a need for CEAs of effective interventions to allow comparison between programs. Suggestions to reduce cost are presented.     

Employee Assistance Program Evaluation in a Federal Government Agency

Demonstrating the efficacy of an Employee Assistance Program is an equally important endeavor in public and private sector work organizations. This paper discusses the cost-benefit and cost-effectiveness evaluation that was conducted by the U.S. Department of Health and Human Services Employee Counseling Services program. Further, it details the components of the 30 month project relative to evaluation design, data collection processes, and strategies used to maintain employee privacy while analyzing personnel data. Results of the evaluation are also presented.     

Monitoring and evaluation design of Malawi’s Right Foods at the Right Time nutrition program

Child stunting is a public health problem in Malawi. In 2014, the Government of Malawi launched the Right Foods at the Right Time (RFRT) program in Ntchisi district delivering nutrition social and behavior change communication, a small-quantity lipid-based nutrient supplement to children 6–23 months, and nutrition sensitive activities. Monitoring and evaluation (M&E) systems are key aspects of successful program implementation. We describe these and the methodology for an impact evaluation that was conducted for this program. Two monitoring systems using traditional and electronic platforms were established to register and track program delivery and processes including number of eligible beneficiaries, worker performance, program participation, and to monitor input, output, and outcome indicators. The impact evaluation used comparative cross-sectional and longitudinal designs to assess impact on anthropometric and infant and young child feeding outcomes. Three cross-sectional surveys (base-, mid-, and end-line) and two longitudinal cohorts of children followed in 6-month intervals from 6 to 24 months of age, were conducted in sampled households in the program and a neighboring comparison district. Additional M&E included qualitative studies, a process evaluation, and a cost-effectiveness study. The current paper describes lessons from this program’s M&E, and demonstrates how multiple implementation research activities can inform course-correction and program scale-up.     

Assessing Employee Assistance Programs:

There is a great need for better understanding of an overall evaluation scheme of EAPs. The author presents four types or phases of EAP evaluation: (a) intervention planning; (b) program monitoring; (c) impact assessment; and (d) economic efficiency. Under this typology, seven common models of evaluation methods are elaborated with some procedural references: (1) need assessment; (2) program development; (3) utilization analysis; (4) outcome evaluation; (5) net effect assessment; (6) cost-benefit analysis; and (7) cost-effectiveness analysis.     

Cost-effectiveness of family-based substance abuse treatment

In order to compete in a financially sensitive health care system, family systems-based treatments must demonstrate effective clinical results as well as cost-effectiveness. Cost-effectiveness research can demonstrate to health care insurers and policy makers which treatments are viable options for implementation. The present literature review identified eight cost-effectiveness family-based substance abuse treatment studies. The results suggest that certain family-based treatments are cost-effective and warrant consideration for inclusion in health care delivery systems.     

Resolving conflicting demands on evaluation: A test demonstration of videotex for farmers

Evaluating an innovation for federal, state, and local policymakers and program managers alike entails conflicting demands on the evaluation study. Policymakers at federal, state, and local levels are best assisted by impact evaluations, whereas state and local program managers are best assisted by process evaluations. In-house evaluators often have an advantage in conducting process evaluations; external evaluators generally have an advantage in conducting impact evaluations. A cost-effective approach may be to combine in-house process evaluation and external impact evaluation. This dual approach was found to reduce conflicting demands on the evaluation of an experimental videotex system for agricultural producers.     

The economic evaluation of a Participatory Ergonomics programme to prevent low back and neck pain

This study was an economic evaluation conducted alongside a cluster randomised controlled trial with a follow-up of 12 months. The aim was to evaluate the cost-effectiveness and cost-benefit of the Stay@Work Participatory Ergonomics programme (PE) compared to a control group (no PE). In total, 37 departments (n=3047 workers) were randomised into either the intervention (PE) or control group (no PE). During a meeting, working groups followed the steps of PE, and composed and prioritized ergonomic measures aimed to prevent low back pain (LBP) and neck pain (NP). Working groups had to implement the ergonomic measures within three months in their department. Cost data included those directly related to LBP and NP. Cost-effectiveness analyses (CEA) and cost-benefit analyses (CBA) were performed. After 12 months, health care costs and costs of productivity losses were higher in the intervention group than in the control group (the mean total cost difference was $/euro$127; 95% CI $/euro$-164 - $/euro$418). From a societal perspective, the CEA showed that PE was not cost-effective compared to control for LBP and NP prevalence, work performance, and sick leave. The CBA from a company perspective showed a monetary loss of $/euro$78 per worker. The PE programme was neither cost-effective nor cost-beneficial on any of the effect measures.     

Mentoring as a Bridge to Positive Outcomes for Teen Mothers and Their Children

This paper compares historical and psychosocial realities of teenage parenting to prevailing contemporary myths. The authors explore the negative consequences of current remedies based on these myths. Community volunteer mentoring programs are suggested as a cost-effective intervention that can benefit communities and promote positive outcomes for at-risk pregnant teenagers. The authors suggest strategies for implementation of a community volunteer mentoring program based on an existing program in an urban community health center. Detailed information for social workers in community agencies is provided with regard to community education, recruitment, training, and ongoing support and supervision of mentors; funding requirements; and program evaluation.     

A model for a comparative cost-effectiveness evaluation of two mental health partial care programs.

This study provides a model for comparing the cost-effectiveness of two mental health programs. The example cited examines the cost-effectiveness of a hospital-based and a community-based partial care program. Using each group as its own control, the authors compare patients' use of inpatient hospitalisation and emergency room services during one year prior to and one year subsequent to first admission in day care. In both day care programs, patients substantially reduced their number of inpatient admissions and days. Neither program, however, showed a significant decrease in patients' use of the emergency room. Reflecting the substantially lower costs of the community-based program, the benefit-cost and cost-effectiveness ratios were significantly better in the community-based program than in the hospital-based program. Based on this evidence the authors conclude that the lower-cost, equally effective community-based program should be carefully considered as an alternative for more partial care patients.     

Skill acquisition and client movement indices. Implementing cost-effective analysis in rehabilitation programs.

This article outlines the development and implementation of a cost-effective approach to client and program evaluation. The indices presented summarize a client's acquisition of behavioral skills (Skill Acquisition Index) and progression along a continuum of becoming less dependent and more productive (Client Movement Index). Data are presented summarizing how the two non-monetary outcome measures have been used in formative program evaluation. The paper also discusses some of the problems involved in implementing cost-effective analyses in rehabilitation programs and suggests specific strategies to overcome the problems.     

Cost-inclusive evaluation: a banquet of approaches for including costs, benefits, and cost-effectiveness and cost-benefit analyses in your next evaluation

An introduction to the special issue on cost-inclusive evaluation, providing a brief history of the use of costs, benefits, cost-effectiveness, and cost-benefit analyses in the evaluation of human services. Two tables present brief glossaries of terms and analyses common in cost-inclusive program evaluation.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

A Focus on the Program of All-Inclusive Care for the Elderly (PACE)

For over four decades, the Program of All-Inclusive Care for the Elderly (PACE) has been operated by nonprofit organizations. Research has demonstrated that nonprofit PACE provides quality, cost-effective community-based care to older adults who would otherwise require a nursing home level of care. Recently, the U.S. Secretary of the Department of Health and Human Services has authorized for-profit entities to operate PACE, contingent on their ability to demonstrate that they can provide care that is similar to nonprofit PACE with regard to access to care, quality of care, and cost-effectiveness. In 2013, a study was conducted to evaluate how PACE operates under for-profit versus nonprofit status. The results were presented to Congress which, in turn, authorized for-profit PACE providers. This article critiques the 2013 study, offers a comparison to for-profit hospice, and argues that at best there is not enough evidence to conclude that for-profit PACE provides the same quality of care as existing nonprofit operators.     

Evidence from the field: Findings on issues related to planning,implementing and evaluating gender-based programs

An Initiative of the United States Department of Health and Human Services’ Office on Women's Health (OWH), Coalition for a Healthier Community (CHC), supports ten grantees across the U.S. in the implementation of gender-based health interventions targeting women and girls. A national evaluation is assessing whether gender-focused public health systems approaches are sustainable and cost effective in addressing health disparities in women and girls. To inform the evaluation, a systematic examination was conducted of literature in both the public and private sector designed to track, assess, understand, and improve women's health, public health systems approaches, and the cost-effectiveness and sustainability of gender-based programs. A two-person team assured the quality of the results following the review of abstracts and full-text articles. Of 123 articles meeting eligibility criteria (See inclusion criteria described in Section 2.2 below), only 18 met inclusion criteria specific to a focus on a systems approach, cost-effectiveness and/or sustainability. Studies assessing systems approaches suggested their effectiveness in changing perceptions and increasing knowledge within a community; increasing involvement of local decision-makers and other community leaders in women's health issues; and increasing community capacity to address women and girls’ health. Further evaluation of the cost-effectiveness and sustainability of gender-based approaches is needed.     

Approaches to caring, essential elements for a quality service and cost-effectiveness in South Africa

By 2010 there will be close to two million orphans in South Africa, mainly as a result of HIV/AIDS. This paper assesses different approaches to the care and support of children orphaned by AIDS and other vulnerable children, as well as the cost-effectiveness of each approach. Using a typology of care and essential elements of care, six approaches are evaluated: informal, non-statutory foster care; community-based support; home-based care; unregistered residential care; statutory adoption and fostering; and statutory residential care. A cost-effectiveness analysis assessed actual programs and the costs of providing a minimum standard of care for the six approaches. High costs are associated with formal models of care. Informal approaches may lack the resources to meet children's rights. Resources should be largely allocated to the more cost-effective, informal, community-based structures, but formal models will still be needed for those children who cannot be placed elsewhere.     

A strategy for accurate collection of incremental cost data for cost-effectiveness analyses in field trials. Pawtucket's minimal contact cholesterol education intervention

With many community field trials or education interventions, the cost-effectiveness analyses are not given a high priority. However, this type of evaluation is important for purposes of future adoption of the intervention. The accurate measurement of costs can best be served by prospective collection of data. This article describes a methodology for collection of cost data that coincides with the intervention implementation. This cost analysis strategy has seven discrete steps. The Minimal Contact Education for Cholesterol Change study is used as an example of the use of this strategy. This intervention provides cholesterol education at six different levels of intensity at four different sectors. The intensity levels vary along a continuum from very little education input to a maximum level of intervention that might be practical in a screening setting. The cost-effectiveness analysis component of the study will identify the incremental cost-effectiveness of each intervention along the continuum.     

The development and field testing of a vocational education evaluation model

Reported is the development and field testing of an evaluation model used to evaluate the vocational education programs at the Metropolitan Community Colleges (MCC) of Kansas City, Missouri which can be implemented on vocational programs at other institutions. The model evaluated a vocational program on the following six criteria: (1) program's relationship to job market profile; (2) program's level of community support; (3) program's success in meeting vocational aspirations of clientele; (4) program's success in terms of student performance; (5) program 's cost-effectiveness; and (6) program 's success in reaching the handicapped and disadvantaged. The evaluation model provides a strategy and a set of procedures for assessing the quality of existing vocational education programs of study at any post-secondary institution. The model can be easily and inexpensively implemented, thus providing a data base which is useful for administrative planning and decision making.     

Evaluability assessment of a community development leisure program in Spain

This paper presents the results of an evaluability assessment (EA) conducted in Spain. EAs are assessments that minimize the uncertainty and risk inherent in a subsequent evaluation (e.g. an impact or results assessment) that may require excessive resources. We designed our own EA model by combining three theoretical approaches, with a holistic and pedagogic vision of the process and a flexible methodology to enable adaptation to different programs and contexts, thus fostering transferability. The EA was carried using a qualitative methodology: documentary analysis, seven semi-structured interviews with stakeholders, eight direct observation sessions and a check-list for politicians and experts. The outcome report helped promote organizational learning, provided the program with a logical and assessable theoretical model, detected its strengths and helped overcome the barriers to efficient assessment. Based on these achievements, an evaluation of the results was recommended, with repeated pretest-posttest measurements and an equivalent control group.     

Program decisions : Evaluating a bird in the hand versus two in the bush

In new programs that have not reached full effectiveness, or in programs that are designed to produce future benefits (i.e., become “institutionalized”), evaluations that fail properly to assess future performance may bias decisions against the programs. For such programs, consideration of future benefits may significantly enhance the assessment of their cost-effectiveness. This paper describes an evaluation methodology based on decision theory that can take future benefits into account in evaluating a program. The methodology was applied to the Metropolitan Atlanta Crime Commission (MACC) Community Anti-Crime Project and indicated the potential for substantial future benefits subsequent to cessation of federal funding. The implications of use of this methodology and the MACC results for the overall Community Anti-Crime Program are also discussed.