The devil is in the details: Implementing secular outcome measurement methods in faith‐based organizations

Faith‐based organizations (FBOs) are now being encouraged to seek federal funding for the purpose of supporting faith‐based delivery of social services in local communities (that is, the Compassionate Capital Fund). This shift in the federal perspective has resulted in increased attention to the effectiveness of the services provided by FBOs. Though rigorous demonstrations are needed to determine the relative effectiveness of FBO services, other available alternatives offer a viable avenue to aid FBOs in improving their programs and monitoring their success. One such alternative is the use of outcome measurement techniques, such as those that have been infusing the nonprofit sector in the United States since the mid‐1990s, particularly among United Way–funded programs. This chapter discusses how to adapt secular outcomes measurement approaches to faith‐based organizations and their programs and the likely pitfalls that FBOs will face.     

Testing Faith

Abstract

The U.S. federal government, through Charitable Choice, has opened public funding for the delivery of social services to faith-based organizations (FBOs) more than ever before. This increased access to governmental funding at all levels has led to a closer examination of the evidence base on the effectiveness of the services provided by FBOs, and the capacity of FBOs to respond to data needs for accountability and program improvement efforts. This paper discusses the current status of evaluation research on FBO services and the emerging data needs among faith-based providers. Promising avenues for enhancing the current understanding of outcomes of FBO services are explored such as (1) adopting outcome measurement practices in use within the current nonprofit sector, and (2) developing more rigorous research designs tailored to the special contexts of faith-based services. The paper suggests an agenda in which FBO operators, funders, and evaluators work together to move forward in improving the evidence base on faith-based services.     

Assessing child welfare outcomes in Central and Eastern Europe

Despite the declared priority given to children's well-being in Central and Eastern Europe, the effectiveness of services has never been measured. Financial considerations and differences in the objectives of those involved have increased the importance of outcome measurement.     

Accountability revisited: the arrival of client outcome evaluation

In the past few years, "accountability" for public mental health programs has become differentiated in the minds of not only program evaluators, but also program managers and funders, including state and local-level legislators. Increasingly, these officials are becoming concerned with more than just the numbers and targets of services delivered, and the cost involved, and are looking for evidence of positive outcome or impact on clients to justify program implementation and maintenance. This represents a significant move beyond the two accountability models that most recently seemed to be the focus of most formal accountability efforts--performance measurement and quality assurance. Pressures for implementing these two alternatives seem to have been reduced somewhat by the new federal Administration, but even prior to its advent there had been a rapid escalation in awareness of and concern for client outcome measurement among important audiences, including state and local mental health policy-makers and the U.S. Congress. This presents a major new opportunity and challenge for program evaluators at this new accountability focus continues to gather momentum.     

Prevention validation and accounting platform: a framework for establishing accountability and performance measures of substance abuse prevention programs

The field of substance abuse prevention has neither an overarching conceptual framework nor a set of shared terminologies for establishing the accountability and performance outcome measures of substance abuse prevention services rendered. Hence, there is a wide gap between what we currently have as data on one hand and information that are required to meet the performance goals and accountability measures set by the Government Performance and Results Act of 1993 on the other. The task before us is: How can we establish the accountability and performance measures of substance abuse prevention programs and transform the field of prevention into prevention science? The intent of this volume is to serve that purpose and accelerate the processes of this transformation by identifying the requisite components of the transformation (i.e., theory, methodology, convention on terms, and data) and by introducing an open forum called, Prevention Validation and Accounting (PREVA) Platform. The entire PREVA Platform (for short, the Platform) is designed as an analytic framework, which is formulated by a collectivity of common concepts, terminologies, accounting units, protocols for counting the units, data elements, and operationalizations of various constructs, and other summary measures intended to bring about an efficient and effective measurement of process input, program capacity, process output, performance outcome, and societal impact of substance abuse prevention programs. The measurement units and summary data elements are designed to be measured across time and across jurisdictions, i.e., from local to regional to state to national levels. In the Platform, the process input is captured by two dimensions of time and capital. Time is conceptualized in terms of service delivery time and time spent for research and development. Capital is measured by the monies expended for the delivery of program activities during a fiscal or reporting period. Program capacity is captured by fourteen measurement units, tapping into the dimensions of staff resources and community assets. Staff resources are, in turn, operationalized in terms of staff size, staff certification status, staff turnover rate, and the accreditation status of a provider agency. Community assets are operationalized by the number of community centers accessible to the funded agency, number of formalized teams or antidrug coalitions active in the catchment area, and other social/human services providers with whom the prevention agency has formalized networks. The totality of process output from all sources of program activities is reduced to eighteen classes of measures. These are operationalized by thirty-three summary measures. Some of these include: total count of events facilitated; total number of clients served; average number of clients served per event; clients served by single and multiple program sessions; classification of target population in terms of the severity of risk as defined by the Institute of Medicine; age groups and race/ethnicity of clients served; number of program participants retained by recurring programs; number of clients who have completed the program; penetration rates to the target population; client attrition rates; average referral rates per provider per time interval; referral success rates; and so on. All process output measures specified in the Platform are derived from two broad classes of events classified as either products or services. The collectivity of these measures is expected to present a cost-effective, parsimonious, yet comprehensive picture of the entire spectrum of the process output, i.e., "what came out of the program as program activities". For the measurement of performance outcomes, two types of data are incorporated into the Platform: outcome data from individuals and the behavior (or performance) of social indicators from aggregated data bases. Individual data are used to evaluate the outcome of substance abuse programs     

New Zealand Vietnam Veterans’ Family Programme,Nga Whanau a Tu (Families of War): Development and Outcome

This article reviews recent research concerned with the association between combat-related posttraumatic stress disorder (PTSD) and interpersonal functioning, before describing the development of a pilot program established to provide mental health services for Vietnam veteran family members. The results of a brief program evaluation are also presented. Sixty clients provided posttreatment and six-month follow-up data on a variety of outcome measures which were compared with independent ratings returned by 33 therapists. On average, clients reported that counselling had been beneficial and indicated satisfaction with services received. However, at posttest, therapists indicated that approximately 50% of clients were in need of continued treatment. There was a decrease in satisfaction with services over the posttest-follow-up period, but no change on most measures of psychotherapy outcome.     

BIG DATA AND BIG CITIES: THE PROMISES AND LIMITATIONS OF IMPROVED MEASURES OF URBAN LIFE

New, “big data” sources allow measurement of city characteristics and outcome variables at higher collection frequencies and more granular geographic scales than ever before. However, big data will not solve large urban social science questions on its own. Big urban data has the most value for the study of cities when it allows measurement of the previously opaque, or when it can be coupled with exogenous shocks to people or place. We describe a number of new urban data sources and illustrate how they can be used to improve the study and function of cities. We first show how Google Street View images can be used to predict income in New York City, suggesting that similar imagery data can be used to map wealth and poverty in previously unmeasured areas of the developing world. We then discuss how survey techniques can be improved to better measure willingness to pay for urban amenities. Finally, we explain how Internet data is being used to improve the quality of city services. (JEL R1, C8, C18)     

Outcomes of a randomized trial of continuum of care services for children in a child welfare system

The Connecticut Department of Children and Families Title IV-E waiver demonstration evaluated whether the well-being of children approved for residential mental health services could be improved, and lengths of stay in restrictive placements reduced, by providing case rate payments to community agencies to provide continuum of care services. Children between ages 7 and 15 were randomly assigned to either the demonstration group (n = 78) or to usual state-supported services (n = 79). One-year outcome results indicated that in a situation that is less costly, improvement in outcomes occurred in less restrictive settings. Continuum of care services were more effective in 1) returning children to in-home placements, 2) reducing the length of stay in restrictive placements, and (3) utilizing higher levels of case management through coordination among agencies and family support services.     

Crisis nursery effects on child placement after foster care

The results of a study of the relationship between receiving crisis nursery services and the placement outcomes for young children leaving the child welfare system in Illinois are reported in this paper. The placement outcomes for children leaving foster care whose families received crisis nursery support prior to the children's placement in foster care is compared to the placement outcomes for children whose families received only foster care services. The children in two samples were identified by matching crisis nursery children's data from FY 2006 with children's data in the Illinois Child Abuse and Neglect Tracking System and Children Youth and Services Information System databases. After children served by crisis nursery and foster care services were identified, a comparison group of children with like-characteristics whose families received only foster care services was identified using propensity score matching. The children were followed until their out-of-home placement was terminated or until June 30, 2009. The placement outcomes and the length of stay were compared for the two groups. Using logistical regression analysis the results showed that children whose families received crisis nursery services prior to foster care placement were twice as likely to be reunited with their biological families (birth or extended family members) when compared to children whose families received only foster care services. The difference in the length-of-stay in foster care was not statistically significant when the two groups were compared. This preliminary study using administrative data shows that receiving crisis nursery services may have positive effects on the children's ultimate placement outcome after foster care. Additional research is needed to further explore the relationship between placement outcome and crisis nursery services.     

Variables Associated with Positive Treatment Outcomes for Children Surviving Sexual Abuse

This exploratory study examined variables associated with positive treatment outcomes for 47 children who received sexual abuse treatment. Variables associated with a higher percentage of treatment objectives accomplished were: services provided to the pre-adolescent child; child resided at home; perpetrator was criminally charged; support by non-offending caretaker currently and at disclosure; individual, family and group therapy were provided, and termination of services was planned. Variables associated with a positive placement outcome were: services provided to the pre-adolescent child; abuse did not involve body penetration; child resided at home; child was never removed from home; services were not court-ordered; and termination was planned. A higher percentage of applicable treatment objectives accomplished was significantly correlated with a positive placement outcome.     

Quality Assessment Study of Brief Psychotherapy at a University HMO Setting

Abstract

This paper describes the results of a short-term psychotherapy (four to fifteen sessions) quality assessment study conducted at the University of Massachusetts Mental Health Service. The outcome measures devised and the process of their administration are described. The positive outcome results of this study are encouraging and supportive of the impression that high quality brief psychotherapy can be provided at university counseling services. Student and health maintenance organization (HMO) member consumer evaluation opinions were positive and their utilization of services was found to be psychologically sophisticated and appropriate. A frequency table lists the most common presenting problems of this population. Practical modifications are suggested for future quality assessment research of brief psychotherapy.     

Institutional Pressures and Organizational Capacity: The Case of Outcome Measurement1

A central claim of new institutional theory is that organizations in a field come to exhibit shared characteristics over time. Recent literature emphasizes variation across field members, but has yet to concur on why differences occur. This study tests institutional explanations for the uneven implementation of one organizational practice—outcome measurement, an evaluative technique used to assess the impact of an organization’s programs. We analyze data from a new survey investigating the practices of nonprofit organizations (N = 379) and argue for the inclusion of the concept of organizational capacity to account for the uneven implementation of outcome measurement. As predicted by new institutional theory, organizations are more likely to adopt outcome measurement if key actors promulgate its use. However, the implementation of outcome measurement is best explained by the addition of the concept of organizational capacity alongside variables drawn from new institutionalism. Nonprofits with adequate organizational capacity, operationalized—following Weber’s concept of bureaucracy—as the presence of written rules and members with specialized knowledge, are better able to respond to isomorphic pressures to implement a new organizational practice. Our findings expand scholarship that examines the intersection of institutional dynamics and organizational traits in accounting for patterns of implementation of practices across an organizational field.     

Cross-system data linkage for treatment outcome evaluation: lessons learned from the California Treatment Outcome Project

Using administrative data to evaluate health care outcomes has become increasingly common, but the reliability and validity of outcome measures based on cross-system data linkage have been little scrutinized. Applying a deterministic data matching methodology, we linked 6545 Californians admitted to 43 substance abuse treatment programs between 2000 and 2001 to administrative data acquired from three state agency databases. We compared self-reported treatment outcome measures to equivalent measures derived from official records on motor vehicle driving incidents, criminal history, and mental health services utilization. Administrative data provided analogous results in some measures (e.g., percent of people using mental health services, percent ever arrested) and more accurate results in others (e.g., frequency of service utilization, and frequency of arrests). Similar to findings based on the interview data, the administrative data also revealed improvements in several domains 1-year post-treatment compared to 1-year pre-treatment. Experiences with data linkage procedures and strategies for enhancing record linkage accuracy are discussed. Findings illustrate the value of using administrative records for substance abuse treatment outcome evaluation, while highlighting areas for improvement for future cross-system data linkage efforts.     

The impacts of accessible child protection services on services, jobs, and families

This multi-faceted study assessed the differences between accessible and central child protection service delivery sites in Ontario on: (1) family functioning; (2) front-line child protection service providers' satisfaction with services; (3) parents' satisfaction with services; (4) the services and supports available to families; (5) front-line service job satisfaction; and (6) front-line helping relationships in child welfare. A quasi-experimental outcome design gathered data from parents at case-opening and approximately 10 months later. Semi-structured individual or group qualitative interviews were conducted with parents, front-line service providers, and supervisors/managers. Information also was gathered from agency files. Front-line service providers completed an employment questionnaire. Findings indicated that the accessible sites were able to foster a greater proportion of cooperative helping relationships with families and to create more service partnerships. Parents at the accessible sites were more positive about their service involvements and more willing to ask for help again in the future. Front-line child protection service providers at both service delivery models described positive aspects of their jobs as well as frustrations with the expectations of the formal child protection system.     

What makes motherhood so expensive?: The role of social expectations, interdependence, and coordination failure in explaining lower wages of mothers

This paper utilizes expectations imposed by society to explain lower wages of mothers compared to non-mothers in the labor market. Social expectation, interdependence between mothers’ labor supply and childcare services, and lack of coordination between employers, employees (mothers), and childcare services explain how an economy can be caught in a ‘trap’ exhibiting large wage differences. Higher levels of human capital, increased coordination along with affordable childcare services in a more modern economy allows for specialization, increasing returns, and the possibility for a better outcome due to multiple equilibria.     

Assessing outcomes for sufferers of severe mental disorder: A conceptual framework and review

A conceptual framework is presented for classifying the outcomes of services for sufferers of severe mental disorders. The proposed classification framework contains three dimensions: (a) the respondent type; (b) the social context; and (c) the domain of treatment outcomes. Respondent types include the client, the family, the extended social network, the clinician and the scientist. Social contexts include the individual, the family, work/school and the broader community. Domains of treatment outcome include “clinical status,” “functional status,” “life satisfaction and fulfillment,” and “safety and welfare.” This multidimensional framework is used to organize an illustrative review of existing outcome measures. It is concluded that: (a) measures of safety and welfare raise important social and policy questions and these measures require considerable development in order for utility to be realized; (b) although a good beginning has been made, more attention needs to be focused on defining and measuring “life satisfaction and fulfillment” and combining this domain with others to produce “well being” or “quality of life” scales; (c) the functional status area requires additional research and development especially when measurement is conducted in contexts other than work/school; and (d) although measures of clinical status are relatively well developed, the family perspective and context must be integrated into these measures.     

Self‐Help Career Services: A Case Report

Self‐help career services represent an important form of career intervention rarely described in the literature. Using a case‐report approach, the author describes this type of career intervention, which was guided by cognitive information processing theory and Holland's ( 1997 ) typological theory. The report offers an example of appropriately identifying client readiness to engage in career counseling using a 3‐tier model for service delivery composed of direct, brief, and self‐help interventions. It also demonstrates an effective method for translating theory to practice, a particularly critical need in the career development field. The outcome of the intervention supports a differentiated model of career services that increases client accessibility to those services.     

Individual treatment of intimate partner violence perpetrators

This article outlines a rationale for investigating the individual (one-on-one) treatment format and individualized (case-tailored) services for partner abuse perpetrators. Many state standards caution against or prohibit individual services in abuser intervention. However, initial research indicates that motivational interviewing, conducted individually, can increase abusive clients' engagement in the change process. Challenges of using the group format in treatment development are discussed along with potential benefits of individual treatment for this population. Notably, individual treatment can be adapted to the client's stage of change, can address a range of presenting concerns (such as substance abuse and mood disorders) that may influence outcome, and can be used to focus clinical attention on case-specific change targets while avoiding potentially negative and antisocial peer influences in the group format. Nevertheless, individual treatment has been almost entirely ignored to date in clinical research with this population.     

Joint commission evaluation of behavioral health care organizations

The joint Commission on Accreditation of Healthcare Organizations has accredited behavioral health care organizations since 1957. Accredited behavioral health care organizations include organizations providing services to persons with psychiatric disorders, substance abuse disorders, or developmental disabilities/mental retardation; or, provide social services to other related populations. In the late 1980s, the Joint Commission initiated the redesign of accreditation to include performance measurement. Performance measurement initiatives include the IMSystem, Requests for Indicators, National Library of Healthcare Indicators, and a national Council on Performance Measurement. These initiatives are steps toward the integration of performance measurement data into the accreditation process.     

The use of performance measurement in South Carolina nonprofits

Performance measurement is touted by nonprofits as the newest method of ensuring accountability. Knowing the value of the process while having an appreciation for the downside with regard to volume of work can help organizations make more educated decisions about how deeply they want to get involved with performance or outcome measurement. This article provides background information on performance measurement and examples of how it is being used in nonprofit organizations. Next, it presents methods of an exploratory study conducted in South Carolina. Finally, a discussion of results focuses on the effect of outcome measurement on management and funding as well as the need for more in‐depth research.