Using propensity scoring to estimate program-related subgroup impacts in experimental program evaluations

This article discusses the use of propensity scoring in experimental program evaluations to estimate impacts for subgroups defined by program features and participants' program experiences. The authors discuss estimation issues and provide specification tests. They also discuss the use of an overlooked data collection design--obtaining predictions that program intake staff make about applicants' likely program assignments and experiences--that could improve the quality of matched comparison samples. They demonstrate the effectiveness of this approach in producing credible subgroup findings using data from a large-scale experimental evaluation of Job Corps, the nation's largest federal education and training program for disadvantaged youths.     

Does receiving an earnings supplement affect union formation? Estimating effects for program participants using propensity score matching

This article demonstrates a novel application of propensity score matching techniques: to estimate nonexperimental impacts on program participants within the context of an experimental research design. The author examines the relationship between program participation, defined as qualifying for an earnings supplement by working full-time, and marital union formation among low-income mothers in two Canadian provinces. The author finds that receipt of an earnings supplement substantially increased union formation in one province but not the other. A subgroup analysis based on propensities of program participation revealed that the positive effect on unions was concentrated among relatively disadvantaged participants. The techniques demonstrated in the article are broadly applicable to studies in which take-up is less than 100% among those randomly assigned to a program group.     

Maisons Familiales -- Senegal

The local associations of Maisons Familiales (MF) in Senegal periodically conduct participatory evaluations of community projects. 2 evaluations often used include internal exercises by and for the staff, such as a written questionnaire, and an assisted self evaluation. An assisted self evaluation often involves participant subgroups discussing problems and possible solutions with each subgroup later sharing items with a national and/or a foreign evaluation facilitator. The facilitator(s) meets with all the subgroups and brings out important issues in the subgroups, then all the subgroups discuss the issues and form a consensus on what actions should be taken. The training staff at an MF center thought the program was fine based on what a few people said, but, after looking at statistics on the number of trainees over a couple of years, the staff learned that the numbers have declined. The staff then discussed the situation and learned that a barrier had developed between the training staff and villagers. As in any evaluation, one must distinguish between the subjective view (what people say) and objective reality (the actual situation using data). In another type of self evaluation, a group discussed dynamism in a village and came up with 4 different points of view. After visiting a "dynamic" and a "nondynamic" village using the 4 points as measurements, the group learned that its previous impressions of the 2 villages were not completely borne out. This evaluation helped the staff to see villagers' priorities and to listen better. Despite wanting to conduct a real impact evaluation, workers have not yet done so because they do not have time to schedule evaluations, do not have enough base line data, and do not know how to account for influences on changes in the villages other than the MF training programs.     

Program implementation and evaluation results: A reexamination of type III error in a field experiment

This paper reexamines the implementation results obtained from a true experiment in criminal justice. The original analyses found a significant difference in employment outcome between the control group and those experimental clients who received the highest level of treatment. The present analyses attempt to bracket true treatment effects by partitioning both experimental and control groups into treated and untreated subgroups, and by combining data from two independent sources. The original results do not obtain when implementation of the control treatment is also considered in the analysis. Corroborating earlier findings, a positive association was found between magnitude of experimental treatment implementation and employment, but it was unclear that the experimental program was responsible for this relationship. The limitations of the experimental design for disentangling effective components of treatment are discussed. The paper provides an example of the utility of secondary analysis for understanding evaluation results.     

Register data in the evaluation and program planning of addiction treatment programs: Using Sweden as an example

Data from large-scale registers is often underutilized when evaluating addiction treatment programs. Since many programs collect register data regarding clients and interventions, there is a potential to make greater use of such records for program evaluation. The purpose of this article is to discuss the value of using large-scale registers in the evaluation and program planning of addiction treatment systems and programs. Sweden is used as an example of a country where register data is both available and is starting to be used in national evaluation and program planning efforts.The article focuses on possibilities, limitations and practicalities when using large-scale register data to conduct evaluations and program planning of addiction treatment programs. Main conclusions are that using register data for evaluation provides large amounts of data at low cost, limitations associated to the use of register data may be handled statistically, register data can answer important questions in planning of addiction treatment programs, and more accurate measures are needed to account for the diversity of client populations.     

Meta-analysis: an introduction using regression models

Research synthesis of evaluation findings is a multistep process. An investigator identifies a research question, acquires the relevant literature, codes findings from that literature, and analyzes the coded data to estimate the average treatment effect and its distribution in a population of interest. The process of estimating the average treatment effect is meta-analysis. Meta-analysis is gaining popularity across the social sciences, but introductions and even advanced treatments of the subject are often formula-driven. Researchers, who apply formulas without understanding, risk both misapplication and misinterpretation. This article derives formulas used in meta-analysis using standard regression models so this article seeks to introduce readers with an understanding of regression analysis to meta-analysis, intending thereby to motivate and facilitate a deeper reading of the meta-analysis literature.     

The impact profile approach to policy merit. The case of research grants and the university.

In writings on the theory of valuing, many take the position that impacts on the relevant outcome dimensions should be aggregated to arrive at one summary assessment of program merit. A contrary position is taken here, specifying that the impacts should be kept separate and unweighted and expressed only in their own original measurement scales. All impacts, however, should be portrayed, including those for which no rigorous data analysis has been carried out. It is argued that aggregating, even by the individual stakeholder, is both futile and misleading. An extensive evaluation of the effects of research grants on the university is included as a full-scale illustration of the method. Financial impacts are considered, as well as impacts on faculty and student quality, on university prestige, and on the quality of instruction.     

A new method for finding hierarchical subgroups from networks

We present a new method for decomposing a social network into an optimal number of hierarchical subgroups. With a perfect hierarchical subgroup defined as one in which every member is automorphically equivalent to each other, the method uses the REGGE algorithm to measure the similarities among nodes and applies the k-means method to group the nodes that have congruent profiles of dissimilarities with other nodes into various numbers of hierarchical subgroups. The best number of subgroups is determined by minimizing the intra-cluster variance of dissimilarity subject to the constraint that the improvement in going to more subgroups is better than a network whose n nodes are maximally dispersed in the n-dimensional space would achieve. We also describe a decomposability metric that assesses the deviation of a real network from the ideal one that contains only perfect hierarchical subgroups. Four well known network data sets are used to demonstrate the method and metric. These demonstrations indicate the utility of our approach and suggest how it can be used in a complementary way to Generalized Blockmodeling for hierarchical decomposition.     

Social choice with independent subgroup utility scales

In this article, the kinds of utility comparisons that can be made may differ in distinct population subgroups. Within each subgroup, utility is either ordinally or cardinally measurable. Levels and differences of utility may or may not be interpersonally comparable within a subgroup. No utility comparisons are possible between subgroups. Given these informational assumptions, it is shown that any continuous social welfare ordering that satisfies the weak Pareto principle only depends on the utilities of one of the subgroups. The class of social welfare orderings consistent with these assumptions is determined by the scale type of the dictatorial subgroup. Received: 25 May 1999/Accepted: 4 November 1999     

Implemented as intended? Recording family worker activities in a Families First program

In evaluating family preservation services, it is important not only to study the service outcomes and the family characteristics, but also what actually happens during the treatment. This requires a program model. This article describes how a program model works, prescribes how workers should carry it out, and describes how researchers should measure the program's characteristics. The authors use data from Families First in The Netherlands to test the model. Results show that the method of the program meets the specified characteristics. The results are important for treatment, policy, education, and evaluation research.     

A mixed membership model-based measure for subgroup integration in social networks

Social networks analysis often involves quantifying subgroup structure in which tie density is greater among nodes in the same subgroup than between subgroups. One such measure, subgroup insularity or segregation, is the extent that subgroups are separate from each other. We introduce a new measure, γ, which is a parameter from the mixed membership stochastic blockmodel (MMSBM; Airoldi et al., 2008), and differs from many existing measures in that γ does not depend on node membership. We compare this measure to several well-known measures and use simulation studies and real data analysis to provide insight into how this measure can be used in practice.     

Estimating the effects of network covariates on subgroup insularity with a hierarchical mixed membership stochastic blockmodel

Social networks describe the relationships and interactions among a group of individuals. In many peer relationships, individuals tend to associate more often with some members than others, forming subgroups or clusters. Subgroup structure varies across networks; subgroups may be insular, appearing distinct and isolated from one another, or subgroups may be so integrated that subgroup structure is not visually apparent, and there are numerous ways of quantifying these types of structures. We propose a new model that relates the amount of subgroup integration to network attributes, building on the mixed membership stochastic blockmodel (Airoldi et al., 2008) and subsequent work by Sweet and Zheng (2017) and Sweet et al. (2014). We explore some of the operating characteristics of this model with simulated data and apply this model to determine the relationship between teachers’ instructional practices and their classrooms’ peer network subgroup structure.     

The role of postsecondary education in welfare reform. Ohio's JOBS Student Retention Program

The 1996 federal welfare reform legislation encourages quick employment over education and training for the nation's welfare recipients. However, some argue that a one-size-fits-all approach ignores the heterogeneity of this population. This article presents findings from a net impact evaluation of Ohio's JOBS Student Retention Program (JSRP), a program designed to facilitate success for public assistance recipients at 2-year community or technical colleges. The authors evaluate this policy using state administrative data sets. The analyses consist of unadjusted and regression-adjusted comparisons of means for the JSRP group and a constructed comparison group. Outcomes examined include program completion, employment, earnings, and welfare recipiency. Focusing on the most recent 11 of 16 quarters of data available, the average increase in quarterly earnings was 8.45% for program participants and 12.91% for program completers. The results indicate that encouraging postsecondary education for some welfare recipients will boost earnings capacity and therefore long-term self-sufficiency.     

Evaluation of a treatment program for abusive and high-risk families in Spain

This article describes a six-year evaluation of a comprehensive treatment program for abusive families in Spain. The evaluation involved practitioners' clinical judgments, standardized measures, and follow-up data about children's living situations. The results were similar to those obtained by similar programs from the United States and other developed countries. The analysis obtained the lowest rates of success for neglectful and abusive-neglectful families. Dropout and nondropout families differed in two paternal characteristics: alcohol problems and childhood experience of out-of-home care. Rehabilitated and nonrehabilitated families differed in several variables, including time elapsed between case detection and referral to the treatment program. Scores on measures showed significant changes during treatment.     

Evaluating the vocational rehabilitation program using longitudinal data. Evidence for a quasiexperimental research design.

The study presents benefit-cost ratios for 14 disability cohorts served by the Vocational Rehabilitation (VR) Program. The earnings impacts are estimated in a quasiexperimental framework using an internal comparison group. The earnings data are from a unique national panel constructed by linking client data of the Rehabilitative Services Administration with earnings histories from the Social Security Administration. These earnings data accommodate a series of statistical tests that allow us to identify and control for the presence of selection bias when estimating treatment impacts. The results indicate that the VR program is cost-effective in general, although not universally so across specific disabilities.     

Cultivating Compassion for Self and Others: A School-Based Pilot Study for Peer-Nominated Caring Adolescents

Many have called for school-based student programs that teach skills related to self-care and caring for others. Here, such a program for peer-nominated adolescents was developed and piloted virtually at one high school during the COVID-19 pandemic. Results of a longitudinal, quasi-experimental evaluation of the program showed high-quality program implementation and promising program impacts. Effect sizes indicated moderate to large program impacts on improvements in adolescents' self-compassion, sense of interdependence, and perspective-taking, and female adolescents' interoceptive awareness, compared to controls. No group differences in compassion for others were found. The need for more research on programs that help adolescents balance compassion for the self and for others is discussed.     

Evaluation of quality in social care: aplus program

France is not advanced regarding evaluation in social work, despite a law established in January 2002 making evaluation a legal obligation every 5 years. This article presents a software program to help social services evaluate on both individual and group levels. Automatic dashboard results of the program with special emphasis on the main qualities social work would usually present reciprocity and social competence, client ability development, and social care treatment components. The last point is an automatic selection of most efficient treatment components of ability development.     

The differential impact of inflation on key societal interest groups and public policy implications

An inflation problem exists when there is a perceived negative imbalance in the input-output position of a particular group. In this article the inflation problem is disaggregated by type of input-output imbalance, by degree of severity of the imbalance, and by vested interest group. The disaggregated model captures the inverse relationship between price changes and the inflation problem for subgroups, and the duality of the impact of price changes on particular subgroups. The empirical specification of the model makes explicit the diverse and often conflicting impacts of inflation on various subgroups. It is argued that macroeconomic policy makers often implement the wrong type of anti-inflation measures at the wrong time. The proposed input-output model provides a rationale and a framework for implementing microeconomic, targeted policies. With targeted policies, we can attempt to remedy particular inflationary problems without adversely affecting prosperous components of the economy.     

An evaluation case: The implementation and evaluation of a problem-solving training program for adolescents

This study recounts the implementation and evaluation of a problem-solving skills training program for adolescents. Sixty-six students from the seventh, eighth, and ninth grades of a local junior high school were identified by guidance counselors as needing problem-solving training—48 male and 18 female students between the ages of 11 and 16. The hypothesis that students included in the training sessions would demonstrate better problem-solving skills, a more internal locus of control, and a higher level of self-esteem than control group students was tested via a pretest-posttest Control Group Design. The results of a stepwise discriminant analysis of the data supported the hypothesis. An analysis of grade level differences indicated that the program had less effect on the seventh-grade training group. Implications for program modification and the limitations of the evaluation findings are discussed.