Using a community-created multisite evaluation to promote evaluation use across a sector

Multisite evaluation has been cited as a strategy for building knowledge about a program's success and enhancing evaluation capacity and evaluation use. Even so, the extent to which data from these efforts are made available to individual project participants varies. This article describes a community-created multisite evaluation that was designed to overcome this limitation. Based on the negotiated centralized evaluation model and implemented within the context of a community of practice, the EvalFest community-created multisite evaluation has used shared measures to collect data from 30,000 participants from partner sites across the community over three years. This article describes the design characteristics of the community-created multisite approach, and presents results related to partners’ evaluation use that demonstrate the early success of this method. We conclude by considering how this strategy might be applied to other sectors.     

Evaluating Organizations Through an Employee Assistance Program Using an Organization Development Model

This article describes one of the first accounts of an Employee Assistance Program (EAP) staffed by a social worker in industry in the United States. The author was the social worker employed in this EAP (called the Industrial Social Work Project) which was originally motivated by a vice president's concern over employee turnover. The article describes in detail how the EAP was operationalized and used to evaluate the organizational culture by using an organizational development model which involved five overlapping phases. These were: (1) orientation/information gathering; (2) individual treatment; (3) research on cultural change; (4) team building and O.D. projects; and (5) O.D. management training. The main contention of this article is that organizations do in fact contribute to the "professional suicide" of employees, and those who plan to evaluate ENS are remiss not considering the organization context of such programs and their possible impact on employees.     

Managing Multisite Nonprofits

Multisite nonprofit systems are organized in a variety of ways, ranging from central control to local control. Regardless of where a system falls along this organizational continuum, inevitable tensions emerge in the relationship between headquarters and the local organization. The purpose of this article is to understand the causes of these tensions and to offer management action that, regardless of the organizational structure, leads to greater system cohesion. Our exploration was facilitated by intense field research at five multisite systems: Outward Bound USA, Planned Parenthood, Habitat for Humanity, SOS Kinderdorf, and The Nature Conservancy. Our investigation suggests that multisite system behavior may be mapped on two dimensions—one that exerts forces toward unit autonomy and the other influencing the degree of organizational affiliation. These forces are associated with a host of organizational functions that are the traditional domains of management to influence and shape and thereby enable a multisite nonprofit to simultaneously optimize the benefits of organizational affiliation and unit autonomy.     

An Introduction and Illustration of Bayesian Modeling in Couple and Family Therapy Research

Bayesian modeling is becoming increasing popular as a method for data analyses in the social sciences and can move couple, marriage, and family therapy (C/MFT) research forward. Bayesian modeling helps researchers better understand the uncertainty of findings and incorporate previous research into analyses. Other benefits of Bayesian modeling are the straightforward interpretation of findings, high-quality inferences even with small samples (in combination with an informative prior), and the ability to work with complex data structures (observations nested in relationships and time points) which are common in C/MFT research. This article introduces the benefits of Bayesian modeling and provides an example of an Actor–Partner Interdependence Model using R. Information on how to conduct the same analyses using Stata and MPlus is provided in the Supplemental Information.     

Team effects on compensation: an application to salary determination in the National Hockey League

Studies of salary determination largely model pay as a function of the attributes of the individual and the workplace (i.e., employer size, job characteristics, and so forth). This article empirically investigates an additional factor that may influence individual pay, specifically coworker productivity. Data from professional sports are used to evaluate this question because both salary and teammate performance measures are readily available. We find that team attributes have both direct effects on an individual's pay, and indirect effects through altering the rates at which individual player productive characteristics are valued.     

Lessons for Multisite Nonprofits from the United Church of Christ

This article describes the restructuring process currently underway at the United Church of Christ (UCC)—a 1.4 million member Protestant denomination. Three main questions are addressed: What are the central goals of the UCC restructure? What factors might explain the particular strategic direction pursued in UCC restructuring? and How might the UCC case inform recent thinking about the structure and performance of multisite nonprofits?     

Explaining variation in the effects of welfare-to-work programs

Evaluations of government-funded training programs often combine results from similar operations in multiple sites. Findings inevitably vary. It is common to relate site-to-site variations in outcomes to variations in program design, participant characteristics, and the local environment. Frequently, such connections are constructed in a narrative synthesis of multisite results. This article uses findings from the evaluations of California's Greater Avenues for Independence (GAIN) program and the National Evaluation of Welfare-to-Work Strategies (NEWWS) to illustrate why it is important to question the legitimacy of such syntheses. The discussion is carried out using a simple multilevel evaluation model that incorporates models of both individual outcomes within sites and variation in program effects across sites. The results indicate that tempting generalizations about GAIN and NEWWS effects are statistically unjustified but that significant progress might be made in identifying the determinants of program effects in future demonstrations with some changes in evaluation strategy.     

Using interviews to understand the assignment mechanism in a nonexperimental study: the case of eighth grade algebra

Many inquiries regarding the causal effects of policies or programs are based on research designs where the treatment assignment process is unknown, and thus valid inferences depend on tenuous assumptions about the assignment mechanism. This article draws attention to the importance of understanding the assignment mechanism in policy and program evaluation studies, and illustrates how information collected through interviews can develop a richer understanding of the assignment mechanism. Focusing on the issue of student assignment to algebra in 8th grade, I show how a preliminary data collection effort aimed at understanding the assignment mechanism is particularly beneficial in multisite observational studies in education. The findings, based on ten interviews and administrative data from a large school district, draw attention to the often ignored heterogeneity in the assignment mechanism across schools. These findings likely extend beyond the current research project in question to related educational policy issues such as ability grouping, tracking, differential course taking, and curricular intensity, as well as other social programs in which the assignment mechanism can differ across sites.     

DISCRIMINATION, BAYESIAN UPDATING OF EMPLOYER BELIEFS, AND HUMAN CAPITAL ACCUMULATION

This article proposes a dynamic labor market discrimination model based on Bayesian updating of beliefs by the employer. The employer forms initial beliefs of worker ability conditional on group and updates his beliefs each period based on observed output. Inaccurate initial priors lead to diminished human capital investment among members of the undervalued group and may generate inequities lasting many periods or even permanently. Statistical discrimination models in which differing variances drive the inequity are special cases of the model.     

Expectation formation in step-level public good games

This article focuses on the process of expectation formation. Specifically, the question is addressed whether individuals think strategically when they form beliefs about other players'behavior. Most belief learning models assume that people abstract from strategic considerations. Using an incentive-compatible mechanism, experimental data are obtained on subjects'expectations in a step-level public good game and in a game against nature. Beliefs in the interactive games develop in the same way as in the game against nature, providing evidence that strategic considerations do not play a role. The evidence is consistent with predictions derived from the naive Bayesian model.     

The revised safe at home instrument for assessing readiness to change intimate partner violence

This article describes the development and factor structure of the Revised Safe At Home instrument, a 35-item self-report measure designed to assess individuals' readiness to change their intimate partner violence behaviors. Seven new items have been added, representing content specific to the Maintenance stage, and other items have been revised to strengthen the assessment of earlier stages and address gender concerns. Confirmatory factor analysis using multisite data (two sites, a total of 281 men at intake) supported the conclusion that a four-factor model (Precontemplation, Contemplation, Preparation/Action, and Maintenance stages) was consistent with the observed covariances. A high degree of correlation between the Preparation/Action and Maintenance scales was observed, but subsequent testing indicated a need to treat the two as distinct factors in the model. It is recommended that scoring include only 31 items that perform well.     

Bayesian estimation of the network autocorrelation model

The network autocorrelation model has been extensively used by researchers interested modeling social influence effects in social networks. The most common inferential method in the model is classical maximum likelihood estimation. This approach, however, has known problems such as negative bias of the network autocorrelation parameter and poor coverage of confidence intervals. In this paper, we develop new Bayesian techniques for the network autocorrelation model that address the issues inherent to maximum likelihood estimation. A key ingredient of the Bayesian approach is the choice of the prior distribution. We derive two versions of Jeffreys prior, the Jeffreys rule prior and the Independence Jeffreys prior, which have not yet been developed for the network autocorrelation model. These priors can be used for Bayesian analyses of the model when prior information is completely unavailable. Moreover, we propose an informative as well as a weakly informative prior for the network autocorrelation parameter that are both based on an extensive literature review of empirical applications of the network autocorrelation model across many fields. Finally, we provide new and efficient Markov Chain Monte Carlo algorithms to sample from the resulting posterior distributions. Simulation results suggest that the considered Bayesian estimators outperform the maximum likelihood estimator with respect to bias and frequentist coverage of credible and confidence intervals.     

Union wage premiums in an instrumental variables framework

A number of studies have used simultaneous-equations models, and the Inverse-Mills Ratio procedure in particular, to calculate union wage premiums. No study, however, has attempted to build up and extend results using an innovative instrumental variables (IV) model recently developed by Duncan and Leigh. This article fills this void by not only using the IV model to update Duncan and Leigh’s estimates, but also computes union wage premiums for as many as 24 different groups of workers (as opposed to only five by Duncan and Leigh). IV estimates are also compared with results computed from the more traditional IMR and single-equation OLS models.     

From nightlife conventions to daytime hidden agendas: dynamics of urban sexual territories in the South of France

This article focuses on the space-time dynamics of sexual encounters in city neighborhoods associated with sex work and cruising. The data were collected as part of a multisite, longitudinal ethnographic study conducted in major cities of southern France. The results suggest that there is a continuous interaction between these sexualized public spaces and urban planning. Urban planners, however, are not the only actors shaping the location and the use of the space. Participants create a dialectical movement from night normative worlds to occult practices interwoven with daytime activities. They also create subtle social rules governing actors and their practices.     

Disability and employment: towards an understanding of discourse and policy

This article explores the nature of disability employment policy drawing on evidence from fifteen countries. In line with earlier sociological approaches to disability it describes two ways of framing policy in the area. These two paradigms are developed and used to critically evaluate employment policy under the headings: legislative measures, open employment-financial employment support services and sheltered/supported provision. The dilemmas that may arise during implementation of policy are discussed and the consequences for the construction of an emancipatory employment policy. Finally, the article suggests some themes or dimensions that a coherent disability policy should have.     

A comparison of estimators for the network autocorrelation model based on observed social networks

Network autocorrelation models (NAMs) are widely used to study a response variable of interest among subjects embedded within a network. Although the NAM is highly useful for studying such networked observational units, several simulation studies have raised concerns about point estimation. Specifically, these studies have consistently demonstrated a negative bias of maximum likelihood estimators (MLEs) of the network effect parameter. However, in order to gain a practical understanding of point estimation in the NAM, these findings need to be expanded in three important ways. First, these simulation studies are based on relatively simple network generative models rather than observed networks, thereby leaving as an open question how realistic network topologies may affect point estimation in practice. Second, although there has been strong work done in developing two-stage least squares estimators as well as Bayesian estimators, only the MLE has received extensive attention in the literature, thus leaving practitioners in question as to best practices. Third, the performance of these estimators need to be compared using both bias and variance, as well as the coverage rate of each estimator's corresponding confidence or credible interval. In this paper we describe a simulation study which aims to overcome these shortcomings in the following way. We first fit real social networks using the exponential random graph model and used the Bayesian predictive posterior distribution to generate networks with realistic topologies. We then compared the performance of the three different estimators mentioned above.     

A Country‐wide Study of Consumer Choice for an Emerging Supermarket Sector: A Case Study of Nicaragua

Studies portraying and quantifying supermarket clientele based on country‐wide survey data are scarce in development economics literature. This article studies the choice of outlet of Nicaraguan consumers in 1998 and 2001 when supermarkets started to emerge and gain in importance. It applies comparative statistics and a multinomial logit model to countrywide data on 4,000 households. The results show an emerging supermarket sector with a slowly growing clientele, especially among the better endowed and more highly educated families. Small grocery shops or pulperias and the daily and weekly markets continue to serve most clients.     

Construction and application of an indicator system to assess the ergonomic performance of large and medium-sized construction companies

This article proposes a system of indicators to evaluate the performance of companies in ergonomics for buildings. The system was developed based primarily on studies related to the performance evaluation of the construction industry and on Brazilian standards of ergonomics and work safety and had also the contribution of national and international indicators related to ergonomics, work safety, quality, sustainability, quality of work life and to organizational behavior. The indicators were named, classified and their components were assigned to compose the theoretical model SIDECE--System of Performance Indicators in Ergonomics for Building Construction (as for the Portuguese acronym), serving the major goals of ergonomics: health, safety and workers' satisfaction and production efficiency. The SIDECE is being validated along with the building construction companies in the city of Natal, Brazil, whose practical results, deriving from the application of instruments to collect field data, are under process, to be presented on the occasion of the 18th World Congress on Ergonomics. It is intended that the SIDECE be used by building construction companies as a support tool for excellence management.