Portfolio assessment in a collaborative program evaluation: the reliability and validity of a family literacy portfolio

Evaluations increasingly use open-ended assessments, such as portfolios, to gather information about program impact. Portfolios, however, present challenges in achieving acceptable levels of reliability and often lack evidence to support validity. This article describes the components of a portfolio assessment system developed in a collaborative evaluation of a family literacy program. Staff of a family literacy program and two evaluators collaborated to develop the assessment and the tools that frame the review of the portfolios. Rubrics, exemplar portfolios, portfolio logs, and lists of ‘look-fors’ were developed to guide raters in the scoring and to improve the reliability of scores and validity of decisions about the program. Inter-rater reliability for the portfolio assessment was 0.79 for holistic scores and 0.74 for analytic scores. With respect to validity, holistic and analytic scores correlated with criterion measures from 0.50 to 0.59, providing initial evidence of a common construct.     

FACILITATING INTIMACY: INTERVENTIONS AND EFFECTS

The Present study investigated by effects of two sets of marital interventions taken from Emotionally Focused Therapy (EFT) and Cognitive Marital Therapy (CMT) on level of marital intimacy, dyadic trust, and dyadic adjustment. It was hypothesized that both EFT and CMT would have a positive effect on levels of intimacy, trust, and adjustment as compared to a wait-list control group and that there would be a differential effect in favor of EFT as compared to CMT. Thirty-six couples free of marital distress and seeking to enhance theor intimate relationship were randomaly assigned to EFT, CMT, or a wai-list control group. Therapists' interventions were monitored and found to be faithfully implemented. Groups were equivalent on demographic variables and the quality of the therapeutic alliance. Both EFT and CMT group posttest means were found to be significantly higher than controls on the self-report measures of intimacy, Observational measures of intimacy revealed differential effects in favor of EFT as comopared to CMT at posttest. At a 10-week follow-up, EFT group means were significantly higher than CMT on self-reported intimacy and adjustment.     

DEREGULATION AND THE RELIGIOUS MARKET IN TAIWAN: A Research Note

The religious economy model predicts that when encountered with competition, loosely organized religions will fail or will be transformed into congregational religions. Over time, competition will drive congregational religions to establish an extended relationship with their consumers by generating exclusivist claims and exclusivist socialization. And thus exclusive religion tends to occupy the biggest market share. This model, however, has rarely been tested empirically. By analyzing religious trends in Taiwan after the deregulation of the late 1980s, we find that state suppression contributed to the weakness of organized religion while enhancing the popularity of unchurched congregational religions in Taiwan. We also find that deregulation is associated with the rise of organized and congregational faiths. But whether or not these newly organized religions become exclusive or not needs further studies. Implications of the findings for future research are offered.     

Two‐Wave Panel Analysis: Comparing Statistical Methods for Studying the Effects of Transitions

Study of the effect of transitions on individual and family outcomes is central to understanding families over the life course. There is little consensus, however, on the appropriate statistical methods needed to study transitions in panel data. This article compares lagged dependent variable (LDV) and change score (CS) methods for analyzing the effect of events in two‐wave panel data. The methods are described, and their performances are compared both with a simulation and a substantive example using the National Survey of Families and Households two‐wave panel. The results suggest that CS methods have advantages over LDV techniques in estimating the effect of events on outcomes in two‐wave panel data.     

Response Effects in Sex Surveys

Two potential problems in sex surveys—respondent refusalsand invalid reporting—are examined for national and areaprobability samples of males and females. Little support isfound for the contention that sexual topics constitute a sensitiveresearch area in terms of respondent cooperation and candor.It is found that (1) respondent cooperation, as indicated byrefusals and terminations, is neither more nor less problematicin sex research than in surveys on other topics; (2) respondentreports of sexual matters are not systematically related tointerviewer characteristics such as gender, sexual experience,or technical competence; (3) respondent candor, when self-reported,also is not associated systematically with either interviewercharacteristics or demographic and sexual characteristics ofrespondents. This analysis does identify, however, evidencethat (4) sexual matters often are sensitive topics for interviewersand principal investigators, leading both to interviewer turnoverand the creative construction of unobtrusive data collectionprocedures.     

Let's Call it Quits: Break‐Even Effects in the Decision to Stop Taking Risks

“Chasing” behavior, whereby individuals, driven by a desire to break even, continue a risky activity (RA) despite incurring large losses, is a commonly observed phenomenon. We examine whether the desire to break even plays a wider role in decisions to stop engaging in financially motivated RA in a naturalistic setting. We test hypotheses, motivated by this research question, using a large data set: 707,152 transactions of 5,379 individual financial market spread traders between September 2004 and April 2013. The results indicate strong effects of changes in wealth around the break‐even point on the decision to cease an RA. An important mediating factor was the individual's historical long‐term performance. Those with a more profitable trading history were less affected by a fall in cash balance below the break‐even point compared to those who had been less profitable. We observe that break‐even points play an important role in the decision of nonpathological risk takers to stop RAs. It is possible, therefore, that these nonpathological cognitive processes, when occurring in extrema, may result in pathological gambling behavior such as “chasing.” Our data set focuses on RAs in financial markets and, consequently, we discuss the implications for institutions and regulators in the effective management of risk taking in markets. We also suggest that there may be a need to consider carefully the nature and role of “break‐even points” associated with a broader range of nonfinancially‐focused risk‐taking activities, such as smoking and substance abuse.     

In Search of ‘Managerial Work’: Past,Present and Future of an Analytical Category

Based on a comprehensive review of literature, the paper examines how ‘managerial work’ as a fluid analytical category has been approached methodologically, theoretically and empirically for more than 60 years. In particular, it highlights the existence of competing scholarly understandings regarding its nature, performance, meaning and politics. The authors suggest that subsequent empirical investigations have too often worked, methodologically and theoretically, to slot in, and thus effectively reduce, the term to a particular pre‐existing box, rather than exploring open‐endedly the what and how, but also the why of ‘managerial work’ as a distinct mode of situated ordering. Having represented the concept's past and present by identifying four distinct research approaches reflected in representative publications, the authors suggest that more attention should be devoted to a mode of analytical departure that promises to address directly the suggested shortcomings in the literature. Specifically, it is argued that much could be gained if contemporary notions of practice were brought into the study of managerial work. To this end, the authors outline the contours of a practice‐based approach as a sensitizing framework for understanding managerial work by highlighting the situated, relational, sociomaterial, meaning‐making and consequence‐oriented analytical foci the approach suggests, and suggesting a number of conjoint research questions, as well as acknowledging subsequent limitations.     

Acculturation,Ethnicity, and Air Pollution Perceptions

A globalizing world increases immigration between nations, raising the question of how acculturation (or its lack) of immigrants and their descendants to host societies affects risk perceptions. A survey of Paterson, New Jersey, residents tested acculturation's associations with attitudes to air pollution and its management, and knowledge of and self‐reported behaviors concerning air pollution. Linguistic and temporal proxy measures for acculturation were independent variables along with ethnicity, plus controls for gender, age, education, and income in multivariate analyses. About one‐fifth of contrasts between non‐Hispanic whites, non‐Hispanic blacks, English‐interviewed Hispanics, and Spanish‐interviewed Hispanics were statistically significant (Bonferroni‐corrected) and of medium or higher affect size, with most featuring the Spanish‐interviewed Hispanics. Knowledge variables featured the most significant differences. Specifically, Spanish‐interviewed Hispanics reported less concern, familiarity with pollution, recognition of high pollution, and vigorous outdoor activity, and greater belief that government overregulates pollution than English‐interviewed Hispanics (and than the other two groups on most of these variables too). English‐interviewed Hispanics did not differ from non‐Hispanic whites, but did on several variables from non‐Hispanic blacks. Temporal proxies of acculturation among the foreign‐born were far less significant, but concern and familiarity with air pollution increased with time spent in the United States, while belief in overregulation and a positive trend in New Jersey pollution increased with time in the nation of origin. Implications of these acculturation and ethnicity findings for risk perception/communication research and practice are discussed.