The impact of respondent attentiveness on reliability and validity

This study aims to assess whether respondent inattentiveness causes systematic and unsystematic measurement error that influences survey data quality. To determine the impact of (in)attentiveness on the reliability and validity of target measures, we compared respondents from a German online survey (N = 5205) who had passed two attention checks with those who had failed. Our results show that inattentiveness induces both random and systematic measurement error, which impacts estimates of the reliability and validity of multi-item scales. In addition, we conducted a sensitivity analysis, which revealed that the impact of inattentiveness on analyses can be substantial.     

Toward Comprehensive Health Status Measures*

While health, itself, is the central concept in the sociology of health, little agreement exists on what constitutes an appropriate definition of health. This paper develops two composite health status measures (one continuous summary measure, and one set of eight discrete health state measures) based on the World Health Organization's conceptualization of the physical, social, and psychological dimensions of health. Data from a 1978 regional survey demonstrates the continuous summary measure's reliability (alpha = .70), and validity (factor analytic support for the hypothesized dimensions). The results also indicate that although either of the comprehensive measures significantly increase the goodness of fit of structural models of health service utilization, the set of discrete health state measures increases more the goodness of fit, bringing the explained variance of physician visits up from 16 percent to 30 percent. The magnitude of the increments clearly demonstrates the considerable utility of the comprehensive health status measurement approach.     

Union wage differentials for covered members and nonmembers in Great Britain

I present the first longitudinal estimates of covered union member and covered nonmember wage differentials in Great Britain. Crosssectional estimates show that covered union members receive a premium of about 10 percent over other workers, but covered nonmembers have no significant wage differential. Longitudinal estimates that accounted for fixed effects, selectivity, and measurement error indicated that OLS estimates are downward biased. The “true” union wage differential is likely about 30–35 percent. There is a large negative selection effect to union membership. No robust estimate could be found for covered nonmembers. This paper has benefited from conversations with Martyn Andrews, Wiji Arulampalam, Alison Booth, Tim Hatton, John Hutton, George Jakubson, Costas Meghir, and Andrew Oswald. Alison Booth kindly made extensive comments on an earlier draft, which greatly improved the exposition. John Budd, Hank Farber, and Larry Katz directed me towards related work in the U.S. I am also grateful to seminar participants at the University of York University of Essex, and the CEPR Workshop on Labour Market Issues. Thanks to the ESRC Data Archive for supplying the Family Expenditure Survey (FES) data. All data, computer programs, and results alluded to in the text are available on request. The British Household Panel Study (BHPS) and FES data are available from the ESRC Data Archive at the University of Essex. The Panel Study of Manufacturing Establishments (PSME) data are available from the author on request.     

Reconsidering union wage effects: Surveying new evidence on an old topic

I examine evidence on private sector union wage gaps in the United States. The consensus opinion among labor economists of an average union premium of roughly 15 percent is called into question. Two forms of measurement error bias downward standard wage gap estimates. Match bias results from Census earnings imputation procedures that do not include union status as a match criterion. Downward bias is roughly equal to the proportion of workers with imputed earnings, currently about 30 percent. Misclassification of union status causes additional attenuation in union gap measures. This bias has worsened as private sector density has declined, since an increasing proportion of workers designated as union are instead nonunion workers. Corrections for misclassification and match bias lead to estimated union gaps substantially higher than standard estimates, but with less of a downward trend since the mid 1980s. Private sector union gaps corrected for these biases are estimated from the CPS for 1973–2001. The uncorrected estimate for 2001 is .13 log points. Correction for match bias increases the gap to .18 log points; further correction for misclassification bias, based on an assumed 2 percent error rate, increases the gap to .24. Reexamination of the skill-upgrading hypothesis leads to the conclusion that higher union gap estimates are plausible. The conventional wisdom of a 15 percent union wage premium warrants reexamination.     

Assessing the Reliability of the Gambling Functional Assessment: Revised

Dixon and Johnson (Anal Gambl Behav 1: 44–49, 2007) proposed the Gambling Functional Assessment as a tool to identify the consequences maintaining the respondent’s gambling behavior, but subsequent studies on its psychometric properties suggested that it could use improvement. The present study investigated the internal consistency of the Gambling Functional Assessment—Revised using the responses of 1,060 undergraduate students. Temporal reliability was assessed by a second administration of the measure four (n = 87) or twelve (n = 98) weeks after the first administration. Temporal reliability was also compared to the South Oaks Gambling Screen (Lesieur and Blume in Am J Psychiatry 144: 1184–1188, 1987), which was also administered at both time points. Internal consistency measures were good to excellent, even when potential non-gamblers were excluded from the analyses. Temporal stability was also very good, with the possible exception of the consequence of “escape” at 12 weeks. The Gambling Functional Assessment—Revised represents a potentially useful tool for researchers and therapists interested in why respondents are gambling.     

Using Community-Level Correlates to Evaluate Nonresponse Effects in a Telephone Survey

Understanding the relationship between nonresponse processesand key research variables is central to evaluating if and hownonresponse introduces bias into survey estimates. In most telephonesurveys, however, little information is available with whichto estimate these effects. We report a procedure for examiningthe potential effects of nonresponse via analyses that (1) investigatethe linkages between community-level (zip code) variables andsurvey nonresponse and (2) examine the associations betweenthese community-level variables and key survey measures. Wedemonstrate these procedures using hierarchical modeling toanalyze data from a state-wide telephone survey in Illinois.One zip code–level indicator of concentrated disadvantage—thepercentage of the population below poverty level—was foundto be positively associated with nonresponse and, among respondents,with both current physical disability status and lack of healthinsurance coverage, suggesting that both may have been underestimatedin this survey. This inexpensive approach has the potentialof enabling researchers to routinely evaluate nonresponse effectsin their survey data.     

Assessment of Cigarette Smoking and Smoking Susceptibility among Youth: Telephone Computer-Assisted Self-Interviews versus Computer-Assisted Telephone Interviews

This experimental study assesses the effect of two survey methods,telephone audio computer-assisted self-interviewing (T-ACASI)and computer-assisted telephone interviewing (CATI), on self-reportsof smoking behavior and smoking susceptibility among adolescents12–17 years of age in California. In T-ACASI, participantslisten to prerecorded, computer-controlled questions and respondby pressing the keypad on a touch-tone telephone. In CATI, interviewersadminister the questions and enter responses into a computer.Prior research suggests that youth may be more likely to reportsensitive behaviors in a self-administered survey like T-ACASIcompared to an interviewer-administered survey like CATI, dueto greater perceived confidentiality. Logistic regression analyseswere conducted on unweighted data, controlling for demographicdifferences. Adjusted estimates of current smoking (past 30days) were significantly greater in T-ACASI (8.3 percent) thanCATI (4.5 percent). Smoking susceptibility (i.e., lack of afirm commitment not to smoke among those who have never smoked)was also greater in T-ACASI (45.0 percent) than CATI (34.9 percent).In both surveys, social desirability response bias was negativelyassociated with smoking, which suggests that response bias wasproblematic for both modes. Many respondents reported that aparent was present during the interview (59.4 percent in CATI;42.0 percent in T-ACASI). In both surveys, parental presencewas negatively associated with smoking, which suggests thatthis factor could also contribute to underreporting. Applicationof sample weights to the data eliminated the survey mode effects;however, the CATI current smoking estimate (9.3 percent) fromthis study was significantly less than an estimate (14.2 percent)obtained from a self-administered, school-based survey conductedthe same year on California adolescents.     

Determinants of housing quality: A comparison of white,African-American,and Hispanic single-parent women

This study examines housing quality among three groups of single-parent women: white, African-American, and Hispanic. Three indicators of housing quality—crowding, affordability, and satisfaction—are used to discover the extent to which these groups experience housing problems. This study also explores differences and similarities in the factors that precipitate problems in housing quality for these three groups of single parents. Findings suggest important differences and similarities in the nature of housing quality problems among white, African-American, and Hispanic single-parent women. The specified variables explained about 20% of the variance in crowding, housing affordability, and housing satisfaction. On measures of objective and subjective housing quality, white single mothers are better housed than their African-American or Hispanic counterparts. Hispanic single mothers have the largest housing cost burden and average more persons per household than the other groups. African-Americans are twice as likely as either group to live in low-quality housing and reported the lowest average satisfaction with their housing. Her research interests include housing and neighborhood assessment and women and public policy issues. She has recently published articles inUrban Affairs Quarterly andHousing and Society on housing and neighborhood assessment criteria among black urban households and the housing cost burden of various groups of female-headed households. She received her Ph.D. from Ohio State University. Her research interests include the housing adjustment behavior of women and public policy. She has recently completed her thesis on the role of noncustodial parents in determining the quality of life of their children. She is also the recipient of the Tessie Agan Award for research excellence from the American Association of Housing Educators for a paper titled “Unterstanding Constraints and Residential Satisfaction among Low-Income Single-Parent Families.”     

The Next-Birthday Method of Respondent Selection

Four procedures for selecting telephone survey respondents withina household unit—Troldahl-Carter, male/female alternation,next-birthday, and no-selection methods—were comparedin a small-sample survey. The data indicate that the next birthdaymethod is a relatively efficient procedure for selecting a samplethat is representative of all household members.     

MEASUREMENT ERROR IN STYLIZED AND DIARY DATA ON TIME USE

We investigate the nature of measurement error in time-use data. Analysis of "stylized" recall questionnaire estimates and diary-based estimates of housework time from the same respondents of a British survey gives evidence of systematic biases in the stylized estimates and large random errors in both types of estimates. We examine the effect of these measurement problems on three common types of statistical analyses in which the time-use variable is used as: (1) a dependent variable, (2) an explanatory variable, and (3) a basis for cross-tabulations. We develop methods to correct the biases induced by these measurement errors.     

Comparing Thistles and Roses: The Application of Governmental–Voluntary Sector Relations Theory to Scotland and England

Typologies of government–voluntary sector relations mostly categorize nation-level situations, and omit consideration of intra-national differences. They are also rarely subject to empirical testing and subsequent theoretical development. This paper reports on research to apply Kuhnle and Selle's (2002) relational typology, which characterizes the experience of “Britain,” to a study of Scottish and English voluntary organizations' relations with government. Contrary to Kuhnle and Selle, the paper demonstrates — through survey data — a divergence between Scotland and England, within the “British” context, and advances the case for further testing of such conceptual typologies. The methodological challenges in moving from a theoretical framework to a research design capable of field application, using survey methods, are illustrated and discussed. Further directions for theoretically led work are suggested.     

Measuring inequality of opportunity with imperfect data: the case of Turkey

The measurement of inequality of opportunity has hitherto not been attempted in a number of countries because of data limitations. This paper proposes two alternative approaches to circumventing the missing data problems in countries where a demographic and health survey (DHS) and an ancillary household expenditure survey are available. One method relies only on the DHS, and constructs a wealth index as a measure of economic advantage. The alternative method imputes consumption from the ancillary survey into the DHS. In both cases, we compute a lower bound estimator of the share of (ex-ante) inequality of opportunity in total inequality. Parametric and non-parametric estimates are calculated for each method, and the parametric approach is shown to yield preferable lower-bound measures. In an application to the sample of ever-married women aged 30–49 in Turkey, inequality of opportunity accounts for at least 26% (31%) of overall inequality in imputed consumption (the wealth index).     

Regression artifacts in time-series and longitudinal data

The problem of “regression artifacts” in causal inference, otherwise known as the problem of error and reliable irrelevant variance in “independent” variables used for matching or regression adjustment, is illustrated first in the time-series data where a treatment is triggered by an extreme measure. The “offset effect” in psychotherapy, Underwood's scalloped learning curve, and potential pseudo-effects in AIDS therapies are used as illustrations. The magnitude of such artifacts is computable if the autocorrelation pattern for various lags is known, and thus could be distinguished from genuine effects. For longitudinal studies in which a population of respondents is repeatedly measured, the problem of anchoring the matching or regression adjustments on a single wave of measurement (usually the first) is illustrated as affected by the proximally autocorrelated nature of such measures. Data from a famous study of the effects of job training are reinterpreted in light of this consideration.     

The Prevalence of Problem Gambling Among U.S. Adolescents and Young Adults: Results from a National Survey

A random telephone survey was conducted with a representative sample of 2,274 U.S. residents aged 14–21. The prevalence of problem gambling, as measured by the SOGS-RA, was 2.1%. Sixty-eight percent (68%) of the respondents had gambled in the past year, and 11% had gambled more often than twice per week. Males had much higher gambling involvement than females, and gambling involvement increased among older respondents. Blacks were less likely than average to have gambled in the past year, but if they gambled, they were more likely to do so frequently. Low SES respondents were less likely to have gambled in the past year, but if they gambled, they were more likely to be problem gamblers. Life transitions that are associated with assuming adult roles (employment, living independently of parents, non-student status) are also associated with greater gambling involvement. The rates of problem and pathological gambling were lower than those in an adult survey conducted earlier, when measured with the same questionnaire.     

Active Commuting Patterns at a Large,Midwestern College Campus

Abstract

Objective: To understand patterns and influences on active commuting (AC) behavior. Participants: Students and faculty/staff at a university campus. Methods: In April–May 2008, respondents answered an online survey about mode of travel to campus and influences on commuting decisions. Hierarchical regression analyses predicted variance in walking and biking using sets of demographic, psychological, and environmental variables. Results: Of 898 respondents, 55.7% were female, 457 were students (50.4%). Students reported more AC than faculty/staff. For students, the models explained 36.2% and 29.1% of the variance in walking and biking, respectively. Among faculty/staff, the models explained 45% and 25.8% of the variance in walking and biking. For all models, the psychological set explained the greatest amount of variance. Conclusions: With current economic and ecological concerns, AC should be considered a behavior to target for campus health promotion.     

PLURALISTIC IGNORANCE AND WHITE ESTIMATES OF WHITE SUPPORT FOR RACIAL SEGREGATION

This article presents data, taken from a secondary analysisof a national survey, indicating that in 1968 most white Americanadults grossly exaggerated the support among other whites forracial segregation. Several correlates of this pattern of pluralisticignorance—the extent to which the respondents led segregatedlives, their racial attitudes, and their racial values—areexamined. Finally, the relationship between this form of pluralisticignorance and the willingness of these white respondents toendorse racially restricted housing is discussed.     

Survey Participation, Nonresponse Bias, Measurement Error Bias, and Total Bias

A common hypothesis about practices to reduce survey nonresponseis that those persons brought into the respondent pool throughpersuasive efforts may provide data filled with measurementerror. Two questions flow from this hypothesis. First, doesthe mean square error of a statistic increase when sample personswho are less likely to be contacted or cooperate are incorporatedinto the respondent pool? Second, do nonresponse bias estimatesmade on the respondents, using survey reports instead of records,provide accurate information about nonresponse bias? Using aunique data set, the Wisconsin Divorce Study, with divorce recordsas the frame and questions about the frame information includedin the questionnaire, this article takes a first look into thesetwo issues. We find that the relationship between nonresponsebias, measurement error bias, and response propensity is statistic-specificand specific to the type of nonresponse. Total bias tends tobe lower on estimates calculated using all respondents, comparedwith those with only the highest contact and cooperation propensities,and nonresponse bias analyses based on respondents yield conclusionssimilar to those based on records. Finally, we find that errorproperties of statistics may differ from error properties ofthe individual variables used to calculate the statistics.     

LITTLE WHITE LIES AND SOCIAL SCIENCE MODELS CORRELATED RESPONSE ERRORS IN A PANEL STUDY OF VOTING

One interpretation for the common survey finding that the backgroundcharacteristics of vote overreporters resemble those of actualvoters is that misreporters usually vote. This hypothesis—thatmisreporters regularly voted in earlier elections—is testedwith data from the 1972–74–76 Michigan ElectionPanel. It receives no support: the 1972 and 1974 validated turnoutof the 1976 misreporters was very low. Moreover, misreportingwas a fairly stable respondent characteristic: misreportingabout an election in one interview was correlated with misreportingabout the remaining elections in each of the other two interviews.A comparison of regressions predicting turnout using the validatedreports versus the self-reports shows that the respondent errorscan distort conclusions about the correlates of voting. Forexample, controlling for three other variables, education wasrelated to self-reported voting but not to validated voting.Here, as well as in surveys of other socially desirable or undesirableissues, respondent self-reports may bias survey data in favorof commonsense models of the world.     

Evaluating the Problem Gambling Severity Index

A large, integrated survey data set provided by the Ontario Problem Gambling Centre was used to investigate psychometric properties of the Problem Gambling Severity Index (PGSI). This nine-item self-report instrument was designed to measure a single, problem gambling construct. Unlike its nearest competitor—the South Oaks Gambling Screen (SOGS)—the PGSI was designed specifically for use with a general population rather than in a clinical context. The present analyses demonstrated that the PGSI does assess a single, underlying, factor, but that this is complicated by different, multiple factor structures for respondents with differing levels of problem gambling severity. The PGSI also demonstrated small to moderate correlations with measures of gambling frequency and faulty cognitions. Overall, the PGSI presents a viable alternative to the SOGS for assessing degrees of problem gambling severity in a non-clinical context.

Antidiscrimination measures of the 1960s and occupational mobility: Evidence for black American men

Based on an economywide index, I estimate that the occupational status of U.S. black men, relative to white men, rose an average of .5 percent per year over the 1965–1981 period. After accounting for pre-existing trends, education, and censoring supply factors, approximately 40 percent of the increase remains. I argue that these residual post– 1964 effects may be attributed to the antidiscrimination measures of the times, particularly Title VII of the Civil Rights Act of 1964. Although there seems to be some cyclical censoring, I uncover no evidence in support of the secular censoring hypothesis that observed post– 1964 black male occupational progress results largely from black male labor supply declines. I assign about one-half of the 21 percent relative earnings gains by black men during 1965–1981 to occupational mobility. Compared with previous findings for black women, however, the results suggest substantially lower gains for black men. In addition, occupational advancement appears to explain a much smaller proportion of the earnings increases for black men than for black women. An earlier version of this paper was revised while the author was Visiting Associate Professor at the Department of Economics and the Frederick Douglass Institute, University of Rochester, New York. Useful comments were provided by members of the Institute and the Department of Economics. Grant support by the Oakland University Research Committee and the School of Business Administration is gratefully acknowledged.