Experiments in Producing Nonresponse Bias

While nonresponse rates in household surveys are increasingin most industrialized nations, the increasing rates do notalways produce nonresponse bias in survey estimates. The linkagebetween nonresponse rates and nonresponse bias arises from thepresence of a covariance between response propensity and thesurvey variables of interest. To understand the covariance term,researchers must think about the common influences on responsepropensity and the survey variable. Three variables appear tobe especially relevant in this regard: interest in the surveytopic, reactions to the survey sponsor, and the use of incentives.A set of randomized experiments tests whether those likely tobe interested in the stated survey topic participate at higherrates and whether nonresponse bias on estimates involving variablescentral to the survey topic is affected by this. The experimentsalso test whether incentives disproportionately increase theparticipation of those less interested in the topic. The experimentsshow mixed results in support of these key hypotheses.     

Nonresponse in the American Time Use Survey: Who Is Missing from the Data and How Much Does It Matter?

This article examines nonresponse in a large government survey,the American Time Use Survey (ATUS), which interviews personsin households previously interviewed in the Current PopulationSurvey. The response rate for the ATUS has been below 60 percentfor the first two years of its existence, raising questionsabout whether the results can be generalized to the target population.The article begins with an analysis of the types of nonresponseencountered in the ATUS. Noncontact accounts for roughly 60percent of ATUS nonresponse, with refusals accounting for roughly40 percent. We find little support for the hypothesis that busypeople are less likely to respond to the ATUS but find considerablesupport for the hypothesis that people who are weakly integratedinto their communities are less likely to respond, mostly becausethey are less likely to be contacted. When we compare aggregateestimates of time use calculated using the ATUS base weightswithout any adjustment for nonresponse, estimates calculatedusing the ATUS final weights with a nonresponse adjustment,and estimates calculated using weights that incorporate ourown nonresponse adjustment based on a propensity model, we findsome modest differences, but the three sets of estimates arebroadly similar. The article ends with suggestions for furtherresearch and analysis.     

Nonresponse Rates and Nonresponse Bias in Household Surveys

Many surveys of the U.S. household population are experiencinghigher refusal rates. Nonresponse can, but need not, inducenonresponse bias in survey estimates. Recent empirical findingsillustrate cases when the linkage between nonresponse ratesand nonresponse biases is absent. Despite this, professionalstandards continue to urge high response rates. Statisticalexpressions of nonresponse bias can be translated into causalmodels to guide hypotheses about when nonresponse causes bias.Alternative designs to measure nonresponse bias exist, providingdifferent but incomplete information about the nature of thebias. A synthesis of research studies estimating nonresponsebias shows the bias often present. A logical question at thismoment in history is what advantage probability sample surveyshave if they suffer from high nonresponse rates. Since postsurveyadjustment for nonresponse requires auxiliary variables, theanswer depends on the nature of the design and the quality ofthe auxiliary variables.     

Pemoline in Adult Attention Deficit Hyperactivity Disorder: Predictors of Nonresponse

Abstract

The psychostimulants are the most widely used treatment modality for attention deficit/hyperactivity disorder (ADHD). Although positive responses to psychostimulants have been demonstrated, 46% of adults with ADHD, on average, are nonresponsive or cannot tolerate the medications because of adverse effects. We reviewed the charts of 43 students treated with pemoline and studied parameters of nonresponse. Sixteen (37%) met the study definition of nonresponse. Of these, 5 (12%) were symptom nonresponders and 11 (25%) were nonresponders because of adverse effects. We found a significant relationship between age and nonresponse resulting from adverse effects, yet no other factors predicted nonresponse to pemoline. Clinicians should consider using smaller starting doses of pemoline in adults to minimize adverse effects.     

Survey nonresponse and the distribution of income

The paper examines the distributional implications of selective compliance in sample surveys, whereby households with different incomes are not equally likely to participate. Poverty and inequality measurement implications are discussed for monotonically decreasing and inverted-U compliance-income relationships. We demonstrate that the latent income effect on the probability of compliance can be estimated from information on response rates across geographic areas. On implementing the method on the Current Population Survey for the U.S. we find that the compliance probability falls monotonically as income rises. Correcting for nonresponse appreciably increases mean income and inequality, but has only a small impact on poverty incidence up to poverty lines common in the U.S. Martin Ravallion: Corresponding author.     

Estimating the Nonresponse BiasDue to Refusals in Telephone Surveys

This paper attempts to aid the process of accumulating the necessaryinformation for making more informed judgments about the effectsof nonresponse under different conditions. Two measures, whichpermit quantifiable nonsubjective assessment of the effectsof nonresponse on sample estimates, are introduced and are usedto examine the effects of respondent refusals in a random-digit-dialedgeneral population telephone survey of over 1,200 householdsas the response rate isincreased from 74.5 percent to 86.8 percent.By applying these measures under a wide range of conditions,the adequacy of various response rates may be assessed and morerational decisions made about the costs and benefits of devotingextraordinary resources to minimizing nonresponse.     

The Hidden 25 Percent: An Analysis of Nonresponse on the 1980 General Social Survey

Methods for estimating nonresponse bias are reviewed and severalmethods are tried on the 1980 GSS. The results indicate thatvarious estimating procedures are inappropriate and that eventhe more promising techniques can provide faulty estimates ofnonresponse bias. By its nature, nonresponse bias is very difficultto assess accurately and no simple, certain method exists.     

The Role of Topic Interest in Survey Participation Decisions

While a low survey response rate may indicate that the riskof nonresponse error is high, we know little about when nonresponsecauses such error and when nonresponse is ignorable. Leverage-saliencetheory of survey participation suggests that when the surveytopic is a factor in the decision to participate, noncooperationwill cause nonresponse error. We test three hypotheses derivedfrom the theory: (1) those faced with a survey request on atopic of interest to them cooperate at higher rates than dothose less interested in the topic; (2) this tendency for the"interested" to cooperate more readily is diminished when monetaryincentives are offered; and (3) the impact of interest on cooperationhas nonignorability implications for key statistics. The datacome from a three-factor experiment examining the impact oncooperation with surveys on (a) five different topics, using(b) samples from five different populations that have knownattributes related to the topics, with (c) two different incentiveconditions.     

Survey Participation, Nonresponse Bias, Measurement Error Bias, and Total Bias

A common hypothesis about practices to reduce survey nonresponseis that those persons brought into the respondent pool throughpersuasive efforts may provide data filled with measurementerror. Two questions flow from this hypothesis. First, doesthe mean square error of a statistic increase when sample personswho are less likely to be contacted or cooperate are incorporatedinto the respondent pool? Second, do nonresponse bias estimatesmade on the respondents, using survey reports instead of records,provide accurate information about nonresponse bias? Using aunique data set, the Wisconsin Divorce Study, with divorce recordsas the frame and questions about the frame information includedin the questionnaire, this article takes a first look into thesetwo issues. We find that the relationship between nonresponsebias, measurement error bias, and response propensity is statistic-specificand specific to the type of nonresponse. Total bias tends tobe lower on estimates calculated using all respondents, comparedwith those with only the highest contact and cooperation propensities,and nonresponse bias analyses based on respondents yield conclusionssimilar to those based on records. Finally, we find that errorproperties of statistics may differ from error properties ofthe individual variables used to calculate the statistics.     

Using Community-Level Correlates to Evaluate Nonresponse Effects in a Telephone Survey

Understanding the relationship between nonresponse processesand key research variables is central to evaluating if and hownonresponse introduces bias into survey estimates. In most telephonesurveys, however, little information is available with whichto estimate these effects. We report a procedure for examiningthe potential effects of nonresponse via analyses that (1) investigatethe linkages between community-level (zip code) variables andsurvey nonresponse and (2) examine the associations betweenthese community-level variables and key survey measures. Wedemonstrate these procedures using hierarchical modeling toanalyze data from a state-wide telephone survey in Illinois.One zip code–level indicator of concentrated disadvantage—thepercentage of the population below poverty level—was foundto be positively associated with nonresponse and, among respondents,with both current physical disability status and lack of healthinsurance coverage, suggesting that both may have been underestimatedin this survey. This inexpensive approach has the potentialof enabling researchers to routinely evaluate nonresponse effectsin their survey data.     

The Effects of Branching on Item Nonresponse

The analysis of responses to a large sample survey instrumentindicates that branching instructions significantly increasethe rate of item nonresponse for items immediately followingthe branch. While older respondents did have a higher incidenceof item nonresponse, none of six other factors (education, gender,distance to next question, number of previous branches, futurebehavior, or nature of response) hypothesized to affect itemnonresponse were significantly related to the rate of nonresponse.     

Trends in Nonresponse Rates, 1952-1979

Reported increases in nonresponse rates to sample surveys havenot been systematically documented to date. Data from the NationalElection Studies and the Surveys of Consumer Attitudes, twowell-known continuing studies conducted by the Survey ResearchCenter at the University of Michigan, permit the assessmentof long-term trends for the two major components of nonresponse,refusals and other noninterviews, by urban subgroups using timeseries regression techniques. The analyses clearly demonstratethat there have been substantial increases in total nonresponsedue primarily to increases in the percentages of respondentswho refuse to be interviewed, and that these trends are relatedto the level of urbanization.     

Evaluating the Effects of Screening for Telephone Service in Dual Frame RDD Surveys

The high costs and largely unknown error properties of cellulartelephone interviews make screening for cell-only adults a potentiallyattractive option in dual frame RDD surveys. Screening out adultswith landline telephones from the cellular sample does not affectthe coverage properties of a dual frame survey, but it may affectother sources of error, especially nonresponse. In this study,data from a 2006 dual frame RDD survey conducted for the PewResearch Center, the Associated Press, and AOL are used to evaluatethe effects of implementing a cell-only screener on both thebias and variance of weighted survey estimates. The effect ofscreening appears to be minimal so long as an adjustment fortelephone service is included in the weighting method. Resultsof an attempt to correct for residual nonresponse due to inaccessibilityare also discussed.     

Eating Disorder Symptoms Among College Students: Prevalence,Persistence, Correlates,and Treatment-Seeking

Abstract

Objective: To examine the prevalence, correlates, persistence, and treatment-seeking related to symptoms of eating disorders (EDs) in a random sample of college students. Participants: A random sample of students at a large university were recruited for an Internet survey in Fall 2005 and a follow-up survey in Fall 2007. Methods: ED symptoms were measured using the SCOFF screen and adjusted for nonresponse using administrative data and a nonresponse survey. Results: 2,822 (56%) students completed the baseline survey. Among undergraduates the prevalence of positive screens was 13.5% for women and 3.6% for men. Among students with positive screens, 20% had received past-year mental health treatment. In the follow-up sample (N = 753), ED symptoms at baseline significantly predicted symptoms 2 years later. Conclusions: Symptoms of EDs were prevalent and persistent among college students in this study. These findings suggest that brief screens can identify a large number of students with untreated EDs.     

Effects of the social dominance orientation on environmental attitudes of Chilean university students / Efectos de la orientación a la dominancia social sobre las actitudes ambientales de universitarios chilenos

By adopting a multidimensional approach to environmental attitudes (apathy, anthropocentrism, connectedness and emotional affinity with nature), this study explores the relationships between these dimensions, social dominance orientation (SDO) and political orientation. Two hundred and sixty-two Chilean university students filled out an ad hoc online questionnaire. Two confirmatory factorial analyses showed the four typologies of environmental attitudes and the two factors for SDO, which are labelled group dominance and opposition to equality. Through hierarchical regressions, group dominance showed higher explanatory power of environmental attitudes than political orientation. Furthermore, using the bootstrap procedure we showed that group dominance mediated between political orientation and attitudes of apathy, anthropocentrism and connectedness, while opposition to equality mediated between apathy and emotional affinity. We conclude that SDO is an important variable for measuring political ideology while also providing new nuances when analysing the relationships with environmental attitudes.     

The Influence of Advance Letters on Response in Telephone Surveys: A Meta-Analysis

Recently, the leading position of telephone surveys as the majormode of data collection has been challenged. Telephone surveyssuffer from a growing nonresponse, partly due to the generalnonresponse trend for all surveys and partly due to changesin society and technology influencing contactability and willingnessto answer. One way to counteract the increasing nonresponseis the use of an advance letter. In mail and face-to-face surveys,advance letters have been proven effective. Based on the proveneffectiveness in face-to-face and mail surveys, survey handbooksadvise the use of advance letters in telephone surveys. Thisstudy reviews the evidence for this advice and presents a quantitativesummary of empirical studies on the effectiveness of advanceletters in raising the response rate for telephone surveys.The major conclusion is that advance letters are also an effectivetool in telephone surveys, with an average increase in responserate (RR1) from 58 percent (no letter) to 66 percent (advanceletter), and an average increase in cooperation rate (COOP1)from 64 percent (no letter) to 75 percent (advance letter).     

Inequality,Unemployment and Crime: A Cross-National Analysis*

In order to contribute to the development of an international perspective on crime and to examine a central tenet similar to many theoretical perspectives on the etiology of crime, cross-national data on inequality, unemployment, and crime rates were analyzed. It was hypothesized that nations having a high rate of unemployment and an inequitable distribution of income would have a high crime rate. The results of the correlational analyses indicate a moderate positive relationship between the rate of unemployment and homicide rates, whereas there are small negative relationships between unemployment rates and (1) the rates of property crime, and (2) the total crime rates. The variable of inequality is strongly related to the three indices of crime and the directions of the relationships are consistent with those involving unemployment rates. The results were further investigated to examine the possibility that the observed relationships were due to the effects of industrialization. The direction of the zero-order correlations involving property crime rates and total crime rates are not changed in the partial correlations and the strength of the relationships are not consistently reduced. These results are discussed in reference to their implications for criminological theory and the development of a comparative criminology.     

Testing for the Potential for Nonresponse Bias in Sample Surveys

This article proposes using a statistical hypothesis test tocheck for the nonresponse bias that results when there is acorrrelation between strength of opinion and reply on issuepreference. This idea is developed through an example of twomail polls conducted by a Columbus, Ohio newspaper to ascertainvoter preference prior to the November 1983 election.     

Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey

Declining contact and cooperation rates in random digit dial(RDD) national telephone surveys raise serious concerns aboutthe validity of estimates drawn from such research. While researchin the 1990s indicated that nonresponse bias was relativelysmall, response rates have continued to fall since then. Thecurrent study replicates a 1997 methodological experiment thatcompared results from a "Standard" 5-day survey employing thePew Research Center’s usual methodology with results froma "Rigorous" survey conducted over a much longer field periodand achieving a significantly higher response rate. As withthe 1997 study, there is little to suggest that unit nonresponsewithin the range of response rates obtained seriously threatensthe quality of survey estimates. In 77 out of 84 comparableitems, the two surveys yielded results that were statisticallyindistinguishable. While the "Rigorous" study respondents tendedto be somewhat less politically engaged, they did not reportconsistently different behaviors or attitudes on other kindsof questions. With respect to sample composition, the Standardsurvey was closely aligned with estimates from the U.S. Censusand other large government surveys on most variables. We extendour analysis of nonresponse to include comparisons with thehardest-to-reach respondents and with respondents who terminatedthe interview prior to completion.     

Measuring depression in the community: a comparison of telephone and personal interviews

Interview method effects in response to the Center for Epidemiologic Studies-Depression Scale (CES-D) are compared among a community sample (N = 546) of adults randomly assigned to either telephone or in-person interviews. The interviews were conducted in Los Angeles during 1980. No significant differences were observed between the two interview methods in nonresponse to symptom items, preference for specific response categories, reliability, mean level of depression, or proportion classified as depressed. Furthermore, no significant interactions were found between sociodemographic characteristics and the method of interview.