Experiments in Producing Nonresponse Bias

While nonresponse rates in household surveys are increasingin most industrialized nations, the increasing rates do notalways produce nonresponse bias in survey estimates. The linkagebetween nonresponse rates and nonresponse bias arises from thepresence of a covariance between response propensity and thesurvey variables of interest. To understand the covariance term,researchers must think about the common influences on responsepropensity and the survey variable. Three variables appear tobe especially relevant in this regard: interest in the surveytopic, reactions to the survey sponsor, and the use of incentives.A set of randomized experiments tests whether those likely tobe interested in the stated survey topic participate at higherrates and whether nonresponse bias on estimates involving variablescentral to the survey topic is affected by this. The experimentsalso test whether incentives disproportionately increase theparticipation of those less interested in the topic. The experimentsshow mixed results in support of these key hypotheses.     

Face-to-Face Interviews and Mailed Questionnaires: The Net Difference in Response Rate

Traditionally, as reflected in methodology textbook reviewsof the survey nonresponse literature, interviews have been thoughtto achieve substantially higher response rates than mailed questionnaires.The paper describes a correlational design for assessing thetypical response difference between these two forms of datacollection, after controls for other factors known to affectsurvey response. Attention is given to change over time in theresponse difference between interviews and questionnaires. Inthe primary data set, the net response due to data collectionamounts to some 7.5 percentage points, in the middle (30–70)range of survey response. The analysis quantifies the declineover time in interview response, and shows mailed questionnairesto be free from decline. A second data set reproduces the essentialfindings.     

The Role of Topic Interest in Survey Participation Decisions

While a low survey response rate may indicate that the riskof nonresponse error is high, we know little about when nonresponsecauses such error and when nonresponse is ignorable. Leverage-saliencetheory of survey participation suggests that when the surveytopic is a factor in the decision to participate, noncooperationwill cause nonresponse error. We test three hypotheses derivedfrom the theory: (1) those faced with a survey request on atopic of interest to them cooperate at higher rates than dothose less interested in the topic; (2) this tendency for the"interested" to cooperate more readily is diminished when monetaryincentives are offered; and (3) the impact of interest on cooperationhas nonignorability implications for key statistics. The datacome from a three-factor experiment examining the impact oncooperation with surveys on (a) five different topics, using(b) samples from five different populations that have knownattributes related to the topics, with (c) two different incentiveconditions.     

Survey Participation, Nonresponse Bias, Measurement Error Bias, and Total Bias

A common hypothesis about practices to reduce survey nonresponseis that those persons brought into the respondent pool throughpersuasive efforts may provide data filled with measurementerror. Two questions flow from this hypothesis. First, doesthe mean square error of a statistic increase when sample personswho are less likely to be contacted or cooperate are incorporatedinto the respondent pool? Second, do nonresponse bias estimatesmade on the respondents, using survey reports instead of records,provide accurate information about nonresponse bias? Using aunique data set, the Wisconsin Divorce Study, with divorce recordsas the frame and questions about the frame information includedin the questionnaire, this article takes a first look into thesetwo issues. We find that the relationship between nonresponsebias, measurement error bias, and response propensity is statistic-specificand specific to the type of nonresponse. Total bias tends tobe lower on estimates calculated using all respondents, comparedwith those with only the highest contact and cooperation propensities,and nonresponse bias analyses based on respondents yield conclusionssimilar to those based on records. Finally, we find that errorproperties of statistics may differ from error properties ofthe individual variables used to calculate the statistics.     

Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey

Declining contact and cooperation rates in random digit dial(RDD) national telephone surveys raise serious concerns aboutthe validity of estimates drawn from such research. While researchin the 1990s indicated that nonresponse bias was relativelysmall, response rates have continued to fall since then. Thecurrent study replicates a 1997 methodological experiment thatcompared results from a "Standard" 5-day survey employing thePew Research Center’s usual methodology with results froma "Rigorous" survey conducted over a much longer field periodand achieving a significantly higher response rate. As withthe 1997 study, there is little to suggest that unit nonresponsewithin the range of response rates obtained seriously threatensthe quality of survey estimates. In 77 out of 84 comparableitems, the two surveys yielded results that were statisticallyindistinguishable. While the "Rigorous" study respondents tendedto be somewhat less politically engaged, they did not reportconsistently different behaviors or attitudes on other kindsof questions. With respect to sample composition, the Standardsurvey was closely aligned with estimates from the U.S. Censusand other large government surveys on most variables. We extendour analysis of nonresponse to include comparisons with thehardest-to-reach respondents and with respondents who terminatedthe interview prior to completion.     

Nonresponse Rates and Nonresponse Bias in Household Surveys

Many surveys of the U.S. household population are experiencinghigher refusal rates. Nonresponse can, but need not, inducenonresponse bias in survey estimates. Recent empirical findingsillustrate cases when the linkage between nonresponse ratesand nonresponse biases is absent. Despite this, professionalstandards continue to urge high response rates. Statisticalexpressions of nonresponse bias can be translated into causalmodels to guide hypotheses about when nonresponse causes bias.Alternative designs to measure nonresponse bias exist, providingdifferent but incomplete information about the nature of thebias. A synthesis of research studies estimating nonresponsebias shows the bias often present. A logical question at thismoment in history is what advantage probability sample surveyshave if they suffer from high nonresponse rates. Since postsurveyadjustment for nonresponse requires auxiliary variables, theanswer depends on the nature of the design and the quality ofthe auxiliary variables.     

Evaluating the Effects of Screening for Telephone Service in Dual Frame RDD Surveys

The high costs and largely unknown error properties of cellulartelephone interviews make screening for cell-only adults a potentiallyattractive option in dual frame RDD surveys. Screening out adultswith landline telephones from the cellular sample does not affectthe coverage properties of a dual frame survey, but it may affectother sources of error, especially nonresponse. In this study,data from a 2006 dual frame RDD survey conducted for the PewResearch Center, the Associated Press, and AOL are used to evaluatethe effects of implementing a cell-only screener on both thebias and variance of weighted survey estimates. The effect ofscreening appears to be minimal so long as an adjustment fortelephone service is included in the weighting method. Resultsof an attempt to correct for residual nonresponse due to inaccessibilityare also discussed.     

Pemoline in Adult Attention Deficit Hyperactivity Disorder: Predictors of Nonresponse

Abstract

The psychostimulants are the most widely used treatment modality for attention deficit/hyperactivity disorder (ADHD). Although positive responses to psychostimulants have been demonstrated, 46% of adults with ADHD, on average, are nonresponsive or cannot tolerate the medications because of adverse effects. We reviewed the charts of 43 students treated with pemoline and studied parameters of nonresponse. Sixteen (37%) met the study definition of nonresponse. Of these, 5 (12%) were symptom nonresponders and 11 (25%) were nonresponders because of adverse effects. We found a significant relationship between age and nonresponse resulting from adverse effects, yet no other factors predicted nonresponse to pemoline. Clinicians should consider using smaller starting doses of pemoline in adults to minimize adverse effects.     

Gambling Behavior Among Adolescents with Attention Deficit/Hyperactivity Disorder

Impulsivity is inherent to both problem gambling and ADHD. The purpose of this study is to examine ADHD key symptoms, and gambling behaviors and problem severity among adolescents. Additionally, internalizing and externalizing behaviors exhibited among these individuals and the role of these symptoms in gambling are examined. We used a cross-sectional study design and survey 1,130 adolescents aged 12–19. Results indicated that adolescents who screened positive for ADHD were significantly more likely than non-ADHD adolescents to engage in gambling and significantly more likely to develop gambling problems. Those who screened positive as predominantly inattentive and those who screened positive for ADHD Combined (Inattention and Hyperactivity–Impulsivity) were equally likely to gamble, but the latter were twice as likely to have gambling problems. However, we found no significant interaction between the key ADHD symptoms and gambling as the severity of hyperactivity–impulsivity or inattention did not significantly differ with respect to gambling pathology. Emotional problems and depressive affect were the only variables that could significantly differentiate the ADHD types and gambling severity. Our Results highlight the clinical importance of considering the subtype of ADHD among gamblers and the greater association of depressive affect and emotional problems with gambling among adolescents.     

Effects and Costs of Tracing Strategies on Nonresponse Bias in a Survey of Workers with Low-Back Injury

Declining rates of participation are an increasing challengefor studies that involve telephone surveys. This study examinedthe costs of a telephone survey methodology that used increasinglyintensive tracing methods to track a pool of claimants who hadsustained occupational back injuries. It also compared the respondentsample to people who refused the survey and/or were not locatedor contacted. 3,181 claimants were drawn from a database maintainedby the Missouri Division of Workers’ Compensation (DWC)and 1,475 completed a telephone interview. The DWC databaseprovided data reflecting monetary and disability outcomes forall potential participants; telephone interviews provided additionalself-reported data. More intensive tracing strategies improvedthe representativeness of the sample, yielding more women andminority participants. Relative to less intensive techniques,advanced tracing efforts located people whose DWC records weremore similar to claimants not located. While the hands-on tracingefforts reduced the apparent bias of the respondent sample,costs increased as tracing strategies intensified – over$98,000 was spent to trace 1,027 claimants who were never locatedor contacted. The results suggest some guidelines that may facilitatedecision-making for researchers and funders who must balancethe trade-off between costs and nonresponse bias when planningsurveys.     

SUBSTANCE USE ESTIMATES FROM RDD AND AREA PROBABILITY SAMPLES: IMPACT OF DIFFERENTIAL SCREENING METHODS AND UNIT NONRESPONSE

Survey mode effects linked to unit nonresponse and screeningprocedures were estimated in a study of self-reported to bacco,alcohol, and illicit drug use. Two samples of the same population were drawn: a random digit dial (RDD) sample with telephonescreening and a multistage area probability sample with in-personscreening. Respondents in both samples were interviewed by telephoneto control mode of communication effects. Households with outtelephones were excluded from both samples to control samplecoverage effects. Although results show substantially lowerresponse rates in the telephone survey due to screening refusals,the two survey procedures produced samples that were very similarin demographic characteristics and in substance use profiles.Differential screening operations and differential nonresponserates between the RDD and area probability designs did not appearto be a major source of mode effects in comparing telephoneand face-to-face surveys of substance use.     

Self-Reported versus Proxy-Reported Voter Turnout in the Current Population Survey

The Current Population Survey (CPS) Voter Supplement is a premiersource of data on turnout in the United States. A little-knownaspect of the survey is that for a sizable proportion of people,turnout is measured by proxy—one member of a householdreports for another member—rather than self-reports ofhaving voted. The purpose of this research note is to investigatehow the use of proxy-reported turnout affects conclusions aboutthe determinants of turnout in America. The results suggesta generally optimistic assessment. Although proxy-reported turnoutis about 4 percentage points lower than self-reported turnout,the individual-level correlates of turnout and interstate turnoutdifferences are mostly similar for the two measures.     

THEORIES OF ETHNIC SOCIAL DISTANCE: COMPARATIVE EMPIRICAL TESTS FOR THREE DISTINCT ETHNIC GROUPS

Social science theories of ethnic division and antipathy are tested empirically using survey and zip code data for representative samples of whites, African Americans, and Mexican Americans in Texas. Ordered logistic regression tests estimate the effects of theoretically relevant variables on probabilities of racial and ethnic out-group social distance. Competing social science theories of ethnic and racial social distance are tested for multiple groups. Statistical support is found for group attachment/identity theories and realistic group conflict theories in explaining variance in out-group social closeness. The socioeconomic theory was not found to be an important predictor of out-group social closeness.     

Does Paid Work Enhance Women's Access to Welfare? Evidence from Selected Industrializing Countries

Citizenship has become increasingly associated—both symbolicallyand programmatically—with a person's capacity to performpaid work. For feminists this raises a key question: can women'sparticipation in paid work become the basis for their inclusionin welfare settlements? The question is examined here by exploringthe extent to which women's absorption into export-orientedmanufacturing industries has created the conditions for theirsocial citizenship in three industrializing countries (SouthKorea, China, and Mexico). The social rights considered arelimited to pensions and health care.     

Proceedings of the Sixty-Second Annual Conference of the American Association for Public Opinion Research

The American Association for Public Opinion Research (AAPOR)held its sixty-second annual conference during May 17–20,2007 at the Hyatt Regency Orange County Hotel in Anaheim, California. Attracting 825 attendees, the conference comprised 84 sessionsand three poster sessions dealing with methodological and substantiveissues in public opinion and survey research. Sessions addressedareas of research related to the interests of AAPOR members– methodology, nonresponse, sampling, voting and elections,to     

Effect of Interviewer Experience on Interview Pace and Interviewer Attitudes

Traditional statistical analyses of interviewer effects on surveydata do not examine whether these effects change over a fieldperiod. However, the nature of the survey interview is dynamic.Interviewers’ behaviors and perceptions may evolve asthey gain experience, thus potentially affecting data quality.This paper looks at how interview length and interviewer evaluationsof respondents change over interviewers’ workloads. Multilevelmodels with random interviewer effects are used to account forthe clustering of cases within interviewers and individual interviewercharacteristics in the 1984, 1988, and 2000 National ElectionStudies. The 1984 and 1988 NES released sample in four replicates,minimizing the confound between order in an interviewers’workload and sample composition. We find that over the courseof the studies, both measures change significantly. Interviewerprior survey experience also was significantly negatively relatedto the length of the interview. These findings have implicationsfor interviewer training prior to and during studies, as wellas suggesting future research to reveal why these behaviorsand perceptions change.     

SAME-GENDER SEX IN THE UNITED STATES IMPACT OF T-ACASI ON PREVALENCE ESTIMATES

Well-conducted telephone surveys provide an economical meansof estimating the prevalence of sexual and reproductive behaviorsin a population. There is, however, a nontrivial potential forbias since respondents must report sensitive information toa human interviewer. The National STD and Behavior MeasurementExperiment (NSBME) evaluates a new survey technology—telephoneaudio computer-assisted self-interviewing (T-ACASI)—thateliminates this requirement. The NSBME embedded a randomizedexperiment in a survey of probability samples of 1,543 U.S.and 744 Baltimore adults ages 18 to 45. Compared with NSBMErespondents interviewed by human interviewers, respondents interviewedby T-ACASI were 1.5 to 1.6 times more likely to report same-gendersexual attraction, experience, and genital contact. The impactof T-ACASI was more pronounced (odds ratio = 2.5) for residentsof locales that have historically been less tolerant of same-gendersexual behaviors and for respondents in households with children(odds ratio = 3.0).     

Estimating the Nonresponse BiasDue to Refusals in Telephone Surveys

This paper attempts to aid the process of accumulating the necessaryinformation for making more informed judgments about the effectsof nonresponse under different conditions. Two measures, whichpermit quantifiable nonsubjective assessment of the effectsof nonresponse on sample estimates, are introduced and are usedto examine the effects of respondent refusals in a random-digit-dialedgeneral population telephone survey of over 1,200 householdsas the response rate isincreased from 74.5 percent to 86.8 percent.By applying these measures under a wide range of conditions,the adequacy of various response rates may be assessed and morerational decisions made about the costs and benefits of devotingextraordinary resources to minimizing nonresponse.     

A model based approach for predicting annual poverty rates without expenditure data

The primus inter pares of the UN-approved Millennium Development Goals is to reduce poverty. The only internationally accepted method of estimating poverty requires a measurement of total consumption based on a time-consuming and resource-demanding measure of household expenditure in an integrated survey over 12 months. Rather than measuring poverty, say, only every fifth year, a model is presented to predict poverty based on a small set of household variables to be collected annually between two 12-monthly household surveys. Information obtained from these “light” surveys might then be used to predict poverty rates. The key question is whether the inaccuracy in these predictions is acceptable. It is recommended that these models be tested at a country level and if the test results are similar to those found here, that this approach be adopted.      

Response-Order Effects and the Early Gallup Split-Ballots

A meta-analysis of split-ballots conducted by the Gallup Organizationin the 1930s, 1940s, and early 1950s shows that response-ordereffects were generally small in magnitude when averaged acrossa great variety of topics and questions—and as comparedwith many of those reported in the response-effects literaturetoday. When analyzed by various question characteristics, theresults provided some support for predictions derived from currentcognitive models of response-order effects, particularly thosefrom satisficing theory. As predicted, questions asked orallywere more likely to generate a statistically significant patternof recency effects if the response alternatives or the questionsas a whole were longer rather than shorter. Other predictedpatterns of primacy and recency effects failed to materialize,however, perhaps largely because of the inherent design limitationsand partial confounding of question attributes in any such secondaryanalysis of archival survey data, but perhaps, too, becauseof simple chance variations. The data from these early experimentsnonetheless provide a partial, albeit limited, test of rivalhypotheses and explanations of response-order effects in theliterature.