Consequences of reducing nonresponse in a national telephone survey

Critics of public opinion polls often claim that methodological shortcuts taken to collect timely data produce biased results. This study compares two random digit dial national telephone surveys that used identical questionnaires but very different levels of effort: a "Standard" survey conducted over a 5-day period that used a sample of adults who were home when the interviewer called, and a "Rigorous" survey conducted over an 8-week period that used random selection from among all adult household members. Response rates, computed according to AAPOR guidelines, were 60.6 percent for the Rigorous and 36.0 percent for the Standard study. Nonetheless, the two surveys produced similar results. Across 91 comparisons, no difference exceeded 9 percentage points, and the average difference was about 2 percentage points. Most of the statistically significant differences were among demographic items. Very few significant differences were found on attention to media and engagement in politics, social trust and connectedness, and most social and political attitudes, including even those toward surveys.     

Effects of screening questionnaires on response in a two-phase postal survey

For surveys targeting specific population groups, the two-phase postal approach (screener followed by a topical survey sent to eligible households) has been demonstrated to be more effective at identifying population domains of interest than random digit dial telephone methods considering cost, coverage, and response. An important question is how best to motivate screener response from eligible households. In 2011, we conducted a large-scale field test to empirically test a number of methods for motivating response. We fielded screening surveys that varied content-influencing relevance, and also switched screener questionnaires for following up nonrespondents to the initial postal survey – an approach we have labeled responsive tailoring. In another experiment, we tested the effect of asking for first names in the screener questionnaire. In this article, we describe the effects of these experimental treatments on response to both the screener and the topical survey.     

The impact of touch‐tone data entry on reports of HIV and STD risk behaviors in telephone interviews

Respondents’ concerns about privacy can decrease reporting of HIV and STD risk behaviors in general population telephone surveys. The purpose of this paper is to describe the results of an experimental study evaluating whether one method for increasing privacy, touch‐tone data entry (TTDE), is effective in increasing estimates of sexual behaviors from a population‐based survey. We conducted a random‐digit‐dial telephone survey of adults in New Jersey (n = 405), with half the respondents using TTDE for answering sexual behavior questions. TTDE led to increased reports of same‐sex sexual behavior, certain HIV and STD risk factors, and concern about one's risk for HIV and STD transmission. TTDE also narrowed the difference between men's and women's reports of the number of different sexual partners over the past 10 years. The feasibility and limitations of TTDE are discussed, along with possible alternative interpretations that consider the impact of TTDE on the dynamics of the interaction between the respondent and the interviewer.     

Telephone versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires: Comparisons of Respondent Satisficing and Social Desirability Response Bias

The last 50 years have seen a gradual replacement of face-to-faceinterviewing with telephone interviewing as the dominant modeof survey data collection in the United States. But some ofthe most expensive and large-scale nationally funded, long-termsurvey research projects involving national area-probabilitysamples and long questionnaires retain face-to-face interviewingas their mode. In this article, we propose two ways in whichshifting such surveys to random digit dialing (RDD) telephoneinterviewing might affect the quality of data acquired, andwe test these hypotheses using data from three national modeexperiments. Random digit dialing telephone respondents weremore likely to satisfice (as evidenced by no-opinion responding,nondifferentiation, and acquiescence), to be less cooperativeand engaged in the interview, and were more likely to expressdissatisfaction with the length of the interview than were face-to-facerespondents, despite the fact that the telephone interviewswere completed more quickly than the face-to-face interviews.Telephone respondents were also more suspicious about the interviewprocess and more likely to present themselves in socially desirableways than were face-to-face respondents. These findings shedlight on the nature of the survey response process, on the costsand benefits associated with particular survey modes, and onthe nature of social interaction generally.     

Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey

Declining contact and cooperation rates in random digit dial(RDD) national telephone surveys raise serious concerns aboutthe validity of estimates drawn from such research. While researchin the 1990s indicated that nonresponse bias was relativelysmall, response rates have continued to fall since then. Thecurrent study replicates a 1997 methodological experiment thatcompared results from a "Standard" 5-day survey employing thePew Research Center’s usual methodology with results froma "Rigorous" survey conducted over a much longer field periodand achieving a significantly higher response rate. As withthe 1997 study, there is little to suggest that unit nonresponsewithin the range of response rates obtained seriously threatensthe quality of survey estimates. In 77 out of 84 comparableitems, the two surveys yielded results that were statisticallyindistinguishable. While the "Rigorous" study respondents tendedto be somewhat less politically engaged, they did not reportconsistently different behaviors or attitudes on other kindsof questions. With respect to sample composition, the Standardsurvey was closely aligned with estimates from the U.S. Censusand other large government surveys on most variables. We extendour analysis of nonresponse to include comparisons with thehardest-to-reach respondents and with respondents who terminatedthe interview prior to completion.     

Nonresponse Bias in a Dual Frame Sample of Cell and Landline Numbers

We conducted a dual frame survey of landline and cell phonenumbers in 2004 to evaluate the feasibility of including cellphone numbers in random digit dial telephone surveys in theUnited States. By sampling cell phone numbers, the coveragebias associated with households that have only cell phones iseliminated. However, we discovered two major sources of nonresponsebias in the dual frame sample. In an attempt to reduce thesebiases, we applied several different estimation schemes. Buta comparison to the 2004 Current Population Survey Cell PhoneSupplement showed that none of the estimation schemes substantiallyreduced the nonresponse bias of the estimates. We suggest othermethods that might be used in future surveys that include cellphones and discuss the need for additional data collection andresearch on this issue.     

Problems Relating to Declining Response Rates to Social Survey Research in Japan: Trends After 2000

Abstract: Sociologists in Japan recognize that the circumstances under which social survey research is conducted have been deteriorating and have become critical since 2000. Social survey research is understood as a large-scale survey that mainly draws samples by a random sampling method, using a questionnaire as a measurement tool. The crises of social survey research are summarized as following: (1) the increasing difficulty of conducting accurate sampling and (2) the decline in response rates to drawn samples (planned samples). This paper focuses on the decline in response rates to social survyes by examining recent trends in relation to social surveys conducted by the Japanese government and local government (prefectures). In summary, there has been a landslidedecline of the response rate between 2005 and 2006 in surveys conducted by Japanese government. This decline was thought to mainly come from the declining of response rates among the elderly and women, groups that had previously had high response rates. This seems to be the result of the growing level of the crime involving the abuse of the personal information. However, there did not appear to be a serious decline in the response rate for surveys conducted by local governments. Given that it is not easy to restore response rates, it is important to develop research methods and techniques to cope with the decline. Further, informing respondents of the purpose of the research and its benefit to may help raise response rates.     

SUBSTANCE USE ESTIMATES FROM RDD AND AREA PROBABILITY SAMPLES: IMPACT OF DIFFERENTIAL SCREENING METHODS AND UNIT NONRESPONSE

Survey mode effects linked to unit nonresponse and screeningprocedures were estimated in a study of self-reported to bacco,alcohol, and illicit drug use. Two samples of the same population were drawn: a random digit dial (RDD) sample with telephonescreening and a multistage area probability sample with in-personscreening. Respondents in both samples were interviewed by telephoneto control mode of communication effects. Households with outtelephones were excluded from both samples to control samplecoverage effects. Although results show substantially lowerresponse rates in the telephone survey due to screening refusals,the two survey procedures produced samples that were very similarin demographic characteristics and in substance use profiles.Differential screening operations and differential nonresponserates between the RDD and area probability designs did not appearto be a major source of mode effects in comparing telephoneand face-to-face surveys of substance use.     

Estimating Residency Rates for Undetermined Telephone Numbers

The method for estimating residency rates in random digit dial(RDD) telephone surveys is important for computing responserates. This article reviews existing methods of estimating residencyrates and introduces a new "survival" method that takes advantageof more information to provide improved estimates. Examplesof applying this to large RDD samples are given along with suggestionsfor use of the method in other surveys.     

From Agenda Setting to Refusal Setting: Survey Nonresponse as a Function of Media Coverage Across the 2004 Election Cycle

Past research suggests that there is a relationship betweensurvey response and topic salience, namely that individualsresponding to a survey are likely to find the survey topic moresalient than nonrespondents do. For election surveys, nonresponseresulting from a lack of salience can influence findings becauserespondents may be more interested in politics than nonrespondents.The agenda-setting model suggests that media coverage shouldheighten salience. Thus, as media coverage of political campaignsincreases over the course of an election, refusals to a politicalsurvey should decline. Using data from the National AnnenbergElection Survey (NAES), which was conducted nearly continuouslyin 2004, this study investigates the issue of nonresponse ina random digit dial telephone survey across the election cycleby examining daily changes in the refusal rates using time-seriesanalysis. Content analyses of the frequencies of presidentialcampaign stories mentioned in the New York Times and three networknews broadcasts were matched against a time series from theNAES to demonstrate that increases in media coverage of theelection were negatively related to the survey refusal rate. Received for publication January 15, 2005. Revision received March 18, 2007. Accepted for publication May 14, 2007.     

The Current Efficiency of List-Assisted Telephone Sampling Designs

List-assisted random digit dialing (RDD) designs became popularin the late 1980s and early 1990s. Work done by the Bureau ofLabor Statistics and the University of Michigan in 1993 resultedin the development of the underlying theory for these designsas well as the evaluation of various alternative sampling plansto optimize the method. Recent research to reevaluate thesedesigns in light of the significant changes in the telephonesystem over the past decade is presented in this article. Thearticle provides background on the development of list-assisteddesigns, and recent changes in the U.S. telephone system arereviewed. Using 1999 data from Survey Sampling, Inc., an analysisof the current state of the telephone system is presented, anda reoptimization of the earlier designs is undertaken. Resultsfrom the earlier work are compared with findings from the 1999data.     

Crime Victimization in Latin America and Intentions to Migrate to the United States

Among the challenges faced by Latin America at the onset of the 21st century is the increase in crime and violence that began in the mid-1980s, and which, to one degree or another, has afflicted most countries in the region. In this study we explore the potential implications of the upsurge in crime on migration by testing the hypothesis that crime victimization in Latin America increases the probability that people have given serious thought to the prospect of migrating with their families to the United States. Using Latinobarometro public opinion surveys of approximately 49,000 respondents residing in 17 countries in 2002, 2003, and 2004, the results of a Hierarchical Generalized Linear Model found that, net of individual and country-level control variables, the probability of seriously considering family migration to the United States was around 30 percent higher among respondents who reported that they or a member of their family was a victim of a crime sometime during the year prior to the survey. Evidence that victimization promotes the propensity to emigrate is a finding that contributes to an understanding of the transnational consequences of the increase in crime in Latin America, and adds a new variable to the inventory of factors that encourage people to migrate to the United States.     

Mitofsky Waksberg: Learning From The Past

Warren Mitofsky and Joseph Waksberg were two of the most influentialresearchers in telephone surveys and survey research methods.Their development of the Mitofsky–Waksberg method forrandom digit dialing sampling revolutionized telephone samplingand surveys. Jointly and separately, Waksberg and Mitofsky wereinstrumental in the development of many innovative methods andapplications. This article reviews some of their contributionsand discusses how their approaches to these problems may bea model for addressing issues that are being encountered currentlyin the United States in cell phone research.     

Cell Phone Survey Feasibility in The U.S.: Sampling and Calling Cell Numbers Versus Landline Numbers

In 2004, we conducted a nationwide dual frame survey of landlineand cell phone numbers to evaluate the feasibility of includingcell phone numbers in a random digit dial telephone survey.Households with both landline and cell phones were eligiblefor selection in both samples. This article describes our designand data collection methods; compares the results from the twosamples (with an emphasis on operational characteristics); andpresents the outcomes of two experimental manipulations designedto improve the cell phone response rate.     

Monitoring youth behavior in response to structural changes. Alternative approaches for measuring adolescent drinking

Random digit dial (RDD) telephone and self-administered school-based surveys of drinking and drinking and driving were compared for adolescents from two Northern California communities. The RDD and school-based surveys resulted in very different samples. The telephone sample contained a greater proportion of European Americans and a smaller proportion of Asian Americans and "other" ethnicities. Respondents to the telephone sample also tended to be older and of higher socioeconomic status (SES). The telephone survey captured very few school dropouts. Moreover, it resulted in lower self-reports of drinking and drinking and driving. Survey mode appeared to influence respondents equally, regardless of their gender, age, ethnicity, or SES.     

Comparison of web and mail surveys in collecting illicit drug use data: a randomized experiment

This randomized experiment examined survey mode effects for self-reporting illicit drug use by comparing prevalence estimates between a Web-based survey and a mail-based survey. A random sample of 7,000 traditional-aged undergraduate students attending a large public university in the United States was selected to participate in the spring of 2001. Students were randomly assigned to self-administer a survey via the Web or U.S. mail. The Web survey produced a significantly higher response rate than the U.S. mail survey. The prevalence estimates of illicit drug use (lifetime and past year) did not differ significantly between the two survey modes. The findings provide preliminary evidence that Web and U.S. mail surveys produce similar results regarding illicit drug use among undergraduate students. Although additional research is needed involving more diverse samples, these findings bode well for using Web surveys in college-based research.     

USING DUAL FRAME DESIGNS TO REDUCE NONRESPONSE IN TELEPHONE SURVEYS

This article reports on the results of a series of experimentsdesigned to improve response rates for telephone surveys. Inthree surveys telephone households were selected using bothstandard random digit dialing (RDD) techniques and lists oftelephone numbers purchased from a commercial firm. In the RDDportions of the samples "cold contact" interviewing methodswere used; in the list frame portions advance letters were mailed,and the listed household name was used in the introduction.Experiments were designed to test the effects on response ratesof the advance letters and use of the listed household nameas a means of establishing rapport. The advance letters increasedresponse rates, but no difference could be attributed to theuse of names. The mixture of RDD and list sampling techniquesis also used to evaluate the effects of relative response rateson substantive findings. The cost consequences of these dualframe designs are assessed along a number of dimensions, andthe cost and error components of these designs are discussed.     

The Influence of Incremental Increases in Token Cash Incentives on Mail Survey Response: Is There an Optimal Amount?

The research reported is from a large-scale, national, mixed-modesurvey (N = 49,675 addresses), in which token incremental, noncontingentincentives used in the second (mail) stage of the survey weretested in an experimental design. The levels of the incentivestested ranged from $0 to $10, in one-dollar increments, withthe exception that there was no $9 condition. Furthermore, theexperiment was conducted using three types of household addressesthat originally were sampled in the first stage of the mixed-modesurvey via a random digit dial (RDD) frame: (1) households thatagreed to participate in the mail stage of the survey; (2) householdsthat were never contacted in the RDD stage despite numerouscontact attempts; and (3) households that had refused to participatewhen contacted in the RDD stage. It was determined that amongthose households that had previously agreed to participate,even those that received no cash incentive ($0) were more likelyto return completed surveys than the groups of previously noncontactedand previously refusing households that received $10. It wasalso found that the nature of the outcome of the prior contactwith the household mediated (that is, interacted with) the proportionaleffects of the incremental incentives, in ways only partiallyconsistent with Dillman’s social exchange theory. Thefindings, which are consistent with leverage-salience theory,suggest that if there is an optimal amount for an incentiveto use in a mail survey, then it is a variable amount best targetedto the individual household.     

Comparison of telephone RDD and online panel survey modes on CPGI scores and co-morbidities

The purpose of this study was to compare the influence of two survey modes – telephone random digit dialling (RDD) and stratified sampling of an online panel – on Canadian Problem Gambling Index (CPGI) scores and co-morbidities. An identical gambling survey was administered to 4000 South Koreans selected via random dialling of cell phone numbers and to 4330 South Korean members of an online panel. The online panelists were found to have significantly higher levels of tobacco use, drug or alcohol problems and mental health problems, higher employment status and lower happiness levels. However, even when controlling for these variables, CPGI scores for the online panel were significantly higher than those for the telephone interview. This appears to be due to higher levels of pathology among online panelists combined with potentially more valid self-reports due to enhanced anonymity.     

Locating a Special Population Using Random Digit Dialing

Frequently, researchers are only interested in specific subgroupsof the total population. Waksberg (1978) has designed a two-stagemethod of selecting equal-probability samples of the generalpopulation using random digit dialing. The present researchproposes and illustrates a modification of Waksberg's procedureto locate special populations which cluster geographically.Situations arising from the proposed modifications are discussedand alternative solutions suggested.