What's Missing from National Landline RDD Surveys?: The Impact of the Growing Cell-Only Population

The number of cell phone only households has continued to grow– 12.8 percent of all households by the end of 2006, accordingto the National Health Interview Survey (NHIS). To assess theextent and nature of the potential bias in landline telephonesamples created by the absence of cell-only households, thePew Research Center conducted four independent dual frame studiesin 2006, three of which included at least 200 interviews withcell-only respondents. Despite the fact that there are significantdifferences between cell-only and landline respondents on manyimportant variables, across the four surveys with differentsubstantive content we find that including a cell-only samplewith a landline RDD sample produces general population estimatesthat are nearly identical to those from the landline samplealone. Yet, while the noncoverage problem is currently not damagingestimates for the entire population, we find evidence that itdoes create biased estimates on certain variables for youngadults, 25 percent of whom are cell-only according to the mostrecent government estimate.     

Survey Participation, Nonresponse Bias, Measurement Error Bias, and Total Bias

A common hypothesis about practices to reduce survey nonresponseis that those persons brought into the respondent pool throughpersuasive efforts may provide data filled with measurementerror. Two questions flow from this hypothesis. First, doesthe mean square error of a statistic increase when sample personswho are less likely to be contacted or cooperate are incorporatedinto the respondent pool? Second, do nonresponse bias estimatesmade on the respondents, using survey reports instead of records,provide accurate information about nonresponse bias? Using aunique data set, the Wisconsin Divorce Study, with divorce recordsas the frame and questions about the frame information includedin the questionnaire, this article takes a first look into thesetwo issues. We find that the relationship between nonresponsebias, measurement error bias, and response propensity is statistic-specificand specific to the type of nonresponse. Total bias tends tobe lower on estimates calculated using all respondents, comparedwith those with only the highest contact and cooperation propensities,and nonresponse bias analyses based on respondents yield conclusionssimilar to those based on records. Finally, we find that errorproperties of statistics may differ from error properties ofthe individual variables used to calculate the statistics.     

A Comparison of Address-Based Sampling (ABS) Versus Random-Digit Dialing (RDD) for General Population Surveys

Valid and reliable public health data are becoming more difficultto obtain through random-digit dial (RDD) telephone surveys.As a result, researchers are evaluating different survey designs(i.e., sampling frame and survey mode combinations) as complementsor alternatives to RDD. Traditionally, mail surveys of the generalpublic have been limited due to a lack of a complete samplingframe of households. Recent advances in electronic record keeping,however, have allowed researchers to develop a sample from aframe of addresses (e.g., the U.S. Postal Service Delivery SequenceFile, which appears to provide coverage which rivals or possiblyexceeds that obtained through RDD sampling methods). To testthe use of this frame for surveying adults aged 18 years andolder across a wide geographic area, a pilot study was conductedas part of the 2005 Behavioral Risk Factor Surveillance System(BRFSS). The pilot compared use of a traditional, RDD telephonesurvey methodology to an approach using a mail version of thequestionnaire completed by a random sample of households drawnfrom an address-based frame. The findings indicate that themail survey approach can achieve higher response rates in low-response-ratestates (<40%) than RDD (particularly when two mailings aresent). Additionally, the address frame with mail survey designprovides access to cell phone only households and offers costsavings over the telephone approach. The resulting sample, however,significantly overrepresents non-Hispanic whites and peoplewith higher levels of education.     

SUBSTANCE USE ESTIMATES FROM RDD AND AREA PROBABILITY SAMPLES: IMPACT OF DIFFERENTIAL SCREENING METHODS AND UNIT NONRESPONSE

Survey mode effects linked to unit nonresponse and screeningprocedures were estimated in a study of self-reported to bacco,alcohol, and illicit drug use. Two samples of the same population were drawn: a random digit dial (RDD) sample with telephonescreening and a multistage area probability sample with in-personscreening. Respondents in both samples were interviewed by telephoneto control mode of communication effects. Households with outtelephones were excluded from both samples to control samplecoverage effects. Although results show substantially lowerresponse rates in the telephone survey due to screening refusals,the two survey procedures produced samples that were very similarin demographic characteristics and in substance use profiles.Differential screening operations and differential nonresponserates between the RDD and area probability designs did not appearto be a major source of mode effects in comparing telephoneand face-to-face surveys of substance use.     

Nonresponse Bias in a Dual Frame Sample of Cell and Landline Numbers

We conducted a dual frame survey of landline and cell phonenumbers in 2004 to evaluate the feasibility of including cellphone numbers in random digit dial telephone surveys in theUnited States. By sampling cell phone numbers, the coveragebias associated with households that have only cell phones iseliminated. However, we discovered two major sources of nonresponsebias in the dual frame sample. In an attempt to reduce thesebiases, we applied several different estimation schemes. Buta comparison to the 2004 Current Population Survey Cell PhoneSupplement showed that none of the estimation schemes substantiallyreduced the nonresponse bias of the estimates. We suggest othermethods that might be used in future surveys that include cellphones and discuss the need for additional data collection andresearch on this issue.     

Household Telephone Service and Usage Patterns in the United states in 2004: Implications for Telephone Samples

Changes in the U.S. telephone system, especially the rapid growthin the prevalence and use of cell phones, raise concerns aboutundercoverage error in random digit dial (RDD) telephone samples.A supplement to the Current Population Survey (CPS) was conductedin 2004 to examine telephone service and usage in U.S. households.This article explores the potential for biases in RDD surveysresulting from the increases in cell phones by presenting estimatesof the percentage of households with different types of telephoneservice, including the percentage of cell-only households, andgiving demographic profiles of households by type of telephoneservice. Logistic regression models examine variables that predictwhether households are without a telephone or only have cellphones. These predictors may be used for weighting adjustmentsto reduce undercoverage biases. We address some additional issues,including the wording of questions for measuring telephone service,that are relevant if telephone-sampling methods are revisedto include cell phones. The estimates from the CPS supplementare also used to help understand some of the new sampling andweighting problems associated with selecting samples from cellphone numbers.     

The Role of Topic Interest in Survey Participation Decisions

While a low survey response rate may indicate that the riskof nonresponse error is high, we know little about when nonresponsecauses such error and when nonresponse is ignorable. Leverage-saliencetheory of survey participation suggests that when the surveytopic is a factor in the decision to participate, noncooperationwill cause nonresponse error. We test three hypotheses derivedfrom the theory: (1) those faced with a survey request on atopic of interest to them cooperate at higher rates than dothose less interested in the topic; (2) this tendency for the"interested" to cooperate more readily is diminished when monetaryincentives are offered; and (3) the impact of interest on cooperationhas nonignorability implications for key statistics. The datacome from a three-factor experiment examining the impact oncooperation with surveys on (a) five different topics, using(b) samples from five different populations that have knownattributes related to the topics, with (c) two different incentiveconditions.     

Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey

Declining contact and cooperation rates in random digit dial(RDD) national telephone surveys raise serious concerns aboutthe validity of estimates drawn from such research. While researchin the 1990s indicated that nonresponse bias was relativelysmall, response rates have continued to fall since then. Thecurrent study replicates a 1997 methodological experiment thatcompared results from a "Standard" 5-day survey employing thePew Research Center’s usual methodology with results froma "Rigorous" survey conducted over a much longer field periodand achieving a significantly higher response rate. As withthe 1997 study, there is little to suggest that unit nonresponsewithin the range of response rates obtained seriously threatensthe quality of survey estimates. In 77 out of 84 comparableitems, the two surveys yielded results that were statisticallyindistinguishable. While the "Rigorous" study respondents tendedto be somewhat less politically engaged, they did not reportconsistently different behaviors or attitudes on other kindsof questions. With respect to sample composition, the Standardsurvey was closely aligned with estimates from the U.S. Censusand other large government surveys on most variables. We extendour analysis of nonresponse to include comparisons with thehardest-to-reach respondents and with respondents who terminatedthe interview prior to completion.     

Sampling Telephone Numbers and Adults, Interview Length, and Weighting in The California Health Interview Survey Cell Phone Pilot Study

This article describes several features included in a CaliforniaHealth Interview Survey cell phone pilot study that differ fromearlier cell phone surveys conducted in the United States. Onedifference is that the study used a screening design and onlyadults living in cell-only households were interviewed. Mostof the previous studies used dual frame designs. Another differencewas the development and implementation of a within-householdadult sampling procedure to cover adults when cell phones wereshared in the household. The study was also intended to determineif conducting a cell phone interview of the same scope and lengthas a regular telephone interview was feasible. Most previoussurveys were focused on cell phone topics and were shorter thanthe comparable landline surveys. We then explore some interestingproblems that arise in weighting a cell phone survey using ascreening design. We conclude with a discussion of planninga new cell phone survey in 2007 that incorporates findings fromthis study.     

The Influence of Incremental Increases in Token Cash Incentives on Mail Survey Response: Is There an Optimal Amount?

The research reported is from a large-scale, national, mixed-modesurvey (N = 49,675 addresses), in which token incremental, noncontingentincentives used in the second (mail) stage of the survey weretested in an experimental design. The levels of the incentivestested ranged from $0 to $10, in one-dollar increments, withthe exception that there was no $9 condition. Furthermore, theexperiment was conducted using three types of household addressesthat originally were sampled in the first stage of the mixed-modesurvey via a random digit dial (RDD) frame: (1) households thatagreed to participate in the mail stage of the survey; (2) householdsthat were never contacted in the RDD stage despite numerouscontact attempts; and (3) households that had refused to participatewhen contacted in the RDD stage. It was determined that amongthose households that had previously agreed to participate,even those that received no cash incentive ($0) were more likelyto return completed surveys than the groups of previously noncontactedand previously refusing households that received $10. It wasalso found that the nature of the outcome of the prior contactwith the household mediated (that is, interacted with) the proportionaleffects of the incremental incentives, in ways only partiallyconsistent with Dillman’s social exchange theory. Thefindings, which are consistent with leverage-salience theory,suggest that if there is an optimal amount for an incentiveto use in a mail survey, then it is a variable amount best targetedto the individual household.     

USING DUAL FRAME DESIGNS TO REDUCE NONRESPONSE IN TELEPHONE SURVEYS

This article reports on the results of a series of experimentsdesigned to improve response rates for telephone surveys. Inthree surveys telephone households were selected using bothstandard random digit dialing (RDD) techniques and lists oftelephone numbers purchased from a commercial firm. In the RDDportions of the samples "cold contact" interviewing methodswere used; in the list frame portions advance letters were mailed,and the listed household name was used in the introduction.Experiments were designed to test the effects on response ratesof the advance letters and use of the listed household nameas a means of establishing rapport. The advance letters increasedresponse rates, but no difference could be attributed to theuse of names. The mixture of RDD and list sampling techniquesis also used to evaluate the effects of relative response rateson substantive findings. The cost consequences of these dualframe designs are assessed along a number of dimensions, andthe cost and error components of these designs are discussed.     

Experiments in Producing Nonresponse Bias

While nonresponse rates in household surveys are increasingin most industrialized nations, the increasing rates do notalways produce nonresponse bias in survey estimates. The linkagebetween nonresponse rates and nonresponse bias arises from thepresence of a covariance between response propensity and thesurvey variables of interest. To understand the covariance term,researchers must think about the common influences on responsepropensity and the survey variable. Three variables appear tobe especially relevant in this regard: interest in the surveytopic, reactions to the survey sponsor, and the use of incentives.A set of randomized experiments tests whether those likely tobe interested in the stated survey topic participate at higherrates and whether nonresponse bias on estimates involving variablescentral to the survey topic is affected by this. The experimentsalso test whether incentives disproportionately increase theparticipation of those less interested in the topic. The experimentsshow mixed results in support of these key hypotheses.     

Using Community-Level Correlates to Evaluate Nonresponse Effects in a Telephone Survey

Understanding the relationship between nonresponse processesand key research variables is central to evaluating if and hownonresponse introduces bias into survey estimates. In most telephonesurveys, however, little information is available with whichto estimate these effects. We report a procedure for examiningthe potential effects of nonresponse via analyses that (1) investigatethe linkages between community-level (zip code) variables andsurvey nonresponse and (2) examine the associations betweenthese community-level variables and key survey measures. Wedemonstrate these procedures using hierarchical modeling toanalyze data from a state-wide telephone survey in Illinois.One zip code–level indicator of concentrated disadvantage—thepercentage of the population below poverty level—was foundto be positively associated with nonresponse and, among respondents,with both current physical disability status and lack of healthinsurance coverage, suggesting that both may have been underestimatedin this survey. This inexpensive approach has the potentialof enabling researchers to routinely evaluate nonresponse effectsin their survey data.     

Cellular-Only Substitution in the United States as Lifestyle Adoption: Implications for Telephone Survey Coverage

Historically, the coverage bias from excluding the United Statescell-only population from survey samples has been minimal dueto the relatively small size of this group. However, the unrelentinggrowth of this segment has sparked growing concern that telephonesurveys of the general public in the United States will becomeincreasingly subject to coverage bias. While there is evidencethat demographic weighting can be used to eliminate this bias,the availability of the weights lag behind the rapidly changingcell-only population. To explain the extent of the problem,we propose a reliable model to forecast cell-only populationsize and demographics. This model posits that a stable behavioralprocess, the rate of habit retention, can be estimated fromprior wireless lifestyle adoption in the United States and mayalso describe adoption of the cell-only lifestyle. Using measuresof incentive and habituation, we test this assumption by predictingchanges in the cell-only population size and changes in agedemographics. The accuracy of predictions confirms the two adoptionbehaviors are similar. We then develop forecasts of age demographicsthrough 2009, and show how cell-only lifestyle adoption leadsto potential coverage bias that is better addressed throughthis type of modeling rather than weighting from historicaldata.     

The Effect of Interviewer Characteristics and Expectations on Response

This study reports on two sets of findings related to interviewereffects, derived from a national RDD sample of the adult population.The first of these concerns the effect of interviewer characteristicsand expectations on overall cooperation rates; the second, theeffect of interviewer characteristics and expectations on itemnonresponse and response quality. We found that interviewers'age, the size of the interviewing assignment, and interviewers'expectations all had a strong effect on overall cooperationrates; the relation of experience to response rate was curvilinearin this sample. Age and education have consistent but statisticallyinsignificant effects on item nonresponse. The effect of interviewers'expectations on responses within the interview resembles thatin earlier studies, but is less pronounced and less consistent.     

Are Divorce Studies Trustworthy? The Effects of Survey Nonresponse and Response Errors

Researchers rely on relationship data to measure the multifaceted nature of families. This article speaks to relationship data quality by examining the ramifications of different types of error on divorce estimates, models predicting divorce behavior, and models employing divorce as a predictor. Comparing matched survey and divorce certificate information from the 1995 Life Events and Satisfaction Study (N = 1,811) showed that nonresponse error is responsible for the majority of the error in divorce data. Misreporting the divorce event was rare, and more than two thirds of respondents provided a divorce date within 6 months of the actual date. Nevertheless, divorce date error attenuated effects of time since divorce on outcomes. Gender, child custody, marital history, and education were associated with divorce error.     

Nonresponse Rates and Nonresponse Bias in Household Surveys

Many surveys of the U.S. household population are experiencinghigher refusal rates. Nonresponse can, but need not, inducenonresponse bias in survey estimates. Recent empirical findingsillustrate cases when the linkage between nonresponse ratesand nonresponse biases is absent. Despite this, professionalstandards continue to urge high response rates. Statisticalexpressions of nonresponse bias can be translated into causalmodels to guide hypotheses about when nonresponse causes bias.Alternative designs to measure nonresponse bias exist, providingdifferent but incomplete information about the nature of thebias. A synthesis of research studies estimating nonresponsebias shows the bias often present. A logical question at thismoment in history is what advantage probability sample surveyshave if they suffer from high nonresponse rates. Since postsurveyadjustment for nonresponse requires auxiliary variables, theanswer depends on the nature of the design and the quality ofthe auxiliary variables.     

UNDERSTANDING THE DECISION TO PARTICIPATE IN A SURVEY

The lack of full participation in sample surveys threatens theinferential value of the survey method. We review a set of conceptualdevelopments and experimental findings that appear to be informativeabout causes of survey participation; offer an integration ofthat work with findings from the more traditional statisticaland survey methodological literature on nonresponse; and, giventhe theoretical structure, deduce potentially promising pathsof research toward the understanding of survey participation.     

Nonresponse in the American Time Use Survey: Who Is Missing from the Data and How Much Does It Matter?

This article examines nonresponse in a large government survey,the American Time Use Survey (ATUS), which interviews personsin households previously interviewed in the Current PopulationSurvey. The response rate for the ATUS has been below 60 percentfor the first two years of its existence, raising questionsabout whether the results can be generalized to the target population.The article begins with an analysis of the types of nonresponseencountered in the ATUS. Noncontact accounts for roughly 60percent of ATUS nonresponse, with refusals accounting for roughly40 percent. We find little support for the hypothesis that busypeople are less likely to respond to the ATUS but find considerablesupport for the hypothesis that people who are weakly integratedinto their communities are less likely to respond, mostly becausethey are less likely to be contacted. When we compare aggregateestimates of time use calculated using the ATUS base weightswithout any adjustment for nonresponse, estimates calculatedusing the ATUS final weights with a nonresponse adjustment,and estimates calculated using weights that incorporate ourown nonresponse adjustment based on a propensity model, we findsome modest differences, but the three sets of estimates arebroadly similar. The article ends with suggestions for furtherresearch and analysis.     

Estimating the Nonresponse BiasDue to Refusals in Telephone Surveys

This paper attempts to aid the process of accumulating the necessaryinformation for making more informed judgments about the effectsof nonresponse under different conditions. Two measures, whichpermit quantifiable nonsubjective assessment of the effectsof nonresponse on sample estimates, are introduced and are usedto examine the effects of respondent refusals in a random-digit-dialedgeneral population telephone survey of over 1,200 householdsas the response rate isincreased from 74.5 percent to 86.8 percent.By applying these measures under a wide range of conditions,the adequacy of various response rates may be assessed and morerational decisions made about the costs and benefits of devotingextraordinary resources to minimizing nonresponse.