Advance Letters as a Means of Improving Respondent Cooperation in Random Digit Dial Studies: A Multistate Experiment

Advance letters can potentially reduce the degree of nonresponsein random digit dial (RDD) surveys; however, they can also havea heterogeneous impact on subgroups, disproportionately raisingparticipation rates among certain segments of the populationand thereby having a detrimental effect on nonresponse bias.This is, in part, because advance letters can only be used inRDD surveys with the subset of respondents for whom an addresscan be identified. It may also be related to who in a householdsees the letter. We assess whether the use of advance letterscan improve the level of participation in the Behavioral RiskFactor Surveillance System (BRFSS) without introducing otherpotential data biases. The data reported here corroborate previousfindings, in terms of the positive impact that advance letterscan have on overall response rates (approximately a 6-percentage-pointgain). Moreover, the advance letters were cost-efficient inthat the cost of obtaining a fixed number of completed surveysusing advance letters was lower than the cost without letters.However, the positive impact of the advance letter on reducingnonresponse may have been offset to some extent in that theadvance letter may have biased the sample of those who completedthe interview toward older, white respondents and those of highersocioeconomic status, and away from younger, nonwhite individualsand persons with lower education and income levels. This lattergroup was underrepresented in the final sample and also lesslike to remember having received a letter.     

The Cost-Effectiveness of Alternative Advance Mailings in a Telephone Survey

Advance letters are being used increasingly to combat the declinein response rates, but their effectiveness depends partly ontheir being read. In the face of the post-9/11 anthrax mailingscare, the Office for Survey Research (OSR) converted to usingadvance postcards rather than advance letters. Subsequently,the OSR conducted an experiment to test the effectiveness ofsending letters, postcards, or nothing to directory-listed numbersselected in a large random digit dial (RDD) telephone survey.In terms of response rates, the results clearly indicate thatsending advance letters is more effective than sending postcards,which, in turn, is more effective than sending nothing. A detailedcost-effectiveness evaluation indicates that sending lettersactually saves money, despite the increased costs associatedwith the mailings, compared to sending postcards; sending postcardsis also more cost-effective than not sending anything.     

The Influence of Advance Letters on Response in Telephone Surveys: A Meta-Analysis

Recently, the leading position of telephone surveys as the majormode of data collection has been challenged. Telephone surveyssuffer from a growing nonresponse, partly due to the generalnonresponse trend for all surveys and partly due to changesin society and technology influencing contactability and willingnessto answer. One way to counteract the increasing nonresponseis the use of an advance letter. In mail and face-to-face surveys,advance letters have been proven effective. Based on the proveneffectiveness in face-to-face and mail surveys, survey handbooksadvise the use of advance letters in telephone surveys. Thisstudy reviews the evidence for this advice and presents a quantitativesummary of empirical studies on the effectiveness of advanceletters in raising the response rate for telephone surveys.The major conclusion is that advance letters are also an effectivetool in telephone surveys, with an average increase in responserate (RR1) from 58 percent (no letter) to 66 percent (advanceletter), and an average increase in cooperation rate (COOP1)from 64 percent (no letter) to 75 percent (advance letter).     

Estimating Residency Rates for Undetermined Telephone Numbers

The method for estimating residency rates in random digit dial(RDD) telephone surveys is important for computing responserates. This article reviews existing methods of estimating residencyrates and introduces a new "survival" method that takes advantageof more information to provide improved estimates. Examplesof applying this to large RDD samples are given along with suggestionsfor use of the method in other surveys.     

A Comparison of Address-Based Sampling (ABS) Versus Random-Digit Dialing (RDD) for General Population Surveys

Valid and reliable public health data are becoming more difficultto obtain through random-digit dial (RDD) telephone surveys.As a result, researchers are evaluating different survey designs(i.e., sampling frame and survey mode combinations) as complementsor alternatives to RDD. Traditionally, mail surveys of the generalpublic have been limited due to a lack of a complete samplingframe of households. Recent advances in electronic record keeping,however, have allowed researchers to develop a sample from aframe of addresses (e.g., the U.S. Postal Service Delivery SequenceFile, which appears to provide coverage which rivals or possiblyexceeds that obtained through RDD sampling methods). To testthe use of this frame for surveying adults aged 18 years andolder across a wide geographic area, a pilot study was conductedas part of the 2005 Behavioral Risk Factor Surveillance System(BRFSS). The pilot compared use of a traditional, RDD telephonesurvey methodology to an approach using a mail version of thequestionnaire completed by a random sample of households drawnfrom an address-based frame. The findings indicate that themail survey approach can achieve higher response rates in low-response-ratestates (<40%) than RDD (particularly when two mailings aresent). Additionally, the address frame with mail survey designprovides access to cell phone only households and offers costsavings over the telephone approach. The resulting sample, however,significantly overrepresents non-Hispanic whites and peoplewith higher levels of education.     

Experiments with incentives in telephone surveys

In an effort to counter the problem of noncooperation, survey organizations are offering incentives to respondents with increasing frequency, some at the outset of the survey, as has traditionally been done in mail surveys, and some only after the person has refused, in an attempt to convert the refusal. This article reports on a series of experiments carried out over a period of about 2 years with a monthly telephone survey, the Survey of Consumer Attitudes, in an effort to increase response rates or reduce interviewer effort. We report on experiments with prepaid versus promised incentives; advance letters; and advance letters with prepaid incentives; and we also report on the effects of incentives on response quality, sample composition, response bias, interviewer and respondent expectations, and costs.     

SUBSTANCE USE ESTIMATES FROM RDD AND AREA PROBABILITY SAMPLES: IMPACT OF DIFFERENTIAL SCREENING METHODS AND UNIT NONRESPONSE

Survey mode effects linked to unit nonresponse and screeningprocedures were estimated in a study of self-reported to bacco,alcohol, and illicit drug use. Two samples of the same population were drawn: a random digit dial (RDD) sample with telephonescreening and a multistage area probability sample with in-personscreening. Respondents in both samples were interviewed by telephoneto control mode of communication effects. Households with outtelephones were excluded from both samples to control samplecoverage effects. Although results show substantially lowerresponse rates in the telephone survey due to screening refusals,the two survey procedures produced samples that were very similarin demographic characteristics and in substance use profiles.Differential screening operations and differential nonresponserates between the RDD and area probability designs did not appearto be a major source of mode effects in comparing telephoneand face-to-face surveys of substance use.     

Monitoring youth behavior in response to structural changes. Alternative approaches for measuring adolescent drinking

Random digit dial (RDD) telephone and self-administered school-based surveys of drinking and drinking and driving were compared for adolescents from two Northern California communities. The RDD and school-based surveys resulted in very different samples. The telephone sample contained a greater proportion of European Americans and a smaller proportion of Asian Americans and "other" ethnicities. Respondents to the telephone sample also tended to be older and of higher socioeconomic status (SES). The telephone survey captured very few school dropouts. Moreover, it resulted in lower self-reports of drinking and drinking and driving. Survey mode appeared to influence respondents equally, regardless of their gender, age, ethnicity, or SES.     

Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey

Declining contact and cooperation rates in random digit dial(RDD) national telephone surveys raise serious concerns aboutthe validity of estimates drawn from such research. While researchin the 1990s indicated that nonresponse bias was relativelysmall, response rates have continued to fall since then. Thecurrent study replicates a 1997 methodological experiment thatcompared results from a "Standard" 5-day survey employing thePew Research Center’s usual methodology with results froma "Rigorous" survey conducted over a much longer field periodand achieving a significantly higher response rate. As withthe 1997 study, there is little to suggest that unit nonresponsewithin the range of response rates obtained seriously threatensthe quality of survey estimates. In 77 out of 84 comparableitems, the two surveys yielded results that were statisticallyindistinguishable. While the "Rigorous" study respondents tendedto be somewhat less politically engaged, they did not reportconsistently different behaviors or attitudes on other kindsof questions. With respect to sample composition, the Standardsurvey was closely aligned with estimates from the U.S. Censusand other large government surveys on most variables. We extendour analysis of nonresponse to include comparisons with thehardest-to-reach respondents and with respondents who terminatedthe interview prior to completion.     

Bias in a Directory Sample for a Mail Survey of Rural Households

This study examines bias in a mail survey of rural householdsthat used telephone directories as the sampling frame and comparesit with a parallel survey that used an independent sample selectedfrom field listings. A higher response rate was obtained withthe directory sample, which was primarily attributed to pooreraddress information for the listed sample. Although coverageof the study population was higher for the listed frame thanfor the directory frame, no significant differences were foundfor eight respondent and seven household characteristics. Also,no significant differences were found for seven key contentvariables regarding health behaviors and attitudes. The findingssupport the use of telephone directories as the sampling framefor general population mail surveys of rural areas where suchcoverage of the population is considered adequate.     

Effects of an introductory letter on response rates to a teen/parent telephone health survey

The authors conducted a pilot study in preparation for a larger investigation that will rely on telephone surveys to assess select health behaviors of teens and their parents, with a focus on indoor tanning. This study used a randomized design to assess the impact of a presurvey letter on response rates to a telephone survey, as well as prevalence estimates of ever having used an indoor tanning facility. The letter did not have a significant effect on response rates or prevalence estimates in the two cities under study. Findings suggest that researchers should not assume that a letter sent in advance of a telephone survey will necessarily increase response rates enough to justify their use.     

Effects of Category Order on Answers in Mail and Telephone Surveys1

Abstract Past research suggests that mail surveys encourage a primacy effect, which is a tendency to choose the first answers from a list, whereas telephone surveys encourage a recency effect, a tendency to choose the last answers from a list. This paper summarizes results from 82 new experiments conducted in 12 separate surveys in seven states. Only four of 33 mail survey comparisons exhibited significant primacy effects, while five of 26 experiments in telephone surveys exhibited recency effects. In addition, only three of 23 cross-method comparisons produced a significant primacy/recency effect in the expected manner. The conclusion is that the prevalence of primacy and recency effects has been over-estimated by past research and a new theoretical approach that takes into account multiple causation is needed for examining these effects.     

EFFECTS OF INTERVIEW MODE ON SELF-REPORTED DRUG USE

The effects of interview mode on 18- to 34-year-olds' self-reportedtobacco, alcohol, marijuna, and cocaine use were investigated.RDD telephone and personal drug use surveys of the state ofNew Jersey were conducted in 1986–87. In the personalinterview, respondents recorded their drug use on self-administratedanswer sheets. Compared to the area probability sample in thepersonal interview, RDD yielded a sample of blacks higher inincome and education, and more likely to be married and currentlyemployed; white SES was also higher in the telephone sample,but to a lesser degree than for blacks. Controlling for demographiccharacteristics and RDD's exclusion of non-telephone households,the telephone survey yielded significantly lower estimates ofblacks' alcohol consumption, and lifetime and recent marijuanause. Whites' alcohol consumption was slightly lower by telephone;otherwise, estimates of whites' use of the four substances werenearly identical in the two modes. Sample coverage, respondentdemographic characteristics, and racial matching of interviewerand respondent did not account for the significant mode differences.Characteristics of the interview situation itself, such as provisionof privacy in the self-administered format, may have influencedtendencies toward socially desirable responding to a threateningtopic such as drug use.     

Consequences of reducing nonresponse in a national telephone survey

Critics of public opinion polls often claim that methodological shortcuts taken to collect timely data produce biased results. This study compares two random digit dial national telephone surveys that used identical questionnaires but very different levels of effort: a "Standard" survey conducted over a 5-day period that used a sample of adults who were home when the interviewer called, and a "Rigorous" survey conducted over an 8-week period that used random selection from among all adult household members. Response rates, computed according to AAPOR guidelines, were 60.6 percent for the Rigorous and 36.0 percent for the Standard study. Nonetheless, the two surveys produced similar results. Across 91 comparisons, no difference exceeded 9 percentage points, and the average difference was about 2 percentage points. Most of the statistically significant differences were among demographic items. Very few significant differences were found on attention to media and engagement in politics, social trust and connectedness, and most social and political attitudes, including even those toward surveys.     

MEETING THE CHALLENGE OF ANSWERING MACHINES

Answering machines have become one of the major challenges toobtaining high response rates in telephone surveys. This articlepresents some data that may be helpful in answering two questionsthat frequently arise: (1) What are the chances that additionalcalls to a household known to use an answering machine willresult in a completed interview? (2) When is the best time tocall, in order to minimize the chances of encountering an answeringmachine? The data used to analyze these questions are basedon the calling records of the 1990 California Disability Survey,a large random-digit dialing survey that generated about 330,000calls and completed interviews at over 24,000 house holds. Thislarge number of calls is interesting not only in absolute termsbut because of the high number of callbacks that they represent,designed to bring the response rate up over 80 percent. Therecords of calls analyzed here, consequently, can show convincinglywhat really happens when 10, 20, 30, or even more callbacksare attempted. Such opportunities are rare. Researchers cantake advantage of these results to generate more informed callingstrategies and consequently improve response rates in theirsurveys.     

The Influence of Incremental Increases in Token Cash Incentives on Mail Survey Response: Is There an Optimal Amount?

The research reported is from a large-scale, national, mixed-modesurvey (N = 49,675 addresses), in which token incremental, noncontingentincentives used in the second (mail) stage of the survey weretested in an experimental design. The levels of the incentivestested ranged from $0 to $10, in one-dollar increments, withthe exception that there was no $9 condition. Furthermore, theexperiment was conducted using three types of household addressesthat originally were sampled in the first stage of the mixed-modesurvey via a random digit dial (RDD) frame: (1) households thatagreed to participate in the mail stage of the survey; (2) householdsthat were never contacted in the RDD stage despite numerouscontact attempts; and (3) households that had refused to participatewhen contacted in the RDD stage. It was determined that amongthose households that had previously agreed to participate,even those that received no cash incentive ($0) were more likelyto return completed surveys than the groups of previously noncontactedand previously refusing households that received $10. It wasalso found that the nature of the outcome of the prior contactwith the household mediated (that is, interacted with) the proportionaleffects of the incremental incentives, in ways only partiallyconsistent with Dillman’s social exchange theory. Thefindings, which are consistent with leverage-salience theory,suggest that if there is an optimal amount for an incentiveto use in a mail survey, then it is a variable amount best targetedto the individual household.     

Telephone versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires: Comparisons of Respondent Satisficing and Social Desirability Response Bias

The last 50 years have seen a gradual replacement of face-to-faceinterviewing with telephone interviewing as the dominant modeof survey data collection in the United States. But some ofthe most expensive and large-scale nationally funded, long-termsurvey research projects involving national area-probabilitysamples and long questionnaires retain face-to-face interviewingas their mode. In this article, we propose two ways in whichshifting such surveys to random digit dialing (RDD) telephoneinterviewing might affect the quality of data acquired, andwe test these hypotheses using data from three national modeexperiments. Random digit dialing telephone respondents weremore likely to satisfice (as evidenced by no-opinion responding,nondifferentiation, and acquiescence), to be less cooperativeand engaged in the interview, and were more likely to expressdissatisfaction with the length of the interview than were face-to-facerespondents, despite the fact that the telephone interviewswere completed more quickly than the face-to-face interviews.Telephone respondents were also more suspicious about the interviewprocess and more likely to present themselves in socially desirableways than were face-to-face respondents. These findings shedlight on the nature of the survey response process, on the costsand benefits associated with particular survey modes, and onthe nature of social interaction generally.     

The Current Efficiency of List-Assisted Telephone Sampling Designs

List-assisted random digit dialing (RDD) designs became popularin the late 1980s and early 1990s. Work done by the Bureau ofLabor Statistics and the University of Michigan in 1993 resultedin the development of the underlying theory for these designsas well as the evaluation of various alternative sampling plansto optimize the method. Recent research to reevaluate thesedesigns in light of the significant changes in the telephonesystem over the past decade is presented in this article. Thearticle provides background on the development of list-assisteddesigns, and recent changes in the U.S. telephone system arereviewed. Using 1999 data from Survey Sampling, Inc., an analysisof the current state of the telephone system is presented, anda reoptimization of the earlier designs is undertaken. Resultsfrom the earlier work are compared with findings from the 1999data.     

Sampling Telephone Numbers and Adults, Interview Length, and Weighting in The California Health Interview Survey Cell Phone Pilot Study

This article describes several features included in a CaliforniaHealth Interview Survey cell phone pilot study that differ fromearlier cell phone surveys conducted in the United States. Onedifference is that the study used a screening design and onlyadults living in cell-only households were interviewed. Mostof the previous studies used dual frame designs. Another differencewas the development and implementation of a within-householdadult sampling procedure to cover adults when cell phones wereshared in the household. The study was also intended to determineif conducting a cell phone interview of the same scope and lengthas a regular telephone interview was feasible. Most previoussurveys were focused on cell phone topics and were shorter thanthe comparable landline surveys. We then explore some interestingproblems that arise in weighting a cell phone survey using ascreening design. We conclude with a discussion of planninga new cell phone survey in 2007 that incorporates findings fromthis study.     

Telephone surveys under different frame constructions—evaluating differences in estimates,precision, and cost

Dual-frame, random-digit-dialing (DFRDD) surveys, using a frame of landline telephone numbers and a frame of cell phone numbers, have become the norm for telephone surveys. Two alternative frame constructions sample exclusively from a cell phone frame, referred to as the ‘single-frame cell,’ and sample from the listed landline frame and the cell phone frame, referred to as ‘listed/cell.’ The listed/cell frame construction differs from the DFRDD in that it does not select unlisted-landline frame members. We observe in our simulated data that the listed/cell frame construction results in estimates that more closely approximate the estimates from DFRDD compared with the single-frame cell, results in an improvement in precision compared with the DFRDD that is not as large as the improvement in the single-frame cell, and has lower cost than both the DFRDD and the single-frame cell. We conclude that a listed/cell frame construction might be a better solution than either a DFRDD or single-frame cell frame construction.