Advance Letters as a Means of Improving Respondent Cooperation in Random Digit Dial Studies: A Multistate Experiment

Advance letters can potentially reduce the degree of nonresponsein random digit dial (RDD) surveys; however, they can also havea heterogeneous impact on subgroups, disproportionately raisingparticipation rates among certain segments of the populationand thereby having a detrimental effect on nonresponse bias.This is, in part, because advance letters can only be used inRDD surveys with the subset of respondents for whom an addresscan be identified. It may also be related to who in a householdsees the letter. We assess whether the use of advance letterscan improve the level of participation in the Behavioral RiskFactor Surveillance System (BRFSS) without introducing otherpotential data biases. The data reported here corroborate previousfindings, in terms of the positive impact that advance letterscan have on overall response rates (approximately a 6-percentage-pointgain). Moreover, the advance letters were cost-efficient inthat the cost of obtaining a fixed number of completed surveysusing advance letters was lower than the cost without letters.However, the positive impact of the advance letter on reducingnonresponse may have been offset to some extent in that theadvance letter may have biased the sample of those who completedthe interview toward older, white respondents and those of highersocioeconomic status, and away from younger, nonwhite individualsand persons with lower education and income levels. This lattergroup was underrepresented in the final sample and also lesslike to remember having received a letter.     

Experiments with incentives in telephone surveys

In an effort to counter the problem of noncooperation, survey organizations are offering incentives to respondents with increasing frequency, some at the outset of the survey, as has traditionally been done in mail surveys, and some only after the person has refused, in an attempt to convert the refusal. This article reports on a series of experiments carried out over a period of about 2 years with a monthly telephone survey, the Survey of Consumer Attitudes, in an effort to increase response rates or reduce interviewer effort. We report on experiments with prepaid versus promised incentives; advance letters; and advance letters with prepaid incentives; and we also report on the effects of incentives on response quality, sample composition, response bias, interviewer and respondent expectations, and costs.     

USING DUAL FRAME DESIGNS TO REDUCE NONRESPONSE IN TELEPHONE SURVEYS

This article reports on the results of a series of experimentsdesigned to improve response rates for telephone surveys. Inthree surveys telephone households were selected using bothstandard random digit dialing (RDD) techniques and lists oftelephone numbers purchased from a commercial firm. In the RDDportions of the samples "cold contact" interviewing methodswere used; in the list frame portions advance letters were mailed,and the listed household name was used in the introduction.Experiments were designed to test the effects on response ratesof the advance letters and use of the listed household nameas a means of establishing rapport. The advance letters increasedresponse rates, but no difference could be attributed to theuse of names. The mixture of RDD and list sampling techniquesis also used to evaluate the effects of relative response rateson substantive findings. The cost consequences of these dualframe designs are assessed along a number of dimensions, andthe cost and error components of these designs are discussed.     

Effects of an introductory letter on response rates to a teen/parent telephone health survey

The authors conducted a pilot study in preparation for a larger investigation that will rely on telephone surveys to assess select health behaviors of teens and their parents, with a focus on indoor tanning. This study used a randomized design to assess the impact of a presurvey letter on response rates to a telephone survey, as well as prevalence estimates of ever having used an indoor tanning facility. The letter did not have a significant effect on response rates or prevalence estimates in the two cities under study. Findings suggest that researchers should not assume that a letter sent in advance of a telephone survey will necessarily increase response rates enough to justify their use.     

Effects of an advance telephone call in a personal interview survey.

During October and November 1982, 1,260 Medicare-eligible senior citizens were interviewed in a survey focusing on health care of the elderly. As part of the survey, an experiment was conducted in each of the three survey sites to determine the effects of an advance telephone call to schedule an appointment for a personal interview. One random half sample in each site was sent a lead letter, followed by a telephone call to schedule a personal interview. The other half sample was sent a lead letter followed by a personal contact, with no intervening telephone call. Telephoning to arrange an appointment for a personal interview resulted in a 20 percent saving in data collection costs with only a 1 percent decrease in response rate.     

RESPONSE RATES AND RESPONSE CONTENT IN MAIL VERSUS FACE-TO-FACE SURVEYS

Two surveys were administered based on the same area probabilitysampling frame and with some of the same questions: one samplewas used for hour-long face-to-face interviewing in the 1992Detroit Area Study; the other sample received a much shorterquestionnaire in the mail for sell-administration. The samplesegments had previously been stratified in terms of the percentagethat was black. For the predominantly white stratum, there wasonly a small difference in response rates due to mode of administration.For the predominantly black stratum, the mail survey obtaineda considerably lower response rate then the face-to-face survey.Within the predominantly white stratum, there were no cleardifferences between results for the two modes of administrationin demographic variables or in gross housing characteristics.However, the mail survey respondents expressed more negativeattitudes toward racial integration and affirmative action thandid the face-to-face respondents. Because the mail sample ofthe predominantly black stratum was small, it was not possibleto carry out similar analyses of demographic or attitudinaldifferences, or to determine whether its lower response ratewas due mainly to race, to correlates of race such as incomeor education, or even to problems with mail delivery in centralcities.     

Bias in a Directory Sample for a Mail Survey of Rural Households

This study examines bias in a mail survey of rural householdsthat used telephone directories as the sampling frame and comparesit with a parallel survey that used an independent sample selectedfrom field listings. A higher response rate was obtained withthe directory sample, which was primarily attributed to pooreraddress information for the listed sample. Although coverageof the study population was higher for the listed frame thanfor the directory frame, no significant differences were foundfor eight respondent and seven household characteristics. Also,no significant differences were found for seven key contentvariables regarding health behaviors and attitudes. The findingssupport the use of telephone directories as the sampling framefor general population mail surveys of rural areas where suchcoverage of the population is considered adequate.     

The Cost-Effectiveness of Alternative Advance Mailings in a Telephone Survey

Advance letters are being used increasingly to combat the declinein response rates, but their effectiveness depends partly ontheir being read. In the face of the post-9/11 anthrax mailingscare, the Office for Survey Research (OSR) converted to usingadvance postcards rather than advance letters. Subsequently,the OSR conducted an experiment to test the effectiveness ofsending letters, postcards, or nothing to directory-listed numbersselected in a large random digit dial (RDD) telephone survey.In terms of response rates, the results clearly indicate thatsending advance letters is more effective than sending postcards,which, in turn, is more effective than sending nothing. A detailedcost-effectiveness evaluation indicates that sending lettersactually saves money, despite the increased costs associatedwith the mailings, compared to sending postcards; sending postcardsis also more cost-effective than not sending anything.     

Möglichkeiten und Probleme des Einsatzes postalischer Befragungen

The research focuses on the strategies in mail surveys that lead to high response rates and on the social characteristics of the respondents that are recruited into the survey through reminders. Empirical basis are surveys in the East German cities of Chemnitz, Dresden and Leipzig with an original sample size of ca 2000 persons per city and up to three reminders. The response rate varies between 68 and 70 percent and surpasses the usual response rates in face to face and telephone surveys done in German cities. In contrast to common assumptions in the literature it has no effect on the response rate whether stamps are used on the address and return envelope, neither do original versus scanned signatures make a difference and whether financial incentives are used in the late reminders. Through the reminders especially respondents with low education and of little political interest are recruited into the survey.     

A Comparison of Address-Based Sampling (ABS) Versus Random-Digit Dialing (RDD) for General Population Surveys

Valid and reliable public health data are becoming more difficultto obtain through random-digit dial (RDD) telephone surveys.As a result, researchers are evaluating different survey designs(i.e., sampling frame and survey mode combinations) as complementsor alternatives to RDD. Traditionally, mail surveys of the generalpublic have been limited due to a lack of a complete samplingframe of households. Recent advances in electronic record keeping,however, have allowed researchers to develop a sample from aframe of addresses (e.g., the U.S. Postal Service Delivery SequenceFile, which appears to provide coverage which rivals or possiblyexceeds that obtained through RDD sampling methods). To testthe use of this frame for surveying adults aged 18 years andolder across a wide geographic area, a pilot study was conductedas part of the 2005 Behavioral Risk Factor Surveillance System(BRFSS). The pilot compared use of a traditional, RDD telephonesurvey methodology to an approach using a mail version of thequestionnaire completed by a random sample of households drawnfrom an address-based frame. The findings indicate that themail survey approach can achieve higher response rates in low-response-ratestates (<40%) than RDD (particularly when two mailings aresent). Additionally, the address frame with mail survey designprovides access to cell phone only households and offers costsavings over the telephone approach. The resulting sample, however,significantly overrepresents non-Hispanic whites and peoplewith higher levels of education.     

UNDERSTANDING MAIL SURVEY RESPONSE BEHAVIOR A META-ANALYSIS

A meta-analysis of prior studies of techniques designed to inducemail survey response rates was conducted. Research encompassing184 effects (study outcomes) in 115 studies (articles) for 17predictors of response rate was examined. The average effectsize across all manipulations was r=.065, indicating an averageincrease of about 6.5 percent in response rates for manipulations.Effect sizes for specific predictors and two potential moderatorsof effects were examined. Results indicated that repeated contactsin the form of preliminary notification and follow-ups, appeals,inclusion of a return envelope, postage, and monetary incentives,were effective in increasing survey response rates. Significanteffect sizes for the predictors ranged from an increase in responseof 2 percent to 31 percent. Implications of the results forthe conduct of mail surveys and future research on mail surveyresponse behavior are discussed.     

Personalization of Mail Surveys for General Public and Populations with a Group Identity: Results from Nine Studies*

Abstract The effect of personalization on mail survey response rates was examined in nine studies that included 17 comparisons under several research conditions. A study of this variable across multiple experiments in five agricultural experiment stations was undertaken because of conflicting results from previous research and from concern that the effectiveness of personalization might have decreased over time. Results show that, while response to general public surveys appeared to increase modestly across all treatment groups, there was no positive effect for populations in which a group identity (e.g., Dear Oregon Gardner or Dear ATV Owner) is employed to address respondents in cover letters. Personalization appears to remain useful for improving response in surveys of the general public.     

RRT meets RDD: statistical strategies for assuring response privacy in telephone surveys

Despite its growing role in survey research, the telephone survey has been largely neglected with respect to the design and testing of statistical procedures for assuring response privacy (e.g., randomized response techniques). This paper demonstrates that such procedures are no less feasible to administer in telephone surveys than in face-to-face surveys. Both randomized response techniques and alternatives to randomized response are considered.     

Effects of Category Order on Answers in Mail and Telephone Surveys1

Abstract Past research suggests that mail surveys encourage a primacy effect, which is a tendency to choose the first answers from a list, whereas telephone surveys encourage a recency effect, a tendency to choose the last answers from a list. This paper summarizes results from 82 new experiments conducted in 12 separate surveys in seven states. Only four of 33 mail survey comparisons exhibited significant primacy effects, while five of 26 experiments in telephone surveys exhibited recency effects. In addition, only three of 23 cross-method comparisons produced a significant primacy/recency effect in the expected manner. The conclusion is that the prevalence of primacy and recency effects has been over-estimated by past research and a new theoretical approach that takes into account multiple causation is needed for examining these effects.     

SUBSTANCE USE ESTIMATES FROM RDD AND AREA PROBABILITY SAMPLES: IMPACT OF DIFFERENTIAL SCREENING METHODS AND UNIT NONRESPONSE

Survey mode effects linked to unit nonresponse and screeningprocedures were estimated in a study of self-reported to bacco,alcohol, and illicit drug use. Two samples of the same population were drawn: a random digit dial (RDD) sample with telephonescreening and a multistage area probability sample with in-personscreening. Respondents in both samples were interviewed by telephoneto control mode of communication effects. Households with outtelephones were excluded from both samples to control samplecoverage effects. Although results show substantially lowerresponse rates in the telephone survey due to screening refusals,the two survey procedures produced samples that were very similarin demographic characteristics and in substance use profiles.Differential screening operations and differential nonresponserates between the RDD and area probability designs did not appearto be a major source of mode effects in comparing telephoneand face-to-face surveys of substance use.     

Breaking ground on the virtual frontier: Surveying civic life on the internet

The Internet provides a virtual frontier to expand our access to information and to increase our knowledge and understanding of public opinion, political behavior, social trends and lifestyles through survey research. Comparable to other technological advancements, such as the telephone and the computer, the Internet presents opportunities that will impact significantly on the process and quality of survey research now and in the twenty-first century. Cyberspace permits us to move beyond traditional face-to-face, mail and telephone surveys, yet still to examine basic issues regarding the quality of data collection: sampling, questionnaire design, survey distribution, means of response, and database creation. This surveys include those posted on LISTERV mailing lists, USENET newsgroups, and the World Wide Web. A survey on politics and civic life, conducted by the authors, is used as a case study. She is currently pursuing interests in methodological issues concerning surveying using the Internet, and in policy issues concerning student victimization on college and university campuses. His publications includeViable Democracy (1979);Manipulating Public Opinion (1989);Free Expression, Public Support and Censorship: Examining Government’s Role in the Arts in Canada and the United States (1994); and numerous articles in scholarly and popular journals. His current research interests are contemporary democratic theory and political participation, as well as the political thought of John Locke.     

MAIL SURVEYS FOR ELECTION FORECASTING?: AN EVALUATION OF THE COLUMBUS DISPATCH POLL

Because of slow turnaround time and typically low response rates,mail surveys have generally been considered of little valuein election forecasting. However, statewide mail surveys conductedby the Columbus Dispatch newspaper since 1980 have made remarkablyaccurate forecasts of Ohio election outcomes. In comparisonto statewide surveys by two other organizations employing conventionaltelephone interview methods, the mail surveys were consistentlymore accurate and were generally less susceptible to sourcesof inaccuracy such as high rolloff and low publicity. The mailsurvey's advantage is attributable at least in part to largersample sizes, sampling and response procedures that yieldedmore representative samples of voters, lack of the need to allocateundecided respondents, and superior questionnaire design. Thesefindings suggest that mail surveys not only may be viable alternativesto telephone surveys but may actually be superior to them undersome conditions. Further-more, these results demonstrate thatsurveys with low response rates are not necessarily low in validity.     

RRT Meets RDD: Statistical Strategies for Assuring Response Privacy in Telephone Surveys

Despite its growing role in survey research, the telephone surveyhas been largely neglected with respect to the design and testingof statistical procedures for assuring response privacy (e.g.,randomized response techniques). This paper demonstrates thatsuch procedures are no less feasible to administer in telephonesurveys than in face-to-face surveys. Both randomized responsetechniques and alternatives to randomized response are considered.     

The Effects of Material Incentives in Mail Surveys: Two Studies

Two studies were undertaken on the effects of including a materialnonmonetary incentive in mail surveys, using various samplesof the general public in the Netherlands. The results show thatnonmonetary incentives produce a higher initial response rate,but follow-ups reduced the effect of the incentive to a nonsignificantratio. Inclusion of the incentive produced no response biasand little volunteer bias. Results from these studies offera possible explanation for why past studies on nonmonetary incentiveshave often shown positive effects: they were conducted usingmethods that produced low response rates. When methods thatproduce high response rates are used, the effect of nonmonetaryincentives on response rate disappears. Finally, the use ofmonetary incentives in mail surveys with a high base responseis discussed.