Telephone surveys under different frame constructions—evaluating differences in estimates,precision, and cost

Dual-frame, random-digit-dialing (DFRDD) surveys, using a frame of landline telephone numbers and a frame of cell phone numbers, have become the norm for telephone surveys. Two alternative frame constructions sample exclusively from a cell phone frame, referred to as the ‘single-frame cell,’ and sample from the listed landline frame and the cell phone frame, referred to as ‘listed/cell.’ The listed/cell frame construction differs from the DFRDD in that it does not select unlisted-landline frame members. We observe in our simulated data that the listed/cell frame construction results in estimates that more closely approximate the estimates from DFRDD compared with the single-frame cell, results in an improvement in precision compared with the DFRDD that is not as large as the improvement in the single-frame cell, and has lower cost than both the DFRDD and the single-frame cell. We conclude that a listed/cell frame construction might be a better solution than either a DFRDD or single-frame cell frame construction.     

BIAS IN LIST-ASSISTED TELEPHONE SAMPLES

A number of researchers have suggested list-assisted samplingfor the selection of telephone households to overcome some ofthe operational difficulties associated with the Mitofsky-Waksbergmethods of random digit dialing (RDD). An advantage of a list-assistedmethod of RDD is that an equal probability systematic sampleof telephone numbers can be selected and the variances of estimatesfrom such a sample are usually lower than from a clustered designlike the Mitofsky-Waksberg method. The main disadvantage ofthe list-assisted method is that it excludes some householdsfrom the sample, thus creating a coverage bias in the estimates.This article describes research on the coverage bias for a particularmethod of list-assisted sampling. The two key determinants ofcoverage bias are the proportion of households that are noteligible for the sample and the differences in the characteristicsof the covered and not covered populations. The results showthat about 4 percent of all households are excluded in nationalsamples using this method of sampling. Furthermore, they showthat the differences between the covered and uncovered populationsare generally not large. The coverage bias resulting from theseconditions may often be small.     

USING DUAL FRAME DESIGNS TO REDUCE NONRESPONSE IN TELEPHONE SURVEYS

This article reports on the results of a series of experimentsdesigned to improve response rates for telephone surveys. Inthree surveys telephone households were selected using bothstandard random digit dialing (RDD) techniques and lists oftelephone numbers purchased from a commercial firm. In the RDDportions of the samples "cold contact" interviewing methodswere used; in the list frame portions advance letters were mailed,and the listed household name was used in the introduction.Experiments were designed to test the effects on response ratesof the advance letters and use of the listed household nameas a means of establishing rapport. The advance letters increasedresponse rates, but no difference could be attributed to theuse of names. The mixture of RDD and list sampling techniquesis also used to evaluate the effects of relative response rateson substantive findings. The cost consequences of these dualframe designs are assessed along a number of dimensions, andthe cost and error components of these designs are discussed.     

The Influence of Information Costs on Donor Decision Making

This article explores whether an individual's information costs influence the information‐gathering strategies that he or she turns to prior to donating to a nonprofit. The data for the study come from a telephone survey of residents in a large county in southern California (n = 1,002). The sample was selected using random‐digit‐dialing technology and a computer‐assisted telephone interviewing system. A series of binomial logit models revealed that information costs significantly influenced the information‐gathering strategies individuals turned to in order to learn about nonprofit performance prior to making a donation. Results also revealed that greater confidence in nonprofits did not lead individuals to forgo efforts to obtain information on nonprofit performance altogether. The findings from this study suggest that it is important for nonprofit administrators to recognize, and then take into account, that not all individuals will rely on the same information sources when seeking to learn about nonprofit performance.     

Response Styles in Telephone and Household Interviewing: A Field Experiment

Two independent groups from the Los Angeles metropolitan areawere sampled and interviewed—one was sampled by a formof random digit dialing and interviewed by telephone; the otherwas sampled on an area probability basis and interviewed face-to-facein households. Few sociodemographic differences were found betweenthe telephone and household samples, suggesting that the twomethods sampled essentially the same population. There weredifferences in the quantity and quality of the data obtained,however, and the telephone sample had more missing data forfamily income, more acquiescence, evasiveness, and extremenessresponse bias on attitude questions, and more responses to checklists.     

Nonresponse Bias in a Dual Frame Sample of Cell and Landline Numbers

We conducted a dual frame survey of landline and cell phonenumbers in 2004 to evaluate the feasibility of including cellphone numbers in random digit dial telephone surveys in theUnited States. By sampling cell phone numbers, the coveragebias associated with households that have only cell phones iseliminated. However, we discovered two major sources of nonresponsebias in the dual frame sample. In an attempt to reduce thesebiases, we applied several different estimation schemes. Buta comparison to the 2004 Current Population Survey Cell PhoneSupplement showed that none of the estimation schemes substantiallyreduced the nonresponse bias of the estimates. We suggest othermethods that might be used in future surveys that include cellphones and discuss the need for additional data collection andresearch on this issue.     

Telephone versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires: Comparisons of Respondent Satisficing and Social Desirability Response Bias

The last 50 years have seen a gradual replacement of face-to-faceinterviewing with telephone interviewing as the dominant modeof survey data collection in the United States. But some ofthe most expensive and large-scale nationally funded, long-termsurvey research projects involving national area-probabilitysamples and long questionnaires retain face-to-face interviewingas their mode. In this article, we propose two ways in whichshifting such surveys to random digit dialing (RDD) telephoneinterviewing might affect the quality of data acquired, andwe test these hypotheses using data from three national modeexperiments. Random digit dialing telephone respondents weremore likely to satisfice (as evidenced by no-opinion responding,nondifferentiation, and acquiescence), to be less cooperativeand engaged in the interview, and were more likely to expressdissatisfaction with the length of the interview than were face-to-facerespondents, despite the fact that the telephone interviewswere completed more quickly than the face-to-face interviews.Telephone respondents were also more suspicious about the interviewprocess and more likely to present themselves in socially desirableways than were face-to-face respondents. These findings shedlight on the nature of the survey response process, on the costsand benefits associated with particular survey modes, and onthe nature of social interaction generally.     

Mitofsky Waksberg: Learning From The Past

Warren Mitofsky and Joseph Waksberg were two of the most influentialresearchers in telephone surveys and survey research methods.Their development of the Mitofsky–Waksberg method forrandom digit dialing sampling revolutionized telephone samplingand surveys. Jointly and separately, Waksberg and Mitofsky wereinstrumental in the development of many innovative methods andapplications. This article reviews some of their contributionsand discusses how their approaches to these problems may bea model for addressing issues that are being encountered currentlyin the United States in cell phone research.     

Does Telephone Audio Computer-Assisted Self-Interviewing Improve the Accuracy of Prevalence Estimates of Youth Smoking? Evidence from the UMass Tobacco Study

Despite their advantage for obtaining representative samplesof adolescents, telephone surveys have been regarded as an inferiormethod for collecting data on youth tobacco use because theyyield lower estimates than school-based, self-administered surveys.Although no gold standard for smoking estimates exists, thelower estimates in telephone surveys have been attributed tounderreporting due to youths’ concern that parents orothers may overhear their responses. Telephone audio computer-assistedself-interviewing (T-ACASI) is a cost-effective method for obtaininga representative sample of youths and provides increased privacyfor the respondent. We hypothesize that using T-ACASI wouldencourage youths to report more fully smoking behavior comparedto traditional interviewer-administered telephone methods. Ouranalysis further assesses whether respondent age, gender, race/ethnicity,and parental attitude toward smoking moderated the relationshipbetween survey mode and smoking reports. Using data from a statewidetobacco use survey that randomly assigned youth respondentsto either T-ACASI or interviewer-administered modes, we findthat youths were more likely to report smoking behaviors inT-ACASI mode and that this was especially true for girls, particularlythose who believed their parents would disapprove strongly oftheir smoking. The findings suggest that traditional telephonesurveys may underestimate smoking prevalence in most girls bya factor of two, and that a technique for ensuring privacy forthese respondents is an important component of effective telephonesurvey methodology.     

EFFECTS OF INTERVIEW MODE ON SELF-REPORTED DRUG USE

The effects of interview mode on 18- to 34-year-olds' self-reportedtobacco, alcohol, marijuna, and cocaine use were investigated.RDD telephone and personal drug use surveys of the state ofNew Jersey were conducted in 1986–87. In the personalinterview, respondents recorded their drug use on self-administratedanswer sheets. Compared to the area probability sample in thepersonal interview, RDD yielded a sample of blacks higher inincome and education, and more likely to be married and currentlyemployed; white SES was also higher in the telephone sample,but to a lesser degree than for blacks. Controlling for demographiccharacteristics and RDD's exclusion of non-telephone households,the telephone survey yielded significantly lower estimates ofblacks' alcohol consumption, and lifetime and recent marijuanause. Whites' alcohol consumption was slightly lower by telephone;otherwise, estimates of whites' use of the four substances werenearly identical in the two modes. Sample coverage, respondentdemographic characteristics, and racial matching of interviewerand respondent did not account for the significant mode differences.Characteristics of the interview situation itself, such as provisionof privacy in the self-administered format, may have influencedtendencies toward socially desirable responding to a threateningtopic such as drug use.     

Reaching the U.S. Cell Phone Generation: Comparison of Cell Phone Survey Results with an Ongoing Landline Telephone Survey

Noncoverage rates in U.S. landline-based telephone samples dueto cell phone only households (i.e., households with no landlinebut accessible by cell phone) and the corresponding potentialfor bias in estimates from surveys that sample only from landlineframes are growing issues. Building on some of the few publishedstudies that focus on this problem, a study was conducted inthree states (Georgia, New Mexico, and Pennsylvania) as partof the Behavioral Risk Factor Surveillance System (BRFSS), theworld's largest ongoing public health telephone survey, to evaluatethe effectiveness of conducting the BRFSS interview with a sampledrawn from dedicated cell phone telephone exchanges and mixed-use(landline and cell phone) exchanges. Approximately 600 interviewswere conducted in each of two groups: cell phone only adults(n = 572) and adults with both a landline and a cell phone (n= 592). Making comparisons with data from the ongoing, landline-basedBRFSS survey, we report on response rates, demographic characteristicsof respondents, key survey estimates of health conditions andrisk behaviors, and survey costs. The methods used in this studyhave wide application for other U.S. telephone surveys.     

Telephone sampling bias in surveying disability

Results of the California Disability Survey indicate that telephone interviewing is well suited for undertaking disability studies that provide (1) estimates of subgroups of the disabled population, including those that are statistically rare: (2) information on current and anticipated areas of policy concern; and (3) information for geographic areas important in rehabilitation program planning. Although these objectives necessitated a large sample size and a complex instrument, the costs and timeliness of telephone interviewing enabled the survey objectives to be reached. This paper discusses the advantages of telephone interviewing, as well as biases inherent in its use. The magnitude of the bias from the omission of nontelephone households is assessed, and the results confirm that the omission of nontelephone households introduces only minor biases into estimates for the total working-age population. A method of weighting is developed and illustrated.     

Motivators for Seeking Gambling-Related Treatment Among Ontario Problem Gamblers

A random digit dialing telephone survey was used to interview 8,467 adults in Ontario, Canada. The NODS-CLiP was used to identify a representative sample of 730 gamblers (54.3% male, mean age 45.3 years) with possible past year gambling problems in order to explore factors that might affect disordered gamblers’ motivators for seeking gambling-related help. A final sample of 526 gamblers provided useable data on possible reasons for and barriers to seeking help, awareness of services, self-perception of gambling problems and experience with help-seeking. Financial and relationship issues were the most frequently volunteered motivators. However, over two-thirds of the respondents could not think of a reason for seeking help. Gamblers who had self-admitted or more severe problems, who knew how to get help, who were employed and had more education, and who identified possible barriers to seeking help were more likely to suggest motivators, especially financial ones. More research is recommended on gamblers’ trajectory towards recognition of a gambling problem, the process of overcoming specific barriers to treatment, and the role of social advantage (e.g., education and employment), in order to devise educational campaigns that will encourage earlier help-seeking among disordered gamblers.     

Assessment of Cigarette Smoking and Smoking Susceptibility among Youth: Telephone Computer-Assisted Self-Interviews versus Computer-Assisted Telephone Interviews

This experimental study assesses the effect of two survey methods,telephone audio computer-assisted self-interviewing (T-ACASI)and computer-assisted telephone interviewing (CATI), on self-reportsof smoking behavior and smoking susceptibility among adolescents12–17 years of age in California. In T-ACASI, participantslisten to prerecorded, computer-controlled questions and respondby pressing the keypad on a touch-tone telephone. In CATI, interviewersadminister the questions and enter responses into a computer.Prior research suggests that youth may be more likely to reportsensitive behaviors in a self-administered survey like T-ACASIcompared to an interviewer-administered survey like CATI, dueto greater perceived confidentiality. Logistic regression analyseswere conducted on unweighted data, controlling for demographicdifferences. Adjusted estimates of current smoking (past 30days) were significantly greater in T-ACASI (8.3 percent) thanCATI (4.5 percent). Smoking susceptibility (i.e., lack of afirm commitment not to smoke among those who have never smoked)was also greater in T-ACASI (45.0 percent) than CATI (34.9 percent).In both surveys, social desirability response bias was negativelyassociated with smoking, which suggests that response bias wasproblematic for both modes. Many respondents reported that aparent was present during the interview (59.4 percent in CATI;42.0 percent in T-ACASI). In both surveys, parental presencewas negatively associated with smoking, which suggests thatthis factor could also contribute to underreporting. Applicationof sample weights to the data eliminated the survey mode effects;however, the CATI current smoking estimate (9.3 percent) fromthis study was significantly less than an estimate (14.2 percent)obtained from a self-administered, school-based survey conductedthe same year on California adolescents.     

Effects of an introductory letter on response rates to a teen/parent telephone health survey

The authors conducted a pilot study in preparation for a larger investigation that will rely on telephone surveys to assess select health behaviors of teens and their parents, with a focus on indoor tanning. This study used a randomized design to assess the impact of a presurvey letter on response rates to a telephone survey, as well as prevalence estimates of ever having used an indoor tanning facility. The letter did not have a significant effect on response rates or prevalence estimates in the two cities under study. Findings suggest that researchers should not assume that a letter sent in advance of a telephone survey will necessarily increase response rates enough to justify their use.     

Beliefs About Gambling Problems and Recovery: Results from a General Population Telephone Survey

Respondents were asked their beliefs about gambling abuse as part of a general population telephone survey. The random digit dialing survey consisted of 8,467 interviews of adults, 18 years and older, from Ontario, Canada (45% male; mean age = 46.2). The predominant conception of gambling abuse was that of an addiction, similar to drug addiction. More than half of respondents reported that treatment was necessary and almost three-quarters of respondents felt that problem gamblers would have to give up gambling completely in order to overcome their gambling problem. Problem gamblers (past or current) were less likely than non- or social gamblers to believe that treatment was needed, and current problem gamblers were least likely to believe that abstinence was required, as compared to all other respondents. Strong agreement with conceptions of gambling abuse as disease or addiction were positively associated with belief that treatment is needed, while strong agreement with conceptions of disease or wrongdoing were positively associated with belief that abstinence is required.     

Effects of an advance telephone call in a personal interview survey.

During October and November 1982, 1,260 Medicare-eligible senior citizens were interviewed in a survey focusing on health care of the elderly. As part of the survey, an experiment was conducted in each of the three survey sites to determine the effects of an advance telephone call to schedule an appointment for a personal interview. One random half sample in each site was sent a lead letter, followed by a telephone call to schedule a personal interview. The other half sample was sent a lead letter followed by a personal contact, with no intervening telephone call. Telephoning to arrange an appointment for a personal interview resulted in a 20 percent saving in data collection costs with only a 1 percent decrease in response rate.     

Cell Phone Survey Feasibility in The U.S.: Sampling and Calling Cell Numbers Versus Landline Numbers

In 2004, we conducted a nationwide dual frame survey of landlineand cell phone numbers to evaluate the feasibility of includingcell phone numbers in a random digit dial telephone survey.Households with both landline and cell phones were eligiblefor selection in both samples. This article describes our designand data collection methods; compares the results from the twosamples (with an emphasis on operational characteristics); andpresents the outcomes of two experimental manipulations designedto improve the cell phone response rate.     

Mental health citizen surveys: a comparison of two within household telephone sampling techniques.

Two within household telephone sampling techniques for collecting citizen survey data for community mental health center's needs assessments were compared on the basis of respondent demographic characteristics; response, completion, and refusal rates; and, administrative costs. Initially, all households were chosen via random digit dialing. Then, one technique allowed the surveyor to interview any adult member of a household who answered the phone. The other technique used was developed by Troldahl and Carter and required the use of a reference table procedure to determine the appropriate respondent. While more expensive, the Troldahl-Carter method produced significantly better representativeness of the population for a rural county. Factors relating to a decision regarding which technique to employ are discussed.     

Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey

Declining contact and cooperation rates in random digit dial(RDD) national telephone surveys raise serious concerns aboutthe validity of estimates drawn from such research. While researchin the 1990s indicated that nonresponse bias was relativelysmall, response rates have continued to fall since then. Thecurrent study replicates a 1997 methodological experiment thatcompared results from a "Standard" 5-day survey employing thePew Research Center’s usual methodology with results froma "Rigorous" survey conducted over a much longer field periodand achieving a significantly higher response rate. As withthe 1997 study, there is little to suggest that unit nonresponsewithin the range of response rates obtained seriously threatensthe quality of survey estimates. In 77 out of 84 comparableitems, the two surveys yielded results that were statisticallyindistinguishable. While the "Rigorous" study respondents tendedto be somewhat less politically engaged, they did not reportconsistently different behaviors or attitudes on other kindsof questions. With respect to sample composition, the Standardsurvey was closely aligned with estimates from the U.S. Censusand other large government surveys on most variables. We extendour analysis of nonresponse to include comparisons with thehardest-to-reach respondents and with respondents who terminatedthe interview prior to completion.