A Comparison of Address-Based Sampling (ABS) Versus Random-Digit Dialing (RDD) for General Population Surveys

Valid and reliable public health data are becoming more difficultto obtain through random-digit dial (RDD) telephone surveys.As a result, researchers are evaluating different survey designs(i.e., sampling frame and survey mode combinations) as complementsor alternatives to RDD. Traditionally, mail surveys of the generalpublic have been limited due to a lack of a complete samplingframe of households. Recent advances in electronic record keeping,however, have allowed researchers to develop a sample from aframe of addresses (e.g., the U.S. Postal Service Delivery SequenceFile, which appears to provide coverage which rivals or possiblyexceeds that obtained through RDD sampling methods). To testthe use of this frame for surveying adults aged 18 years andolder across a wide geographic area, a pilot study was conductedas part of the 2005 Behavioral Risk Factor Surveillance System(BRFSS). The pilot compared use of a traditional, RDD telephonesurvey methodology to an approach using a mail version of thequestionnaire completed by a random sample of households drawnfrom an address-based frame. The findings indicate that themail survey approach can achieve higher response rates in low-response-ratestates (<40%) than RDD (particularly when two mailings aresent). Additionally, the address frame with mail survey designprovides access to cell phone only households and offers costsavings over the telephone approach. The resulting sample, however,significantly overrepresents non-Hispanic whites and peoplewith higher levels of education.     

SurveyMotion: what can we learn from sensor data about respondents’ completion and response behavior in mobile web surveys?

Participation in web surveys via smartphones increased continuously in recent years. The reasons for this increase are a growing proportion of smartphone owners and an increase in mobile Internet access. However, research has shown that smartphone respondents are frequently distracted and/or multitasking, which might affect completion and response behavior in a negative way. We propose ‘SurveyMotion (SMotion)’, a JavaScript-based tool for mobile devices that can gather information about respondents’ motions during web survey completion by using sensor data. Specifically, we collect data about the total acceleration (TA) of smartphones. We conducted a lab experiment and varied the form of survey completion (e.g. standing or walking). Furthermore, we employed questions with different response formats (e.g. radio buttons and sliders) and measured response times. The results reveal that SMotion detects higher TAs of smartphones for respondents with comparatively higher motion levels. In addition, respondents’ motion level affects response times and the quality of responses given. The SMotion tool promotes the exploration of how respondents complete mobile web surveys and could be employed to understand how future mobile web surveys are completed.     

Personalization of Mail Surveys for General Public and Populations with a Group Identity: Results from Nine Studies*

Abstract The effect of personalization on mail survey response rates was examined in nine studies that included 17 comparisons under several research conditions. A study of this variable across multiple experiments in five agricultural experiment stations was undertaken because of conflicting results from previous research and from concern that the effectiveness of personalization might have decreased over time. Results show that, while response to general public surveys appeared to increase modestly across all treatment groups, there was no positive effect for populations in which a group identity (e.g., Dear Oregon Gardner or Dear ATV Owner) is employed to address respondents in cover letters. Personalization appears to remain useful for improving response in surveys of the general public.     

Comparing Check-All and Forced-Choice Question Formats in Web Surveys

For survey researchers, it is common practice to use the check-allquestion format in Web and mail surveys but to convert to theforced-choice question format in telephone surveys. The assumptionunderlying this practice is that respondents will answer thetwo formats similarly. In this research note we report resultsfrom 16 experimental comparisons in two Web surveys and a papersurvey conducted in 2002 and 2003 that test whether the check-alland forced-choice formats produce similar results. In all 16comparisons, we find that the two question formats do not performsimilarly; respondents endorse more options and take longerto answer in the forced-choice format than in the check-allformat. These findings suggest that the forced-choice questionformat encourages deeper processing of response options and,as such, is preferable to the check-all format, which may encouragea weak satisficing response strategy. Additional analyses showthat neither acquiescence bias nor item nonresponse seem topose substantial problems for use of the forced-choice questionformat in Web surveys.     

Demystifying the use of mail survey in developing countries: a case of Nepal

ABSTRACT

The mail survey is a popular data collection tool in developed countries, but little is known about its use and efficacy in developing countries. Grounded on social exchange and diffusion of innovations theories, this study sought to ascertain whether the mail survey works in Nepal – one of the developing countries, by looking at how long respondents take to respond to mail surveys, and what the determinants are of the response time. A mail survey was conducted in June–September 2012 among 863 village animal health workers. Analysis of the data derived from 500 usable surveys shows that if properly planned and executed, the mail survey will perfectly work in developing countries as well. However, response period greatly varied by respondent’s home region, program sponsor, income groups, but not by gender, age, and education. The paper concludes by outlining suggestions to improve the mail survey.     

Comparison of web and mail surveys in collecting illicit drug use data: a randomized experiment

This randomized experiment examined survey mode effects for self-reporting illicit drug use by comparing prevalence estimates between a Web-based survey and a mail-based survey. A random sample of 7,000 traditional-aged undergraduate students attending a large public university in the United States was selected to participate in the spring of 2001. Students were randomly assigned to self-administer a survey via the Web or U.S. mail. The Web survey produced a significantly higher response rate than the U.S. mail survey. The prevalence estimates of illicit drug use (lifetime and past year) did not differ significantly between the two survey modes. The findings provide preliminary evidence that Web and U.S. mail surveys produce similar results regarding illicit drug use among undergraduate students. Although additional research is needed involving more diverse samples, these findings bode well for using Web surveys in college-based research.     

Can we predict device use? An investigation into mobile device use in surveys

In this study, we investigate whether mobile device use in surveys can be predicted. We aim to identify possible motives for device use and build a model by drawing on theory from technology acceptance research and survey research. We then test this model with a Structural Equation Modeling approach using data of seven waves of the GESIS panel. We test whether our theoretical model fits the data by focusing on measures of fit, and by studying the standardized effects of the model. Results reveal that intention to use a particular device can predict actual use quite well. Ease of smartphone use is the most meaningful variable: if people use a smartphone for specific tasks, their intention to use a smartphone for survey completion is also more likely. In conclusion, investing in ease of use of mobile survey completion could encourage respondents to use mobile devices. This can foremost be established by building well-designed surveys for mobile devices.     

The effect of personalization on response rates and data quality in web surveys

Personalizing correspondence has often shown to significantly increase survey response rates in mail surveys. This study experimentally tests whether personalization of email invitations acts correspondingly to web survey response rates. Also, it is investigated whether personalization influences data quality. The results of the study, using a large student sample, show that personalization significantly increases the web survey response rate by 8.6 percentage points. The data quality does not appear to be affected in any major way by personalizing the email invitations. However, the analyses do show that respondents of the personalization condition tend to respond with more social desirability bias to sensitive questions. Therefore, it is concluded that personalization has positive effects on the survey response rate, but one should carefully consider whether or not to personalize when a survey on sensitive topics is conducted.     

Portal surveys of time-out drinking locations: a tool for studying binge drinking and AOD use

Portal surveys, defined as assessments occurring proximal to the entry point to a high-risk locale and immediately on exit, can be used in different settings to measure characteristics and behavior of attendees at an event of interest. This methodology has been developed to assess alcohol and other drug (AOD) use at specific events and has included measuring intentions to use collected at entry and reported use on exit, as well as chemical tests for AOD consumption at both entrance and exit. Recent applications of the portal survey procedure to electronic music dance events that occur in established venues (e.g., bars or nightclubs) are discussed.     

The measurement of personal values in survey research: a test of alternative rating procedures

When survey researchers are interested in measuring the personal values of respondents, they often use a rating rather than a ranking method because it is easier and faster to administer and yields data that are amenable to parametric statistical analyses. However, because personal values are inherently positive constructs, respondents often exhibit little differentiation among the values and end-pile their ratings toward the positive end of the scale. Such lack of differentiation may potentially affect the statistical properties of the values and the ability to detect relationships with other variables. Two experiments were conducted via mail surveys to general population samples to test alternative rating methods designed to increase differentiation and reduce end-piling in the rating of personal values. The results suggest that a procedure in which respondents first pick their most and least important values, then rate them (most-least), provides more differentiation and less end-piling than a simple rating procedure (rate-only). Increased differentiation for the most-least method influenced the fit of latent structure and resulted in more robust relations between the values ratings and other criterion variables. These results generalized across type of values scale, number of values rated, and number of rating points.     

Assessment of Cigarette Smoking and Smoking Susceptibility among Youth: Telephone Computer-Assisted Self-Interviews versus Computer-Assisted Telephone Interviews

This experimental study assesses the effect of two survey methods,telephone audio computer-assisted self-interviewing (T-ACASI)and computer-assisted telephone interviewing (CATI), on self-reportsof smoking behavior and smoking susceptibility among adolescents12–17 years of age in California. In T-ACASI, participantslisten to prerecorded, computer-controlled questions and respondby pressing the keypad on a touch-tone telephone. In CATI, interviewersadminister the questions and enter responses into a computer.Prior research suggests that youth may be more likely to reportsensitive behaviors in a self-administered survey like T-ACASIcompared to an interviewer-administered survey like CATI, dueto greater perceived confidentiality. Logistic regression analyseswere conducted on unweighted data, controlling for demographicdifferences. Adjusted estimates of current smoking (past 30days) were significantly greater in T-ACASI (8.3 percent) thanCATI (4.5 percent). Smoking susceptibility (i.e., lack of afirm commitment not to smoke among those who have never smoked)was also greater in T-ACASI (45.0 percent) than CATI (34.9 percent).In both surveys, social desirability response bias was negativelyassociated with smoking, which suggests that response bias wasproblematic for both modes. Many respondents reported that aparent was present during the interview (59.4 percent in CATI;42.0 percent in T-ACASI). In both surveys, parental presencewas negatively associated with smoking, which suggests thatthis factor could also contribute to underreporting. Applicationof sample weights to the data eliminated the survey mode effects;however, the CATI current smoking estimate (9.3 percent) fromthis study was significantly less than an estimate (14.2 percent)obtained from a self-administered, school-based survey conductedthe same year on California adolescents.     

Möglichkeiten und Probleme des Einsatzes postalischer Befragungen

The research focuses on the strategies in mail surveys that lead to high response rates and on the social characteristics of the respondents that are recruited into the survey through reminders. Empirical basis are surveys in the East German cities of Chemnitz, Dresden and Leipzig with an original sample size of ca 2000 persons per city and up to three reminders. The response rate varies between 68 and 70 percent and surpasses the usual response rates in face to face and telephone surveys done in German cities. In contrast to common assumptions in the literature it has no effect on the response rate whether stamps are used on the address and return envelope, neither do original versus scanned signatures make a difference and whether financial incentives are used in the late reminders. Through the reminders especially respondents with low education and of little political interest are recruited into the survey.     

RESPONSE RATES AND RESPONSE CONTENT IN MAIL VERSUS FACE-TO-FACE SURVEYS

Two surveys were administered based on the same area probabilitysampling frame and with some of the same questions: one samplewas used for hour-long face-to-face interviewing in the 1992Detroit Area Study; the other sample received a much shorterquestionnaire in the mail for sell-administration. The samplesegments had previously been stratified in terms of the percentagethat was black. For the predominantly white stratum, there wasonly a small difference in response rates due to mode of administration.For the predominantly black stratum, the mail survey obtaineda considerably lower response rate then the face-to-face survey.Within the predominantly white stratum, there were no cleardifferences between results for the two modes of administrationin demographic variables or in gross housing characteristics.However, the mail survey respondents expressed more negativeattitudes toward racial integration and affirmative action thandid the face-to-face respondents. Because the mail sample ofthe predominantly black stratum was small, it was not possibleto carry out similar analyses of demographic or attitudinaldifferences, or to determine whether its lower response ratewas due mainly to race, to correlates of race such as incomeor education, or even to problems with mail delivery in centralcities.     

Gambling and Alcohol Use: Trends in the State of New Mexico from 1996–1998

Alcohol consumption and its relationship to gambling was examined in a statewide New Mexico survey in 1996 and 1998. Data regarding both drinking habits and gambling behavior were obtained from a stratified random sample of the adult population (N = 2674) across the entire State of New Mexico via phone survey. These surveys were carried out shortly after a period when New Mexico experienced an initial surge in the legalized gaming industry. Fifty-seven percent of survey respondents reported drinking in the past month, while 43.1% reported no drinking in the past 30 days. Results show that while the number of days in the past 30 that a person drinks is significantly correlated with some types of gambling behavior (e.g., in 1998, drinking more days was associated with more card gambling and sports gambling), it is the amount of alcohol consumed per occasion that is associated with more gambling behavior. For example, in both 1996 and 1998, drinking more per occasion was associated with more sports betting, dice gambling, number/lottery gambling, gambling using machines, and paper game gambling (e.g., pull tabs, punchboard). These results suggest that problem gambling behavior is not affected as much by the number of occasions on which one consumes alcohol, as by the amount of alcohol consumed per occasion.     

Worksite e-mail health promotion trial: Early lessons

In preparation for an electronic mail (e-mail) and web-based health promotion intervention across multiple worksites, secure, regulatory-compliant, user-friendly e-mail and Internet applications were used to recruit potential participants across worksites, to enroll participants, and to collect baseline health assessment data. Specific hardware and software information technology environments were required of the 19 participating worksites. Sequential e-mails introduced the study and invited participation. Twenty-four percent of all employees (1106 of approximately 4600) provided consent. E-mail delivered a web link for the baseline study assessment, and reminder e-mails were sent to prompt completion. Of those who consented, 888 (80%) completed baseline health and behavior data surveys. An HTML-native web survey software was more stable across computing environments. Using e-mail and web assessment, this research recruited, enrolled, and collected data from more than 850 participants. Technical and operational challenges emerged at each step. Solutions and recommendations are discussed. Overall, this experience suggests that the use of e-mail and web software can facilitate recruitment, enrollment, and data acquisition through direct contact with study participants. This experience yields a series of lessons learned for using e-mail and the Internet to support multi-site trials.     

RRT meets RDD: statistical strategies for assuring response privacy in telephone surveys

Despite its growing role in survey research, the telephone survey has been largely neglected with respect to the design and testing of statistical procedures for assuring response privacy (e.g., randomized response techniques). This paper demonstrates that such procedures are no less feasible to administer in telephone surveys than in face-to-face surveys. Both randomized response techniques and alternatives to randomized response are considered.     

Effects of Personal Salutations in E-mail Invitations to Participate in a Web Survey

Effects of personalizing survey invitations on response rateshave been extensively researched in the realm of mail surveys.Commonly, it is found that response rates increase when personalizationis applied. Recently, efforts have been made to investigatewhether these findings extend to the field of Web surveys thatuse e-mail invitations. Using data from a Web experiment conductedamong first-year university students, personalization of e-mailinvitations is shown to have significantly increased the responserate by 7.8 percentage points. From the theory that personalizationdecreases the level of anonymity and perceived privacy, differencesin responses to sensitive questions were expected. However,no effects of personalization on the responses to a questionprobing for respondents’ sexual behavior were found.     

Interpretation and interview context: examining the General Social Survey name generator using cognitive methods

This paper reports on a small (N=50) study of how survey respondents interpret the General Social Survey (GSS)'s “discuss important matters” name generator. The study involved concurrent think-aloud interviews, in which respondents were debriefed about their thought processes immediately after answering the name generator. Analyses of these responses indicate that some respondents had difficulty in specifying what was meant by the term “important matters”; sizable minorities understood the question in terms of frequency of contact or intimacy rather than in terms of specific social exchanges. Most of those interviewed said that their “important matters” had to do with personal/intimate relationships or other issues of personal life (e.g., finances, hobbies, or health), but appreciable numbers referred to work and political discussions. An interview context experiment revealed that a respondent's definition of “important matters” can be shaped by the substantive content of the preceding parts of an interview schedule. Notwithstanding these findings, the composition of the networks elicited in the study does not appear to vary substantially across interpretations of the name generator. We conclude that the name generator succeeds in measuring “core” discussion networks, though with somewhat nonspecific content. Implications for the measurement of personal networks in sample surveys are discussed.     

Does Telephone Audio Computer-Assisted Self-Interviewing Improve the Accuracy of Prevalence Estimates of Youth Smoking? Evidence from the UMass Tobacco Study

Despite their advantage for obtaining representative samplesof adolescents, telephone surveys have been regarded as an inferiormethod for collecting data on youth tobacco use because theyyield lower estimates than school-based, self-administered surveys.Although no gold standard for smoking estimates exists, thelower estimates in telephone surveys have been attributed tounderreporting due to youths’ concern that parents orothers may overhear their responses. Telephone audio computer-assistedself-interviewing (T-ACASI) is a cost-effective method for obtaininga representative sample of youths and provides increased privacyfor the respondent. We hypothesize that using T-ACASI wouldencourage youths to report more fully smoking behavior comparedto traditional interviewer-administered telephone methods. Ouranalysis further assesses whether respondent age, gender, race/ethnicity,and parental attitude toward smoking moderated the relationshipbetween survey mode and smoking reports. Using data from a statewidetobacco use survey that randomly assigned youth respondentsto either T-ACASI or interviewer-administered modes, we findthat youths were more likely to report smoking behaviors inT-ACASI mode and that this was especially true for girls, particularlythose who believed their parents would disapprove strongly oftheir smoking. The findings suggest that traditional telephonesurveys may underestimate smoking prevalence in most girls bya factor of two, and that a technique for ensuring privacy forthese respondents is an important component of effective telephonesurvey methodology.     

Does "Yes or No" on the Telephone Mean the Same as "Check-All-That-Apply" on the Web?

Recent experimental research has shown that respondents to forced-choicequestions endorse significantly more options than respondentsto check-all questions. This research has challenged the commonassumption that these two question formats can be used interchangeablybut has been limited to comparisons within a single survey mode.In this paper we use data from a 2004 random sample survey ofuniversity students to compare the forced-choice and check-allquestion formats across web self-administered and telephoneinterviewer-administered surveys as they are commonly used insurvey practice. We find that the within-mode question formateffects revealed by previous research and reaffirmed in thecurrent study appear to persist across modes as well; the telephoneforced-choice format produces higher endorsement than the webcheck-all format. These results provide further support forthe argument that the check-all and forced-choice question formatsdo not produce comparable results and are not interchangeableformats. Additional comparisons show that the forced-choiceformat performs similarly across telephone and web modes.