Consequences of reducing nonresponse in a national telephone survey

Critics of public opinion polls often claim that methodological shortcuts taken to collect timely data produce biased results. This study compares two random digit dial national telephone surveys that used identical questionnaires but very different levels of effort: a "Standard" survey conducted over a 5-day period that used a sample of adults who were home when the interviewer called, and a "Rigorous" survey conducted over an 8-week period that used random selection from among all adult household members. Response rates, computed according to AAPOR guidelines, were 60.6 percent for the Rigorous and 36.0 percent for the Standard study. Nonetheless, the two surveys produced similar results. Across 91 comparisons, no difference exceeded 9 percentage points, and the average difference was about 2 percentage points. Most of the statistically significant differences were among demographic items. Very few significant differences were found on attention to media and engagement in politics, social trust and connectedness, and most social and political attitudes, including even those toward surveys.     

The impact of touch-tone data entry on reports of HIV and STD risk behaviors in telephone interviews

Respondents concerns about privacy can decrease reporting of HIV and STD risk behaviors in general population telephone surveys. The purpose of this paper is to describe the results of an experimental study evaluating whether one method for increasing privacy, touch-tone data entry (TTDE), is effective in increasing estimates of sexual behaviors from a population-based survey. We conducted a random-digit-dial telephone survey of adults in New Jersey (n = 405), with half the respondents using TTDE for answering sexual behavior questions. TTDE led to increased reports of same-sex sexual behavior, certain HIV and STD risk factors, and concern about one s risk for HIV and STD transmission. TTDE also narrowed the difference between men s and women s reports of the number of different sexual partners over the past 10 years. The feasibility and limitations of TTDE are discussed, as well as possible alternative interpretations that consider the impact of TTDE on the dynamics of the interaction between the respondent and the interviewer.     

Experiments with incentives in telephone surveys

In an effort to counter the problem of noncooperation, survey organizations are offering incentives to respondents with increasing frequency, some at the outset of the survey, as has traditionally been done in mail surveys, and some only after the person has refused, in an attempt to convert the refusal. This article reports on a series of experiments carried out over a period of about 2 years with a monthly telephone survey, the Survey of Consumer Attitudes, in an effort to increase response rates or reduce interviewer effort. We report on experiments with prepaid versus promised incentives; advance letters; and advance letters with prepaid incentives; and we also report on the effects of incentives on response quality, sample composition, response bias, interviewer and respondent expectations, and costs.     

The impact of touch‐tone data entry on reports of HIV and STD risk behaviors in telephone interviews

Respondents’ concerns about privacy can decrease reporting of HIV and STD risk behaviors in general population telephone surveys. The purpose of this paper is to describe the results of an experimental study evaluating whether one method for increasing privacy, touch‐tone data entry (TTDE), is effective in increasing estimates of sexual behaviors from a population‐based survey. We conducted a random‐digit‐dial telephone survey of adults in New Jersey (n = 405), with half the respondents using TTDE for answering sexual behavior questions. TTDE led to increased reports of same‐sex sexual behavior, certain HIV and STD risk factors, and concern about one's risk for HIV and STD transmission. TTDE also narrowed the difference between men's and women's reports of the number of different sexual partners over the past 10 years. The feasibility and limitations of TTDE are discussed, along with possible alternative interpretations that consider the impact of TTDE on the dynamics of the interaction between the respondent and the interviewer.     

Race-of-Interviewer Effects in Telephone Interviews

Previous studies have found a race-of-interviewer effect onsurvey questions dealing with racial issues. This effect hasbeen found in both personal interviews and on questionnairesfilled out in the presence of an interviewer. This study examineswhether a race-of-interviewer effect is also present in telephoneinterviews. The results show that a race-of-interviewer effectdoes occur in telephone interviews on racial questions.     

Editorial: Speaking in Tongues—Diversity in Mixed Methods Research

Pre‐testing is a standard tool to increase appropriateness and to capture the audience, yet rarely is this technique employed for population surveys. This study aimed to examine parental views on the content and methodology of a population child health survey. Forty‐eight Australian families of children aged 0–12 years were interviewed about the content and proposed data collection methods of the draft Victorian Population Survey of Child Health and Wellbeing. Concerns with instructions and items of several commonly used child health measures were identified. Parents preferred face‐to‐face methods rather than telephone interviews due to survey legitimacy; however, 72% of parents indicated that they would participate in a telephone survey. This study provided new findings about respondent views on child health measures and the conditions under which CATI versus face‐to‐face methods for child health data collection methods would be acceptable.     

Beyond interviewer effects in the standardized measurement of ego-centric networks

The effects which interviewers exert on the collection of ego-centric networks have recently come into the focus of methodological considerations. Studies consistently show that the size of networks varies depending on the interviewer. We would like to expand on this research strand by pointing to different aspects which have so far gone unremarked in the discussion. First, size is mainly analysed as a network measure which is influenced during data collection, while other common measures such as network density or composition have not received sufficient consideration. Second, large-scale surveys using face-to-face interviews usually allocate interviewers to a single sampling point. Differences between sampling points (locality effects) are attributed to interviewer effects. Hence, we disentangle the effects of the locality and interviewer. Third, the discussion on interviewer effects often follows an “actor-oriented” consideration of how data collection situations are structured by interviewers. Expanding this approach from a relational perspective, we consider the relationship between the interviewers and respondents and whether this relationship influences the collection of network data. To test our hypotheses about the influence of interviewers, the locality and the interviewer-respondent relationship on different network measures, we use data from the 2010 German General Social Survey (n = 2827 respondents, n = 220 interviewers). The multilevel analyses show that the relationship between the interviewer and the respondent is not very relevant. Furthermore, the analyses show that interviewers have an influence on the network size but not on measures of their composition. However, evidence on the prevalence of locality or interviewer effects is mixed. Finally, homophilous interviewer-respondent relationships have very little effect on network characteristics. We find evidence of training and fatigue effects on network size. However, much of the variation in network size caused by the interviewer still remains unexplained. We draw conclusions on how to organize interview situations in surveys.     

The influence of interviewer characteristics on support for democracy and political engagement in Sub-Saharan Africa*

This study investigates how an interviewer’s characteristics affect how respondents answer survey questions about democracy and political engagement. I analyze data from the 2008 Afrobarometer surveys, in which 810 interviewers surveyed 27,713 respondents across 20 countries in sub-Saharan Africa. Using these data, I study how interviewer education, age, and gender affect two outcomes: (1) response distributions to attitudinal and behavioral survey questions and (2) the likelihood of respondents saying ‘don’t know’ to a survey question. The analysis also investigates how the respondent’s perception of who sponsored the survey (NGO, private sector, government) affects attitudes. The results show that these interviewer characteristics affect the quality of survey data on political attitudes and behaviors. In the discussion, I consider the implications for research based on public opinion data about democracy and political engagement.     

METAPHORS FOR PUBLIC OPINION IN LITERATURE

Previous research found substantial effects of the race of theinterviewer on measures of civic attitudes and electoral participationof blacks in NES surveys from 1964, 1976, 1978, 1980, and 1984.This study extends the previous analysis in two ways: it usesdata from two additional NES surveys, 1982 and 1986; and itfocuses on the effects of the race of the interviewer on race-relatedattitudes. Blacks interviewed by whites were much more likelyto express warmth and closeness toward whites than were blacksinterviewed by blacks. But whereas there is no race-of-interviewereffect on blacks' expressions of warmth toward blacks, thereis a clear race-of-interviewer effect on blacks' expressionsof closeness toward blacks. The pattern of responses to thecloseness items appears to result from the format of the question.The observed trend of decreasing expressed closeness of blackstoward blacks in NES surveys between 1976 and 1984 is an artifactof changes in the racial composition of the interviewer staff.     

THE EFFECTS OF THE RACE OF THE INTERVIEWER ON RACE-RELATED ATTITUDES OF BLACK RESPONDENTS IN SRC/CPS NATIONAL ELECTION STUDIES

Previous research found substantial effects of the race of theinterviewer on measures of civic attitudes and electoral participationof blacks in NES surveys from 1964, 1976, 1978, 1980, and 1984.This study extends the previous analysis in two ways: it usesdata from two additional NES surveys, 1982 and 1986; and itfocuses on the effects of the race of the interviewer on race-relatedattitudes. Blacks interviewed by whites were much more likelyto express warmth and closeness toward whites than were blacksinterviewed by blacks. But whereas there is no race-of-interviewereffect on blacks' expressions of warmth toward blacks, thereis a clear race-of-interviewer effect on blacks' expressionsof closeness toward blacks. The pattern of responses to thecloseness items appears to result from the format of the question.The observed trend of decreasing expressed closeness of blackstoward blacks in NES surveys between 1976 and 1984 is an artifactof changes in the racial composition of the interviewer staff.     

Does Telephone Audio Computer-Assisted Self-Interviewing Improve the Accuracy of Prevalence Estimates of Youth Smoking? Evidence from the UMass Tobacco Study

Despite their advantage for obtaining representative samplesof adolescents, telephone surveys have been regarded as an inferiormethod for collecting data on youth tobacco use because theyyield lower estimates than school-based, self-administered surveys.Although no gold standard for smoking estimates exists, thelower estimates in telephone surveys have been attributed tounderreporting due to youths’ concern that parents orothers may overhear their responses. Telephone audio computer-assistedself-interviewing (T-ACASI) is a cost-effective method for obtaininga representative sample of youths and provides increased privacyfor the respondent. We hypothesize that using T-ACASI wouldencourage youths to report more fully smoking behavior comparedto traditional interviewer-administered telephone methods. Ouranalysis further assesses whether respondent age, gender, race/ethnicity,and parental attitude toward smoking moderated the relationshipbetween survey mode and smoking reports. Using data from a statewidetobacco use survey that randomly assigned youth respondentsto either T-ACASI or interviewer-administered modes, we findthat youths were more likely to report smoking behaviors inT-ACASI mode and that this was especially true for girls, particularlythose who believed their parents would disapprove strongly oftheir smoking. The findings suggest that traditional telephonesurveys may underestimate smoking prevalence in most girls bya factor of two, and that a technique for ensuring privacy forthese respondents is an important component of effective telephonesurvey methodology.     

Untersuchungen zur Ursache unterschiedlicher Ergebnisse sehr ähnlicher Viktimisierungssurveys

Two very similar surveys on victimization were conducted in 1997. Although the target population, design, operationalization, field organization and part of the interviewer staff were nearly identical, the resulting estimates of victimization status were remarkably different. Neither correction for design effects nor the use of appropriate weighting eliminated this statistically significant difference. Several methodological details of the surveys are considered as explanations. Quantitative estimates of their possible effects are given. The interviewer working conditions are identified as the most likely source of the discrepancy. This result emphasizes the importance of these seldom documented and even more rarely controlled fieldwork details.     

Reducing Refusal Rates for Telephone Interviews

In two experiments, refusal rates to telephone interviews werenot affected by substantial changes in the introductory remarksof the interviewer. A prior letter significantly lowered refusalrates in a third experiment. In all three, interviewer sex hadno effect.     

Presentation matters: how mode effects in item non-response depend on the presentation of response options

Comparability of data across modes is an important issue in survey research. In this paper we discuss item non-response to attitudinal questions in telephone and web surveys. We present results from a survey experiment conducted in Italy and in Spain that compares different presentations of response options in an online setting with a benchmark telephone survey. In line with earlier studies we find that (A), the share of non substantial answers in the online survey depends on how the response option is presented. Comparing different presentations in an online survey to the standard approach of telephone surveys using propensity score matching, we find that (B), the share and pattern of non substantial answers, is most similar across the two modes in the online survey when the it is captured in an instruction on each screen for the online survey. Our findings are of particular relevance for the design of multiple mode or mix-mode surveys of attitudinal questions using online and telephone modes.     

INTERVIEWING OLDER ADULTS: MODE COMPARISON USING DATA FROM A FACE-TO-FACE SURVEY AND A TELEPHONE RESURVEY

A face-to-face survey conducted in 1984 with a sample of 1491residents of the Detroit metropolitan area (including an oversampleof older adults) and a reinterview of a random subset of theserespondents by telephone were used to compare the two modesof data collection across two age levels. Except for a tendencytoward a disproportionately large number of DK answers and adisproportionately large amount of interviewer assistance onthe telephone, respondents 60 years of age and older did notexhibit larger mode differences than did respondents under 60.For both age groups, response distributions were rather similar,suggesting little effect of mode. Likewise, response style differedlittle by mode, while a higher proportion of missing data (i.e.,"I don't know" answers) was given on the telephone. The responserate for the telephone reinterview was 90%, somewhat lower forolder than younger persons. The findings support the feasibilityof using the telephone for reinterviewing older adults.     

Effects of an introductory letter on response rates to a teen/parent telephone health survey

The authors conducted a pilot study in preparation for a larger investigation that will rely on telephone surveys to assess select health behaviors of teens and their parents, with a focus on indoor tanning. This study used a randomized design to assess the impact of a presurvey letter on response rates to a telephone survey, as well as prevalence estimates of ever having used an indoor tanning facility. The letter did not have a significant effect on response rates or prevalence estimates in the two cities under study. Findings suggest that researchers should not assume that a letter sent in advance of a telephone survey will necessarily increase response rates enough to justify their use.     

Response Order Effects in Dichotomous Categorical Questions Presented Orally: The Impact of Question and Respondent Attributes

Using data from 548 experiments in telephone surveys conductedby the Gallup Organization, we explored how attributes of questionsand respondents moderate response order effects in dichotomouscategorical questions. These effects were predominantly recencyeffects and occurred most in questions that were more difficultto comprehend (especially among respondents with the least education),with response choices that were more difficult to comprehend(because they were complete sentences instead of words or phrasesand because they were not mutually exclusive), and that wereasked after many prior questions. Recency effects were alsomore common in questions that explicitly or implicitly encouragedrespondents to wait until they had heard all the answer choicesbefore formulating a judgment than in questions that inducedrespondents to begin formulating a judgment before all the answerchoices had been read (especially among the least educated respondents).A study of interviewer behavior revealed patterns of pausingbetween and within sentences that help to explain why some typesof questions are especially prone to recency effects and othersare not.     

SAME-GENDER SEX IN THE UNITED STATES IMPACT OF T-ACASI ON PREVALENCE ESTIMATES

Well-conducted telephone surveys provide an economical meansof estimating the prevalence of sexual and reproductive behaviorsin a population. There is, however, a nontrivial potential forbias since respondents must report sensitive information toa human interviewer. The National STD and Behavior MeasurementExperiment (NSBME) evaluates a new survey technology—telephoneaudio computer-assisted self-interviewing (T-ACASI)—thateliminates this requirement. The NSBME embedded a randomizedexperiment in a survey of probability samples of 1,543 U.S.and 744 Baltimore adults ages 18 to 45. Compared with NSBMErespondents interviewed by human interviewers, respondents interviewedby T-ACASI were 1.5 to 1.6 times more likely to report same-gendersexual attraction, experience, and genital contact. The impactof T-ACASI was more pronounced (odds ratio = 2.5) for residentsof locales that have historically been less tolerant of same-gendersexual behaviors and for respondents in households with children(odds ratio = 3.0).     

Epidemiological surveys of pathological gambling: Critique and suggestions for modification

Epidemiological studies of problem and pathological gambling were examined for their accuracy. Fundamental flaws and biases were found in these surveys. These include problems with survey instruments; nonresponses and refusal bias; the exclusion of institutionalized populations; exclusion of other groups; and failure to protect against denial on the part of the respondent when others are present near the telephone. Based on the issues discussed, one can reasonably be expected to assume that most epidemiological surveys seriously underestimate the extent of problem and pathological gambling. Alternative strategies for addressing these issues are discussed. These strategies include the use of field interviews, surveys of institutionalized populations, frequent player surveys and significant other surveys. The value and potential problems of these approaches are also discussed.This is a revision of a paper presented at the 9th International Conference on Risk and Gambling, Las Vegas, Nevada.The author would like to thank Rachel Volberg and Mark Dickerson as well as conference participants for their valuable suggestions on an earlier draft of this paper.     

Telephone survey respondents' reactions to questions regarding interpersonal violence

Concerns have been raised regarding the appropriateness of asking about violence victimization in telephone interviews and whether asking such questions increases respondents' distress or risk for harm. However, no large-scale studies have evaluated the impact of asking such questions during a telephone interview. This study explored respondents' reactions to questions regarding violence in two large recently completed telephone surveys. After respondents were asked about violence, they were asked if they thought surveys should ask such questions and whether they felt upset or afraid because of the questions. In both surveys, the majority of respondents (regardless of their victimization history) were willing to answer questions about violence and were not upset or afraid because of the questions. More than 92% of respondents thought such questions should be asked. These results challenge commonly held beliefs and assumptions and provide some assurance to those concerned with the ethical collection of data on violent victimization.