Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey

Declining contact and cooperation rates in random digit dial(RDD) national telephone surveys raise serious concerns aboutthe validity of estimates drawn from such research. While researchin the 1990s indicated that nonresponse bias was relativelysmall, response rates have continued to fall since then. Thecurrent study replicates a 1997 methodological experiment thatcompared results from a "Standard" 5-day survey employing thePew Research Center’s usual methodology with results froma "Rigorous" survey conducted over a much longer field periodand achieving a significantly higher response rate. As withthe 1997 study, there is little to suggest that unit nonresponsewithin the range of response rates obtained seriously threatensthe quality of survey estimates. In 77 out of 84 comparableitems, the two surveys yielded results that were statisticallyindistinguishable. While the "Rigorous" study respondents tendedto be somewhat less politically engaged, they did not reportconsistently different behaviors or attitudes on other kindsof questions. With respect to sample composition, the Standardsurvey was closely aligned with estimates from the U.S. Censusand other large government surveys on most variables. We extendour analysis of nonresponse to include comparisons with thehardest-to-reach respondents and with respondents who terminatedthe interview prior to completion.     

CARRYOVER EFFECTS IN ATTITUDE SURVEYS

Answers to attitude questions in surveys can vary markedly dependingon the preceding items in the questionnaire. This study concernssuch context effects. More than 1,100 respondents were askedabout six target issues in a telephone survey. Before answeringthe target questions, most of the respondents had been askedabout one of two sets of related issues; the remainder receivedunrelated, "neutral" context items. For five of the six targetissues, the two groups receiving different sets of context itemsdiffered significantly in their answers to the target questions.However, only one of the substantive context groups differedsignificantly from the group that received the neutral contextitems. Additional results indicated that context effects werelarger when respondents reported that their beliefs about thetarget issue were both mixed and important. Coding of openendedresponses to follow-up questions suggested that the contextitems may have changed how respondents thought about the targetissues.     

Assessment of Cigarette Smoking and Smoking Susceptibility among Youth: Telephone Computer-Assisted Self-Interviews versus Computer-Assisted Telephone Interviews

This experimental study assesses the effect of two survey methods,telephone audio computer-assisted self-interviewing (T-ACASI)and computer-assisted telephone interviewing (CATI), on self-reportsof smoking behavior and smoking susceptibility among adolescents12–17 years of age in California. In T-ACASI, participantslisten to prerecorded, computer-controlled questions and respondby pressing the keypad on a touch-tone telephone. In CATI, interviewersadminister the questions and enter responses into a computer.Prior research suggests that youth may be more likely to reportsensitive behaviors in a self-administered survey like T-ACASIcompared to an interviewer-administered survey like CATI, dueto greater perceived confidentiality. Logistic regression analyseswere conducted on unweighted data, controlling for demographicdifferences. Adjusted estimates of current smoking (past 30days) were significantly greater in T-ACASI (8.3 percent) thanCATI (4.5 percent). Smoking susceptibility (i.e., lack of afirm commitment not to smoke among those who have never smoked)was also greater in T-ACASI (45.0 percent) than CATI (34.9 percent).In both surveys, social desirability response bias was negativelyassociated with smoking, which suggests that response bias wasproblematic for both modes. Many respondents reported that aparent was present during the interview (59.4 percent in CATI;42.0 percent in T-ACASI). In both surveys, parental presencewas negatively associated with smoking, which suggests thatthis factor could also contribute to underreporting. Applicationof sample weights to the data eliminated the survey mode effects;however, the CATI current smoking estimate (9.3 percent) fromthis study was significantly less than an estimate (14.2 percent)obtained from a self-administered, school-based survey conductedthe same year on California adolescents.     

The influence of interviewer characteristics on support for democracy and political engagement in Sub-Saharan Africa*

This study investigates how an interviewer’s characteristics affect how respondents answer survey questions about democracy and political engagement. I analyze data from the 2008 Afrobarometer surveys, in which 810 interviewers surveyed 27,713 respondents across 20 countries in sub-Saharan Africa. Using these data, I study how interviewer education, age, and gender affect two outcomes: (1) response distributions to attitudinal and behavioral survey questions and (2) the likelihood of respondents saying ‘don’t know’ to a survey question. The analysis also investigates how the respondent’s perception of who sponsored the survey (NGO, private sector, government) affects attitudes. The results show that these interviewer characteristics affect the quality of survey data on political attitudes and behaviors. In the discussion, I consider the implications for research based on public opinion data about democracy and political engagement.     

Toward a social psychology of survey methodology: An application of the approach and directions for the future

Survey methodology is a relatively new academic discipline focused on understanding sources of survey errors. As an interdisciplinary field, survey methodology borrows theoretical approaches from other disciplines and applies them to understand how survey respondents answer questions. One field in particular, cognitive psychology, has played a central role in the development of survey methodology. The cognitive approach has focused researchers' attentions on the sources of error at each stage of the cognitive process respondents use to answer a survey question: comprehension of the question, recollection of relevant information, estimation and judgment, and reporting an answer. Although this focus on the cognitive response process has been positive and fruitful, potentially strong social and interactional influences on the response process have been underinvestigated and undertheorized. Thus, this essay argues for a revitalized research program in the sociological social psychology of survey methodology, given its rich body of theory and research. The current strengths of social psychological and interactional approaches are highlighted, focusing primarily on recent work using identity theory to understand social desirability biases. Finally, potentially fruitful future directions for research are proposed, matching sociological social psychological theories to the survey errors upon which they may shed light.     

SurveyMotion: what can we learn from sensor data about respondents’ completion and response behavior in mobile web surveys?

Participation in web surveys via smartphones increased continuously in recent years. The reasons for this increase are a growing proportion of smartphone owners and an increase in mobile Internet access. However, research has shown that smartphone respondents are frequently distracted and/or multitasking, which might affect completion and response behavior in a negative way. We propose ‘SurveyMotion (SMotion)’, a JavaScript-based tool for mobile devices that can gather information about respondents’ motions during web survey completion by using sensor data. Specifically, we collect data about the total acceleration (TA) of smartphones. We conducted a lab experiment and varied the form of survey completion (e.g. standing or walking). Furthermore, we employed questions with different response formats (e.g. radio buttons and sliders) and measured response times. The results reveal that SMotion detects higher TAs of smartphones for respondents with comparatively higher motion levels. In addition, respondents’ motion level affects response times and the quality of responses given. The SMotion tool promotes the exploration of how respondents complete mobile web surveys and could be employed to understand how future mobile web surveys are completed.     

A Comparison of Survey Techniques on Sensitive Sexual Behavior in Italy

This article compares two national surveys carried out through the most commonly used procedures in Italy: CATI (computer-assisted telephone interviews) and SAQ-FI (self-answered questionnaires following interviews). Both surveys ask two identical questions concerning sensitive sexual behavior: early age at first intercourse and same-sex attraction. The SAQ-FI survey had both unit non-response and item non-response rates much lower than the CATI survey. Moreover, in the CATI survey, the groups with highest item non-response rates were also the groups with the lowest proportions of early intercourse and homosexual attraction. In addition, a differential analysis of the respondents produced diverse results for the two surveys. This is especially true of results by gender for same-sex attraction: Such behavior is more common among men (3.1%) than women (2.9%), according to the CATI survey, whereas the opposite is true of the SAQ-FI survey (6.1% of men vs. 7.7% women). In Italy at the beginning of the 21st century, CATI surveys reveal a lower level of early intercourse and same-sex attraction than SAQ-FI surveys. This article argues that the CATI survey underestimates the true level of these sensitive sexual behaviors in the Italian population.     

Question Understanding Aid (QUAID): A Web Facility that Tests Question Comprehensibility

When respondents do not understand the meaning of a survey question,they will not supply valid and reliable answers. Survey methodologistsshould therefore benefit from computer tools and other analyticalschemes that help them identify problems with questions withrespect to comprehension difficulty. We developed a Web facilitycalled Question Understanding Aid (QUAID; www.psyc.memphis.edu/quaid.html)that assists survey methodologists in identifying problems withthe wording, syntax, and semantics of questions on questionnaires.The survey methodologist enters the question into the Web facility,along with any context information and answer alternatives thataccompany the question. QUAID quickly returns a list of potentialproblems with question comprehension, including unfamiliar technicalterms, vague or imprecise relative terms, vague or ambiguousnoun phrases, complex syntax, and working memory overload. Thisarticle describes QUAID and some empirical studies that haveassessed the validity and utility of QUAID’s critiquesof questions. The output of QUAID was compared with the judgmentsof experts in language, discourse, and cognition during thedevelopment of the tool. In one evaluation, expert survey methodologistscritiqued and revised problematic questions, whereas in a secondevaluation survey methodologists evaluated the quality of originalproblematic questions, questions revised with the assistanceof QUAID, and questions revised without QUAID. In a third evaluation,eye-tracking data were collected while respondents read questionson a computer screen and answered questions aloud. Respondentshad a tendency to give up processing difficult questions tooearly (called an early exit), which potentially threatens thevalidity of the respondents’ answers. Survey methodologistsare encouraged to use QUAID and further evaluate its validityand utility.     

Do Advance Letters Improve Preelection Forecast Accuracy?

The survey methodology literature has debated whether advanceletters to potential survey respondents will reduce nonresponsebias and thereby improve the accuracy of preelection forecasts.This research note analyzes the results of experiments conductedin Maryland, New York, and Pennsylvania in which advance letterswere sent to a random sample of potential survey respondentsto 2002 preelection surveys. We find a significant increasein the overall response rate, although notably less than inpast studies. However, the advance letters did not improve therepresentativeness of survey respondents or the accuracy ofthe election forecasts.     

Toward an Open-Source Methodology: What We Can Learn from the Blogosphere

During the 2004 election campaign, millions of political enthusiastsdownloaded poll data on the Internet, while "Weblogs" provideda new forum for commentary on survey methodology. At the sametime, traditional public opinion surveys came under pressurefrom declining cooperation, contact, and coverage rates, andmany automated and Internet surveys began to proliferate. Thisarticle provides some examples of "blog" commentary on automatedand Internet polls and then explores the lessons to be learnedfrom the spirit of innovation and openness of the Internet inevaluating new survey methods such as automated polls and thoseconducted over the Internet.     

The development of a CAHPS instrument for Nursing Home Residents (NHCAHPS)

We report on a federal initiative to develop a CAHPS (The Consumer Assessment of Healthcare Providers and Systems) survey to measure residents' experiences with quality-of-care and quality-of-life in nursing homes (known as NHCAHPS). We focus on how we created and tested questions for inclusion in the instrument and tested a possible cognitive screener to determine which residents could participate in a NHCAHPS interview. The major lessons learned were: (1) In contrast to other CAHPS surveys, ratings were more useful than reports because of the difficulty that residents had with summarizing over time and people; (2) consistent with other CAHPS surveys, the 0 to 10 response scale appeared to work well with nursing home residents for many of the quality-of-care questions; however, a different response scale was needed for many of the quality-of-life items; and (3) in contrast with typical survey methodology and other CAHPS surveys where explicit time reference periods are used, a non-specific present reference period in questions seemed to work best.     

The Impact of "No Opinion" Response Options on Data Quality: Non-Attitude Reduction or an Invitation to Satisfice?

According to many seasoned survey researchers, offering a no-opinionoption should reduce the pressure to give substantive responsesfelt by respondents who have no true opinions. By contrast,the survey satisficing perspective suggests that no-opinionoptions may discourage some respondents from doing the cognitivework necessary to report the true opinions they do have. Weaddress these arguments using data from nine experiments carriedout in three household surveys. Attraction to no-opinion optionswas found to be greatest among respondents lowest in cognitiveskills (as measured by educational attainment), among respondentsanswering secretly instead of orally, for questions asked laterin a survey, and among respondents who devoted little effortto the reporting process. The quality of attitude reports obtained(as measured by over-time consistency and responsiveness toa question manipulation) was not compromised by the omissionof no-opinion options. These results suggest that inclusionof no-opinion options in attitude measures may not enhance dataquality and instead may preclude measurement of some meaningfulopinions.     

EDUCATED GUESSES: THE PROCESS OF ANSWERING FACTUAL KNOWLEDGE QUESTIONS IN SURVEYS

Responses to autobiographical questions are known to representmore than simply retrieval of information from memory; inference,cuing, and "availability" all play a role. Using responses toitems in four different surveys, we find that respondent motivationand ability, together with contextual cues, help determine howsurvey respondents answer knowledge questions about the worldaround us. Thus, we extend the domain of factual items for whichthe role of inferential processes is recognized, and we specifymore precisely the kinds of factors that respondents use inanswering such questions. We also find suggestive evidence thatattitudes influence answers to information questions, thus extendingas well the kinds of factors seen as likely to affect reportsabout facts.     

Telephone survey respondents' reactions to questions regarding interpersonal violence

Concerns have been raised regarding the appropriateness of asking about violence victimization in telephone interviews and whether asking such questions increases respondents' distress or risk for harm. However, no large-scale studies have evaluated the impact of asking such questions during a telephone interview. This study explored respondents' reactions to questions regarding violence in two large recently completed telephone surveys. After respondents were asked about violence, they were asked if they thought surveys should ask such questions and whether they felt upset or afraid because of the questions. In both surveys, the majority of respondents (regardless of their victimization history) were willing to answer questions about violence and were not upset or afraid because of the questions. More than 92% of respondents thought such questions should be asked. These results challenge commonly held beliefs and assumptions and provide some assurance to those concerned with the ethical collection of data on violent victimization.     

Eye-tracking methodology: exploring the processing of question formats in web surveys

In social research, the use of agree/disagree (A/D) questions is a popular method for measuring attitudes. Research has shown that A/D questions require complex cognitive processing and are susceptible to response bias. Thus, some researchers recommend the use of item-specific (IS) questions. This study examines the processing of A/D and IS questions, using eye-tracking methodology. By recording respondents’ eye movements, how respondents process survey questions can be evaluated. The results reveal that IS questions cause more and longer fixations. However, this only applies to the response categories. There are no differences regarding the question stems. Altogether, it seems that IS response categories trigger deeper cognitive processing than A/D response categories.     

The effect of personalization on response rates and data quality in web surveys

Personalizing correspondence has often shown to significantly increase survey response rates in mail surveys. This study experimentally tests whether personalization of email invitations acts correspondingly to web survey response rates. Also, it is investigated whether personalization influences data quality. The results of the study, using a large student sample, show that personalization significantly increases the web survey response rate by 8.6 percentage points. The data quality does not appear to be affected in any major way by personalizing the email invitations. However, the analyses do show that respondents of the personalization condition tend to respond with more social desirability bias to sensitive questions. Therefore, it is concluded that personalization has positive effects on the survey response rate, but one should carefully consider whether or not to personalize when a survey on sensitive topics is conducted.     

Nonresponse Bias for Attitude Questions

By separating respondents in a survey of farmers into thosewho refused to be interviewed and were converted, those whowere hard to reach, and those who gave no problems, we can estimatenonresponse bias from refusers not converted and respondentsnot reached. We show that for attitude questions the refusalbias is serious, ranging from 2 to 4 percent on many attitudes,and very often larger than the standard error in surveys ofordinary size.     

Web Survey Design: Paging versus Scrolling

A key choice in the design of Web surveys is whether to placethe survey questions in a multitude of short pages or in longscrollable pages. There are advantages and disadvantages ofeach approach, but little empirical evidence to guide the choice.In 2003 we conducted a survey of over 21,000 undergraduate students.Ten percent of the 10,000 respondents were directed to the scrollableversion of the survey, containing a single form for each ofthe major sections. The balance was assigned to the paging version,in which questions were presented to be visible without scrolling.The instrument contained a maximum of 268 possible questions,including topics that varied in sensitivity and desirability.The survey also permitted comparison of the effect of skip patternsby implementing skip instructions and hyperlinks in the scrollabledesign, and also recorded time at the end of each of the fivetopical sections. Differences between designs are evaluatedin terms of various forms of nonresponse, univariate and bivariatemeasurement properties, and proxies for respondent burden.     

An evaluation of transit procurement training

We evaluated a training course called “Orientation to Transit Procurement”, designed and conducted by the National Transit Institute. This course is designed to provide Federal Transit Administration (FTA) grantees an overview of regulations and best practices related to the procurement process. Our objective in conducting the evaluation was to understand how transit agency staff made changes in procurement practices in response to the course training. The evaluation was mixed mode: an Internet survey followed by in-depth interviews with a small group of respondents. Survey respondents were also provided with an open-ended question providing us with additional context for our evaluation. Results show that the training is substantially successful at meeting the goal of improving procurement practices at transit agencies; indeed, most respondents report making changes at their agencies as the proximate result of the training. This was at odds with our exploration of knowledge of procurement topics, as most respondents gave inaccurate answers on multiple-choice “knowledge questions”. This may have been due to question structure or, more likely, the nature of online surveys. Suitable training on the procurement of information technology was also a main concern. The lack of training in this area is indicative of the broader challenge facing public transit agencies in how to incorporate new forms of technology into their existing practices and bureaucratic structures.     

The effect of interviewer evaluation of respondents’ attitudes and understanding on future survey participation in a panel study

This study explored whether the interviewers’ judgments of respondents’ performances in an initial survey can predict survey participation for a second survey among the same group of respondents in a panel study. Specifically, this study examined two interviewer variables: the interviewers’ judgments of (1) the respondents’ understanding of survey questions and (2) the respondents’ attitudes toward the survey. The analyses revealed that respondents who were rated to have an excellent understanding of the survey questions and friendly attitudes were more likely to respond to the second interview request than those with a poor understanding and unfriendly attitudes, after controlling for the characteristics of the respondents. Of these two, the judgment of the respondents’ attitudes was a stronger predictor. However, interviewer’s judgments are not correlated with the primary survey outcome or the data quality. This suggests limited value of the interviewer observation.