Response Order Effects in Dichotomous Categorical Questions Presented Orally: The Impact of Question and Respondent Attributes

Using data from 548 experiments in telephone surveys conductedby the Gallup Organization, we explored how attributes of questionsand respondents moderate response order effects in dichotomouscategorical questions. These effects were predominantly recencyeffects and occurred most in questions that were more difficultto comprehend (especially among respondents with the least education),with response choices that were more difficult to comprehend(because they were complete sentences instead of words or phrasesand because they were not mutually exclusive), and that wereasked after many prior questions. Recency effects were alsomore common in questions that explicitly or implicitly encouragedrespondents to wait until they had heard all the answer choicesbefore formulating a judgment than in questions that inducedrespondents to begin formulating a judgment before all the answerchoices had been read (especially among the least educated respondents).A study of interviewer behavior revealed patterns of pausingbetween and within sentences that help to explain why some typesof questions are especially prone to recency effects and othersare not.     

Eye-Tracking Data: New Insights on Response Order Effects and Other Cognitive Shortcuts in Survey Responding

Survey researchers since Cannell have worried that respondents may take various shortcuts to reduce the effort needed to complete a survey. The evidence for such shortcuts is often indirect. For instance, preferences for earlier versus later response options have been interpreted as evidence that respondents do not read beyond the first few options. This is really only a hypothesis, however, that is not supported by direct evidence regarding the allocation of respondent attention. In the current study, we used a new method to more directly observe what respondents do and do not look at by recording their eye movements while they answered questions in a Web survey. The eye-tracking data indicate that respondents do in fact spend more time looking at the first few options in a list of response options than those at the end of the list; this helps explain their tendency to select the options presented first regardless of their content. In addition, the eye-tracking data reveal that respondents are reluctant to invest effort in reading definitions of survey concepts that are only a mouse click away or paying attention to initially hidden response options. It is clear from the eye-tracking data that some respondents are more prone to these and other cognitive shortcuts than others, providing relatively direct evidence for what had been suspected based on more conventional measures.     

Experiments in Producing Nonresponse Bias

While nonresponse rates in household surveys are increasingin most industrialized nations, the increasing rates do notalways produce nonresponse bias in survey estimates. The linkagebetween nonresponse rates and nonresponse bias arises from thepresence of a covariance between response propensity and thesurvey variables of interest. To understand the covariance term,researchers must think about the common influences on responsepropensity and the survey variable. Three variables appear tobe especially relevant in this regard: interest in the surveytopic, reactions to the survey sponsor, and the use of incentives.A set of randomized experiments tests whether those likely tobe interested in the stated survey topic participate at higherrates and whether nonresponse bias on estimates involving variablescentral to the survey topic is affected by this. The experimentsalso test whether incentives disproportionately increase theparticipation of those less interested in the topic. The experimentsshow mixed results in support of these key hypotheses.     

Visual Context Effects in Web Surveys

There are many examples of context effects in survey measurement.Responses to survey questions can be shaped by the order ofquestions, the format of response options, the broader surveyenvironment, and so on. For Web surveys, the inclusion of visualimages is a trivial design issue, but may have consequencesfor the responses obtained because they change the visual context.We report a series of experiments examining how responses maybe affected by the use of images in Web surveys. Specifically,we examine the effect that pictures of a healthy woman exercisingversus a sick woman in a hospital bed have on self-rated health.We replicated the experiments in three different surveys, varyingsuch factors as the size and placement of the image and thelocation of the question within the questionnaire. In general,we find that when exposed to a picture of a fit woman, respondentsconsistently rate their own health lower than when exposed toa picture of a sick woman.     

Good item or bad—can latent class analysis tell?: the utility of latent class analysis for the evaluation of survey questions

Summary.  Latent class analysis has been used to model measurement error, to identify flawed survey questions and to estimate mode effects. Using data from a survey of University of Maryland alumni together with alumni records, we evaluate this technique to determine its usefulness for detecting bad questions in the survey context. Two sets of latent class analysis models are applied in this evaluation: latent class models with three indicators and latent class models with two indicators under different assumptions about prevalence and error rates. Our results indicated that the latent class analysis approach produced good qualitative results for the latent class models—the item that the model deemed the worst was the worst according to the true scores. However, the approach yielded weaker quantitative estimates of the error rates for a given item.     

Color, Labels, and Interpretive Heuristics for Response Scales

We carried out two experiments to investigate how the shadingof the options in a response scale affected the answers to thesurvey questions. The experiments were embedded in two web surveys,and they varied whether the two ends of the scale were representedby shades of the same or different hues. The experiments alsovaried the numerical labels for the scale points and examinedresponses to both unipolar scales (assessing frequency) andbipolar scales (assessing favorability). We predicted that theuse of different hues would affect how respondents viewed thelow end of the scale, making responses to that end seem moreextreme than when the two ends were shades of the same hue.This hypothesis was based on the notion that respondents usevarious interpretive heuristics in assigning meaning to thevisual features of survey questions. One such cue is visualsimilarity. When two options are similar in appearance, respondentswill see them as conceptually closer than when they are dissimilarin appearance. The results were generally consistent with thisprediction. When the end points of the scale were shaded indifferent hues, the responses tended to shift toward the highend of the scale, as compared to scales in which both ends ofthe scale were shaded in the same hue. Though noticeable, thisshift was less extreme than the similar shift produced whennegative numbers were used to label one end of the scale; moreover,the effect of color was eliminated when each scale point hada verbal label. These findings suggest that respondents havedifficulty using scales and pay attention even to incidentalfeatures of the response scales in interpreting the scale points.     

Spacing, Position, and Order: Interpretive Heuristics for Visual Features of Survey Questions

We present the results of six experiments that demonstrate theimpact of visual features of survey questions on the responsesthey elicit, the response process they initiate, or both. Allsix experiments were embedded in Web surveys. Experiments 1and 2 investigate the effects of the placement of nonsubstantiveresponse options (for example, "No opinion" and "Don’tknow" answer options) in relation to the substantive options.The results suggest that when these options are not differentiatedvisually (by a line or a space) from the substantive options,respondents may be misled about the midpoint of the scale; respondentsseemed to use the visual rather than the conceptual midpointof the scale as a reference point for responding. Experiment3, which varied the spacing of the substantive options, showeda similar result. Responses were pushed in the direction ofthe visual midpoint when it fell to one side of the conceptualmidpoint of the response scale. Experiment 4 examined the effectsof varying whether the response options, which were arrayedvertically, followed a logical progression from top to bottom.Respondents answered more quickly when the options followeda logical order. Experiment 5 examined the effects of the placementof an unfamiliar item among a series of similar items. For example,one set of items asked respondents to say whether several makesand models of cars were expensive or not. The answers for theunfamiliar items depended on the items that were nearby on thelist. Our last experiment varied whether a battery of relateditems was administered on a single screen, across two screens,or with each item on its own screen. The intercorrelations amongthe items were highest when they were all on the same screen.Respondents seem to apply interpretive heuristics in assigningmeaning to visual cues in questionnaires. They see the visualmidpoint of a scale as representing the typical or middle response;they expect options to be arrayed in a progression beginningwith the leftmost or topmost item; and they expect items thatare physically close to be related to each other conceptually.     

Latent class analysis of response inconsistencies across modes of data collection

Latent class analysis (LCA) has been hailed as a promising technique for studying measurement errors in surveys, because the models produce estimates of the error rates associated with a given question. Still, the issue arises as to how accurate these error estimates are and under what circumstances they can be relied on. Skeptics argue that latent class models can understate the true error rates and at least one paper (Kreuter et al., 2008) demonstrates such underestimation empirically. We applied latent class models to data from two waves of the National Survey of Family Growth (NSFG), focusing on a pair of similar items about abortion that are administered under different modes of data collection. The first item is administered by computer-assisted personal interviewing (CAPI); the second, by audio computer-assisted self-interviewing (ACASI). Evidence shows that abortions are underreported in the NSFG and the conventional wisdom is that ACASI item yields fewer false negatives than the CAPI item. To evaluate these items, we made assumptions about the error rates within various subgroups of the population; these assumptions were needed to achieve an identifiable LCA model. Because there are external data available on the actual prevalence of abortion (by subgroup), we were able to form subgroups for which the identifying restrictions were likely to be (approximately) met and other subgroups for which the assumptions were likely to be violated. We also ran more complex models that took potential heterogeneity within subgroups into account. Most of the models yielded implausibly low error rates, supporting the argument that, under specific conditions, LCA models underestimate the error rates.