THAT WHICH WE CALL WELFARE BY ANY OTHER NAME WOULD SMELL SWEETER AN ANALYSIS OF THE IMPACT OF QUESTION WORDING ON RESPONSE PATTERNS

Responses to survey questions are dependent on the words usedin the questions. Sometimes the alteration of words can completelychange the response distribution without obviously changingthe meaning or intent of the question. This situation occurswhen "welfare" is used instead of "poor." In all contexts examined"welfare" produced much more negative and less generous responsesthan "poor." In addition the two terms appear to tap slightlydifferent dimensions with "welfare" accessing notions of wasteand bureaucracy that are untapped or tapped much less by "poor."     

EFFECTS OF QUESTIONNAIRE DESIGN ON THE QUALITY OF SURVEY DATA

The importance of format, graphic layout, and question routinginstructions on the quality of survey data has been acknowledgedfor self-administered survey instruments, but the significanceof these factors for questionnaires administered by interviewershas not been recognized equally. This paper examines two studiesin which interviewers used different questionnaire designs toadminister the same set of survey questions to randomly selectedadult respondents in housing units that made up area probabilitysamples of the Detroit metropolitan area. The paper presentsempirical evidence of specific effects (questions skipped inerror, unprobed answers) directly attributable to the choiceof questionnaire design. The analysis shows that question nairedesign choices can either help or hurt the quality of data collectedby interviewers. Furthermore, the behaviors of experienced andinexperienced interviewers are affected in similar ways. Inother words, interviewing experience does not compensate forformat deficits in the design of survey instruments.     

ON THE ASSIGNMENT OF INDIVIDUALS TO LATENT CLASSES

Consider an m-way cross-classification table (for m = 3, 4, … ) of m dichotomous variables that describes (1) the 2 ^m possible response patterns to a set of m questions (where the response to each question is binary), and (2) the number of individuals whose responses to the m questions can be described by a particular response pattern, for each of the 2 ^m possible response patterns. Consider the situation where the data in the cross-classification table are analyzed using a particular latent class model having T latent classes (for T = 2, 3, …), and where this model fits the data well. With this latent class model, it is possible to estimate, for an individual who has a particular response pattern, what is the conditional probability that this individual is in a particular latent class, for each of the T latent classes. In this article, the following question is considered: For an individual who has a particular response pattern, can we use the corresponding estimated conditional probabilities to assign this individual to one of the T latent classes? Two different assignment procedures are considered here, and for each of these procedures, two different criteria are introduced to help assess when the assignment procedure is satisfactory and when it is not. In addition, we describe here the particular framework and context in which the two assignment procedures, and the two criteria, are considered. For illustrative purposes, the latent class analysis of a classic set of data, a four-way cross-classification of some survey data, obtained in a two-wave panel study, is discussed; and the two different criteria introduced herein are applied in this analysis to each of the two assignment procedures .     

SUBSEQUENT QUESTIONS MAY INFLUENCE ANSWERS TO PRECEDING QUESTIONS IN MAIL SURVEYS

Data of a mode experiment demonstrate that the emergence ofcontext effects may be independent of question order under mailsurvey conditions. Under telephone interview conditions, substantivelyrelated questions affected responses to the target questiononly when asked first. However, the same questions affectedresponses under mail survey conditions independent of whetherthey preceded or followed the target question.     

Question Understanding Aid (QUAID): A Web Facility that Tests Question Comprehensibility

When respondents do not understand the meaning of a survey question,they will not supply valid and reliable answers. Survey methodologistsshould therefore benefit from computer tools and other analyticalschemes that help them identify problems with questions withrespect to comprehension difficulty. We developed a Web facilitycalled Question Understanding Aid (QUAID; www.psyc.memphis.edu/quaid.html)that assists survey methodologists in identifying problems withthe wording, syntax, and semantics of questions on questionnaires.The survey methodologist enters the question into the Web facility,along with any context information and answer alternatives thataccompany the question. QUAID quickly returns a list of potentialproblems with question comprehension, including unfamiliar technicalterms, vague or imprecise relative terms, vague or ambiguousnoun phrases, complex syntax, and working memory overload. Thisarticle describes QUAID and some empirical studies that haveassessed the validity and utility of QUAID’s critiquesof questions. The output of QUAID was compared with the judgmentsof experts in language, discourse, and cognition during thedevelopment of the tool. In one evaluation, expert survey methodologistscritiqued and revised problematic questions, whereas in a secondevaluation survey methodologists evaluated the quality of originalproblematic questions, questions revised with the assistanceof QUAID, and questions revised without QUAID. In a third evaluation,eye-tracking data were collected while respondents read questionson a computer screen and answered questions aloud. Respondentshad a tendency to give up processing difficult questions tooearly (called an early exit), which potentially threatens thevalidity of the respondents’ answers. Survey methodologistsare encouraged to use QUAID and further evaluate its validityand utility.     

QUESTION-ORDER EFFECTS ON THE DETERMINANTS OF SUBJECTIVE WELL-BEING

Part-whole combinations of questions are believed to be particularlyvulnerable to question-order effects. A split-ballot experimentwas used in a telephone survey to vary the order of a generalwell-being question and a set of questions on well-being ineight specific life domains. A number of order effects on multipleregression parameters were found for married respondents; noorder effects were found for unmarried persons, however. Theseresults show that conclusions about the relative importanceof specific life domains for general well-being and conclusionsabout the effects of unmeasured variables can be substantiallyaffected by the order of the general and specific items. Severalexplanations for the order effects are discussed.     

INTENSIFIERS IN BEHAVIORAL FREQUENCY QUESTIONS

Intensifiers, words such as "very" and "extremely", are usedto magnify the meaning of the phrases to which they are applied.In a series of studies we investigate how intensifiers in questionstems affect response patterns in social surveys. Our researchindicates that even apparently important differences in questionwording may in some situations have little or no impact; addingan intensifier to a root did not create a response shift forseveral intensifier/root combinations in large-scale surveys.Using both field and laboratory techniques we explore the situationsin which shifts do occur and go some lengths toward describingwhy this happens. Response shifts were observed for two situations.First, when "extreme" was applied to "physical pain" there wasa substantial and significant response shift. Second, when respondentswere asked a question without an intensifier and then had thequestion repeated immediately afterward with an intensifier,a response shift was produced. In addition to the practicalsignificance for survey methodologists, these results are importantto cognitive psychologists interested in word meaning.     

Comparing the Quality of Data Obtained by Minimally Balanced and Fully Balanced Attitude Questions

When measuring attitudes with questions that offer dichotomous,mutually exclusive response options, researchers can ask "fullybalanced" questions (which fully state both competing pointsof view) or "minimally balanced" questions (which fully stateone viewpoint and only briefly acknowledge the second viewpoint).The two studies reported here investigated whether the greaterefficiency of the latter approach brought with it reductionsin the quality of the data obtained. Two experiments embeddedin national sample surveys showed that minimally balanced andfully balanced attitude questions yielded similar distributionsof responses and that responses to the two question forms wereequivalent in terms of concurrent validity. These studies suggestthat greater efficiency can be achieved via minimal balancingat no cost in terms of data quality.     

RESPONSE EFFECTS IN MAIL SURVEYS

A variety of response effects that had been found previouslyin interview surveys were tested in a mail survey of a heterogeneouslocal population. These included experiments on question orderresponse order, no-opinion filters, middle-response alternatives,and acquiescence. The results generally supported earlier findingsbased on student samples which had shown that order efects wereeliminated in self-administered surveys but that question-formeffects occurred as in interview surveys. One question-ordereffect, however, was found in the mail survey, and a type ofresponse-order effect (a primacy effect) that had not been previouslytested also occured. Interactions between education and responseeffects that had sometimes been found in interview surveys werenot present in the mail survey.     

The Impact of Question and Respondent Characteristics on Comprehension and Mapping Difficulties

Behavior coding is one technique researchers use to detect problemsin survey questions, but it has been primarily explored as apractical tool rather than a source of insight into the theoreticalunderstanding of the cognitive processes by which respondentsanswer survey questions. The latter is the focus of the currentinvestigation. Using data from a large study in which face-to-faceinterviews were taped and extensive behavior coding was done,we tested whether sets of respondent behavior codes could beused to distinguish respondent difficulties with comprehensionof the question from difficulties associated with mapping ajudgment onto the response format provided, and whether characteristicsof the survey questions and respondents could be used to predictwhen and for whom such difficulties would occur. Sets of behaviorcodes were identified that reflected comprehension and mappingdifficulties, and these two types of difficulties were associatedwith different question and respondent characteristics. Thisevidence suggests that behavior coding shows promise as a toolfor researchers studying the cognitive processes involved inanswering survey questions.     

ISSUE INVOLVEMENT AND RESPONSE EFFECTS IN PUBLIC OPINION SURVEYS

This research tests the widespread assumption that responseeffects due to variations in question form, wording, or contextwill be greatest among respondents who are least involved withan issue. A meta-analysis of results from 15 split-ballot experimentsconducted over a five-year period indicates that the responseeffects of using counterarguments or middle alternatives insurvey questions are significantly larger, as would be expected,among respondents who are less involved with a given issue thanamong those who are highly involved with it. But the effectsof question order and response order appear to be largely unrelatedto how involved a respondent is with a particular issue. Issueinvolvement, then, appears to specify some response effects,but not others.     

THE ANSWERING MACHINE POSES MANY QUESTIONS FOR TELEPHONE SURVEY RESEARCHERS

The proliferation of the telephone answering machine raisesa number of questions about the continued efficiency of thetelephone as a data-gathering mechanism. The most critical ofthese is what effect the use of these machines might have onestablishing contact with potential respondents. If individualsroutinely use these machines to screen calls, then their accessibilityto telephone survey researchers will be restricted. A secondquestion is the degree to which respondents who own answeringmachines and are reachable are likely to participate in a survey.A third question is the extent to which the incidence of theanswering machine as a response disposition might vary by thetime of calling and the respondent's place of residence. Thefindings of this study, based on a nationwide survey, pointto a significant proportion of answering machine owners beingreachable and willing to participate. Furthermore, the answeringmachine appears to be in use more on weekends than on weekdayevenings and in more urbanized areas than in areas with fewerinhabitants.     

RESPONSE LATENCY AS A SIGNAL TO QUESTION PROBLEMS IN SURVEY RESEARCH

This research explores the potential utility of response latencyas an index of question problems in survey research. The timerespondents took to answer three types of bad questions wascompared to the time they took to answer the repaired versionsof the questions. Questions containing a superfluous negativeand double-barreled questions took longer to answer than nearlyidentical questions without these problems. Repaired versionsof questions soliciting frequency estimates, however, took longerto answer than their problematic counter parts. The resultsare discussed in the context of a model of question answering,and their implications for survey methodology are explored.     

Response-Order Effects and the Early Gallup Split-Ballots

A meta-analysis of split-ballots conducted by the Gallup Organizationin the 1930s, 1940s, and early 1950s shows that response-ordereffects were generally small in magnitude when averaged acrossa great variety of topics and questions—and as comparedwith many of those reported in the response-effects literaturetoday. When analyzed by various question characteristics, theresults provided some support for predictions derived from currentcognitive models of response-order effects, particularly thosefrom satisficing theory. As predicted, questions asked orallywere more likely to generate a statistically significant patternof recency effects if the response alternatives or the questionsas a whole were longer rather than shorter. Other predictedpatterns of primacy and recency effects failed to materialize,however, perhaps largely because of the inherent design limitationsand partial confounding of question attributes in any such secondaryanalysis of archival survey data, but perhaps, too, becauseof simple chance variations. The data from these early experimentsnonetheless provide a partial, albeit limited, test of rivalhypotheses and explanations of response-order effects in theliterature.     

Visual Context Effects in Web Surveys

There are many examples of context effects in survey measurement.Responses to survey questions can be shaped by the order ofquestions, the format of response options, the broader surveyenvironment, and so on. For Web surveys, the inclusion of visualimages is a trivial design issue, but may have consequencesfor the responses obtained because they change the visual context.We report a series of experiments examining how responses maybe affected by the use of images in Web surveys. Specifically,we examine the effect that pictures of a healthy woman exercisingversus a sick woman in a hospital bed have on self-rated health.We replicated the experiments in three different surveys, varyingsuch factors as the size and placement of the image and thelocation of the question within the questionnaire. In general,we find that when exposed to a picture of a fit woman, respondentsconsistently rate their own health lower than when exposed toa picture of a sick woman.     

Eye-tracking methodology: exploring the processing of question formats in web surveys

In social research, the use of agree/disagree (A/D) questions is a popular method for measuring attitudes. Research has shown that A/D questions require complex cognitive processing and are susceptible to response bias. Thus, some researchers recommend the use of item-specific (IS) questions. This study examines the processing of A/D and IS questions, using eye-tracking methodology. By recording respondents’ eye movements, how respondents process survey questions can be evaluated. The results reveal that IS questions cause more and longer fixations. However, this only applies to the response categories. There are no differences regarding the question stems. Altogether, it seems that IS response categories trigger deeper cognitive processing than A/D response categories.     

DO SURVEYS OVERSTATE PUBLIC CONCERNS?

Because the item, "How concerned are you about...?" asks respondentsto indicate their level of concern about an issue, some respondentsmay sense it assumes they are concerned or should be concernedabout the issue. Using a filter question to first determineif people are concerned about the issue before asking for theirdegree of concern may help solve this problem. To test thishypothesis, a split-ballot experiment was embedded in a nationalrandom digit dialing telephone survey on food-related issues.For the four items included in the experiment, the group receivingthe filter versions of the questions gave roughly double thepercentages of "not concerned" responses as the group receivingstandard items, and the filter group also gave fewer responsesat the upper end of the response scale.     

Reliability and validity of egocentered network data collected via web: A meta-analysis of multilevel multitrait multimethod studies

Our goal in this article is to assess reliability and validity of egocentered network data collected through web surveys using multilevel confirmatory factor analysis under the multitrait multimethod approach.In this study, we analyze a questionnaire of social support of Ph.D. students in three European countries. The traits used are the frequency of social contact questions. The methods used are web survey design variants.We consider egocentered network data as hierarchical; therefore, a multilevel analysis is required. Within and between-ego reliabilities and validities are defined and interpreted.Afterwards, we proceed to a meta-analysis of the results of the three countries where within and between-ego validities and reliabilities are predicted from survey design variables which have to do with question order (by questions or by alters), response category labels (end labels or all labels) and lay-out of the questionnaire (graphical display or plain text). Results show that question order by questions, all-labeled response categories and a graphical display lay-out with images lead to a better data quality. Our basic approach consisting on multilevel and meta-analysis can be applied to evaluate the quality of any type of egocentered network questionnaire, regardless of the data collection mode.     

The Social Roots of the Partisan Gender Gap

I suggest that the gender gap in party identification is dependenton question wording and asymmetric stereotypes about men's andwomen's partisan preferences. A survey experiment reanalyzesthe gender gap by comparing the standard partisan battery toan alternative version that emphasizes feelings rather thanthoughts. Bringing question wording into closer alignment withtheory causes the gender gap to shrink. This happens in partbecause the "feel" questions find women to be less Democraticthan did the "think" questions. Moreover, reduction of the gendergap occurs mostly among highly sophisticated women and not thoseusually susceptible to question wording effects. Contrary topopular wisdom, men and women appear to be more, not less, alikepolitically when feelings are primed.     

Web survey design and administration.

Many claims are being made about the advantages of conducting surveys on the Web. However, there has been little research on the effects of format or design on the levels of unit and item response or on data quality. In a study conducted at the University of Michigan, a number of experiments were added to a survey of the student population to assess the impact of design features on resulting data quality. A sample of 1,602 students was sent an e-mail invitation to participate in a Web survey on attitudes toward affirmative action. Three experiments on design approaches were added to the survey application. One experiment varied whether respondents were reminded of their progress through the instrument. In a second experiment, one version presented several related items on one screen, while the other version presented one question per screen. In a third experiment, for one series of questions a random half of the sample clicked radio buttons to indicate their answers, while the other half entered a numeric response in a box. This article discusses the overall implementation and outcome of the survey, and it describes the results of the imbedded design experiments.