Effects of Category Order on Answers in Mail and Telephone Surveys1

Abstract Past research suggests that mail surveys encourage a primacy effect, which is a tendency to choose the first answers from a list, whereas telephone surveys encourage a recency effect, a tendency to choose the last answers from a list. This paper summarizes results from 82 new experiments conducted in 12 separate surveys in seven states. Only four of 33 mail survey comparisons exhibited significant primacy effects, while five of 26 experiments in telephone surveys exhibited recency effects. In addition, only three of 23 cross-method comparisons produced a significant primacy/recency effect in the expected manner. The conclusion is that the prevalence of primacy and recency effects has been over-estimated by past research and a new theoretical approach that takes into account multiple causation is needed for examining these effects.     

Reducing Refusal Rates for Telephone Interviews

In two experiments, refusal rates to telephone interviews werenot affected by substantial changes in the introductory remarksof the interviewer. A prior letter significantly lowered refusalrates in a third experiment. In all three, interviewer sex hadno effect.     

Personalization of Mail Surveys for General Public and Populations with a Group Identity: Results from Nine Studies*

Abstract The effect of personalization on mail survey response rates was examined in nine studies that included 17 comparisons under several research conditions. A study of this variable across multiple experiments in five agricultural experiment stations was undertaken because of conflicting results from previous research and from concern that the effectiveness of personalization might have decreased over time. Results show that, while response to general public surveys appeared to increase modestly across all treatment groups, there was no positive effect for populations in which a group identity (e.g., Dear Oregon Gardner or Dear ATV Owner) is employed to address respondents in cover letters. Personalization appears to remain useful for improving response in surveys of the general public.     

Factors that influence reading and comprehension of branching instructions in self–administered questionnaires

Summary: In this paper we examine the tendency for branching instructions to be ignored, misread, or otherwise not appropriately followed so that item nonresponse occurs for follow–up questions. The potential influence on branching errors of seven features of question complexity are examined, including high number of question words, high number of answer categories, last categories branch, all categories branch, write–in responses, location at the bottom of a page, and high distance between the answer box and branching instruction. A logistic regression analysis revealed that question complexity had a tendency to increase certain errors, but not others.* A more detailed version of this paper with additional analysis and discussion is available at http://www.sesrc.wsu.edu/dillman/. The opinions expressed here are those of the authors, not necessarily of the institutions where they presently work or the U.S. Census Bureau, which provided financial support for the collection of these data. We would like to thank Aref Dajani and Yves Thibaudeau for their advice on the analysis used in this paper.     

Toward the Assessment of Public Values

Efforts to measure trends in citizen preferences have been curtailedby the tendency of public opinion analysts to study ephemeralissues, to depend upon single questions, and to analyze issuesindividually rather than in relation to competing concerns.An illustrative analysis uses conceptualization of opinionsin a value framework and factor analysis to overcome these limitations.     

An exploratory analysis of select predictors of concern for law and order

Within the last decade concern for law and order has become a major domestic issue. Presidential candidates have campaigned on the issue. Opinion polls have charted the growing public concern. Crime statistics have reported startling growth in the number of serious crimes committed in the United States. This paper attempts to assess the ability of different variables to predict and explain this high concern. Two aspects of concern will be discussed: (1) awareness of and (2) willingness to do something about crime. Theoretically little is known about how awareness for a social problem can be translated into forms of action to deal with the perceived problem. Methodologically, a technique is needed for assessing the relative importance of different variables related to the problem area. Path analysis is discussed as a possible technique. The data was collected through a random sample of heads-of-households in the State of Washington during the summer of 1970 (N=3101; response rate=75%). Some of the major findings include: (1) people in larger cities are more aware of a crime problem than people in smaller cities and towns, but people in larger cities are less willing to allocate tax dollars to combat crime; (2) the size of city, and implicitly the crime rate, is the most important variable for understanding the public awareness of crime; (3) the elderly and those of conservative orientation are more willing to allocate funds to combat crime than the young and liberal, although there exists no difference in their awarencess of the problem; and (4) relative exposure to mass media, socio-economic status, or identification with one's community make little contribution to the understanding of either awareness of crime or willingness to allocate tax dollars to combat crime.     

Helping Respondents Get It Right the First Time: The Influence of Words, Symbols, and Graphics in Web Surveys

We utilize and apply visual design theory to experimentallytest ways to improve the likelihood that web respondents reportdate answers in a particular format desired by the researcher,thus reducing possible deleterious effects of error messagesor requests for corrections. These experiments were embeddedin a series of web surveys of random samples of university students.We seek to examine the sequential and cumulative effects ofvisually manipulating the size and proximity of the answer spaces,the use of symbols instead of words, the verbal language ofthe question stem, and the graphical location of the symbolicinstruction. Our results show that the successive series ofvisual language manipulations improve respondents’ useof the desired format (two digits for the month and four digitsfor the year) from 45 percent to 96 percent. These results suggestthat writing effective questions for web surveys may dependas much or more on the presentation of the answer categories/spacesas the question wording itself.     

Does "Yes or No" on the Telephone Mean the Same as "Check-All-That-Apply" on the Web?

Recent experimental research has shown that respondents to forced-choicequestions endorse significantly more options than respondentsto check-all questions. This research has challenged the commonassumption that these two question formats can be used interchangeablybut has been limited to comparisons within a single survey mode.In this paper we use data from a 2004 random sample survey ofuniversity students to compare the forced-choice and check-allquestion formats across web self-administered and telephoneinterviewer-administered surveys as they are commonly used insurvey practice. We find that the within-mode question formateffects revealed by previous research and reaffirmed in thecurrent study appear to persist across modes as well; the telephoneforced-choice format produces higher endorsement than the webcheck-all format. These results provide further support forthe argument that the check-all and forced-choice question formatsdo not produce comparable results and are not interchangeableformats. Additional comparisons show that the forced-choiceformat performs similarly across telephone and web modes.     

The Influence of Graphical and Symbolic Language Manipulations on Responses to Self-Administered Questions

This article reports results from 14 experimental comparisonsdesigned to test 7 hypotheses about the effects of two typesof nonverbal languages (symbols and graphics) on responses toself-administered questionnaires. The experiments were includedin a survey of 1,042 university students. Significant differenceswere observed for most comparisons, providing support for allseven hypotheses. These results support the view that respondents’answers to questions in self-administered surveys are influencedby more than words. Thus, the visual presentation of questionsmust be taken into consideration when designing such surveysand, especially, when comparing results across surveys in whichthe visual presentation of questions is varied.     

The Paradox of Discretion and the Case of Elian Gonzalez

The exercise of administrative discretion is inevitable in complex policy-making and law enforcement, including immigration policy generally and the Elian Gonzalez case specifically. Clearly, discretion is in tension with democratic notions of political accountability, the rule of law, and constitutional separation of powers. Yet the Elian case suggests that although discretion poses a threat to democracy it also contributes to a democratic public administration. Furthermore, because discretion is exercised in a sea of safeguards its dangers can be reconciled to its necessity.