Scale direction effects in agree/disagree and item-specific questions: a comparison of question formats

The effect of the response scale direction on response behavior is a well-known phenomenon in survey research. While there are several approaches to explaining how such response order effects occur, the literature reports mixed evidence. Furthermore, different question formats seem to vary in their susceptibility to these effects. We therefore investigate the occurrence of response order effects in Agree/Disagree (A/D) and Item-Specific (IS) questions. We conducted an experiment among n = 930 students in which we varied the scale direction (decremental vs. incremental) within A/D and IS questions and asked respondents to evaluate the questionnaires. The results reveal response order effects within the A/D but not within the IS question format. Furthermore, respondent’s evaluations suggest that completion of the IS questionnaires requires more consideration than the completion of the A/D questionnaires. Altogether, our findings indicate that IS questions are more robust against response order effects than A/D questions.     

Which probes are most useful when undertaking cognitive interviews?

This paper reports the use of verbal probes in cognitive interviews (CIs) undertaken to test the usefulness, validity and reliability of survey questions. Through examining the use of probes by three interviewers undertaking interviews as part the piloting of a cross-national crime survey, we examine which of the various types of probes used in CIs produce the most useful information. Other influences on interview quality are examined, including differences between interviewers and respondents themselves. The analyses rely on multi-level modelling and suggest that anticipated, emergent and conditional probes provide the most useful data. Furthermore, age, gender and educational levels appear to have no bearing on the quality of the data generated.     

Testing different rank order question layouts for PC and smartphone respondents

We studied the impact of different layouts for rank order questions on respondent effort, data quality, and substantive results among PC and smartphone respondents, in an experiment in an opt-in online panel in Spain, using an order-by-click design. We experimentally varied the device, the number of columns, and, for smartphone respondents, the position of the ‘next’ button in questions on trust in institutions.

We found some evidence of lower data quality for smartphone users but no evidence that presenting ranking items in one column performs differently than two columns. We also find little evidence that these effects differ by the number of response options presented or the number to be ranked. The placement of the ‘next’ button had little effect on performance on ranking items. Overall, our findings suggest that the format and layout of order-by-click questions has little effect on data quality, regardless of device used.     

Incorporating eye tracking into cognitive interviewing to pretest survey questions

In this study, we investigated whether incorporating eye tracking into cognitive interviewing is effective when pretesting survey questions. In the control condition, a cognitive interview was conducted using a standardized interview protocol that included pre-defined probing questions for about one-quarter of the questions in a 52-item questionnaire. In the experimental condition, participants’ eye movements were tracked while they completed an online version of the questionnaire. Simultaneously, their reading patterns were monitored for evidence of response problems. Afterward, a cognitive interview was conducted using an interview protocol identical to that in the control condition. We compared both approaches with regard to the number and types of problems they detected. We found support for our hypothesis that cognitive interviewing and eye tracking complement each other effectively. As expected, the hybrid method was more productive in identifying both questionnaire problems and problematic questions than applying cognitive interviewing alone.     

The Viability of Readiness Assessment in Contributing to Improved Career Services: Response to Jepsen (2000)

This article responds to D. A. Jepsen's (2000) commentary on Sampson et al.'s theory‐based approach to using readiness assessment to improve career services (J. P. Sampson, G. W. Peterson, R. C. Reardon, & J. G. Lenz, 2000). Three topics are included in this response: the reliability, validity, and utility of readiness assessment measures; verbal ability and the use of cognitive information‐processing theory in practice; and the potential contribution of reflective evaluation in career decision making. This article concludes with recommended research questions to examine the impact of readiness assessment on the effectiveness of career service delivery.     

The Impact of Question and Respondent Characteristics on Comprehension and Mapping Difficulties

Behavior coding is one technique researchers use to detect problemsin survey questions, but it has been primarily explored as apractical tool rather than a source of insight into the theoreticalunderstanding of the cognitive processes by which respondentsanswer survey questions. The latter is the focus of the currentinvestigation. Using data from a large study in which face-to-faceinterviews were taped and extensive behavior coding was done,we tested whether sets of respondent behavior codes could beused to distinguish respondent difficulties with comprehensionof the question from difficulties associated with mapping ajudgment onto the response format provided, and whether characteristicsof the survey questions and respondents could be used to predictwhen and for whom such difficulties would occur. Sets of behaviorcodes were identified that reflected comprehension and mappingdifficulties, and these two types of difficulties were associatedwith different question and respondent characteristics. Thisevidence suggests that behavior coding shows promise as a toolfor researchers studying the cognitive processes involved inanswering survey questions.     

RESPONSE-TIME MEASUREMENT IN SURVEY RESEARCH A METHOD FOR CATI AND A NEW LOOK AT NONATTITUDES

The paper introduces a methodology for measuring accuratelythe time it takes respondents to answer questions in computer-assistedtelephone surveys. The methodology, which is completely invisibleto respondents, comprises a computer "clock," capable of timingresponses with millisecond accuracy, and a "voice-key" thatconverts sounds emitted by respondents into signals capableof triggering the computer clock. Response times to questionson a range of attitude questions, including stable and unstableattitudes, were measured. The results revealed orderly normsin the latencies associated with various types of survey questions.The latencies associated with the expression of stable and unstableattitudes are discussed in the context of the notion of "nonattitudes"and shed new light on this controversial thesis. Overall, ourresults demonstrate that response latencies can be measuredprecisely and reliably in telephone surveys and that the datafrom such measurement open new windows on the cognitive dynamicsof survey responses.     

Toward a social psychology of survey methodology: An application of the approach and directions for the future

Survey methodology is a relatively new academic discipline focused on understanding sources of survey errors. As an interdisciplinary field, survey methodology borrows theoretical approaches from other disciplines and applies them to understand how survey respondents answer questions. One field in particular, cognitive psychology, has played a central role in the development of survey methodology. The cognitive approach has focused researchers' attentions on the sources of error at each stage of the cognitive process respondents use to answer a survey question: comprehension of the question, recollection of relevant information, estimation and judgment, and reporting an answer. Although this focus on the cognitive response process has been positive and fruitful, potentially strong social and interactional influences on the response process have been underinvestigated and undertheorized. Thus, this essay argues for a revitalized research program in the sociological social psychology of survey methodology, given its rich body of theory and research. The current strengths of social psychological and interactional approaches are highlighted, focusing primarily on recent work using identity theory to understand social desirability biases. Finally, potentially fruitful future directions for research are proposed, matching sociological social psychological theories to the survey errors upon which they may shed light.     

Response Order Effects in Dichotomous Categorical Questions Presented Orally: The Impact of Question and Respondent Attributes

Using data from 548 experiments in telephone surveys conductedby the Gallup Organization, we explored how attributes of questionsand respondents moderate response order effects in dichotomouscategorical questions. These effects were predominantly recencyeffects and occurred most in questions that were more difficultto comprehend (especially among respondents with the least education),with response choices that were more difficult to comprehend(because they were complete sentences instead of words or phrasesand because they were not mutually exclusive), and that wereasked after many prior questions. Recency effects were alsomore common in questions that explicitly or implicitly encouragedrespondents to wait until they had heard all the answer choicesbefore formulating a judgment than in questions that inducedrespondents to begin formulating a judgment before all the answerchoices had been read (especially among the least educated respondents).A study of interviewer behavior revealed patterns of pausingbetween and within sentences that help to explain why some typesof questions are especially prone to recency effects and othersare not.     

Cognitive Characteristics of Strategic and Non-strategic Gamblers

Participation in strategic and non-strategic games is mostly explained in the literature by gender: men gamble on strategic games, while women gamble on non-strategic games. However, little is known about the underlying cognitive factors that could also distinguish strategic and non-strategic gamblers. We suggest that cognitive style and need for cognition also explain participation in gambling subtypes. From a dual-process perspective, cognitive style is the tendency to reject or accept the fast, automatic answer that comes immediately in response to a problem. Individuals that preferentially reject the automatic response use an analytic style, which suggest processing information in a slow way, with deep treatment. The intuitive style supposes a reliance on fast, automatic answers. The need for cognition provides a motivation to engage in effortful activities. One hundred and forty-nine gamblers (53 strategic and 96 non-strategic) answered the Cognitive Reflection Test, Need For Cognition Scale, and socio-demographic questions. A logistic regression was conducted to evaluate the influence of gender, cognitive style and need for cognition on participation in strategic and non-strategic games. Our results show that a model with both gender and cognitive variables is more accurate than a model with gender alone. Analytic (vs. intuitive) style, high (vs. low) need for cognition and being male (vs. female) are characteristics of strategic gamblers (vs. non-strategic gamblers). This study highlights the importance of considering the cognitive characteristics of strategic and non-strategic gamblers in order to develop preventive campaigns and treatments that fit the best profiles for gamblers.     

Eye-Tracking Data: New Insights on Response Order Effects and Other Cognitive Shortcuts in Survey Responding

Survey researchers since Cannell have worried that respondents may take various shortcuts to reduce the effort needed to complete a survey. The evidence for such shortcuts is often indirect. For instance, preferences for earlier versus later response options have been interpreted as evidence that respondents do not read beyond the first few options. This is really only a hypothesis, however, that is not supported by direct evidence regarding the allocation of respondent attention. In the current study, we used a new method to more directly observe what respondents do and do not look at by recording their eye movements while they answered questions in a Web survey. The eye-tracking data indicate that respondents do in fact spend more time looking at the first few options in a list of response options than those at the end of the list; this helps explain their tendency to select the options presented first regardless of their content. In addition, the eye-tracking data reveal that respondents are reluctant to invest effort in reading definitions of survey concepts that are only a mouse click away or paying attention to initially hidden response options. It is clear from the eye-tracking data that some respondents are more prone to these and other cognitive shortcuts than others, providing relatively direct evidence for what had been suspected based on more conventional measures.     

Who Plays What? Participation Profiles in Chance Versus Skill-based Gambling

To determine whether gambling participation falls into skill and chance-based categories and, if so, to determine the socio-demographic characteristics associated with these different categories. A cross-sectional analysis of all respondents to the 2005 Northern Territory Gambling Prevalence Survey who gambled in the 12 months prior to the survey. Factor analysis was employed to determine whether a chance versus skill-based dichotomy described the structure of gambling participation. Gambler preference groups were constructed using the median of rotated factor scores. Multinomial logit regression was then used to determine independent associations between explanatory variables and categories of gambler preferences. The skill and chance-based dichotomy did describe player preferences for the sample of adult gamblers in the Northern Territory, Australia. Gender, age, household income, household structure and the geographic location (access to gambling opportunities) of respondents were all associated with different degrees of participation in skill and chance based gambling activities. Notably, respondents 35 years and over were significantly over-represented in the low-skill/high-chance participation group, and under-represented in the high-skill/low-chance group. It is clear that the term gambling is a confounding rubric that hides differences both in the type of activity and the socio-demographic profiles of participants. An examination of the latter raises important questions about the role of chance in later life, as well as the role of self-determination in gambling for other groups, particularly younger men.     

SurveyMotion: what can we learn from sensor data about respondents’ completion and response behavior in mobile web surveys?

Participation in web surveys via smartphones increased continuously in recent years. The reasons for this increase are a growing proportion of smartphone owners and an increase in mobile Internet access. However, research has shown that smartphone respondents are frequently distracted and/or multitasking, which might affect completion and response behavior in a negative way. We propose ‘SurveyMotion (SMotion)’, a JavaScript-based tool for mobile devices that can gather information about respondents’ motions during web survey completion by using sensor data. Specifically, we collect data about the total acceleration (TA) of smartphones. We conducted a lab experiment and varied the form of survey completion (e.g. standing or walking). Furthermore, we employed questions with different response formats (e.g. radio buttons and sliders) and measured response times. The results reveal that SMotion detects higher TAs of smartphones for respondents with comparatively higher motion levels. In addition, respondents’ motion level affects response times and the quality of responses given. The SMotion tool promotes the exploration of how respondents complete mobile web surveys and could be employed to understand how future mobile web surveys are completed.     

(Un)expected exception: validating acquiescent response style factor in the Czech Republic

ABSTRACT

This study explores the acquiescent response style (ARS) among respondents in the Czech Republic. To analyse ARS, confirmatory factor analysis (CFA) was employed and the response style (RS) was modelled as a latent variable. The RS factor in the CFA model must be validated by its relationship to education and age, i.e. proxies of cognitive ability. The two studies presented in this article use large amount of data as all available balanced batteries of items were analysed. In Study 1, the RS factor showed no correlation with age/education in most of the models. Study 2 employed additional measures of cognitive ability and education and confirmed the overall absence of the relationship. Author concludes that the modelled RS factor then cannot be considered the ARS and deduces that the most likely explanation of the identified systematic error variance is the respondents’ carelessness manifested by automatically choosing the extreme agreement, or choosing the first presented answer as a result of memory effect.     

When near means related: evidence from three web survey experiments on inter-item correlations in grid questions

In this article, we present the results of three question design experiments on inter-item correlations, which tested a grid design against a single-item design. The first and second experiments examined the inter-item correlations of a set with five and seven items, respectively, and the third experiment examined the impact of the question design on inter-item correlations of follow-up questions. We found that the inter-item correlations in grid designs were higher for both the set of five and seven items than for questions in a single-item design. Additionally, the inter-item correlations of the set of follow-up questions were higher if the preceding set of questions was presented in a grid. According to the findings of the present research, respondents seem to apply the interpretative near means related heuristic when answering grid questions, and also when answering follow-up questions preceeded by grid questions.     

Uncertainty and framing in a valuation task

Many stated preference studies report framing effects in responses to valuation questions. Framing in stated preference studies occurs when respondents use irrelevant information contained in the question to help them value the good. This may occur because respondents are uncertain or do not hold well-formed preferences for the good in question. We investigate if respondent certainty explains framing effects in a contingent valuation study, using data from a double bounded dichotomous elicitation format and a follow-up certainty question. We investigate if respondent certainty influences anchoring and the shift effect. We find evidence that the anchoring effect is stronger for respondents who are less certain about their response to the contingent valuation question compared to respondents who are very certain. However, the shift effect is significant and negative only for respondents who are very certain. Our results indicate that certain respondents are more consistent with the predictions of rational behaviour than uncertain respondents.     

Predicting behaviour: comparing the performance of factual versus attitudinal approaches

Despite years of refinement and improvement to questionnaire design, the need remains to identify effective, but ‘user-friendly’ questioning procedures to predict behaviour, without compromising predictive performance. Questionnaires developed to predict behaviour typically use an attitudinal approach. However, those types of questionnaires are associated with increased respondent fatigue, decreased question comprehension, opposition to question tone and demand effects. An alternative approach involves use of factual questions. This study compares ease of application in the field as well as predictive performance of the two approaches to questionnaire design in the context of predicting willingness to donate blood. Results indicate that factual questions are quicker and easier to apply in the field than attitudinal questions, while predictive performance of the approaches is equivalent.     

THE ANSWERING MACHINE POSES MANY QUESTIONS FOR TELEPHONE SURVEY RESEARCHERS

The proliferation of the telephone answering machine raisesa number of questions about the continued efficiency of thetelephone as a data-gathering mechanism. The most critical ofthese is what effect the use of these machines might have onestablishing contact with potential respondents. If individualsroutinely use these machines to screen calls, then their accessibilityto telephone survey researchers will be restricted. A secondquestion is the degree to which respondents who own answeringmachines and are reachable are likely to participate in a survey.A third question is the extent to which the incidence of theanswering machine as a response disposition might vary by thetime of calling and the respondent's place of residence. Thefindings of this study, based on a nationwide survey, pointto a significant proportion of answering machine owners beingreachable and willing to participate. Furthermore, the answeringmachine appears to be in use more on weekends than on weekdayevenings and in more urbanized areas than in areas with fewerinhabitants.     

All aboard the Overland Stage!

Asking surface-level questions is a common practice among educators. Delva Daines's (1986) pilot study indicated that 93 percent of the questions asked during lessons were at the literal level of comprehension and that it was very common for the teachers to restate, rephrase, or answer their own questions before the student had an opportunity to respond. This article provides a foundation for K-12 teachers to begin the implementation of asking higher-level questions in their classrooms and engaging students in critical thinking activities. Using the work of Benjamin Bloom (1956) and Kagan (1999), actual questions and ideas that can be used before, during, and after readings in the classroom will strengthen the ability of all students to think.