Question Understanding Aid (QUAID): A Web Facility that Tests Question Comprehensibility

When respondents do not understand the meaning of a survey question,they will not supply valid and reliable answers. Survey methodologistsshould therefore benefit from computer tools and other analyticalschemes that help them identify problems with questions withrespect to comprehension difficulty. We developed a Web facilitycalled Question Understanding Aid (QUAID; www.psyc.memphis.edu/quaid.html)that assists survey methodologists in identifying problems withthe wording, syntax, and semantics of questions on questionnaires.The survey methodologist enters the question into the Web facility,along with any context information and answer alternatives thataccompany the question. QUAID quickly returns a list of potentialproblems with question comprehension, including unfamiliar technicalterms, vague or imprecise relative terms, vague or ambiguousnoun phrases, complex syntax, and working memory overload. Thisarticle describes QUAID and some empirical studies that haveassessed the validity and utility of QUAID’s critiquesof questions. The output of QUAID was compared with the judgmentsof experts in language, discourse, and cognition during thedevelopment of the tool. In one evaluation, expert survey methodologistscritiqued and revised problematic questions, whereas in a secondevaluation survey methodologists evaluated the quality of originalproblematic questions, questions revised with the assistanceof QUAID, and questions revised without QUAID. In a third evaluation,eye-tracking data were collected while respondents read questionson a computer screen and answered questions aloud. Respondentshad a tendency to give up processing difficult questions tooearly (called an early exit), which potentially threatens thevalidity of the respondents’ answers. Survey methodologistsare encouraged to use QUAID and further evaluate its validityand utility.     

Toward a social psychology of survey methodology: An application of the approach and directions for the future

Survey methodology is a relatively new academic discipline focused on understanding sources of survey errors. As an interdisciplinary field, survey methodology borrows theoretical approaches from other disciplines and applies them to understand how survey respondents answer questions. One field in particular, cognitive psychology, has played a central role in the development of survey methodology. The cognitive approach has focused researchers' attentions on the sources of error at each stage of the cognitive process respondents use to answer a survey question: comprehension of the question, recollection of relevant information, estimation and judgment, and reporting an answer. Although this focus on the cognitive response process has been positive and fruitful, potentially strong social and interactional influences on the response process have been underinvestigated and undertheorized. Thus, this essay argues for a revitalized research program in the sociological social psychology of survey methodology, given its rich body of theory and research. The current strengths of social psychological and interactional approaches are highlighted, focusing primarily on recent work using identity theory to understand social desirability biases. Finally, potentially fruitful future directions for research are proposed, matching sociological social psychological theories to the survey errors upon which they may shed light.     

The influence of interviewer characteristics on support for democracy and political engagement in Sub-Saharan Africa*

This study investigates how an interviewer’s characteristics affect how respondents answer survey questions about democracy and political engagement. I analyze data from the 2008 Afrobarometer surveys, in which 810 interviewers surveyed 27,713 respondents across 20 countries in sub-Saharan Africa. Using these data, I study how interviewer education, age, and gender affect two outcomes: (1) response distributions to attitudinal and behavioral survey questions and (2) the likelihood of respondents saying ‘don’t know’ to a survey question. The analysis also investigates how the respondent’s perception of who sponsored the survey (NGO, private sector, government) affects attitudes. The results show that these interviewer characteristics affect the quality of survey data on political attitudes and behaviors. In the discussion, I consider the implications for research based on public opinion data about democracy and political engagement.     

Comparing Check-All and Forced-Choice Question Formats in Web Surveys

For survey researchers, it is common practice to use the check-allquestion format in Web and mail surveys but to convert to theforced-choice question format in telephone surveys. The assumptionunderlying this practice is that respondents will answer thetwo formats similarly. In this research note we report resultsfrom 16 experimental comparisons in two Web surveys and a papersurvey conducted in 2002 and 2003 that test whether the check-alland forced-choice formats produce similar results. In all 16comparisons, we find that the two question formats do not performsimilarly; respondents endorse more options and take longerto answer in the forced-choice format than in the check-allformat. These findings suggest that the forced-choice questionformat encourages deeper processing of response options and,as such, is preferable to the check-all format, which may encouragea weak satisficing response strategy. Additional analyses showthat neither acquiescence bias nor item nonresponse seem topose substantial problems for use of the forced-choice questionformat in Web surveys.     

WHAT DO OPEN-ENDED QUESTIONS MEASURE?

Open-ended questions are frequently used by survey researchersto measure public opinion. Some scholars, however, have doubtsabout how accurately these kinds of questions measure the viewsof the public. A chief concern is that the questions tap, inpart, people's ability to articulate a response, not their underlyingattitudes. This paper tests whether this concern is warranted.Using open-ended questions from the Center for Political Studies,I show that almost all people respond to open-ended questions.The few individuals who do not respond appear uninterested inthe specific question posed, not unable to answer such questionsin general. These findings should increase our confidence inwork of scholars who have relied on open-ended questions.     

The “Other” Effects of Psychotropic Medication: Social Workers’ Perspectives on the Psychosocial Effects of Medication Treatment on Adolescent Clients

This study explores social workers’ perspectives on the psychosocial effects of psychotropic drug treatment of their adolescent clients. Using a mail survey, we asked a national sample of experienced clinical social workers to answer questions about their perceptions of the effects of such treatment on the sense of self and social well-being of an adolescent client. The survey questions also explored associations between perceived psychosocial medication effects and characteristics of the client and the treatment. The findings suggested that social workers viewed medication treatment as having greater beneficial than harmful psychosocial effects on their adolescent clients, but that both effects existed simultaneously. The most important factors associated with the perceived effects of medication that emerged from social workers’ reports included the etiology of the disorder, the type of drug treatment and its effectiveness in addressing symptoms, the client’s “competence,” and the quality of the relationship between the client and the treating social worker. This study provides direction for future research on a neglected but important question. It would be helpful to broaden the scope of professional discourse on the advantages and disadvantages of psychopharmacologic treatment for youth beyond the effectiveness or safety of the treatment to include questions concerning clinicians’ perceptions of the effects of drug treatment on youths’ sense of self and social well-being.     

Documenting Pornography Use in America: A Comparative Analysis of Methodological Approaches

Estimates of pornography use in the United States range widely. We explore the reasons for the variation in such estimates among U.S. adults using data from four different recent nationally representative samples—each of which asked a different type of question about pornography use. We attribute the notable variation in estimates to differences in question wording and answer options, and assert that a survey question asking respondents about their most recent use of pornography minimizes recall bias and is better poised to assess the overall prevalence of pornography in a population than is the more common approach of asking respondents about their historical general-use pattern. When we privileged the most-recent-use approach, survey data from 2014 reveal that 46% of men and 16% of women between the ages of 18 and 39 intentionally viewed pornography in a given week. These numbers are notably higher than most previous population estimates employing different types of questions. The results have ramifications for methods of surveying sensitive self-reported behaviors and for contextualizing scholars’ claims as well as popular conversations about the reach and implications of pornography use in the United States.     

Democratizing European Science and Technology Policies? On Users,Problems and the Quest for Legitimacy

The claim, that Science and Technology Policies (S&TP) are entering a new phase (or mode) is not restricted to the field of Science Studies, it can also be found in political discourses. A prime example is the EU Framework Programme 5 and the comments around it. Similar shifts can be found in many European states, under headings like 'user orientation', 'problem orientation' or 'sustainability'. These new concepts are often linked to problems of priority setting, to problems of how to increase the effectiveness of research funding, and to questions of acceptability and acceptance of developments in science and technology. The open question is whether this development can be understood as the emergence of a new phase in S&TP that can be characterized as a democratization of science and technology. Framework Programme 5 is used to answer this question.     

Investigating validity and effectiveness of cognitive interviewing as a pretesting method for non-English questionnaires: Findings from Korean cognitive interviews

This study aims to empirically investigate whether cognitive interviewing is effective as a pretesting method for detecting problems in translated survey questionnaires as in the English language source questionnaire. The validity of non-English cognitive interviews is of concern among researchers because the cognitive interviewing technique was developed based on the communicative norms of Western cultures where directness and openness in expressing one’s opinion is encouraged. However, different communicative norms are present in Asian languages and cultures. To date, little research has been conducted on this topic. Using qualitative and quantitative analysis, we compared survey question problems identified through the cognitive interviews conducted in English and in Korean and found that Korean cognitive interviewing was indeed effective for detecting problematic survey questions.     

‘It's About People and Their Environment’: Student Social Workers' Definitions of Social Work Research

This article considers constructions of social work research from the perspectives of student social workers in New Zealand. There have been many academic discussions of the unique epistemology that can be called social work research but little is known of students and/or practitioner views. Are they interested in social work research? Do they even care about debates on epistemology? Forty-three student social workers considered two questions while attending a social work research methods course: ‘What is social work research?’ and ‘What kind of social work researcher might I be?’. A subset of 18 distance students explored a third question: ‘Should social work research be part of everyday practice or not?’. To answer these questions students provided comments in a short survey, material from their written student assignments and comments from online discussion board activities. The results suggest that student social workers have a preference for social work research that is compatible with their clearly articulated social work value base, and that social work research primarily should benefit the client group with which a social worker is closely linked. Student social workers also recognise the importance of research for their everyday practice, yet at the same time feel there are organisational constraints to this happening.     

SURVEY QUESTIONS ABOUT THE FREQUENCY OF VAGUELY DEFINED EVENTS: THE EFFECTS OF RESPONSE ALTERNATIVES

Experimental research has shown that the choice of responsealternatives can influence responses to questions about thefrequency of vaguely defined target events (e.g., feeling annoyed);the set of response alternatives is treated as information thatcontributes to the interpretation of the question. In a seriesof split ballot experiments we investigate whether such effectsoccur in a large-scale survey context for sets of response alternativesthat might be used interchangeably by survey researchers. Thepredicted response shifts were found in our field experiments.Those presented with response alternatives discriminating atlow frequencies reported fewer of the target episodes than thosepresented with higher-frequency response alternatives. However,the size of the observed shifts varied from zero to 13 percentand depended on a number of characteristics of the survey, forexample, the presence of "priming" questions and the orientationof the response scales. Response alternatives are not neutraland therefore must be carefully considered when constructinga survey question.     

Identifying Participation in the Informal Economy Using Survey Research Methods1

Abstract There is growing interest in the extent of informal economic activity as a form of income generating or subsistence activity within first world economies, but little agreement exists in the literature as to what constitutes informal work, what activities in addition to monetary exchanges make up the informal economy, and where these activities are located. The lack of consensus on the proper definition of the activities ,that make up the informal economy is reflected in the lack of systematic data available to investigate these forms of work. This paper investigates using telephone survey research techniques to develop inexpensive and efficient data sources to answer questions about the forms, prevalence, and location of informal work. The research uncovers problems associated with developing survey based measures of informal activity, but it also demonstrates the viability of this approach as well as the generally high level of participation in both rural and urban places.     

Does "Yes or No" on the Telephone Mean the Same as "Check-All-That-Apply" on the Web?

Recent experimental research has shown that respondents to forced-choicequestions endorse significantly more options than respondentsto check-all questions. This research has challenged the commonassumption that these two question formats can be used interchangeablybut has been limited to comparisons within a single survey mode.In this paper we use data from a 2004 random sample survey ofuniversity students to compare the forced-choice and check-allquestion formats across web self-administered and telephoneinterviewer-administered surveys as they are commonly used insurvey practice. We find that the within-mode question formateffects revealed by previous research and reaffirmed in thecurrent study appear to persist across modes as well; the telephoneforced-choice format produces higher endorsement than the webcheck-all format. These results provide further support forthe argument that the check-all and forced-choice question formatsdo not produce comparable results and are not interchangeableformats. Additional comparisons show that the forced-choiceformat performs similarly across telephone and web modes.     

The Impact of Question and Respondent Characteristics on Comprehension and Mapping Difficulties

Behavior coding is one technique researchers use to detect problemsin survey questions, but it has been primarily explored as apractical tool rather than a source of insight into the theoreticalunderstanding of the cognitive processes by which respondentsanswer survey questions. The latter is the focus of the currentinvestigation. Using data from a large study in which face-to-faceinterviews were taped and extensive behavior coding was done,we tested whether sets of respondent behavior codes could beused to distinguish respondent difficulties with comprehensionof the question from difficulties associated with mapping ajudgment onto the response format provided, and whether characteristicsof the survey questions and respondents could be used to predictwhen and for whom such difficulties would occur. Sets of behaviorcodes were identified that reflected comprehension and mappingdifficulties, and these two types of difficulties were associatedwith different question and respondent characteristics. Thisevidence suggests that behavior coding shows promise as a toolfor researchers studying the cognitive processes involved inanswering survey questions.     

Using egocentered networks in survey research. A methodological preview on an application of social network analysis in the area of family research

Can questions about families be adequately answered with household survey data? This question leads us to the problem to defining the family. To help us answer this question we use network analysis as a tool to examine the family without a preconceived definition. To get information about the population of family configurations in Federal Republic of Germany, we choose a survey technique that measured ego-centered networks (10–12 network generators, 4–6 network interpreters). Based on two pretests (n₁ = 98, n₂ = 534) and the main survey (n = 10,000), we discuss some problems encountered during data collection. We also examine the validity and reliability of the data. We used variations of the measurement instrument and asked a group of 99 respondents twice to test reliability. Then we compare our results with census data and compatible US and German studies to test validity. The results show: (1) It is possible to get reliable network data by using survey technique. (2) Size of networks, named persons, named relations are reasonably stable. (3) The instrument to measure networks is constant against minor changes in formulation of questionnaire and against variation of collecting techniques. These findings hold both on the macro level (aggregate data), and also on the micro level (person related data).     

Scale direction effects in agree/disagree and item-specific questions: a comparison of question formats

The effect of the response scale direction on response behavior is a well-known phenomenon in survey research. While there are several approaches to explaining how such response order effects occur, the literature reports mixed evidence. Furthermore, different question formats seem to vary in their susceptibility to these effects. We therefore investigate the occurrence of response order effects in Agree/Disagree (A/D) and Item-Specific (IS) questions. We conducted an experiment among n = 930 students in which we varied the scale direction (decremental vs. incremental) within A/D and IS questions and asked respondents to evaluate the questionnaires. The results reveal response order effects within the A/D but not within the IS question format. Furthermore, respondent’s evaluations suggest that completion of the IS questionnaires requires more consideration than the completion of the A/D questionnaires. Altogether, our findings indicate that IS questions are more robust against response order effects than A/D questions.     

Eye-tracking methodology: exploring the processing of question formats in web surveys

In social research, the use of agree/disagree (A/D) questions is a popular method for measuring attitudes. Research has shown that A/D questions require complex cognitive processing and are susceptible to response bias. Thus, some researchers recommend the use of item-specific (IS) questions. This study examines the processing of A/D and IS questions, using eye-tracking methodology. By recording respondents’ eye movements, how respondents process survey questions can be evaluated. The results reveal that IS questions cause more and longer fixations. However, this only applies to the response categories. There are no differences regarding the question stems. Altogether, it seems that IS response categories trigger deeper cognitive processing than A/D response categories.     

Predicting data quality of proxy reports in egocentric network studies

Egocentric network studies and many general population surveys rely on proxy reports about network contacts of study participants that are asked in name interpreter questions. A central concern is the extent to which proxy reports match the answers these contacts would give themselves if they would be directly interviewed. Based on the theory of survey satisficing, the present research proposes a theoretical framework that allows predicting when proxy reports are likely to match self-reports. Congruence is higher if respondents possess the motivation and ability to answer a proxy question effortfully, and if the task is not too difficult. Moreover, the theory of survey satisficing states that motivation, abilities, and task difficulty are not independent of each other, which provides an explanation for inconsistent findings in the literature. Results from two egocentric network studies study among German adults (N = 756) and among Dutch middle school students (N = 679), in which network contacts were also interviewed, are in line with these hypotheses. Design recommendations for egocentric network studies are provided.     

'If you could change one thing...' What community-based researchers wish they could have done differently

Community-based research (CBR) is rapidly gaining recognitionsas an important tool in addressing complex environmental, healthand social problems. CBR attempts to marry community developmentwith knowledge generation. However, little is known about theCanadian CBR context. This paper reports on the analysis ofan element within a web-based survey to investigate the statusof CBR in Canada. The survey included 25 questions. This paperfocuses on responses to one open-ended question (Q20): ‘Ifthere was one thing you could change about [your last CBR] project,what would it be?’ A content analysis approach guidedanalysis and responses were coded into main theme and subthemes.Our collective reflections offer other CBR practitioners anopportunity to more effectively plan future partnerships.