The influence of interviewer characteristics on support for democracy and political engagement in Sub-Saharan Africa*

This study investigates how an interviewer’s characteristics affect how respondents answer survey questions about democracy and political engagement. I analyze data from the 2008 Afrobarometer surveys, in which 810 interviewers surveyed 27,713 respondents across 20 countries in sub-Saharan Africa. Using these data, I study how interviewer education, age, and gender affect two outcomes: (1) response distributions to attitudinal and behavioral survey questions and (2) the likelihood of respondents saying ‘don’t know’ to a survey question. The analysis also investigates how the respondent’s perception of who sponsored the survey (NGO, private sector, government) affects attitudes. The results show that these interviewer characteristics affect the quality of survey data on political attitudes and behaviors. In the discussion, I consider the implications for research based on public opinion data about democracy and political engagement.     

Which probes are most useful when undertaking cognitive interviews?

This paper reports the use of verbal probes in cognitive interviews (CIs) undertaken to test the usefulness, validity and reliability of survey questions. Through examining the use of probes by three interviewers undertaking interviews as part the piloting of a cross-national crime survey, we examine which of the various types of probes used in CIs produce the most useful information. Other influences on interview quality are examined, including differences between interviewers and respondents themselves. The analyses rely on multi-level modelling and suggest that anticipated, emergent and conditional probes provide the most useful data. Furthermore, age, gender and educational levels appear to have no bearing on the quality of the data generated.     

The effect of interviewer evaluation of respondents’ attitudes and understanding on future survey participation in a panel study

This study explored whether the interviewers’ judgments of respondents’ performances in an initial survey can predict survey participation for a second survey among the same group of respondents in a panel study. Specifically, this study examined two interviewer variables: the interviewers’ judgments of (1) the respondents’ understanding of survey questions and (2) the respondents’ attitudes toward the survey. The analyses revealed that respondents who were rated to have an excellent understanding of the survey questions and friendly attitudes were more likely to respond to the second interview request than those with a poor understanding and unfriendly attitudes, after controlling for the characteristics of the respondents. Of these two, the judgment of the respondents’ attitudes was a stronger predictor. However, interviewer’s judgments are not correlated with the primary survey outcome or the data quality. This suggests limited value of the interviewer observation.     

Reliability and validity of egocentered network data collected via web: A meta-analysis of multilevel multitrait multimethod studies

Our goal in this article is to assess reliability and validity of egocentered network data collected through web surveys using multilevel confirmatory factor analysis under the multitrait multimethod approach.In this study, we analyze a questionnaire of social support of Ph.D. students in three European countries. The traits used are the frequency of social contact questions. The methods used are web survey design variants.We consider egocentered network data as hierarchical; therefore, a multilevel analysis is required. Within and between-ego reliabilities and validities are defined and interpreted.Afterwards, we proceed to a meta-analysis of the results of the three countries where within and between-ego validities and reliabilities are predicted from survey design variables which have to do with question order (by questions or by alters), response category labels (end labels or all labels) and lay-out of the questionnaire (graphical display or plain text). Results show that question order by questions, all-labeled response categories and a graphical display lay-out with images lead to a better data quality. Our basic approach consisting on multilevel and meta-analysis can be applied to evaluate the quality of any type of egocentered network questionnaire, regardless of the data collection mode.     

Propensity to consent to data linkage: experimental evidence on the role of three survey design features in a UK longitudinal panel

When performing data linkage, survey respondents need to provide their informed consent. Since not all respondents agree to this request, the linked data-set will have fewer observations than the survey data-set alone and bias may be introduced. By focusing on the role that survey design features play in gaining respondents’ consent, this paper provides an innovative contribution to the studies in this field. Analysing experimental data collected in a nationally representative household panel survey of the British population, we find that interview features such as question format (dependent/independent questions) and placement of the consent question within the questionnaire have an impact on consent rates.     

Surveying a Chicago public housing development: methodological challenges and lessons learned

During the winter and spring of 2003, the Survey Research Laboratory conducted a survey of residents of the Ida Wells and Madden Park housing developments on Chicago's near-south side. This article reviews existing research findings on using indigenous interviewers and reports the findings about the benefits and challenges of using residents as interviewers versus drawing from a regular pool of interviewers. Specifically, the authors talk about the challenges faced in using indigenous interviewers and then look at some data quality indicators among these two groups of interviewers, including cooperation rates, item nonresponse, and differences in responses to sensitive questions.     

社会调查质量研究:访员臆答与干预效果

本文利用中国家庭动态跟踪调查并行数据和问卷数据,采用事后准实验设计方法,对面访调查中访员臆答及对臆答进行干预的效果进行了分析,指出访员臆答是面访调查中不可忽视的直接影响调查数据质量的因素;访员的职业特征与臆答的发生比有显著关联,计算机辅助面访调查可以有效监测并干预访员臆答;干预可使臆答显著减少,减少的幅度因访员年龄不同而有显著差异,但不因访员性别、教育和职业因素而变化。     

Question Understanding Aid (QUAID): A Web Facility that Tests Question Comprehensibility

When respondents do not understand the meaning of a survey question,they will not supply valid and reliable answers. Survey methodologistsshould therefore benefit from computer tools and other analyticalschemes that help them identify problems with questions withrespect to comprehension difficulty. We developed a Web facilitycalled Question Understanding Aid (QUAID; www.psyc.memphis.edu/quaid.html)that assists survey methodologists in identifying problems withthe wording, syntax, and semantics of questions on questionnaires.The survey methodologist enters the question into the Web facility,along with any context information and answer alternatives thataccompany the question. QUAID quickly returns a list of potentialproblems with question comprehension, including unfamiliar technicalterms, vague or imprecise relative terms, vague or ambiguousnoun phrases, complex syntax, and working memory overload. Thisarticle describes QUAID and some empirical studies that haveassessed the validity and utility of QUAID’s critiquesof questions. The output of QUAID was compared with the judgmentsof experts in language, discourse, and cognition during thedevelopment of the tool. In one evaluation, expert survey methodologistscritiqued and revised problematic questions, whereas in a secondevaluation survey methodologists evaluated the quality of originalproblematic questions, questions revised with the assistanceof QUAID, and questions revised without QUAID. In a third evaluation,eye-tracking data were collected while respondents read questionson a computer screen and answered questions aloud. Respondentshad a tendency to give up processing difficult questions tooearly (called an early exit), which potentially threatens thevalidity of the respondents’ answers. Survey methodologistsare encouraged to use QUAID and further evaluate its validityand utility.     

EFFECTS OF INTERVIEWER GENDER, INTERVIEWER CHOICE, AND ITEM WORDING ON RESPONSES TO QUESTIONS CONCERNING SEXUAL BEHAVIOR

We examined factors influencing responses to questions on sexualbehavior among adult respondents 18–49 years old (unweightedN = 2,030) obtained through a random-digit dialing survey. Basedon self-disclosure and perceived control the ory, we hypothesizedthat giving people a choice in selecting the gender of theirinterviewer rather than being assigned an inter viewer, andusing questions that are "supportive" of what may be perceivedof as nonnormative behavior (enhanced items), would increasedata quality relative to, respectively, matched-or opposite-genderinterviewer conditions and standard worded items. The enhanceditems facilitated responding to a number of sensitive topics.However, the effects of item wording on item response are oftenmediated by interviewer conditions. The "choice" results suggestthat giving respondents greater control decreases question threat.However, the overall findings argue for matching respondentsand interviewers on gender over opposite-gender interviewersor allowing respondents to select their interviewer's gender.Wording and interviewer manipulations reduced the discrepanciesbetween men's and women's self-reports of sexual behavior, butthey did not eliminate them, and in some cases they had no effect.The present findings suggest that males tend to be influencedby variations in item wording, interviewer gender, and respondentcontrol across a somewhat wider range of sexual topics. In general,the findings recommend matching respondents and interviewerson gender and the use of more supportive wording in sexual behaviorquestions. However, for assessment of some topics (e.g., sexualviolence) in particular segments of the population (e.g., men),other procedures, such as increasing respondent control, maybe a better choice. Overall, the data support the view thatin terms of preferred procedures, not all sexual topics arecreated equal.     

Developing a Low‐Cost Technique for Parallel Cross‐Cultural Instrument Development: The Question Appraisal System (QAS‐04)

Many approaches are used to prepare instruments for multicultural administration, depending on the scope, schedule and budget of the study. Sequential questionnaire development, the most common approach to developing cross‐cultural instruments, is also the most affordable. Designers formulate and pretest an instrument in the source language, then translate it into the target language(s) using culture‐specific tailoring. In contrast, parallel development incorporates target cultures throughout the design and pretesting process. The disadvantages to parallel development are that it is expensive, time‐consuming and subject to version control problems. Question Appraisal System (QAS) is a coding tool for pretesting instruments. The QAS is supported by an item taxonomy of the cognitive demands of a question and documents the features that may lead to response error. Results of the appraisal are used to revise question wording, response wording, questionnaire format and question ordering. This article describes research conducted to update the QAS to identify problems due to cross‐cultural and cross‐linguistic application of questions.     

INTERVIEWER GENDER AND GENDER ATTITUDES

This research addresses the effects of interviewer gender onresponses to a broad array of gender-related survey questions,using data from a probability sample of adults in the UnitedStates. We focus on whether gender-of-interviewer effects areevident, whether they vary by respondent gender, and whetherthey vary across several attitudinal domains relevant to genderinequality. While many items do not show statistically significantgender-of-interviewer effects, we document significant effectsacross a variety of items. When such effects are evident, theytend to involve both male and female respondents expressingmore egalitarian gender-related attitudes or greater criticismof existing gender inequalities to female interviewers. Malerespondents offer significantly different responses to maleand female interviewers on questions dealing with gender inequalityin employment. For female respondents, interviewer-gender effectsare evident for items addressing gender-related collective action,policy, and group interests. Using multivariate models thatallow us to represent both respondent-level and interviewer-levelvariables, we find that interviewer-gender effects are statisticallysignificant in most attitudinal domains but that the interactionbetween interviewer gender and respondent gender does not tendto be statistically significant. We consider the implicationsof these findings both for understanding the survey processand for understanding gender relations more generally.     

Ethnicity-of-Interviewer Effects Among Mexican-Americans and Anglos

A general interpersonal deference explanation for ethnicity-of-interviewereffects is tested with a random survey of Anglos and Mexican-Americansusing Anglo and Hispanic interviewers and questions relatedto Mexican-American culture. The theory was most clearly supportedamong Anglo respondents. The mixed support found among Mexican-Americanssuggests that deference is strongest when questions deal withthe culture represented by the interviewer, not the respondent.     

Measuring ego-centered social networks on the web: Questionnaire design issues

Collecting survey data on ego-centered social networks is a difficult task, owing to the complex questionnaire format. Usually, the interviewer handles the dynamics of the question–answer exchange, motivates the respondent and ensures the proper recording of the data. Self-administered modes of data collection, especially web data collection, are more problematic, as the respondents are left alone with a complex and burdensome questionnaire. Therefore, questionnaire layout is crucial for ensuring cooperation and data quality. In this paper we examined three key components of the corresponding web questionnaire: the number of name boxes using a single name generator, question format for assessing alter characteristics (i.e. alter-wise vs. question-wise) and number of name interpreters (i.e. alter characteristics). The number of name boxes was found to be essential for the reported size of social networks and also for some aspects of data quality. Specific data quality effects were also found with respect to variations in question format, where question-wise format performed better than alter-wise. The number of name interpreters had a relatively minor effect. Suggestions for possible standardization of the web interface layout are also given, so that equivalence with other data collection modes can be established.     

Does "Yes or No" on the Telephone Mean the Same as "Check-All-That-Apply" on the Web?

Recent experimental research has shown that respondents to forced-choicequestions endorse significantly more options than respondentsto check-all questions. This research has challenged the commonassumption that these two question formats can be used interchangeablybut has been limited to comparisons within a single survey mode.In this paper we use data from a 2004 random sample survey ofuniversity students to compare the forced-choice and check-allquestion formats across web self-administered and telephoneinterviewer-administered surveys as they are commonly used insurvey practice. We find that the within-mode question formateffects revealed by previous research and reaffirmed in thecurrent study appear to persist across modes as well; the telephoneforced-choice format produces higher endorsement than the webcheck-all format. These results provide further support forthe argument that the check-all and forced-choice question formatsdo not produce comparable results and are not interchangeableformats. Additional comparisons show that the forced-choiceformat performs similarly across telephone and web modes.     

SURVEY QUESTIONS ABOUT THE FREQUENCY OF VAGUELY DEFINED EVENTS: THE EFFECTS OF RESPONSE ALTERNATIVES

Experimental research has shown that the choice of responsealternatives can influence responses to questions about thefrequency of vaguely defined target events (e.g., feeling annoyed);the set of response alternatives is treated as information thatcontributes to the interpretation of the question. In a seriesof split ballot experiments we investigate whether such effectsoccur in a large-scale survey context for sets of response alternativesthat might be used interchangeably by survey researchers. Thepredicted response shifts were found in our field experiments.Those presented with response alternatives discriminating atlow frequencies reported fewer of the target episodes than thosepresented with higher-frequency response alternatives. However,the size of the observed shifts varied from zero to 13 percentand depended on a number of characteristics of the survey, forexample, the presence of "priming" questions and the orientationof the response scales. Response alternatives are not neutraland therefore must be carefully considered when constructinga survey question.     

The Impact of Question and Respondent Characteristics on Comprehension and Mapping Difficulties

Behavior coding is one technique researchers use to detect problemsin survey questions, but it has been primarily explored as apractical tool rather than a source of insight into the theoreticalunderstanding of the cognitive processes by which respondentsanswer survey questions. The latter is the focus of the currentinvestigation. Using data from a large study in which face-to-faceinterviews were taped and extensive behavior coding was done,we tested whether sets of respondent behavior codes could beused to distinguish respondent difficulties with comprehensionof the question from difficulties associated with mapping ajudgment onto the response format provided, and whether characteristicsof the survey questions and respondents could be used to predictwhen and for whom such difficulties would occur. Sets of behaviorcodes were identified that reflected comprehension and mappingdifficulties, and these two types of difficulties were associatedwith different question and respondent characteristics. Thisevidence suggests that behavior coding shows promise as a toolfor researchers studying the cognitive processes involved inanswering survey questions.     

Clarifying question meaning in a household telephone survey

This study contrasts two interviewing techniques that reflectdifferent tacit assumptions about communication. In one, strictlystandardized interviewing, interviewers leave the interpretationof questions up to respondents. In the other, conversationalinterviewing, interviewers say whatever it takes to make surethat questions are interpreted uniformly and as intended. Respondentsfrom a national sample were interviewed twice. Each time theywere asked the same factual questions from ongoing governmentsurveys, five about housing and five about recent purchases.The first interview was strictly standardized; the second wasstandardized for half the respondents and conversational forthe others. Respondents in a second conversational interviewanswered differently than in the first interview more often,and for reasons that conformed more closely to official definitions,than respondents in a second standardized interview. This suggeststhat conversational interviewing improved comprehension, althoughit also lengthened interviews. We conclude that respondentsin a national sample may misinterpret certain questions frequentlyenough to compromise data quality and that such misunderstandingscannot easily be eliminated by pretesting and rewording questionsalone. More standardized comprehension may require less standardizedinterviewer behavior.     

ASSIMILATION AND CONTRAST EFFECTS IN PART-WHOLE QUESTION SEQUENCES: A CONVERSATIONAL LOGIC ANALYSIS

A theoretical model of the emergence of assimilation and contrasteffects in part-whole question sequences is presented. Whenone specific question precedes a general question and the twoare not assigned to the same conversational context, respondentsuse the information primed by the specific question to formthe general judgment. This results in part-whole assimilationeffects. If both questions are perceived as belonging to gether,however, conversational norms of nonredundancy prohibit therepeated use of information that has already been provided inresponse to the specific question when making the general judgment.Accordingly, respondents interpret the general question to referto aspects other than the ones covered by the specific question.Contrast effects may emerge in that case under specified conditions.If several specific questions precede the general question,however, the general one is always interpreted as a requestfor a summary judgment. This results in assimilation effects,even under conditions that would foster contrast effects ifonly one specific question is asked. The model is supportedby experimental data and provides a coherent account of apparentlycontradictory findings previously reported in the survey literature.     

Investigating validity and effectiveness of cognitive interviewing as a pretesting method for non-English questionnaires: Findings from Korean cognitive interviews

This study aims to empirically investigate whether cognitive interviewing is effective as a pretesting method for detecting problems in translated survey questionnaires as in the English language source questionnaire. The validity of non-English cognitive interviews is of concern among researchers because the cognitive interviewing technique was developed based on the communicative norms of Western cultures where directness and openness in expressing one’s opinion is encouraged. However, different communicative norms are present in Asian languages and cultures. To date, little research has been conducted on this topic. Using qualitative and quantitative analysis, we compared survey question problems identified through the cognitive interviews conducted in English and in Korean and found that Korean cognitive interviewing was indeed effective for detecting problematic survey questions.     

Context Effects on Survey Responses to Questions About Abortion

The proportion of favorable responses to a general questionabout abortion was significantly smaller in an NORC survey thanwere responses to the same item in an SRC survey. We hypothesizedthat questionnaire context was the main source of the difference–inone survey the general item followed a more specific questionabout abortion–and carried out two split-ballot questionorder experiments that provide strong support for the hypothesis.The context effect seems to arise from the generality of theabortion question; the specific question itself showed no changedue to context. Furthermore, there is some evidence that ambivalencetoward the issue of abortion is a factor in the sensitivityof the general question to context. The context effect in theseexperiments is also notable for producing contrast rather thanconsistency between responses to adjacent questions on the sameissue. Implications of the findings for the study of socialchange are emphasized.