Sex differences in how heterosexuals think about lesbians and gay men: Evidence from survey context effects

Two experiments were embedded in a 1997 telephone survey of U.S. households to assess possible differences in how heterosexuals think about lesbians versus gay men. In each experiment, one half of the sample first responded to one or more attitude items about lesbians, followed by comparable items about gay men. The other half received the gay male item(s) first. Results are reported separately for White (N = 976) and Black (N = 479) heterosexuals. For White and Black men alike, self‐reported attitudes toward lesbians tended to be more favorable when they were assessed without reference to gay men (i.e., lesbian items presented first). White men's reactions to gay men tended to be less negative when assessed after the questions about lesbians were presented, but Black men's responses did not consistently show this pattern. For some items, women gave more favorable ratings of lesbians and less favorable ratings of gay men when the lesbian items were presented first. The findings suggest possible gender differences in the cognitive organization of heterosexuals ‘ attitudes toward lesbians and gay men.     

Investigating the impact of violations of the “left and top means first” heuristic on response behavior and data quality

ABSTRACT

Web surveys are an established data collection mode that use written language to provide information. The written language is accompanied by visual elements, such as presentation formats and shapes. However, research has shown that visual elements influence response behavior because respondents sometimes use interpretive heuristics to make sense of the visual elements. One such heuristic is the ‘left and top means first’ (LTMF) heuristic, which suggests that respondents tend to believe that a response scale consistently runs from left to right or from top to bottom. We conducted a web survey experiment to investigate how violations of the LTMF heuristic affect response behavior and data quality. For this purpose, a random half of respondents received response options that followed a consistent order and the other half received response options that followed an inconsistent order. The results reveal significantly different response distributions between the two groups. We also found that inconsistently ordered response options significantly increase response times and decrease data quality in terms of criterion validity. We, therefore, recommend using options that follow the design strategies of the LTMF heuristic.     

Question Understanding Aid (QUAID): A Web Facility that Tests Question Comprehensibility

When respondents do not understand the meaning of a survey question,they will not supply valid and reliable answers. Survey methodologistsshould therefore benefit from computer tools and other analyticalschemes that help them identify problems with questions withrespect to comprehension difficulty. We developed a Web facilitycalled Question Understanding Aid (QUAID; www.psyc.memphis.edu/quaid.html)that assists survey methodologists in identifying problems withthe wording, syntax, and semantics of questions on questionnaires.The survey methodologist enters the question into the Web facility,along with any context information and answer alternatives thataccompany the question. QUAID quickly returns a list of potentialproblems with question comprehension, including unfamiliar technicalterms, vague or imprecise relative terms, vague or ambiguousnoun phrases, complex syntax, and working memory overload. Thisarticle describes QUAID and some empirical studies that haveassessed the validity and utility of QUAID’s critiquesof questions. The output of QUAID was compared with the judgmentsof experts in language, discourse, and cognition during thedevelopment of the tool. In one evaluation, expert survey methodologistscritiqued and revised problematic questions, whereas in a secondevaluation survey methodologists evaluated the quality of originalproblematic questions, questions revised with the assistanceof QUAID, and questions revised without QUAID. In a third evaluation,eye-tracking data were collected while respondents read questionson a computer screen and answered questions aloud. Respondentshad a tendency to give up processing difficult questions tooearly (called an early exit), which potentially threatens thevalidity of the respondents’ answers. Survey methodologistsare encouraged to use QUAID and further evaluate its validityand utility.     

Effects of an advance telephone call in a personal interview survey.

During October and November 1982, 1,260 Medicare-eligible senior citizens were interviewed in a survey focusing on health care of the elderly. As part of the survey, an experiment was conducted in each of the three survey sites to determine the effects of an advance telephone call to schedule an appointment for a personal interview. One random half sample in each site was sent a lead letter, followed by a telephone call to schedule a personal interview. The other half sample was sent a lead letter followed by a personal contact, with no intervening telephone call. Telephoning to arrange an appointment for a personal interview resulted in a 20 percent saving in data collection costs with only a 1 percent decrease in response rate.     

An Experiment with a Confidentiality Reminder in a Telephone Survey

The effect of a confidentiality reminder was tested during atelephone survey of the general public of a metropolitan area.One-half of the sample was administered the reminder just priorto the demographic questions; the other half of the sample didnot receive the treatment. The results showed no significantdifferences on the distribution of nonsubstantive responsesbetween the two groups. There was some evidence that the confidentialityreminder had a negative effect on data quality.     

Web Survey Design: Paging versus Scrolling

A key choice in the design of Web surveys is whether to placethe survey questions in a multitude of short pages or in longscrollable pages. There are advantages and disadvantages ofeach approach, but little empirical evidence to guide the choice.In 2003 we conducted a survey of over 21,000 undergraduate students.Ten percent of the 10,000 respondents were directed to the scrollableversion of the survey, containing a single form for each ofthe major sections. The balance was assigned to the paging version,in which questions were presented to be visible without scrolling.The instrument contained a maximum of 268 possible questions,including topics that varied in sensitivity and desirability.The survey also permitted comparison of the effect of skip patternsby implementing skip instructions and hyperlinks in the scrollabledesign, and also recorded time at the end of each of the fivetopical sections. Differences between designs are evaluatedin terms of various forms of nonresponse, univariate and bivariatemeasurement properties, and proxies for respondent burden.     

Spacing, Position, and Order: Interpretive Heuristics for Visual Features of Survey Questions

We present the results of six experiments that demonstrate theimpact of visual features of survey questions on the responsesthey elicit, the response process they initiate, or both. Allsix experiments were embedded in Web surveys. Experiments 1and 2 investigate the effects of the placement of nonsubstantiveresponse options (for example, "No opinion" and "Don’tknow" answer options) in relation to the substantive options.The results suggest that when these options are not differentiatedvisually (by a line or a space) from the substantive options,respondents may be misled about the midpoint of the scale; respondentsseemed to use the visual rather than the conceptual midpointof the scale as a reference point for responding. Experiment3, which varied the spacing of the substantive options, showeda similar result. Responses were pushed in the direction ofthe visual midpoint when it fell to one side of the conceptualmidpoint of the response scale. Experiment 4 examined the effectsof varying whether the response options, which were arrayedvertically, followed a logical progression from top to bottom.Respondents answered more quickly when the options followeda logical order. Experiment 5 examined the effects of the placementof an unfamiliar item among a series of similar items. For example,one set of items asked respondents to say whether several makesand models of cars were expensive or not. The answers for theunfamiliar items depended on the items that were nearby on thelist. Our last experiment varied whether a battery of relateditems was administered on a single screen, across two screens,or with each item on its own screen. The intercorrelations amongthe items were highest when they were all on the same screen.Respondents seem to apply interpretive heuristics in assigningmeaning to visual cues in questionnaires. They see the visualmidpoint of a scale as representing the typical or middle response;they expect options to be arrayed in a progression beginningwith the leftmost or topmost item; and they expect items thatare physically close to be related to each other conceptually.     

Comparison of web and mail surveys in collecting illicit drug use data: a randomized experiment

This randomized experiment examined survey mode effects for self-reporting illicit drug use by comparing prevalence estimates between a Web-based survey and a mail-based survey. A random sample of 7,000 traditional-aged undergraduate students attending a large public university in the United States was selected to participate in the spring of 2001. Students were randomly assigned to self-administer a survey via the Web or U.S. mail. The Web survey produced a significantly higher response rate than the U.S. mail survey. The prevalence estimates of illicit drug use (lifetime and past year) did not differ significantly between the two survey modes. The findings provide preliminary evidence that Web and U.S. mail surveys produce similar results regarding illicit drug use among undergraduate students. Although additional research is needed involving more diverse samples, these findings bode well for using Web surveys in college-based research.     

DNA COLLECTION IN A RANDOMIZED SOCIAL SCIENCE STUDY OF COLLEGE PEER EFFECTS

We describe the DNA collection processes of an initial pilot and full study, which is designed to investigate joint peer and genetic effects on health behaviors and attitudes in a college campus setting. In the main study, 2664 (79.5%) students completed a Web survey and 2080 (78.7% of the survey completers after adjusting for the ineligible) provided a saliva DNA sample. The response rate for completing both the survey and the DNA portion of the study is 62.5%. Our DNA yields are of high quality. Overall, our experiences and results demonstrate that genetic data can be successfully collected as a part of traditional social science survey research projects. To aid others in doing so, we provide extensive details of our data collection experiences and offer recommendations to future researchers seeking to do or evaluate similar work.     

Presentation matters: how mode effects in item non-response depend on the presentation of response options

Comparability of data across modes is an important issue in survey research. In this paper we discuss item non-response to attitudinal questions in telephone and web surveys. We present results from a survey experiment conducted in Italy and in Spain that compares different presentations of response options in an online setting with a benchmark telephone survey. In line with earlier studies we find that (A), the share of non substantial answers in the online survey depends on how the response option is presented. Comparing different presentations in an online survey to the standard approach of telephone surveys using propensity score matching, we find that (B), the share and pattern of non substantial answers, is most similar across the two modes in the online survey when the it is captured in an instruction on each screen for the online survey. Our findings are of particular relevance for the design of multiple mode or mix-mode surveys of attitudinal questions using online and telephone modes.     

Eating disorders in graduate students: exploring the SCOFF questionnaire as a simple screening tool

The results of several studies have established the validity of the SCOFF questionnaire (a 5-question screening tool for eating disorders), but researchers need to explore further replicability using the US version in the graduate school population. In this study, the authors asked 335 graduate students attending the Northwestern student health clinic on the Chicago campus to complete a written survey anonymously. A total of 305 (91%) patients completed the survey. The sensitivity and specificity for the SCOFF was 53.3% and 93.2%, respectively. This produced a PPV (the proportion who tested positive on the screen and actually had an eating disorder) of 66.7% and an NPV (the proportion of those who tested negative on the screen and actually did not have an eating disorder) of 88.7%. More than 80% of respondents were dissatisfied with their shape and weight, with over one third having a moderate to severe body image disturbance. The SCOFF is an easy instrument to administer that health care providers can use to screen for eating disorders in the primary care setting.     

Management of Allergic Respiratory Disease by the Nurse Practitioner

The results of several studies have established the validity of the SCOFF questionnaire (a 5-question screening tool for eating disorders), but researchers need to explore further replicability using the US version in the graduate school population. In this study, the authors asked 335 graduate students attending the Northwestern student health clinic on the Chicago campus to complete a written survey anonymously. A total of 305 (91%) patients completed the survey. The sensitivity and specificity for the SCOFF was 53.3% and 93.2%, respectively. This produced a PPV (the proportion who tested positive on the screen and actually had an eating disorder) of 66.7% and an NPV (the proportion of those who tested negative on the screen and actually did not have an eating disorder) of 88.7%. More than 80% of respondents were dissatisfied with their shape and weight, with over one third having a moderate to severe body image disturbance. The SCOFF is an easy instrument to administer that health care providers can use to screen for eating disorders in the primary care setting.     

Effect of the content of a news item on Spaniards’ attitudes towards Palestinians and Israelis / Efecto del contenido de una noticia sobre la actitud de los españoles hacia palestinos e israelíes

Based on the effects of framing on audience opinion, the attitudes towards Palestinians and Israelis are evaluated when a news item reports on a violent event between the two sides. To do so, a short inter-group experiment was conducted. A 3 (manipulation of the news item) x 2 (group evaluated) design was applied. Three news formats were presented: (1) about a Palestinian attack on Israel; (2) about an Israeli attack on Palestine; (3) no news item. Half of the participants evaluated the Israelis while the other half evaluated the Palestinians. The results of the study, which was performed with a Spanish sample (N = 319), showed that there were no differences in the evaluation of the group according to the group evaluated, although there were differences according to the manipulation of the news item: in both conditions in which one group attacked another, the group evaluated was rated more negatively than the control group. The results also showed that both emotions and stereotypes mediated the effects of framing on prejudice. These results suggest that the participants’ attitude seems to depend more on the violent framing of the news items than on the role assigned to each group and that emotional processes and stereotypes are what regulate these attitudes.     

Visual Context Effects in Web Surveys

There are many examples of context effects in survey measurement.Responses to survey questions can be shaped by the order ofquestions, the format of response options, the broader surveyenvironment, and so on. For Web surveys, the inclusion of visualimages is a trivial design issue, but may have consequencesfor the responses obtained because they change the visual context.We report a series of experiments examining how responses maybe affected by the use of images in Web surveys. Specifically,we examine the effect that pictures of a healthy woman exercisingversus a sick woman in a hospital bed have on self-rated health.We replicated the experiments in three different surveys, varyingsuch factors as the size and placement of the image and thelocation of the question within the questionnaire. In general,we find that when exposed to a picture of a fit woman, respondentsconsistently rate their own health lower than when exposed toa picture of a sick woman.     

Predictors of Cycling in College Students

Abstract

Objectives: To (1) assess cycling-related questions that have been added to the electronic version of the American College Health Association National College Health Assessment II (ACHA-NCHA), (2) examine cycling prevalence, and (3) identify predictors of cycling in college students. Participants: Predominately female (69%), undergraduate (89%), and white (85%) students (N = 949) from a large, urban, northwestern, bicycle-friendly university completed the electronic version of the ACHA-NCHA II. Methods: Thirty cycling-related questions were added to the ACHA-NCHA II and a subsample of questions was analyzed. Results: Cycling questions added to the ACHA-NCHA II scale were reliable and valid, based on the psychometric data analysis. More than half (59%) of this sample cycled; of those, 58% cycled for transportation and 44% for recreation. Facilitators and barriers to cycling were different for cycling in general and cycling for transportation. Conclusions: Cycling questions added to the ACHA-NCHA II can be utilized to enhance knowledge relative to cycling on college campuses.     

Estimating network properties from snowball sampled data

This article addresses the estimation of topological network parameters from data obtained with a snowball sampling design. An approximate expression for the probability of a vertex to be included in the sample is derived. Based on this sampling distribution, estimators for the mean degree, the degree correlation, and the clustering coefficient are proposed. The performance of these estimators and their sensitivity with respect to the response rate are validated through Monte Carlo simulations on several test networks. Our approach has no complex computational requirements and is straightforward to apply to real-world survey data. In a snowball sample design, each respondent is typically enquired only once. Different from the widely used estimator for Respondent-Driven Sampling (RDS), which assumes sampling with replacement, the proposed approach relies on sampling without replacement and is thus also applicable for large sample fractions. From the simulation experiments, we conclude that the estimation quality decreases with increasing variance of the network degree distribution. Yet, if the degree distribution is not to broad, our approach results in good estimates for the mean degree and the clustering coefficient, which, moreover, are almost independent from the response rate. The estimates for the degree correlation are of moderated quality.     

WebQuests and Web 2.0 Screen Design

This study compared the designs of a traditional style WebQuest and a Web 2.0 style WebQuest in terms of their effectiveness as a teaching tool. The sample included 104 university sophomore students. Students were randomly assigned to two groups, with one group using the traditional style WebQuest and the other used the Web 2.0 style WebQuest. Data were collected (a) using a test developed by the author and (b) by conducting unstructured interviews. The results showed that the mean score of the students using the Web 2.0 style WebQuest was significantly higher than the traditional style WebQuest group on a test measuring the understanding of the WebQuest content. Furthermore, interview results indicated that while students from both groups enjoyed the activity, the responses from the Web 2.0 style group were more positive.     

Analyzing Web pages visual scanpaths: between and within tasks variability

In this paper, we propose a new method for comparing scanpaths in a bottom-up approach, and a test of the scanpath theory. To do so, we conducted a laboratory experiment in which 113 participants were invited to accomplish a set of tasks on two different websites. For each site, they had to perform two tasks that had to be repeated ounce. The data were analyzed using a procedure similar to the one used by Duchowski et al. [8]. The first step was to automatically identify, then label, AOIs with the mean-shift clustering procedure [19]. Then, scanpaths were compared two by two with a modified version of the string-edit method, which take into account the order of AOIs visualizations [2]. Our results show that scanpaths variability between tasks but within participants seems to be lower than the variability within task for a given participant. In other words participants seem to be more coherent when they perform different tasks, than when they repeat the same tasks. In addition, participants view more of the same AOI when they perform a different task on the same Web page than when they repeated the same task. These results are quite different from what predicts the scanpath theory.     

Eye-Tracking Data: New Insights on Response Order Effects and Other Cognitive Shortcuts in Survey Responding

Survey researchers since Cannell have worried that respondents may take various shortcuts to reduce the effort needed to complete a survey. The evidence for such shortcuts is often indirect. For instance, preferences for earlier versus later response options have been interpreted as evidence that respondents do not read beyond the first few options. This is really only a hypothesis, however, that is not supported by direct evidence regarding the allocation of respondent attention. In the current study, we used a new method to more directly observe what respondents do and do not look at by recording their eye movements while they answered questions in a Web survey. The eye-tracking data indicate that respondents do in fact spend more time looking at the first few options in a list of response options than those at the end of the list; this helps explain their tendency to select the options presented first regardless of their content. In addition, the eye-tracking data reveal that respondents are reluctant to invest effort in reading definitions of survey concepts that are only a mouse click away or paying attention to initially hidden response options. It is clear from the eye-tracking data that some respondents are more prone to these and other cognitive shortcuts than others, providing relatively direct evidence for what had been suspected based on more conventional measures.     

The impact of schools self-administering substance abuse surveys: an evaluation

The literature suggests that administering drug surveys to public school students is best done by persons outside of the school system (or at least unfamiliar to the students). This is the approach used by the long-time Monitoring the Future project. Because of the increased costs that administration by outside research associates requires (both time and transportation), we decided to design an experiment that would compare it to administration by school personnel. Six school systems were split, and 3756 surveys were administered, about half by outside research associates and half by school personnel. The results indicate that there were minimal and non-significant differences between the two groups. On the basis of these results, we draw the conclusion that school personnel are able to administer survey instruments without affecting the quality of the results and, thus, that school districts may safely use such data in the evaluation of their drug education and drug prevention programs.