How unclear terms affect survey data

Although writing clear questions is accepted as a general goal in surveys, procedures to ensure that each key term is consistently understood are not routine. Researchers who do not adequately test respondent understanding of questions must assume that ambiguity will not have a large or systematic effect on their results. Seven questions that were drawn from questions used in national health surveys were subjected to special pretest procedures and found to contain one or more poorly defined terms. When the questions were revised to clarify the definition of key terms, significantly different estimates resulted. The implication is that unclear terms are likely to produce biased estimates. The results indicate that evaluation of survey questions to identify key terms that are not consistently understood and defining unclear terms are ways to reduce systematic error in survey measurement.     

SURVEY QUESTIONS ABOUT THE FREQUENCY OF VAGUELY DEFINED EVENTS: THE EFFECTS OF RESPONSE ALTERNATIVES

Experimental research has shown that the choice of responsealternatives can influence responses to questions about thefrequency of vaguely defined target events (e.g., feeling annoyed);the set of response alternatives is treated as information thatcontributes to the interpretation of the question. In a seriesof split ballot experiments we investigate whether such effectsoccur in a large-scale survey context for sets of response alternativesthat might be used interchangeably by survey researchers. Thepredicted response shifts were found in our field experiments.Those presented with response alternatives discriminating atlow frequencies reported fewer of the target episodes than thosepresented with higher-frequency response alternatives. However,the size of the observed shifts varied from zero to 13 percentand depended on a number of characteristics of the survey, forexample, the presence of "priming" questions and the orientationof the response scales. Response alternatives are not neutraland therefore must be carefully considered when constructinga survey question.     

Presentation matters: how mode effects in item non-response depend on the presentation of response options

Comparability of data across modes is an important issue in survey research. In this paper we discuss item non-response to attitudinal questions in telephone and web surveys. We present results from a survey experiment conducted in Italy and in Spain that compares different presentations of response options in an online setting with a benchmark telephone survey. In line with earlier studies we find that (A), the share of non substantial answers in the online survey depends on how the response option is presented. Comparing different presentations in an online survey to the standard approach of telephone surveys using propensity score matching, we find that (B), the share and pattern of non substantial answers, is most similar across the two modes in the online survey when the it is captured in an instruction on each screen for the online survey. Our findings are of particular relevance for the design of multiple mode or mix-mode surveys of attitudinal questions using online and telephone modes.     

Question Order and Fair Play: Evidence of Even-Handedness in Rural Surveys1

The norm of even-handedness is a question-order effect known to occur in face-to-face interviews when respondents favor one of two complementary or competing opinion questions, both written at the same level of specificity. After defining and illustrating the concept, evidence of even-handedness in opinion questions is explored in a Montana mail survey and in recall of recent behavior in Iowa telephone interviews. The Montana survey provides evidence of education effects that are not consistent with previous theories of cognitive sophistication, suggesting that future research must consider the interaction between substantive issues and cognitive processes. The Iowa poll provides evidence that respondents' recall of their recent behaviors and the behaviors of their partner also are subject to context effects. This poll also provides evidence that even-handedness operates independently of social desirablity, especially among women.     

Letter to the Editor

In a telephone survey 18 new social work educators responded to questions about their perceptions of mentoring in their first years as faculty members. Results indicate that new social work faculty believe mentoring relationships provide them with multiple benefits, including improved teaching and research performance. Data also suggest that programs must be carefully developed and supported if protégés, mentors, and their organizations are to fully realize these benefits. Factors such as mentor-protégé matching, mentor characteristics, the roles of mentors, organizational support, and the mentoring process must be considered if a successful mentoring program is to be implemented.     

Survey Participation, Nonresponse Bias, Measurement Error Bias, and Total Bias

A common hypothesis about practices to reduce survey nonresponseis that those persons brought into the respondent pool throughpersuasive efforts may provide data filled with measurementerror. Two questions flow from this hypothesis. First, doesthe mean square error of a statistic increase when sample personswho are less likely to be contacted or cooperate are incorporatedinto the respondent pool? Second, do nonresponse bias estimatesmade on the respondents, using survey reports instead of records,provide accurate information about nonresponse bias? Using aunique data set, the Wisconsin Divorce Study, with divorce recordsas the frame and questions about the frame information includedin the questionnaire, this article takes a first look into thesetwo issues. We find that the relationship between nonresponsebias, measurement error bias, and response propensity is statistic-specificand specific to the type of nonresponse. Total bias tends tobe lower on estimates calculated using all respondents, comparedwith those with only the highest contact and cooperation propensities,and nonresponse bias analyses based on respondents yield conclusionssimilar to those based on records. Finally, we find that errorproperties of statistics may differ from error properties ofthe individual variables used to calculate the statistics.     

Missing Data in Sociological Research: An Overview of Recent Trends and an Illustration for Controversial Questions, Active Nonrespondents and Targeted Samples

In an age of telemarketers, spam emails, and pop-up advertisements, sociologists are finding it increasingly difficult to achieve high response rates for their surveys. Compounding these issues, the current political and social climate has decreased many survey respondents’ likelihood of responding to controversial questions, which are often at the heart of much research in the discipline. Here we discuss such implications for survey research in sociology using: a content analysis of the prevalence of missing data and survey research methods in the most cited articles in top sociology journals, a case study highlighting the extraction of meaningful information through an example of potential mechanisms driving the non-random missing data patterns in the Religion Among Academic Scientists dataset, and qualitative responses from non-responders in this same case. Implications are likely to increase in importance given the ubiquitous nature of survey research, missing data, and privacy concerns in sociological research.     

Social Survey Research1 in Japanese Sociology

Abstract: In this paper, we look at the history of social survey development in Japanese sociology. First, the history of social research in Japan before World War II is explored. Second, the introduction of survey research to Japan during the American occupation after World War II is examined, and third, the present state and roles of social survey research in Japanese sociology is discussed. Social research was introduced as an administrative tool for the government. Sociology and social research were developed under British empiricism and American pragmatism, but Japanese academia has been based on a metaphysical approach. Social research introduced as a practical tool long had difficulty in being accepted by Japanese academia. For this reason, most sociologists in universities did not use social survey research for practical purposes, but pursued qualitative methodologies for analyzing data to gain academic prestige even after Social Stratification and Mobility (SSM) and Sabro Yasuda's research projects spread social survey methods in the field of Japanese sociology. Such academics did not think that findings acquired through qualitative case studies had to be confirmed through quantitative data to serve a practical purpose, nor did they believe that quantitative data could be better understood when examined along side qualitative data. Social survey methods have been opposed by those who have favored case‐study analysis methods in Japanese sociology. Needless to say, this opposition is fruitless. I propose that professional sociologists in Japanese universities should use social survey research for practical problems more frequently. This is the best way to establish sociology and social research as a science in Japanese society.     

Eye-tracking methodology: exploring the processing of question formats in web surveys

In social research, the use of agree/disagree (A/D) questions is a popular method for measuring attitudes. Research has shown that A/D questions require complex cognitive processing and are susceptible to response bias. Thus, some researchers recommend the use of item-specific (IS) questions. This study examines the processing of A/D and IS questions, using eye-tracking methodology. By recording respondents’ eye movements, how respondents process survey questions can be evaluated. The results reveal that IS questions cause more and longer fixations. However, this only applies to the response categories. There are no differences regarding the question stems. Altogether, it seems that IS response categories trigger deeper cognitive processing than A/D response categories.     

Does "Yes or No" on the Telephone Mean the Same as "Check-All-That-Apply" on the Web?

Recent experimental research has shown that respondents to forced-choicequestions endorse significantly more options than respondentsto check-all questions. This research has challenged the commonassumption that these two question formats can be used interchangeablybut has been limited to comparisons within a single survey mode.In this paper we use data from a 2004 random sample survey ofuniversity students to compare the forced-choice and check-allquestion formats across web self-administered and telephoneinterviewer-administered surveys as they are commonly used insurvey practice. We find that the within-mode question formateffects revealed by previous research and reaffirmed in thecurrent study appear to persist across modes as well; the telephoneforced-choice format produces higher endorsement than the webcheck-all format. These results provide further support forthe argument that the check-all and forced-choice question formatsdo not produce comparable results and are not interchangeableformats. Additional comparisons show that the forced-choiceformat performs similarly across telephone and web modes.     

The Role of Migration Policy Changes in Europe for Return Migration to Senegal

This study questions the role of migration policy changes in France, Italy, and Spain for return migration to Senegal, by analyzing biographic data from the Migration between Africa and Europe (MAFE‐Senegal) survey and the contextual data of the Determinants of International Migration (DEMIG) VISA and DEMIG POLICY databases that cover major changes in migration policies in these destination countries for the different categories of migrants. Event history logistic regressions reveal that Senegalese migrants are less likely to return when the entry restrictions have become tighter. This result suggests that the decision to return depends on the possibility of migrating again after the return, which is crucial for both theory and policy regarding Western democracies' attempts to regulate migration.     

Supplementing a survey with respondent Twitter data to measure e-cigarette information exposure

ABSTRACT

Social media data are increasingly used by researchers to gain insights on individuals’ behaviors and opinions. Platforms like Twitter provide access to individuals’ postings, networks of friends and followers, and the content to which they are exposed. This article presents the methods and results of an exploratory study to supplement survey data with respondents’ Twitter postings, networks of Twitter friends and followers, and information to which they were exposed about e-cigarettes. Twitter use is important to consider in e-cigarette research and other topics influenced by online information sharing and exposure. Further, Twitter metadata provide direct measures of user’s friends and followers as opposed to survey self-reports. We find that Twitter metadata provide similar information to survey questions on Twitter network size without inducing recall error or other measurement issues. Using sentiment coding and machine learning methods, we find Twitter can elucidate on topics difficult to measure via surveys such as online expressed opinions and network composition. We present and discuss models predicting whether respondents’ tweet positively about e-cigarettes using survey and Twitter data, finding the combined data to provide broader measures than either source alone.     

Confessions of a Jobbing Researcher

The relationship between research and policy making is complex and not always direct. This paper describes disability-related research for policy makers which is carried out at the Social Policy Research Unit (SPRU), University of York. A distinction between the ways in which research is formed and the ways in which it is informed is made. The informing of research influences how it is carried out; the research questions themselves, the choice of design and methods, the form of analysis, and the pattern of dissemination. The seminar series has helped SPRU reappraise its disability-related research. Disabled people must be able to influence the framing and elaboration of research questions more directly; there must be an increased commitment to employing and to training disabled researchers; dissemination must be wider; 'crucial gateways' to policy makers and practitioners have to be identified; and the debate on informal care must be moved further on.     

Evaluating the Quality of Online Survey Data Collected in 2018 in the USA: Univariate,Bivariate, and Multivariate Analyses

Collecting survey data in Japan and the USA is difficult for many reasons, including the difficulty of creating complete sampling frames and low response rates. Online surveys, which collect data through online access panels and recruit members using nonprobability sampling, is becoming a popular alternative to traditional survey techniques, both in market and social science research. However, sociological researchers are hesitant to adopt online survey data collection techniques as it is unclear how representative such data are. This article examines the quality of online survey data by comparing online surveys with the two datasets collected using probability sampling, the American Community Survey and the General Social Survey. We conducted univariate, bivariate, and multivariate analyses and investigated whether similarities improve from the univariate analyses to bivariate relationships, and to bivariate relationships with controls. The results show that similarities improved from univariate to bivariate and multivariate analyses, but the results of bivariate and multivariate analyses did not differ. We concluded that data collected via volunteer‐based access panels yielded similar results to the benchmark data, especially when the relationships among variables are examined. We argue that automatically discrediting the data collected via volunteer‐based access panels is not good practice. With careful considerations of data quality, online survey data with nonprobability sampling could advance sociological research by giving researchers more freedom to ask questions at a relatively low cost, with a quick turnaround time, and more opportunities to conduct cross‐national research.     

Who Is Adopted?

ABSTRACT

There are several advantages of using large representative samples to study adoption-related issues, and existing databases are increasingly used for this purpose. Large scale surveys are often limited, however, in the number and type of questions that are asked regarding adoption. Conceptual and methodological issues in measuring adoption status are discussed, and a rich new database-the National Longitudinal Survey of Adolescent Health (Add Health)-is used to illustrate these complexities. In Add Health school survey data, adolescents were asked a direct question about whether or not they were adopted and whether they lived with biological parents. During home interviews, adolescents were not asked about adoption status per se, but they could be classified as adopted if they lived with adoptive rather than biological parents. Parents' and adolescents' reports of adoption status and living arrangements are compared in Add Health data to illustrate the complexities of measuring adoption status when using archival data. For research to be scientifically credible and to enhance the accumulation of knowledge, investigators need to be precise about the concepts and measures they use in studying adoption-related issues, particularly when using previously collected survey data.     

U.S. Data Sources on the Foreign Born and Immigration

Migration data can be divided into two broad types: “stock” or census and survey data and “flow” or administrative data. Both stock and flow data are valuable resources for analyzing the migration process. In the statistical system of the United States, the U.S. Census Bureau is the primary source for census and survey data on the foreign born. The Department of Homeland Security and the U.S. State Department provide several different administrative sources for studying immigration. The goal of this study is to review the best sources of government data available for analyzing (1) the size, distribution, and characteristics of the foreign‐born population and their households and (2) the level of immigration into the United States, and the distribution and characteristics of immigrants by status.     

Integrating qualitative and quantitative methods in survey research on aging: An assessment

The benefits of triangulation have gained widespread acceptance and several notable studies have taken a combined-methods approach. However, to date there have been few analyses of attempts to extend survey research to include a qualitative component. This paper reports on what is essentially a survey research design in which three types of interviewing are triangulated: close-ended questions in a structured interview; open-ended questions embedded in the interview; and follow-up conversational interviews employing open-ended questions. This combined-methods approach is assessed in relation to elaboration of answers, interpretation and context, discovery in the field, and divergence. The author concludes that each source of data, as well as the divergence in findings among them, makes a valuable contribution to the research process.     

Gender and Sexual Motivation

The literature on sexual motivation consistently depicts males as primarily motivated by physical and women by emotional factors. Supporting research has been almost exclusively limited to predominantly unmarried teenagers and college aged students, implicitly assuming the absence of age differences in sexual motivation. There has also been a tendency to infer personal values from stated motivations. The present study questions these two assumptions by analyzing two questions included in a survey of 179 adults aged 22 to 57 years, one on respondents' usual motive for engaging in sexual intercourse and anoth6r on their assessment of its most important benefit. An Analysis of Variance reveals significant gender differences in usual motivation but nor in ideal benefits and significant interactions between gender and age in both usual and ideal motivations. Further, we found only a moderate relationship between usual motive for engaging in sexual intercourse and respondent's assessment of its most important benefit. These findings lead us to conclude that caution must be used in making generalizations regarding motives for engaging in sexual intercourse either across age or from sexual realities to sexual ideals.     

Migration and the population census in Asia and the Pacific: issues,questions and debate

Skeldon discusses the main issues to be resolved when designing questions and strategies to collect migration-related data. The strengths and weaknesses of the various approaches are assessed in the light of the data collected by countries in the Asia-Pacific region during the 1980 round of Censuses. Each country has its own particular needs and must devise its questions accordingly. However, there are a few general principles that are likely to improve the quality of the basic data collected. Some of these principles follow. 1) Detailed information on migration can only be collected through specialized surveys. The census data must be able to provide information on the basic spatial and temporal structures of migration and only upon these can detailed subsequent work proceed. 2) De facto enumeration will not only provide a better quality of information, but will also capture a higher proportion of total human mobility than de jure enumeration. A question of last previous residence and a finely coded duration of residence question are likely to generate the most useful migration data. For reasons of simplicity, continuity, and utility, the question on birthplace is a fundamental census question. 3) In order not to overload census questionnaires, additional information on migration is best collected either through specialized census sample modules or through separate sample surveys.     

大学生兼职现象及影响因素研究

目前,大学生兼职已是一种普遍现象,对大学生社会化起到促进作用,但一系列负面现象相伴而生。对3469名大学生的问卷调查发现,学生兼职行为受到个人特征、经济特征、学校机会及地区特征等因素影响,兼职的负面影响对大学生成长不利,因此高校应合理引导和管理大学生兼职行为。