Complex sample data recommendations and troubleshooting

Complex survey data, as highlighted in this issue of Evaluation Review, provide a wealth of opportunities for answering methodological and/or applied research questions. However, the analytic issues of nonindependence and unequal selection probability must be addressed when analyzing this type of data. Thus, to ensure that research questions are accurately answered when using complex survey data, researchers must take extra precautions to ensure complex survey data are correctly analyzed. The purpose of this article is to provide software recommendations for analysis of and tips on troubleshooting when analyzing complex sample data.     

Assessing comprehension in a survey of public reactions to complex issues

This paper reports results of research to develop householdinterview procedures for obtaining reasonably informed publicjudgments about ethical issues in biomedical research. We describe(1) an interviewing strategy designed to enhance comprehensionamong survey respondents, and (2) a composite comprehensionindex based on three types of information from the interview.Using data from a cross-section sample, we identify subgroupsthat differ significantly with respect to scores on the comprehensionindex, and we then describe the observed levels of comprehensionin terms of the operations defining the index. The results demonstraleboth the feasibility and limits of obtaining informed ratherthan naive judgments about complex issues from lay respondents.     

Response Effects in the Electronic Survey

This report examines the electronic survey as a research tool.In an electronic survey, respondents use a text processing programto self-administer a computer-based questionnaire. As more peoplehave access to computers, electronic surveys may become widespread.The electronic survey can reduce processing costs because itautomates the transformation of raw data into computer-readableform. It can combine advantages of interviews (e.g., prompts,complex branching) with those of paper mail surveys (e.g., standardization,anonymity). An important issue is how the electronic surveyaffects the responses of people who use it. We conducted anexperimental sample survey on health attitudes, behaviors, andpersonal traits using two forms of administration: electronicand paper mail. Closed-end responses in the electronic surveywere less socially desirable and tended to be more extreme thanwere responses in the paper survey. Open-ended responses thatcould be edited by respondents were relatively long and disclosing.These findings are consistent with other research on computer-mediatedcommunication, raising general issues about using computersto collect self-report data.     

Studying marriage and family therapists in the 21st century: methodological and technological issues

In this article, I present data from two waves of research on demographic characteristics and practice patterns of marriage and family therapists (MFTs) conducted in 2000 and 2002. The research focuses on the methodological and technological issues in studying this population. Specifically, an online survey with MFTs obtained lower response rates and was therefore more expensive relative to phone and mail methods. Questions about evidence-based therapies (EBTs) were asked two different ways with significantly different results based on how the questions were phrased. A discussion of the relative advantages and disadvantages of using different survey methodologies and consideration of issues pertinent to studying EBTs are included.     

Who Is Adopted?

ABSTRACT

There are several advantages of using large representative samples to study adoption-related issues, and existing databases are increasingly used for this purpose. Large scale surveys are often limited, however, in the number and type of questions that are asked regarding adoption. Conceptual and methodological issues in measuring adoption status are discussed, and a rich new database-the National Longitudinal Survey of Adolescent Health (Add Health)-is used to illustrate these complexities. In Add Health school survey data, adolescents were asked a direct question about whether or not they were adopted and whether they lived with biological parents. During home interviews, adolescents were not asked about adoption status per se, but they could be classified as adopted if they lived with adoptive rather than biological parents. Parents' and adolescents' reports of adoption status and living arrangements are compared in Add Health data to illustrate the complexities of measuring adoption status when using archival data. For research to be scientifically credible and to enhance the accumulation of knowledge, investigators need to be precise about the concepts and measures they use in studying adoption-related issues, particularly when using previously collected survey data.     

Missing Data in Sociological Research: An Overview of Recent Trends and an Illustration for Controversial Questions, Active Nonrespondents and Targeted Samples

In an age of telemarketers, spam emails, and pop-up advertisements, sociologists are finding it increasingly difficult to achieve high response rates for their surveys. Compounding these issues, the current political and social climate has decreased many survey respondents’ likelihood of responding to controversial questions, which are often at the heart of much research in the discipline. Here we discuss such implications for survey research in sociology using: a content analysis of the prevalence of missing data and survey research methods in the most cited articles in top sociology journals, a case study highlighting the extraction of meaningful information through an example of potential mechanisms driving the non-random missing data patterns in the Religion Among Academic Scientists dataset, and qualitative responses from non-responders in this same case. Implications are likely to increase in importance given the ubiquitous nature of survey research, missing data, and privacy concerns in sociological research.     

Supplementing a survey with respondent Twitter data to measure e-cigarette information exposure

ABSTRACT

Social media data are increasingly used by researchers to gain insights on individuals’ behaviors and opinions. Platforms like Twitter provide access to individuals’ postings, networks of friends and followers, and the content to which they are exposed. This article presents the methods and results of an exploratory study to supplement survey data with respondents’ Twitter postings, networks of Twitter friends and followers, and information to which they were exposed about e-cigarettes. Twitter use is important to consider in e-cigarette research and other topics influenced by online information sharing and exposure. Further, Twitter metadata provide direct measures of user’s friends and followers as opposed to survey self-reports. We find that Twitter metadata provide similar information to survey questions on Twitter network size without inducing recall error or other measurement issues. Using sentiment coding and machine learning methods, we find Twitter can elucidate on topics difficult to measure via surveys such as online expressed opinions and network composition. We present and discuss models predicting whether respondents’ tweet positively about e-cigarettes using survey and Twitter data, finding the combined data to provide broader measures than either source alone.     

Life Course Analysis—The Courage to Search for Something More

Abstract

In this paper we discuss middle-to-late life issues of the transgender and intersex communities. We demonstrate that these mid-to-late life issues are richly complex, full of courage, coping, risk, and resilience, and are grounded in a socio-ecological landscape of systemic actual and perceived violence and abuse. We examine how this socio-ecological environment affects the “normative” mid-life cycle processes. Practical examples are drawn from the author's field interviews and survey research over the past decade. We close by examining the effects of such a landscape on the middle-age life stage and examine its potential ramifications for old age as well.     

Perceptions of geography students in the USA and Egypt on global issues

The study aimed to compare the perceptions of Egyptian and US students on global issues. The authors developed a survey of global issues and administered it to sample of 321 Geography students in the USA and Egypt. The survey tapped five issues: global citizenship, cultural diversity, global conflicts, nuclear arms race, and global warming. The results showed that there were significant differences in Egyptian and US students’ perception of each of the global issues.     

Modelling Chinese manufacturers' technology adoption behaviour

Abstract

This paper uses firm-level survey data from Hong Kong to identify determinants of technology adoption. The data analysis was guided by proposing a new model that was built upon the Theory of Planned Behaviour and Technology Acceptance Model. Three types of determinants are considered. They are determinants controllable by the technology suppliers and technology buyers, respectively; an external determinant that is not controllable by both the technology suppliers and buyers. This paper reviews research in related areas and the proposed model suggests ways in which behavioural, environmental and organisational issues may be important to technology adoption. The result reveals that behavioural and organisational issues are relevant to technology adoption decision, whereas the relevance of environmental issues is questionable.     

Opinions on Fictitious Issues: The Pressure to Answer Survey Questions

This research shows, as have previous studies, that a substantialnumber of people will offer opinions on fictitious topics inthe context of a survey interview. The results of a split-ballotexperiment (and a replication) with three fictitious publicaffairs issues suggest that people give opinions on such mattersin large part because of the pressure to answer survey questionsthat is created by the way in which the questions are askedand by the manner in which "don't know" responses are handledby the interviewer. The findings also indicate that there maybe little or no relationship between an individual's willingnessto admit ignorance and his or her inclination to volunteer opinionson fictitious issues, as measured by at least one item fromthe Marlowe-Crowne Social Desirability Scale. In addition, thedata demonstrate that blacks and less well-educated respondentsare more likely than their counterparts to offer opinions onfictitious issues.     

Public relations education in a divided society: PR, terrorism and critical pedagogy in post-conflict Northern Ireland

Most students entering higher education in Northern Ireland's two universities have experienced a highly segregated educational system since they were five years old. This context creates a specific set of issues and problems for the educator who wishes to engage with student groups in a critical analysis of complex and controversial topics within the curriculum. This article presents the results of a research project which was designed to assess student (and educator) experiences of studying the issue of ‘PR and terrorism’ at Northern Ireland's largest university. Data was gathered from undergraduate public relations students using survey questionnaires (administered before and after the teaching and learning experience). The educators engaged in this activity also report on their own ‘participant observation’ experiences and reflect critically upon the role academic staff might play in developing pedagogic practices which are accepting of diversity in respect to controversial and complex curriculum topics.     

Restructuring of American Religion: Further Evidence*

Data are analyzed from three national surveys conducted in 1984, 1989, and 1992, supplemented by other data from a 1991 and a 1992 survey, to examine the distribution of religious conservatives, moderates, and liberals, and to consider the social and ideological correlates of these religious orientations. The results suggest overall stability in the distribution of these orientations and offer modest support for status group, religious socialization, and religious organization interpretations of their sources. The results also indicate that religious views correspond with positions on a number of contested social policy issues, but cast doubt on arguments about deeper differences in worldview and moral perspectives. The ways in which the data support and help to refine arguments presented in The Restrucruring of American Religion are discussed.     

Evaluation of the U.S. Air Force Family Support Center Program

This article provides an overview of the implementation and evaluation of Family Support Centers (FSC) in the U.S. Air Force. After describing the FSC model, the quantitative aspect of the evaluation is discussed, including the evaluation design, implementation methods, survey response rates, and procedures for data analysis. A summary of evaluation results also are presented as well as a discussion of administrative and methodological issues encountered in performing the evaluation. The results of the evaluation suggest that the FSC has become an integral part of the base human service delivery system, especially for married members and civilian spouses.     

A Contradiction in the Health Care Delivery System: Problems of Access*

State-wide sample survey data regarding issues and problems frequently incurred in gaining access to health care delivery systems, as well as reports of illness, are utilized in an analysis covariance. The results indicate that the black population reports receiving more preventive health care services than the white population when controlling for all relevant independent variables. This surprising reversal of the most frequently found relationship between black and white populations, coupled with the reporting by the black population of experiencing more difficulty in gaining access to the system for needed help, presents evidence of a contradiction in the delivery of care.     

Regulating older drivers: how are the states coping?

The nation's population is growing older and its dependence upon the automobile for mobility has never been higher. Traffic safety data indicate that older operators have more fatal crashes than other age groups. As more people approach their senior years the safety issues surrounding older drivers will become even more important. Findings from a 1995 survey are reported identifying the practices of states and the perceptions of licensing officials concerning the older driver issue. The results indicate a diversity in approaches and a series of institutional and behavioral conflicts in the older driver policy subsystem that contribute to imperfect licensing controls.     

Suitability for practice guidelines for students: A survey of Canadian social work programmes

This paper reports on the development of a 'suitability for practice' policy in a large Canadian school of social work. Pertinent gatekeeping issues are reviewed and the results of a survey on suitability polices in Canadian social work schools are presented. The paper concludes with a discussion of the University of Calgary's Faculty of Social Work Suitability Policy. This policy integrates current research on gatekeeping, the results of the Canadian schools survey and input from primary stakeholders: the students, faculty members, the professional association and the community.     

The big data divide and its consequences

Recent advances in technology and the increasing volumes of data that they enable have led to a wave of scholarly and popular attention to big data. While big data is often heralded for its ability to provide insight, the data, its analysis, and its outcomes are not evenly distributed. Currently, scholarship on big data is extending past work on the digital divide, theorizing a new big data divide. While this work most directly addresses the issues of ownership and access to big data, some work extends the divide to issues relating to skill and use. This extension opens up new complications relating to identity, social sorting, use, agency, and global development that are inextricably related to the issues above and to the study of big data. These issues go beyond the simple language of the digital divide extending inquiries into the realm of digital inequalities more generally. Any work on big data and the big data divide needs to engage with a more broad‐based notion of digital inequalities to be better equipped to handle the complex issues above, as well as the material, democratic, and identity problems that big data bring about.     

Investigating call record data using sequence analysis to inform adaptive survey designs

Researchers have become increasingly interested in better understanding the survey data collection process in interviewer-administered surveys. However, tools for analysing paradata capturing information about field processes, also called call record data, are still not yet fully explored. This paper introduces sequence analysis as a simple tool for investigating such data with the aim of better understanding and improving survey processes. A novel approach is to use sequence analysis within interviewers, which allows the identification of unusual interviewer calling behaviours, and may provide guidance on interviewer performance. Combining the technique with clustering, optimal matching and multidimensional scaling, the method offers a way of visualising, displaying and summarising complex call record data. The method is introduced to inform survey management and survey monitoring. The method is hence informative for adaptive survey designs and will help to identify unusual behaviour and outliers and to improve survey processes. Sequence analysis is applied to call record data from the UK Understanding Society survey. The findings inform further modelling of call record data to increase efficiency in call scheduling.     

Possible Falsification of Survey Data: An Analysis of a Mail Survey in the Soviet Union

Analysis of data from a mall survey in the Soviet Union, carriedout by the agency responsible for the distribution of AmericaIllustrated, strongly suggests that the data were falsified.Information about Soviet society (including the results of anumber of other surveys of the Soviet population) and probabilitytheory were used to generate a set of expected results for thissurvey. The actual results differed dramatically from the expected—theresponse rate was much too high, the sample was too young andcontained too many workers, and too many people had managedto see almost all of the previous year's issues. These discrepancies,combined with the fact that the results served the interestsof the Soviet officials and that it would have been easy forthem to falsify the data, lead us to suspect that they did so.The conclusion discusses the conditions under which variouskinds of falsification are most likely and general strategiesfor detecting falsification.