Telephone survey respondents' reactions to questions regarding interpersonal violence

Concerns have been raised regarding the appropriateness of asking about violence victimization in telephone interviews and whether asking such questions increases respondents' distress or risk for harm. However, no large-scale studies have evaluated the impact of asking such questions during a telephone interview. This study explored respondents' reactions to questions regarding violence in two large recently completed telephone surveys. After respondents were asked about violence, they were asked if they thought surveys should ask such questions and whether they felt upset or afraid because of the questions. In both surveys, the majority of respondents (regardless of their victimization history) were willing to answer questions about violence and were not upset or afraid because of the questions. More than 92% of respondents thought such questions should be asked. These results challenge commonly held beliefs and assumptions and provide some assurance to those concerned with the ethical collection of data on violent victimization.     

The context of program implementation and evaluation: A pilot study of interorganizational differences to improve child welfare reform efforts

This article contributes to the growing literature on evaluation and implementation science by examining the interaction between staff perceptions of organizational strength with perceptions and indicators of program fidelity. As part of a pilot project related to the evaluation of the Family to Family initiative, a survey was distributed to employees within two urban child welfare agencies with a total of 410 respondents across both sites, for a combined response rate of 72.2%. Survey results were analyzed both in terms or respondents' perception of their agency as well as in relation to measures of program performance and workload. Multivariate models show that organizational indicators are the most significant and positive predictors of perceived program implementation. Specifically, staff who positively perceived the availability of information within their agency also believed that the programs were well implemented in their agency. These findings suggest that as the value of program changes are articulated within an organization, the implementation of the initiative is perceived to improve.     

EDUCATED GUESSES: THE PROCESS OF ANSWERING FACTUAL KNOWLEDGE QUESTIONS IN SURVEYS

Responses to autobiographical questions are known to representmore than simply retrieval of information from memory; inference,cuing, and "availability" all play a role. Using responses toitems in four different surveys, we find that respondent motivationand ability, together with contextual cues, help determine howsurvey respondents answer knowledge questions about the worldaround us. Thus, we extend the domain of factual items for whichthe role of inferential processes is recognized, and we specifymore precisely the kinds of factors that respondents use inanswering such questions. We also find suggestive evidence thatattitudes influence answers to information questions, thus extendingas well the kinds of factors seen as likely to affect reportsabout facts.     

PROBING "DONT KNOW" ANSWERS: EFFECTS ON SURVEY ESTIMATES AND VARIABLE RELATIONSHIPS

Don't Know responses to survey questions are ambiguous becausethe same words are used by respondents to mean different things—ignorance,indecision, or uncertainty about the meaning of the questionasked. In order to clarify the meaning of such answers, surveyinterviewers are frequently trained to probe Don't Know answersat least once before the answer is considered final. We arguethat unconditional probing of Don't Know answers may not bea desirable practice, particularly as regards knowledge items.Large unintended effects on responses to four knowledge itemsresulted when two groups of interviewers, who administered thesame survey questions, probed Don't Know responses at differentrates. Strong evidence is provided in the article that the probingof Don't Know answers encouraged guesswork on the part of uninformedrespondents, giving rise to significant distributional differencesand differences in means across half samples for the affectedvariables. However, relationships between variables appear tobe largely unaffected by probing effects. In order to formulatevalid probing guidelines for interviewer training, further researchis needed to establish whether the findings for knowledge questionsgeneralize to factual questions.     

The Influence of Graphical and Symbolic Language Manipulations on Responses to Self-Administered Questions

This article reports results from 14 experimental comparisonsdesigned to test 7 hypotheses about the effects of two typesof nonverbal languages (symbols and graphics) on responses toself-administered questionnaires. The experiments were includedin a survey of 1,042 university students. Significant differenceswere observed for most comparisons, providing support for allseven hypotheses. These results support the view that respondents’answers to questions in self-administered surveys are influencedby more than words. Thus, the visual presentation of questionsmust be taken into consideration when designing such surveysand, especially, when comparing results across surveys in whichthe visual presentation of questions is varied.     

An evaluation of transit procurement training

We evaluated a training course called “Orientation to Transit Procurement”, designed and conducted by the National Transit Institute. This course is designed to provide Federal Transit Administration (FTA) grantees an overview of regulations and best practices related to the procurement process. Our objective in conducting the evaluation was to understand how transit agency staff made changes in procurement practices in response to the course training. The evaluation was mixed mode: an Internet survey followed by in-depth interviews with a small group of respondents. Survey respondents were also provided with an open-ended question providing us with additional context for our evaluation. Results show that the training is substantially successful at meeting the goal of improving procurement practices at transit agencies; indeed, most respondents report making changes at their agencies as the proximate result of the training. This was at odds with our exploration of knowledge of procurement topics, as most respondents gave inaccurate answers on multiple-choice “knowledge questions”. This may have been due to question structure or, more likely, the nature of online surveys. Suitable training on the procurement of information technology was also a main concern. The lack of training in this area is indicative of the broader challenge facing public transit agencies in how to incorporate new forms of technology into their existing practices and bureaucratic structures.     

The Possibility of Mixed-Mode Surveys in Sociological Studies

Abstract:  This paper explores the possibility of the mixed-mode survey in Japan in order to ensure a wider range of respondents, focusing on the specific type of mixture that is designed to make contact with a respondent by more than one mode and in which a respondent can choose an alternative mode convenient for him or her. The dataset mainly used comes from the survey Study on the Social Composition of the Urban Local Community conducted in Shinagawa Ward of Tokyo in 1999. Three other studies are introduced for additional information. As an indicator of the effectiveness of a mixed-mode design, the response rates of four studies are examined and some improvement in the response rates is observed, compared to that of a single-mode design. Another indicator is the representativeness of a sample and it is evaluated using the data of the Shinagawa survey. When the discrepancy between valid respondents and the population on some demographic characteristics is regarded as representation bias, a mixed-mode design is found to reduce the bias. An inherent problem in a mixed-mode design is "mode effect." In order to control for the mode effect in analyses, the inclusion of an additional variable indicating "mode" in the model is recommended. It is further suggested that the interaction effects between the "mode" and the respondents' traits should be checked, on the assumption that the survey mode affects respondents differently. The attempts are made and discussed in this paper using two variables drawn from the Shinagawa survey.     

CARRYOVER EFFECTS IN ATTITUDE SURVEYS

Answers to attitude questions in surveys can vary markedly dependingon the preceding items in the questionnaire. This study concernssuch context effects. More than 1,100 respondents were askedabout six target issues in a telephone survey. Before answeringthe target questions, most of the respondents had been askedabout one of two sets of related issues; the remainder receivedunrelated, "neutral" context items. For five of the six targetissues, the two groups receiving different sets of context itemsdiffered significantly in their answers to the target questions.However, only one of the substantive context groups differedsignificantly from the group that received the neutral contextitems. Additional results indicated that context effects werelarger when respondents reported that their beliefs about thetarget issue were both mixed and important. Coding of openendedresponses to follow-up questions suggested that the contextitems may have changed how respondents thought about the targetissues.     

Personalization of Mail Surveys for General Public and Populations with a Group Identity: Results from Nine Studies*

Abstract The effect of personalization on mail survey response rates was examined in nine studies that included 17 comparisons under several research conditions. A study of this variable across multiple experiments in five agricultural experiment stations was undertaken because of conflicting results from previous research and from concern that the effectiveness of personalization might have decreased over time. Results show that, while response to general public surveys appeared to increase modestly across all treatment groups, there was no positive effect for populations in which a group identity (e.g., Dear Oregon Gardner or Dear ATV Owner) is employed to address respondents in cover letters. Personalization appears to remain useful for improving response in surveys of the general public.     

Color, Labels, and Interpretive Heuristics for Response Scales

We carried out two experiments to investigate how the shadingof the options in a response scale affected the answers to thesurvey questions. The experiments were embedded in two web surveys,and they varied whether the two ends of the scale were representedby shades of the same or different hues. The experiments alsovaried the numerical labels for the scale points and examinedresponses to both unipolar scales (assessing frequency) andbipolar scales (assessing favorability). We predicted that theuse of different hues would affect how respondents viewed thelow end of the scale, making responses to that end seem moreextreme than when the two ends were shades of the same hue.This hypothesis was based on the notion that respondents usevarious interpretive heuristics in assigning meaning to thevisual features of survey questions. One such cue is visualsimilarity. When two options are similar in appearance, respondentswill see them as conceptually closer than when they are dissimilarin appearance. The results were generally consistent with thisprediction. When the end points of the scale were shaded indifferent hues, the responses tended to shift toward the highend of the scale, as compared to scales in which both ends ofthe scale were shaded in the same hue. Though noticeable, thisshift was less extreme than the similar shift produced whennegative numbers were used to label one end of the scale; moreover,the effect of color was eliminated when each scale point hada verbal label. These findings suggest that respondents havedifficulty using scales and pay attention even to incidentalfeatures of the response scales in interpreting the scale points.     

INTERVIEWING OLDER ADULTS: MODE COMPARISON USING DATA FROM A FACE-TO-FACE SURVEY AND A TELEPHONE RESURVEY

A face-to-face survey conducted in 1984 with a sample of 1491residents of the Detroit metropolitan area (including an oversampleof older adults) and a reinterview of a random subset of theserespondents by telephone were used to compare the two modesof data collection across two age levels. Except for a tendencytoward a disproportionately large number of DK answers and adisproportionately large amount of interviewer assistance onthe telephone, respondents 60 years of age and older did notexhibit larger mode differences than did respondents under 60.For both age groups, response distributions were rather similar,suggesting little effect of mode. Likewise, response style differedlittle by mode, while a higher proportion of missing data (i.e.,"I don't know" answers) was given on the telephone. The responserate for the telephone reinterview was 90%, somewhat lower forolder than younger persons. The findings support the feasibilityof using the telephone for reinterviewing older adults.     

Presentation matters: how mode effects in item non-response depend on the presentation of response options

Comparability of data across modes is an important issue in survey research. In this paper we discuss item non-response to attitudinal questions in telephone and web surveys. We present results from a survey experiment conducted in Italy and in Spain that compares different presentations of response options in an online setting with a benchmark telephone survey. In line with earlier studies we find that (A), the share of non substantial answers in the online survey depends on how the response option is presented. Comparing different presentations in an online survey to the standard approach of telephone surveys using propensity score matching, we find that (B), the share and pattern of non substantial answers, is most similar across the two modes in the online survey when the it is captured in an instruction on each screen for the online survey. Our findings are of particular relevance for the design of multiple mode or mix-mode surveys of attitudinal questions using online and telephone modes.     

Survey research with preliterate adult populations in post conflict situations using researcher-assisted self-completion questionnaires

This paper presents an innovative and effective approach for conducting surveys with preliterate adults in a context of limited infrastructure in post conflict Burundi, Africa. We conducted surveys with parents/carers of children with a disability and community members, as a component of mixed methods research with children and adults to provide baseline data for Handicap International in developing programs to prevent sexual abuse among children with a disability. To meet the challenges of conducting surveys on the sensitive topic in such field conditions, we developed specifically adapted researcher-assisted self-completion questionnaires (RASCQ) using images and verbal guidance. RASCQ allowed researchers to provide respondents with supports needed to complete surveys and the privacy and confidentiality needed to encourage honest answers. This paper discusses the development process, details of survey implementation, and quality of data obtained. RASCQ proposed in the current study can be used in various studies with marginalised groups and people in developing countries.     

Monitoring youth behavior in response to structural changes. Alternative approaches for measuring adolescent drinking

Random digit dial (RDD) telephone and self-administered school-based surveys of drinking and drinking and driving were compared for adolescents from two Northern California communities. The RDD and school-based surveys resulted in very different samples. The telephone sample contained a greater proportion of European Americans and a smaller proportion of Asian Americans and "other" ethnicities. Respondents to the telephone sample also tended to be older and of higher socioeconomic status (SES). The telephone survey captured very few school dropouts. Moreover, it resulted in lower self-reports of drinking and drinking and driving. Survey mode appeared to influence respondents equally, regardless of their gender, age, ethnicity, or SES.     

EFFECTS OF INTERNET USE AND SOCIAL RESOURCES ON CHANGES IN DEPRESSION

We examine how people's different uses of the Internet predict their later scores on a standard measure of depression, and how their existing social resources moderate these effects. In a longitudinal US survey conducted in 2001 and 2002, almost all respondents reported using the Internet for information, and entertainment and escape; these uses of the Internet had no impact on changes in respondents' level of depression. Almost all respondents also used the Internet for communicating with friends and family, and they showed lower depression scores six months later. Only about 20 percent of this sample reported using the Internet to meet new people and talk in online groups. Doing so changed their depression scores depending on their initial levels of social support. Those having high or medium levels of social support showed higher depression scores; those with low levels of social support did not experience these increases in depression. Our results suggest that individual differences in social resources and people's choices of how they use the Internet may account for the different outcomes reported in the literature.     

Interpretation and interview context: examining the General Social Survey name generator using cognitive methods

This paper reports on a small (N=50) study of how survey respondents interpret the General Social Survey (GSS)'s “discuss important matters” name generator. The study involved concurrent think-aloud interviews, in which respondents were debriefed about their thought processes immediately after answering the name generator. Analyses of these responses indicate that some respondents had difficulty in specifying what was meant by the term “important matters”; sizable minorities understood the question in terms of frequency of contact or intimacy rather than in terms of specific social exchanges. Most of those interviewed said that their “important matters” had to do with personal/intimate relationships or other issues of personal life (e.g., finances, hobbies, or health), but appreciable numbers referred to work and political discussions. An interview context experiment revealed that a respondent's definition of “important matters” can be shaped by the substantive content of the preceding parts of an interview schedule. Notwithstanding these findings, the composition of the networks elicited in the study does not appear to vary substantially across interpretations of the name generator. We conclude that the name generator succeeds in measuring “core” discussion networks, though with somewhat nonspecific content. Implications for the measurement of personal networks in sample surveys are discussed.     

LOVE OR THE "LECHEROUS PROFESSOR":

We conducted telephone interviews with twenty-five professors and students, or former students, who have been involved in consensual sexual relationships. We examined the extent to which respondents' experiences are consistent with "lecherous professor' stereotypes. We also assessed differences in the perceptions and experiences of professors and students, and how these may be influenced by gender, race/ethnicity, sexual orientation, and undergraduate or graduate student status. Finally, we evaluated whether the experiences and perceptions of respondents suggest that these relationships should be controlled by institutional policies. We found little support for the "lecherous professor' stereotype based on our respondents' experiences. Two-thirds of the relationships were initiated by a student or were mutually initiated by student and professor. All students said they entered freely into these relationships. For many of our respondents, issues related to race/ethnicity, sexuality, and age brought greater challenges to the relationship than did the professor/student status difference. We found that negative reactions from others tended to be most extreme for those who crossed multiple status boundaries. Students, in particular, cited the power difference between professors and students as the greatest detriment in these relationships, although both students and professors tended to refer to power issues in abstract rather than personal terms. Although most students and professors said they would enter into such a relationship again, most advised others against doing so. Nearly all respondents thought consensual relationships were inappropriate in supervisory situations, and they supported institutional policies that prohibit them in such circumstances.     

Ecometrics: Toward a Science of Assessing Ecological Settings, With Application to the Systematic Social Observation of Neighborhoods

This paper considers the quantitative assessment of ecological settings such as neighborhoods and schools. Available administrative data typically provide useful but limited information on such settings. We demonstrate how more complete information can be reliably obtained from surveys and observational studies. Survey-based assessments are constructed by aggregating over multiple item responses of multiple informants within each setting. Item and rater inconsistency produce uncertainty about the setting being assessed, with definite implications for research design. Observation-based assessments also have a multilevel error structure. The paper describes measures constructed from interviews, direct observations, and videotapes of Chicago neighborhoods and illustrates an "ecometric" analysis—a study of bias and random error in neighborhood assessments. Using the observation data as an illustrative example, we present a three-level hierarchical statistical model that identifies sources of error in aggregating across items within face-blocks and in aggregating across face-blocks to larger geographic units such as census tracts. Convergent and divergent validity are evaluated by studying associations between the observational measures and theoretically related measures obtained from the U.S. Census, and a citywide survey of neighborhood residents.     

Uncertainty and framing in a valuation task

Many stated preference studies report framing effects in responses to valuation questions. Framing in stated preference studies occurs when respondents use irrelevant information contained in the question to help them value the good. This may occur because respondents are uncertain or do not hold well-formed preferences for the good in question. We investigate if respondent certainty explains framing effects in a contingent valuation study, using data from a double bounded dichotomous elicitation format and a follow-up certainty question. We investigate if respondent certainty influences anchoring and the shift effect. We find evidence that the anchoring effect is stronger for respondents who are less certain about their response to the contingent valuation question compared to respondents who are very certain. However, the shift effect is significant and negative only for respondents who are very certain. Our results indicate that certain respondents are more consistent with the predictions of rational behaviour than uncertain respondents.