The Effect of Interviewer Characteristics and Expectations on Response

This study reports on two sets of findings related to interviewereffects, derived from a national RDD sample of the adult population.The first of these concerns the effect of interviewer characteristicsand expectations on overall cooperation rates; the second, theeffect of interviewer characteristics and expectations on itemnonresponse and response quality. We found that interviewers'age, the size of the interviewing assignment, and interviewers'expectations all had a strong effect on overall cooperationrates; the relation of experience to response rate was curvilinearin this sample. Age and education have consistent but statisticallyinsignificant effects on item nonresponse. The effect of interviewers'expectations on responses within the interview resembles thatin earlier studies, but is less pronounced and less consistent.     

CONTEXT EFFECTS FOR CENSUS MEASURES OF RACE AND HISPANIC ORIGIN

This paper reports on the results of a split-ballot experimentconducted in 1987 to test alternative versions of the decennialcensus long form. Two forms were randomly assigned and self-administeredin group sessions involving a total of 515 respondents. Theorder of race and Hispanic origin items was experimentally manipulated.The standard long form asks race, then Hispanic origin. Theexperimental form reversed the order of the items in order toreduce perceived redundancy, and to create a more restrictedframe of reference for the race item. The objectives of thecontext manipulation were (1) to reduce item nonresponse forthe Hispanic origin item, and (2) to reduce reporting of "Otherrace" by Hispanics in the race item. Objective (1) was met.Objective (2) was met for Hispanics born in a U.S. state, butnot for immigrants. The results are interpreted as reflectinga process of acculturation that affects how Hispanic respondentsapply U.S. racial categories "white" and "black" in the census.     

House Effects and the Reproducibility of Survey Measurements: A Comparison of the 1980 GSS and the 1980 American National Election Study

This paper examines a collaborative experiment between the GeneralSocial Survey, National Opinion Research Center, and the AmericanNational Election Study, Center for Political Studies. Identicalitems were asked by both surveys, thereby permitting a testof reproducibility and house effects. Significant differenceswere found between the houses, with the largest and most systematicdifferences involving item nonresponse. Various explanationsfor the differences are examined and a general assessment ofthe importance of the differences is presented.     

Estimating the Nonresponse BiasDue to Refusals in Telephone Surveys

This paper attempts to aid the process of accumulating the necessaryinformation for making more informed judgments about the effectsof nonresponse under different conditions. Two measures, whichpermit quantifiable nonsubjective assessment of the effectsof nonresponse on sample estimates, are introduced and are usedto examine the effects of respondent refusals in a random-digit-dialedgeneral population telephone survey of over 1,200 householdsas the response rate isincreased from 74.5 percent to 86.8 percent.By applying these measures under a wide range of conditions,the adequacy of various response rates may be assessed and morerational decisions made about the costs and benefits of devotingextraordinary resources to minimizing nonresponse.     

Surveying a Chicago public housing development: methodological challenges and lessons learned

During the winter and spring of 2003, the Survey Research Laboratory conducted a survey of residents of the Ida Wells and Madden Park housing developments on Chicago's near-south side. This article reviews existing research findings on using indigenous interviewers and reports the findings about the benefits and challenges of using residents as interviewers versus drawing from a regular pool of interviewers. Specifically, the authors talk about the challenges faced in using indigenous interviewers and then look at some data quality indicators among these two groups of interviewers, including cooperation rates, item nonresponse, and differences in responses to sensitive questions.     

Nonresponse in the American Time Use Survey: Who Is Missing from the Data and How Much Does It Matter?

This article examines nonresponse in a large government survey,the American Time Use Survey (ATUS), which interviews personsin households previously interviewed in the Current PopulationSurvey. The response rate for the ATUS has been below 60 percentfor the first two years of its existence, raising questionsabout whether the results can be generalized to the target population.The article begins with an analysis of the types of nonresponseencountered in the ATUS. Noncontact accounts for roughly 60percent of ATUS nonresponse, with refusals accounting for roughly40 percent. We find little support for the hypothesis that busypeople are less likely to respond to the ATUS but find considerablesupport for the hypothesis that people who are weakly integratedinto their communities are less likely to respond, mostly becausethey are less likely to be contacted. When we compare aggregateestimates of time use calculated using the ATUS base weightswithout any adjustment for nonresponse, estimates calculatedusing the ATUS final weights with a nonresponse adjustment,and estimates calculated using weights that incorporate ourown nonresponse adjustment based on a propensity model, we findsome modest differences, but the three sets of estimates arebroadly similar. The article ends with suggestions for furtherresearch and analysis.     

The Role of Topic Interest in Survey Participation Decisions

While a low survey response rate may indicate that the riskof nonresponse error is high, we know little about when nonresponsecauses such error and when nonresponse is ignorable. Leverage-saliencetheory of survey participation suggests that when the surveytopic is a factor in the decision to participate, noncooperationwill cause nonresponse error. We test three hypotheses derivedfrom the theory: (1) those faced with a survey request on atopic of interest to them cooperate at higher rates than dothose less interested in the topic; (2) this tendency for the"interested" to cooperate more readily is diminished when monetaryincentives are offered; and (3) the impact of interest on cooperationhas nonignorability implications for key statistics. The datacome from a three-factor experiment examining the impact oncooperation with surveys on (a) five different topics, using(b) samples from five different populations that have knownattributes related to the topics, with (c) two different incentiveconditions.     

Nonresponse Rates and Nonresponse Bias in Household Surveys

Many surveys of the U.S. household population are experiencinghigher refusal rates. Nonresponse can, but need not, inducenonresponse bias in survey estimates. Recent empirical findingsillustrate cases when the linkage between nonresponse ratesand nonresponse biases is absent. Despite this, professionalstandards continue to urge high response rates. Statisticalexpressions of nonresponse bias can be translated into causalmodels to guide hypotheses about when nonresponse causes bias.Alternative designs to measure nonresponse bias exist, providingdifferent but incomplete information about the nature of thebias. A synthesis of research studies estimating nonresponsebias shows the bias often present. A logical question at thismoment in history is what advantage probability sample surveyshave if they suffer from high nonresponse rates. Since postsurveyadjustment for nonresponse requires auxiliary variables, theanswer depends on the nature of the design and the quality ofthe auxiliary variables.     

Experiments in Producing Nonresponse Bias

While nonresponse rates in household surveys are increasingin most industrialized nations, the increasing rates do notalways produce nonresponse bias in survey estimates. The linkagebetween nonresponse rates and nonresponse bias arises from thepresence of a covariance between response propensity and thesurvey variables of interest. To understand the covariance term,researchers must think about the common influences on responsepropensity and the survey variable. Three variables appear tobe especially relevant in this regard: interest in the surveytopic, reactions to the survey sponsor, and the use of incentives.A set of randomized experiments tests whether those likely tobe interested in the stated survey topic participate at higherrates and whether nonresponse bias on estimates involving variablescentral to the survey topic is affected by this. The experimentsalso test whether incentives disproportionately increase theparticipation of those less interested in the topic. The experimentsshow mixed results in support of these key hypotheses.     

Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey

Declining contact and cooperation rates in random digit dial(RDD) national telephone surveys raise serious concerns aboutthe validity of estimates drawn from such research. While researchin the 1990s indicated that nonresponse bias was relativelysmall, response rates have continued to fall since then. Thecurrent study replicates a 1997 methodological experiment thatcompared results from a "Standard" 5-day survey employing thePew Research Center’s usual methodology with results froma "Rigorous" survey conducted over a much longer field periodand achieving a significantly higher response rate. As withthe 1997 study, there is little to suggest that unit nonresponsewithin the range of response rates obtained seriously threatensthe quality of survey estimates. In 77 out of 84 comparableitems, the two surveys yielded results that were statisticallyindistinguishable. While the "Rigorous" study respondents tendedto be somewhat less politically engaged, they did not reportconsistently different behaviors or attitudes on other kindsof questions. With respect to sample composition, the Standardsurvey was closely aligned with estimates from the U.S. Censusand other large government surveys on most variables. We extendour analysis of nonresponse to include comparisons with thehardest-to-reach respondents and with respondents who terminatedthe interview prior to completion.     

Comparing Check-All and Forced-Choice Question Formats in Web Surveys

For survey researchers, it is common practice to use the check-allquestion format in Web and mail surveys but to convert to theforced-choice question format in telephone surveys. The assumptionunderlying this practice is that respondents will answer thetwo formats similarly. In this research note we report resultsfrom 16 experimental comparisons in two Web surveys and a papersurvey conducted in 2002 and 2003 that test whether the check-alland forced-choice formats produce similar results. In all 16comparisons, we find that the two question formats do not performsimilarly; respondents endorse more options and take longerto answer in the forced-choice format than in the check-allformat. These findings suggest that the forced-choice questionformat encourages deeper processing of response options and,as such, is preferable to the check-all format, which may encouragea weak satisficing response strategy. Additional analyses showthat neither acquiescence bias nor item nonresponse seem topose substantial problems for use of the forced-choice questionformat in Web surveys.     

Using Community-Level Correlates to Evaluate Nonresponse Effects in a Telephone Survey

Understanding the relationship between nonresponse processesand key research variables is central to evaluating if and hownonresponse introduces bias into survey estimates. In most telephonesurveys, however, little information is available with whichto estimate these effects. We report a procedure for examiningthe potential effects of nonresponse via analyses that (1) investigatethe linkages between community-level (zip code) variables andsurvey nonresponse and (2) examine the associations betweenthese community-level variables and key survey measures. Wedemonstrate these procedures using hierarchical modeling toanalyze data from a state-wide telephone survey in Illinois.One zip code–level indicator of concentrated disadvantage—thepercentage of the population below poverty level—was foundto be positively associated with nonresponse and, among respondents,with both current physical disability status and lack of healthinsurance coverage, suggesting that both may have been underestimatedin this survey. This inexpensive approach has the potentialof enabling researchers to routinely evaluate nonresponse effectsin their survey data.     

Pemoline in Adult Attention Deficit Hyperactivity Disorder: Predictors of Nonresponse

Abstract

The psychostimulants are the most widely used treatment modality for attention deficit/hyperactivity disorder (ADHD). Although positive responses to psychostimulants have been demonstrated, 46% of adults with ADHD, on average, are nonresponsive or cannot tolerate the medications because of adverse effects. We reviewed the charts of 43 students treated with pemoline and studied parameters of nonresponse. Sixteen (37%) met the study definition of nonresponse. Of these, 5 (12%) were symptom nonresponders and 11 (25%) were nonresponders because of adverse effects. We found a significant relationship between age and nonresponse resulting from adverse effects, yet no other factors predicted nonresponse to pemoline. Clinicians should consider using smaller starting doses of pemoline in adults to minimize adverse effects.     

Survey Participation, Nonresponse Bias, Measurement Error Bias, and Total Bias

A common hypothesis about practices to reduce survey nonresponseis that those persons brought into the respondent pool throughpersuasive efforts may provide data filled with measurementerror. Two questions flow from this hypothesis. First, doesthe mean square error of a statistic increase when sample personswho are less likely to be contacted or cooperate are incorporatedinto the respondent pool? Second, do nonresponse bias estimatesmade on the respondents, using survey reports instead of records,provide accurate information about nonresponse bias? Using aunique data set, the Wisconsin Divorce Study, with divorce recordsas the frame and questions about the frame information includedin the questionnaire, this article takes a first look into thesetwo issues. We find that the relationship between nonresponsebias, measurement error bias, and response propensity is statistic-specificand specific to the type of nonresponse. Total bias tends tobe lower on estimates calculated using all respondents, comparedwith those with only the highest contact and cooperation propensities,and nonresponse bias analyses based on respondents yield conclusionssimilar to those based on records. Finally, we find that errorproperties of statistics may differ from error properties ofthe individual variables used to calculate the statistics.     

Changes in Telephone Survey Nonresponse over the Past Quarter Century

We correct, update, and elaborate Curtin, Presser, and Singer’s(2000) report that the University of Michigan’s Surveyof Consumer Attitudes (SCA) experienced only a small responserate decline between 1979 and 1996, contrary to the widespreadperception of plunging response rates. Our aims are to (1) correcterrors in the SCA response rate data that affected Curtin, Presser,and Singer’s (2000) result, (2) examine the trend in SCAresponse rates after 1996, when caller identification technologybecame widespread, and (3) describe the roles played by thevarious sources of SCA nonresponse over time. The results showthat the response rate decline from 1979 to 1996 was largerthan described by Curtin, Presser, and Singer (2000); the responserate drop was significantly steeper from 1996 to 2003 than from1979 to 1996; and the 1979 to 2003 trends differed substantiallyfor refusals and noncontacts.     

Nonresponse Effects on Relationships Between Variables

To examine nonresponse effects on variable relationships, datafrom a panel study of 1,870 older Iowa residents are reported.When data are accumulated by wave of response, differences areminor on variable distributions (percentage, mean, standarddeviation), bivariate relationships (correlation), and multivariaterelationships (regression), suggesting that the assumed impactof nonresponse bias on variable relationships has been overstated.     

The Hidden 25 Percent: An Analysis of Nonresponse on the 1980 General Social Survey

Methods for estimating nonresponse bias are reviewed and severalmethods are tried on the 1980 GSS. The results indicate thatvarious estimating procedures are inappropriate and that eventhe more promising techniques can provide faulty estimates ofnonresponse bias. By its nature, nonresponse bias is very difficultto assess accurately and no simple, certain method exists.     

Trends in Nonresponse Rates, 1952-1979

Reported increases in nonresponse rates to sample surveys havenot been systematically documented to date. Data from the NationalElection Studies and the Surveys of Consumer Attitudes, twowell-known continuing studies conducted by the Survey ResearchCenter at the University of Michigan, permit the assessmentof long-term trends for the two major components of nonresponse,refusals and other noninterviews, by urban subgroups using timeseries regression techniques. The analyses clearly demonstratethat there have been substantial increases in total nonresponsedue primarily to increases in the percentages of respondentswho refuse to be interviewed, and that these trends are relatedto the level of urbanization.     

Eating Disorder Symptoms Among College Students: Prevalence,Persistence, Correlates,and Treatment-Seeking

Abstract

Objective: To examine the prevalence, correlates, persistence, and treatment-seeking related to symptoms of eating disorders (EDs) in a random sample of college students. Participants: A random sample of students at a large university were recruited for an Internet survey in Fall 2005 and a follow-up survey in Fall 2007. Methods: ED symptoms were measured using the SCOFF screen and adjusted for nonresponse using administrative data and a nonresponse survey. Results: 2,822 (56%) students completed the baseline survey. Among undergraduates the prevalence of positive screens was 13.5% for women and 3.6% for men. Among students with positive screens, 20% had received past-year mental health treatment. In the follow-up sample (N = 753), ED symptoms at baseline significantly predicted symptoms 2 years later. Conclusions: Symptoms of EDs were prevalent and persistent among college students in this study. These findings suggest that brief screens can identify a large number of students with untreated EDs.     

Evaluating the Effects of Screening for Telephone Service in Dual Frame RDD Surveys

The high costs and largely unknown error properties of cellulartelephone interviews make screening for cell-only adults a potentiallyattractive option in dual frame RDD surveys. Screening out adultswith landline telephones from the cellular sample does not affectthe coverage properties of a dual frame survey, but it may affectother sources of error, especially nonresponse. In this study,data from a 2006 dual frame RDD survey conducted for the PewResearch Center, the Associated Press, and AOL are used to evaluatethe effects of implementing a cell-only screener on both thebias and variance of weighted survey estimates. The effect ofscreening appears to be minimal so long as an adjustment fortelephone service is included in the weighting method. Resultsof an attempt to correct for residual nonresponse due to inaccessibilityare also discussed.