Survey Participation, Nonresponse Bias, Measurement Error Bias, and Total Bias

A common hypothesis about practices to reduce survey nonresponseis that those persons brought into the respondent pool throughpersuasive efforts may provide data filled with measurementerror. Two questions flow from this hypothesis. First, doesthe mean square error of a statistic increase when sample personswho are less likely to be contacted or cooperate are incorporatedinto the respondent pool? Second, do nonresponse bias estimatesmade on the respondents, using survey reports instead of records,provide accurate information about nonresponse bias? Using aunique data set, the Wisconsin Divorce Study, with divorce recordsas the frame and questions about the frame information includedin the questionnaire, this article takes a first look into thesetwo issues. We find that the relationship between nonresponsebias, measurement error bias, and response propensity is statistic-specificand specific to the type of nonresponse. Total bias tends tobe lower on estimates calculated using all respondents, comparedwith those with only the highest contact and cooperation propensities,and nonresponse bias analyses based on respondents yield conclusionssimilar to those based on records. Finally, we find that errorproperties of statistics may differ from error properties ofthe individual variables used to calculate the statistics.     

Experiments in Producing Nonresponse Bias

While nonresponse rates in household surveys are increasingin most industrialized nations, the increasing rates do notalways produce nonresponse bias in survey estimates. The linkagebetween nonresponse rates and nonresponse bias arises from thepresence of a covariance between response propensity and thesurvey variables of interest. To understand the covariance term,researchers must think about the common influences on responsepropensity and the survey variable. Three variables appear tobe especially relevant in this regard: interest in the surveytopic, reactions to the survey sponsor, and the use of incentives.A set of randomized experiments tests whether those likely tobe interested in the stated survey topic participate at higherrates and whether nonresponse bias on estimates involving variablescentral to the survey topic is affected by this. The experimentsalso test whether incentives disproportionately increase theparticipation of those less interested in the topic. The experimentsshow mixed results in support of these key hypotheses.     

Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey

Declining contact and cooperation rates in random digit dial(RDD) national telephone surveys raise serious concerns aboutthe validity of estimates drawn from such research. While researchin the 1990s indicated that nonresponse bias was relativelysmall, response rates have continued to fall since then. Thecurrent study replicates a 1997 methodological experiment thatcompared results from a "Standard" 5-day survey employing thePew Research Center’s usual methodology with results froma "Rigorous" survey conducted over a much longer field periodand achieving a significantly higher response rate. As withthe 1997 study, there is little to suggest that unit nonresponsewithin the range of response rates obtained seriously threatensthe quality of survey estimates. In 77 out of 84 comparableitems, the two surveys yielded results that were statisticallyindistinguishable. While the "Rigorous" study respondents tendedto be somewhat less politically engaged, they did not reportconsistently different behaviors or attitudes on other kindsof questions. With respect to sample composition, the Standardsurvey was closely aligned with estimates from the U.S. Censusand other large government surveys on most variables. We extendour analysis of nonresponse to include comparisons with thehardest-to-reach respondents and with respondents who terminatedthe interview prior to completion.     

Using Community-Level Correlates to Evaluate Nonresponse Effects in a Telephone Survey

Understanding the relationship between nonresponse processesand key research variables is central to evaluating if and hownonresponse introduces bias into survey estimates. In most telephonesurveys, however, little information is available with whichto estimate these effects. We report a procedure for examiningthe potential effects of nonresponse via analyses that (1) investigatethe linkages between community-level (zip code) variables andsurvey nonresponse and (2) examine the associations betweenthese community-level variables and key survey measures. Wedemonstrate these procedures using hierarchical modeling toanalyze data from a state-wide telephone survey in Illinois.One zip code–level indicator of concentrated disadvantage—thepercentage of the population below poverty level—was foundto be positively associated with nonresponse and, among respondents,with both current physical disability status and lack of healthinsurance coverage, suggesting that both may have been underestimatedin this survey. This inexpensive approach has the potentialof enabling researchers to routinely evaluate nonresponse effectsin their survey data.     

Are Divorce Studies Trustworthy? The Effects of Survey Nonresponse and Response Errors

Researchers rely on relationship data to measure the multifaceted nature of families. This article speaks to relationship data quality by examining the ramifications of different types of error on divorce estimates, models predicting divorce behavior, and models employing divorce as a predictor. Comparing matched survey and divorce certificate information from the 1995 Life Events and Satisfaction Study (N = 1,811) showed that nonresponse error is responsible for the majority of the error in divorce data. Misreporting the divorce event was rare, and more than two thirds of respondents provided a divorce date within 6 months of the actual date. Nevertheless, divorce date error attenuated effects of time since divorce on outcomes. Gender, child custody, marital history, and education were associated with divorce error.     

Evaluating the Effects of Screening for Telephone Service in Dual Frame RDD Surveys

The high costs and largely unknown error properties of cellulartelephone interviews make screening for cell-only adults a potentiallyattractive option in dual frame RDD surveys. Screening out adultswith landline telephones from the cellular sample does not affectthe coverage properties of a dual frame survey, but it may affectother sources of error, especially nonresponse. In this study,data from a 2006 dual frame RDD survey conducted for the PewResearch Center, the Associated Press, and AOL are used to evaluatethe effects of implementing a cell-only screener on both thebias and variance of weighted survey estimates. The effect ofscreening appears to be minimal so long as an adjustment fortelephone service is included in the weighting method. Resultsof an attempt to correct for residual nonresponse due to inaccessibilityare also discussed.     

Nonresponse Rates and Nonresponse Bias in Household Surveys

Many surveys of the U.S. household population are experiencinghigher refusal rates. Nonresponse can, but need not, inducenonresponse bias in survey estimates. Recent empirical findingsillustrate cases when the linkage between nonresponse ratesand nonresponse biases is absent. Despite this, professionalstandards continue to urge high response rates. Statisticalexpressions of nonresponse bias can be translated into causalmodels to guide hypotheses about when nonresponse causes bias.Alternative designs to measure nonresponse bias exist, providingdifferent but incomplete information about the nature of thebias. A synthesis of research studies estimating nonresponsebias shows the bias often present. A logical question at thismoment in history is what advantage probability sample surveyshave if they suffer from high nonresponse rates. Since postsurveyadjustment for nonresponse requires auxiliary variables, theanswer depends on the nature of the design and the quality ofthe auxiliary variables.     

Individual determinants of social behavior

By using unique data from the section on social behavior of the Bank of Italy's 2004 Survey of Household Income and Wealth (SHIW), the paper studies the individual determinants of several aspects of social behavior: attitudes to cooperating with anonymous others; interest in politics; participation in groups and associations; and propensity to rely on favoritism both in finding a job and in dealing with government red tape. Our findings suggest that these different aspects of social behavior are only weakly correlated to each other and are explained by different individual determinants. We find that older and more educated individuals display a greater willingness to cooperate, a stronger interest in politics, and more intense association activity. By contrast, the likelihood of relying on favoritism does not depend on age and education. We also find that home-ownership is associated with good social conduct, while urban residence has mostly a negative impact on public behavior. Finally, having left-wing political opinions increases the interest in politics, while it does not affect the other aspects of social behavior.     

Eliciting survey cooperation: incentives, self-interest, and norms of cooperation

Given the weaker ties to community as noted by scholars such as Robert Putnam, survey researchers should not be surprised by a decline in survey participation over the past 10 years. This research analyzes the use of incentives coupled with introductory themes emphasizing cooperation and helpfulness--cooperative norms in American society--to understand their effects on survey response. This article analyzes two separate experiments (one phone and one mail) that provide evidence that norms of cooperation matter in the decision to participate in a survey, suggesting that this is particularly true at the refusal conversion stage. These results indicate that survey researchers may use such themes to their advantage, especially when conducting a nonresponse follow-up in a mail survey.     

Estimating the Nonresponse BiasDue to Refusals in Telephone Surveys

This paper attempts to aid the process of accumulating the necessaryinformation for making more informed judgments about the effectsof nonresponse under different conditions. Two measures, whichpermit quantifiable nonsubjective assessment of the effectsof nonresponse on sample estimates, are introduced and are usedto examine the effects of respondent refusals in a random-digit-dialedgeneral population telephone survey of over 1,200 householdsas the response rate isincreased from 74.5 percent to 86.8 percent.By applying these measures under a wide range of conditions,the adequacy of various response rates may be assessed and morerational decisions made about the costs and benefits of devotingextraordinary resources to minimizing nonresponse.     

UNDERSTANDING THE DECISION TO PARTICIPATE IN A SURVEY

The lack of full participation in sample surveys threatens theinferential value of the survey method. We review a set of conceptualdevelopments and experimental findings that appear to be informativeabout causes of survey participation; offer an integration ofthat work with findings from the more traditional statisticaland survey methodological literature on nonresponse; and, giventhe theoretical structure, deduce potentially promising pathsof research toward the understanding of survey participation.     

The impact of respondent attentiveness on reliability and validity

This study aims to assess whether respondent inattentiveness causes systematic and unsystematic measurement error that influences survey data quality. To determine the impact of (in)attentiveness on the reliability and validity of target measures, we compared respondents from a German online survey (N = 5205) who had passed two attention checks with those who had failed. Our results show that inattentiveness induces both random and systematic measurement error, which impacts estimates of the reliability and validity of multi-item scales. In addition, we conducted a sensitivity analysis, which revealed that the impact of inattentiveness on analyses can be substantial.     

The Effects of Branching on Item Nonresponse

The analysis of responses to a large sample survey instrumentindicates that branching instructions significantly increasethe rate of item nonresponse for items immediately followingthe branch. While older respondents did have a higher incidenceof item nonresponse, none of six other factors (education, gender,distance to next question, number of previous branches, futurebehavior, or nature of response) hypothesized to affect itemnonresponse were significantly related to the rate of nonresponse.     

Web Survey Design: Paging versus Scrolling

A key choice in the design of Web surveys is whether to placethe survey questions in a multitude of short pages or in longscrollable pages. There are advantages and disadvantages ofeach approach, but little empirical evidence to guide the choice.In 2003 we conducted a survey of over 21,000 undergraduate students.Ten percent of the 10,000 respondents were directed to the scrollableversion of the survey, containing a single form for each ofthe major sections. The balance was assigned to the paging version,in which questions were presented to be visible without scrolling.The instrument contained a maximum of 268 possible questions,including topics that varied in sensitivity and desirability.The survey also permitted comparison of the effect of skip patternsby implementing skip instructions and hyperlinks in the scrollabledesign, and also recorded time at the end of each of the fivetopical sections. Differences between designs are evaluatedin terms of various forms of nonresponse, univariate and bivariatemeasurement properties, and proxies for respondent burden.     

The State of Surveying Cell Phone Numbers in the United States: 2007 and Beyond

By the late 1970s, household telephone coverage grew to exceed90 percent in the United States, and by the mid-1980s telephonesurveying of the general public had become commonplace. Nevertheless,20 years later, the ability of researchers to reach representativesamples of the U.S. public via landline (wired) telephone surveysand gather reliable data is being seriously challenged for manyreasons, especially those related to cell phones and the growthof the "cell phone only" population. However, at present thereexists no widely accepted set of Cell Phone Surveying "bestpractices" for U.S. survey researchers to follow. Despite whatsome appear to believe, surveying persons reached on cell phonenumbers in the United States currently is a very complex undertakingif one wants to do it "right," i.e. to do it legally, ethically,and in ways that optimally allocate one's finite resources togather the highest quality data, and to analyze and interpretthose data accurately. This final "wrap-up" article in the specialissue provides a review of the empirical articles in the issuewith a focus on their practical implications for the decisionsthat researchers need to make regarding sampling, coverage,nonresponse, measurement, and weighting in surveys that includeinterviews with persons reached on cell phones. The articlealso highlights the practical implications of a number of legal,ethical, and other issues that relate to surveys in the UnitedStates that include cell phone numbers. Surveying the U.S. cellphone population is possible, if at a higher cost than surveyingtheir landline counterparts, and if with less precision thancurrently can be done surveying the landline population. Thenext five years should see a considerable growth in the methodologicaland statistical know-how that the survey community uses to plan,implement, and interpret cell phone surveys. There is a greatdeal that still must be learned.     

DO RESPONDENTS REALLY "MARK ALL THAT APPLY" ON SELF-ADMINISTERED QUESTIONS?

An experiment was conducted to assess the effect of using "markall that apply" question instructions on survey reporting aspart of the field test for the Second Follow-up of the NationalEducation Longitudinal Study of 1988 Eighth Graders (NELS:88).Mark-all-that-apply instructions were compared with instructionsasking respondents to indicate "yes" or "no" to each responseoption on responses to three items dispersed throughout thequestionnaire and consisting of different topics and numbersof response options. For the three items, significantly fewerresponse options were selected with the mark-all-that-applyinstructions than with the yes/no instructions, but becauseexternal validity criteria were not available, overreportingto the yes/no instructions cannot be ruled out. Instructiondependentprimacy effects, predicted under the hypothesis that respondentswould engage in more superficial processing when given the mark-all-that-applyinstructions, were not found.     

Eating Disorder Symptoms Among College Students: Prevalence,Persistence, Correlates,and Treatment-Seeking

Abstract

Objective: To examine the prevalence, correlates, persistence, and treatment-seeking related to symptoms of eating disorders (EDs) in a random sample of college students. Participants: A random sample of students at a large university were recruited for an Internet survey in Fall 2005 and a follow-up survey in Fall 2007. Methods: ED symptoms were measured using the SCOFF screen and adjusted for nonresponse using administrative data and a nonresponse survey. Results: 2,822 (56%) students completed the baseline survey. Among undergraduates the prevalence of positive screens was 13.5% for women and 3.6% for men. Among students with positive screens, 20% had received past-year mental health treatment. In the follow-up sample (N = 753), ED symptoms at baseline significantly predicted symptoms 2 years later. Conclusions: Symptoms of EDs were prevalent and persistent among college students in this study. These findings suggest that brief screens can identify a large number of students with untreated EDs.     

Nonresponse in the American Time Use Survey: Who Is Missing from the Data and How Much Does It Matter?

This article examines nonresponse in a large government survey,the American Time Use Survey (ATUS), which interviews personsin households previously interviewed in the Current PopulationSurvey. The response rate for the ATUS has been below 60 percentfor the first two years of its existence, raising questionsabout whether the results can be generalized to the target population.The article begins with an analysis of the types of nonresponseencountered in the ATUS. Noncontact accounts for roughly 60percent of ATUS nonresponse, with refusals accounting for roughly40 percent. We find little support for the hypothesis that busypeople are less likely to respond to the ATUS but find considerablesupport for the hypothesis that people who are weakly integratedinto their communities are less likely to respond, mostly becausethey are less likely to be contacted. When we compare aggregateestimates of time use calculated using the ATUS base weightswithout any adjustment for nonresponse, estimates calculatedusing the ATUS final weights with a nonresponse adjustment,and estimates calculated using weights that incorporate ourown nonresponse adjustment based on a propensity model, we findsome modest differences, but the three sets of estimates arebroadly similar. The article ends with suggestions for furtherresearch and analysis.     

ESTIMATING BIAS DUE TO NONRESPONSE IN MAIL SURVEYS

A procedure for minimizing nonresponse error in a self-administeredmail waterfowl harvest survey was tested on a stratified sampleof 3,360 Canada Migratory Game Bird Hunting Permit purchasersin Ontario. On the assumption that follow-ups probe deeper intothe core of nonrespondents, a linear regression model for estimatingparameter values of the population while correcting for nonresponsebias was devised using cumulated responses over three successivemailings. It was estimated that nonrespondents who tended tohave a significantly lower level of participation and involvementin the topic investigated were younger and resided in ruralareas of the province. Nonresponse bias was as high as 14.4percent for waterfowl kill per day of hunting and 11.1 percentfor age of hunters. Results confirm the usefulness of follow-upsof nonrespondents as a means of exploring and correcting fornonresponse error