The Impact of Cell Phone Noncoverage Bias on Polling in the 2004 Presidential Election

Despite concerns that the accuracy of preelection telephonepolls would be harmed by the omission of voters who could bereached only by cell phone, most national polls performed wellin predicting President George W. Bush’s reelection in2004, and state polls were generally accurate as well. The nationalexit poll conducted by the National Election Pool found that7 percent of Election Day voters had cell phone service butno land line; younger voters were far more likely to be cell-only:19 percent among those age 18–24 and 20 percent amongthose age 25–29. Within these two youngest age cohorts,cell-only voters were significantly more likely to be singleand childless. While cell-only voters were more supportive ofJohn Kerry than voters overall, they were similar to other voterswithin their own age cohort. Because of this, preelection telephonesurveys that weighted their data appropriately by age were notsignificantly biased by the absence of the cell-only voters.     

What's Missing from National Landline RDD Surveys?: The Impact of the Growing Cell-Only Population

The number of cell phone only households has continued to grow– 12.8 percent of all households by the end of 2006, accordingto the National Health Interview Survey (NHIS). To assess theextent and nature of the potential bias in landline telephonesamples created by the absence of cell-only households, thePew Research Center conducted four independent dual frame studiesin 2006, three of which included at least 200 interviews withcell-only respondents. Despite the fact that there are significantdifferences between cell-only and landline respondents on manyimportant variables, across the four surveys with differentsubstantive content we find that including a cell-only samplewith a landline RDD sample produces general population estimatesthat are nearly identical to those from the landline samplealone. Yet, while the noncoverage problem is currently not damagingestimates for the entire population, we find evidence that itdoes create biased estimates on certain variables for youngadults, 25 percent of whom are cell-only according to the mostrecent government estimate.     

Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey

Declining contact and cooperation rates in random digit dial(RDD) national telephone surveys raise serious concerns aboutthe validity of estimates drawn from such research. While researchin the 1990s indicated that nonresponse bias was relativelysmall, response rates have continued to fall since then. Thecurrent study replicates a 1997 methodological experiment thatcompared results from a "Standard" 5-day survey employing thePew Research Center’s usual methodology with results froma "Rigorous" survey conducted over a much longer field periodand achieving a significantly higher response rate. As withthe 1997 study, there is little to suggest that unit nonresponsewithin the range of response rates obtained seriously threatensthe quality of survey estimates. In 77 out of 84 comparableitems, the two surveys yielded results that were statisticallyindistinguishable. While the "Rigorous" study respondents tendedto be somewhat less politically engaged, they did not reportconsistently different behaviors or attitudes on other kindsof questions. With respect to sample composition, the Standardsurvey was closely aligned with estimates from the U.S. Censusand other large government surveys on most variables. We extendour analysis of nonresponse to include comparisons with thehardest-to-reach respondents and with respondents who terminatedthe interview prior to completion.     

Consequences of reducing nonresponse in a national telephone survey

Critics of public opinion polls often claim that methodological shortcuts taken to collect timely data produce biased results. This study compares two random digit dial national telephone surveys that used identical questionnaires but very different levels of effort: a "Standard" survey conducted over a 5-day period that used a sample of adults who were home when the interviewer called, and a "Rigorous" survey conducted over an 8-week period that used random selection from among all adult household members. Response rates, computed according to AAPOR guidelines, were 60.6 percent for the Rigorous and 36.0 percent for the Standard study. Nonetheless, the two surveys produced similar results. Across 91 comparisons, no difference exceeded 9 percentage points, and the average difference was about 2 percentage points. Most of the statistically significant differences were among demographic items. Very few significant differences were found on attention to media and engagement in politics, social trust and connectedness, and most social and political attitudes, including even those toward surveys.