Is the web a promising tool for data collection in developing countries? An analysis of the sample bias of 10 web and face-to-face surveys from Africa,Asia, and South America

Whereas the sample composition biases of web surveys have been discussed extensively for developed countries, studies for developing countries are scarce. This article helps to fill that gap by comparing similar non-probability-based web surveys (WEB) and probability-based face-to-face (F2F) surveys both to each other and to the labor force. An analysis of WageIndicator data on work and wages derived from surveys held in 2009–2013 in 10 developing countries (WEB-sample N = 9135; F2F-sample N = 14,659), shows that F2F samples resemble the labor force to a larger extent than web samples do. In both cases, individuals in their 20s and early 30s are overrepresented, and younger and older respondents are underrepresented. This trend is more pronounced in WEB than in F2F samples. However, the differences converge in countries with higher Internet usage. A comparison of the WEB and F2F samples shows that compositions differ greatly, with web respondents being younger, more often male, more often living alone, and higher educated, although these differences are smaller in countries with higher Internet usage. Given the cost differences between the two survey modes, one should nevertheless consider the potential of web surveys as an instrument to gain explorative insights, specifically when searching for individuals with particular characteristics.     

The effect of personalization on response rates and data quality in web surveys

Personalizing correspondence has often shown to significantly increase survey response rates in mail surveys. This study experimentally tests whether personalization of email invitations acts correspondingly to web survey response rates. Also, it is investigated whether personalization influences data quality. The results of the study, using a large student sample, show that personalization significantly increases the web survey response rate by 8.6 percentage points. The data quality does not appear to be affected in any major way by personalizing the email invitations. However, the analyses do show that respondents of the personalization condition tend to respond with more social desirability bias to sensitive questions. Therefore, it is concluded that personalization has positive effects on the survey response rate, but one should carefully consider whether or not to personalize when a survey on sensitive topics is conducted.     

Estimating development cost of an interactive website based cancer screening promotion program

ObjectivesThe aim of this study was to estimate the initial development costs for an innovative talk show format tailored intervention delivered via the interactive web, for increasing cancer screening in women 50–75 who were non-adherent to screening guidelines for colorectal cancer and/or breast cancer.MethodsThe cost of the intervention development was estimated from a societal perspective. Micro costing methods plus vendor contract costs were used to estimate cost. Staff logs were used to track personnel time. Non-personnel costs include all additional resources used to produce the intervention.ResultsDevelopment cost of the interactive web based intervention was $.39 million, of which 77% was direct cost. About 98% of the cost was incurred in personnel time cost, contract cost and overhead cost.ConclusionsThe new web-based disease prevention medium required substantial investment in health promotion and media specialist time. The development cost was primarily driven by the high level of human capital required. The cost of intervention development is important information for assessing and planning future public and private investments in web-based health promotion interventions.     

Measuring expectations of inflation: Effects of survey mode,wording, and opportunities to revise

Several national surveys aim to elicit consumers’ inflation expectations. Median expectations tend to track objective inflation estimates over time, although responses display large dispersion. Medians also tend to differ between surveys, possibly reflecting survey design differences. Using a nationally representative Dutch sample, we evaluate the importance of three survey design features in explaining observed differences: mode (face-to-face vs. web), question wording (‘prices in general’ vs. ‘inflation’), and the explicit opportunity to revise responses. We examine effects on item non-responses, revisions, reported inflation expectations and their deviation from the CPI inflation rate. We discuss implications of our findings for survey design.     

Worksite e-mail health promotion trial: Early lessons

In preparation for an electronic mail (e-mail) and web-based health promotion intervention across multiple worksites, secure, regulatory-compliant, user-friendly e-mail and Internet applications were used to recruit potential participants across worksites, to enroll participants, and to collect baseline health assessment data. Specific hardware and software information technology environments were required of the 19 participating worksites. Sequential e-mails introduced the study and invited participation. Twenty-four percent of all employees (1106 of approximately 4600) provided consent. E-mail delivered a web link for the baseline study assessment, and reminder e-mails were sent to prompt completion. Of those who consented, 888 (80%) completed baseline health and behavior data surveys. An HTML-native web survey software was more stable across computing environments. Using e-mail and web assessment, this research recruited, enrolled, and collected data from more than 850 participants. Technical and operational challenges emerged at each step. Solutions and recommendations are discussed. Overall, this experience suggests that the use of e-mail and web software can facilitate recruitment, enrollment, and data acquisition through direct contact with study participants. This experience yields a series of lessons learned for using e-mail and the Internet to support multi-site trials.     

SurveyMotion: what can we learn from sensor data about respondents’ completion and response behavior in mobile web surveys?

Participation in web surveys via smartphones increased continuously in recent years. The reasons for this increase are a growing proportion of smartphone owners and an increase in mobile Internet access. However, research has shown that smartphone respondents are frequently distracted and/or multitasking, which might affect completion and response behavior in a negative way. We propose ‘SurveyMotion (SMotion)’, a JavaScript-based tool for mobile devices that can gather information about respondents’ motions during web survey completion by using sensor data. Specifically, we collect data about the total acceleration (TA) of smartphones. We conducted a lab experiment and varied the form of survey completion (e.g. standing or walking). Furthermore, we employed questions with different response formats (e.g. radio buttons and sliders) and measured response times. The results reveal that SMotion detects higher TAs of smartphones for respondents with comparatively higher motion levels. In addition, respondents’ motion level affects response times and the quality of responses given. The SMotion tool promotes the exploration of how respondents complete mobile web surveys and could be employed to understand how future mobile web surveys are completed.     

Response Results of Lottery Buyer Behavior Surveys: In-Home vs. Point-of-Purchase Interviews

This paper reports the response results of surveys on lotterypurchasing behavior. Data are based on answers generated fromtwo probability samples: a national, random sample of 1,646households (in-home survey) and a point-of-purchase sample of520 respondents. Results show that answers to the point-of-purchasesurvey were more accurate at half the cost of in-home interviews.The surveys were undertaken in Israel, but have implicationsfor those wishing to sample lottery purchasers in other countries.     

Presentation matters: how mode effects in item non-response depend on the presentation of response options

Comparability of data across modes is an important issue in survey research. In this paper we discuss item non-response to attitudinal questions in telephone and web surveys. We present results from a survey experiment conducted in Italy and in Spain that compares different presentations of response options in an online setting with a benchmark telephone survey. In line with earlier studies we find that (A), the share of non substantial answers in the online survey depends on how the response option is presented. Comparing different presentations in an online survey to the standard approach of telephone surveys using propensity score matching, we find that (B), the share and pattern of non substantial answers, is most similar across the two modes in the online survey when the it is captured in an instruction on each screen for the online survey. Our findings are of particular relevance for the design of multiple mode or mix-mode surveys of attitudinal questions using online and telephone modes.     

Does "Yes or No" on the Telephone Mean the Same as "Check-All-That-Apply" on the Web?

Recent experimental research has shown that respondents to forced-choicequestions endorse significantly more options than respondentsto check-all questions. This research has challenged the commonassumption that these two question formats can be used interchangeablybut has been limited to comparisons within a single survey mode.In this paper we use data from a 2004 random sample survey ofuniversity students to compare the forced-choice and check-allquestion formats across web self-administered and telephoneinterviewer-administered surveys as they are commonly used insurvey practice. We find that the within-mode question formateffects revealed by previous research and reaffirmed in thecurrent study appear to persist across modes as well; the telephoneforced-choice format produces higher endorsement than the webcheck-all format. These results provide further support forthe argument that the check-all and forced-choice question formatsdo not produce comparable results and are not interchangeableformats. Additional comparisons show that the forced-choiceformat performs similarly across telephone and web modes.     

Factors that influence acceptance of web-based e-learning systems for the in-service education of junior high school teachers in Taiwan

Web-based e-learning is not restricted by time or place and can provide teachers with a learning environment that is flexible and convenient, enabling them to efficiently learn, quickly develop their professional expertise, and advance professionally. Many research reports on web-based e-learning have neglected the role of the teacher's perspective in the acceptance of using web-based e-learning systems for in-service education. We distributed questionnaires to 402 junior high school teachers in central Taiwan. This study used the Technology Acceptance Model (TAM) as our theoretical foundation and employed the Structure Equation Model (SEM) to examine factors that influenced intentions to use in-service training conducted through web-based e-learning. The results showed that motivation to use and Internet self-efficacy were significantly positively associated with behavioral intentions regarding the use of web-based e-learning for in-service training through the factors of perceived usefulness and perceived ease of use. The factor of computer anxiety had a significantly negative effect on behavioral intentions toward web-based e-learning in-service training through the factor of perceived ease of use. Perceived usefulness and motivation to use were the primary reasons for the acceptance by junior high school teachers of web-based e-learning systems for in-service training.     

Methodologic challenges of e-health research

In this article, research-based strategies used in a web-based study to assess the impact of participation in cancer-related mailing lists on cancer survivors and caregivers are highlighted. The paper presents alternative methods of measuring response rates in Internet surveys. Response rate estimation, based on American Association for Public Opinion Research guidelines, ranged between 15% and 24.9% for the study. Several strategies were implemented to increase survey participation and reduce item nonresponse. Few participants opted for a telephone survey as an alternative to a web-administered design (mixed-mode administration). Using time-stamp data to diagnose places in the questionnaire where respondents prematurely exited or broke off survey participation, we were able to decrease premature break offs. A lottery was also implemented to increase response rates. The experience of obtaining institutional review board approval for the lottery is also described. Preliminary data indicate slight increases in response rates subsequent to implementing the lottery.     

Reliability and validity of egocentered network data collected via web: A meta-analysis of multilevel multitrait multimethod studies

Our goal in this article is to assess reliability and validity of egocentered network data collected through web surveys using multilevel confirmatory factor analysis under the multitrait multimethod approach.In this study, we analyze a questionnaire of social support of Ph.D. students in three European countries. The traits used are the frequency of social contact questions. The methods used are web survey design variants.We consider egocentered network data as hierarchical; therefore, a multilevel analysis is required. Within and between-ego reliabilities and validities are defined and interpreted.Afterwards, we proceed to a meta-analysis of the results of the three countries where within and between-ego validities and reliabilities are predicted from survey design variables which have to do with question order (by questions or by alters), response category labels (end labels or all labels) and lay-out of the questionnaire (graphical display or plain text). Results show that question order by questions, all-labeled response categories and a graphical display lay-out with images lead to a better data quality. Our basic approach consisting on multilevel and meta-analysis can be applied to evaluate the quality of any type of egocentered network questionnaire, regardless of the data collection mode.     

Perception of alcoholism among Jewish, Moslem and Christian teachers in Israel.

This article describes a survey which investigated perception of alcoholism among Jewish, Moslem and Christian teachers in the north of Israel during the spring of 1991. Data were obtained from a sample of 553 teachers. The teachers were asked to agree or disagree with items associated with the disease concept or with the moral concept of alcoholism. The findings indicate differences in the perception of alcoholism among teachers of different religions, education levels, gender, and drinking practices. No differences were found among teachers of different ages or among those working in different types of schools.     

Comparison of web and mail surveys in collecting illicit drug use data: a randomized experiment

This randomized experiment examined survey mode effects for self-reporting illicit drug use by comparing prevalence estimates between a Web-based survey and a mail-based survey. A random sample of 7,000 traditional-aged undergraduate students attending a large public university in the United States was selected to participate in the spring of 2001. Students were randomly assigned to self-administer a survey via the Web or U.S. mail. The Web survey produced a significantly higher response rate than the U.S. mail survey. The prevalence estimates of illicit drug use (lifetime and past year) did not differ significantly between the two survey modes. The findings provide preliminary evidence that Web and U.S. mail surveys produce similar results regarding illicit drug use among undergraduate students. Although additional research is needed involving more diverse samples, these findings bode well for using Web surveys in college-based research.     

Counting the uninsured: a comparison of national surveys

This Issue Brief reviews surveys that provide estimates of the uninsured population in the United States. It includes a discussion of why the estimates from the various surveys differ. It is important to understand the differences in the estimates of the uninsured population. The projected cost of implementing policy proposals depends on the estimates of the number of people affected by the proposals; for instance, the allocation of funding for the State Children's Health Insurance Program (S-CHIP) depends heavily on the available estimates. In addition, the estimated effectiveness of policy proposals to reduce the uninsured population will be accurate only if the correct count is known and the precise make-up of the uninsured population is understood. Currently, seven surveys can be used to make nationally representative estimates of the number of people without health insurance coverage. Some of the surveys collect health insurance information in the context of obtaining general information on health care, while other surveys are focused on other topics such as labor force participation and public assistance program participation. The most widely used survey that collects information on health insurance coverage is the Current Population Survey (CPS), conducted by the Census Bureau. The most recent estimates from the CPS suggests that 44.3 million Americans were uninsured in 1998. Besides the CPS, a number of other surveys collect information on the uninsured population. They include the Survey of Income and Program Participation (SIPP), Behavioral Risk Factor Surveillance System (BRFSS), Community Tracking Study (CTS), Medical Expenditure Panel Survey (MEPS), National Health Interview Survey (NHIS), and the National Survey of America's Families (NSAF). Estimates of the uninsured from these surveys range from 19 million to 44 million and vary depending on the time frame the survey covers. A number of states have started to question the validity of the uninsured estimates from the CPS, and other surveys, because of the small sample size in many states. As a result, some states have begun to conduct their own surveys to determine the number of uninsured residents. States that regularly conduct their own surveys include Florida, Massachusetts, Minnesota, New Mexico, Oregon, Vermont, and Wisconsin. Unfortunately, the various state surveys are not easily comparable. Research needs to continue to increase understanding of the differences among the surveys and to improve on methodologies to count the uninsured, as the future of public programs, such as S-CHIP and other state and local initiatives to expand health insurance coverage, depends on the accuracy of these estimates. Whatever survey is used, the results show that a substantial number of Americans do not have any health insurance coverage, and the number has been growing.     

A Comparison of Address-Based Sampling (ABS) Versus Random-Digit Dialing (RDD) for General Population Surveys

Valid and reliable public health data are becoming more difficultto obtain through random-digit dial (RDD) telephone surveys.As a result, researchers are evaluating different survey designs(i.e., sampling frame and survey mode combinations) as complementsor alternatives to RDD. Traditionally, mail surveys of the generalpublic have been limited due to a lack of a complete samplingframe of households. Recent advances in electronic record keeping,however, have allowed researchers to develop a sample from aframe of addresses (e.g., the U.S. Postal Service Delivery SequenceFile, which appears to provide coverage which rivals or possiblyexceeds that obtained through RDD sampling methods). To testthe use of this frame for surveying adults aged 18 years andolder across a wide geographic area, a pilot study was conductedas part of the 2005 Behavioral Risk Factor Surveillance System(BRFSS). The pilot compared use of a traditional, RDD telephonesurvey methodology to an approach using a mail version of thequestionnaire completed by a random sample of households drawnfrom an address-based frame. The findings indicate that themail survey approach can achieve higher response rates in low-response-ratestates (<40%) than RDD (particularly when two mailings aresent). Additionally, the address frame with mail survey designprovides access to cell phone only households and offers costsavings over the telephone approach. The resulting sample, however,significantly overrepresents non-Hispanic whites and peoplewith higher levels of education.     

UNDERSTANDING THE DECISION TO PARTICIPATE IN A SURVEY

The lack of full participation in sample surveys threatens theinferential value of the survey method. We review a set of conceptualdevelopments and experimental findings that appear to be informativeabout causes of survey participation; offer an integration ofthat work with findings from the more traditional statisticaland survey methodological literature on nonresponse; and, giventhe theoretical structure, deduce potentially promising pathsof research toward the understanding of survey participation.     

Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey

Declining contact and cooperation rates in random digit dial(RDD) national telephone surveys raise serious concerns aboutthe validity of estimates drawn from such research. While researchin the 1990s indicated that nonresponse bias was relativelysmall, response rates have continued to fall since then. Thecurrent study replicates a 1997 methodological experiment thatcompared results from a "Standard" 5-day survey employing thePew Research Center’s usual methodology with results froma "Rigorous" survey conducted over a much longer field periodand achieving a significantly higher response rate. As withthe 1997 study, there is little to suggest that unit nonresponsewithin the range of response rates obtained seriously threatensthe quality of survey estimates. In 77 out of 84 comparableitems, the two surveys yielded results that were statisticallyindistinguishable. While the "Rigorous" study respondents tendedto be somewhat less politically engaged, they did not reportconsistently different behaviors or attitudes on other kindsof questions. With respect to sample composition, the Standardsurvey was closely aligned with estimates from the U.S. Censusand other large government surveys on most variables. We extendour analysis of nonresponse to include comparisons with thehardest-to-reach respondents and with respondents who terminatedthe interview prior to completion.     

Monitoring youth behavior in response to structural changes. Alternative approaches for measuring adolescent drinking

Random digit dial (RDD) telephone and self-administered school-based surveys of drinking and drinking and driving were compared for adolescents from two Northern California communities. The RDD and school-based surveys resulted in very different samples. The telephone sample contained a greater proportion of European Americans and a smaller proportion of Asian Americans and "other" ethnicities. Respondents to the telephone sample also tended to be older and of higher socioeconomic status (SES). The telephone survey captured very few school dropouts. Moreover, it resulted in lower self-reports of drinking and drinking and driving. Survey mode appeared to influence respondents equally, regardless of their gender, age, ethnicity, or SES.     

Older Adults and the Digital Divide: Assessing Results of a Web-Based Survey

SUMMARY

This study used an on-line, web-based survey to assess the significance of computer and Internet technology in the lives of adults over age 60. A convenience sample of 110 individuals from the United States, Canada and other countries responded to a 20-question survey regarding individual use, opportunities for learning, family and social connectivity and preferences for and barriers to effectual use. Particular focus was made on the self-described perceptions of personal control and life satisfaction within the responding population.