A framework for evaluating eHealth research

Health care is in the midst of a consumer-oriented technology explosion. Individuals of all ages and backgrounds have discovered eHealth. But the challenges of implementing and evaluating eHealth are just beginning to surface, and, as technology changes, new challenges emerge. Evaluation is critical to the future of eHealth. This article addresses four dimensions of eHealth evaluation: (1) design and methodology issues; (2) challenges related to the technology itself; (3) environmental issues that are not specific to eHealth but pose special problems for eHealth researchers; and (4) logistic or administrative concerns of the evaluation methodology selected. We suggest that these four dimensions must be integrated to provide a holistic framework for designing and implementing eHealth research projects, as well as for understanding the totality of the eHealth intervention. The framework must be flexible enough to adapt to a variety of end users, regardless of whether the end user is a healthcare organization, a for-profit business, a community organization, or an individual. The framework is depicted as a puzzle with four interlocking pieces.     

Worksite e-mail health promotion trial: Early lessons

In preparation for an electronic mail (e-mail) and web-based health promotion intervention across multiple worksites, secure, regulatory-compliant, user-friendly e-mail and Internet applications were used to recruit potential participants across worksites, to enroll participants, and to collect baseline health assessment data. Specific hardware and software information technology environments were required of the 19 participating worksites. Sequential e-mails introduced the study and invited participation. Twenty-four percent of all employees (1106 of approximately 4600) provided consent. E-mail delivered a web link for the baseline study assessment, and reminder e-mails were sent to prompt completion. Of those who consented, 888 (80%) completed baseline health and behavior data surveys. An HTML-native web survey software was more stable across computing environments. Using e-mail and web assessment, this research recruited, enrolled, and collected data from more than 850 participants. Technical and operational challenges emerged at each step. Solutions and recommendations are discussed. Overall, this experience suggests that the use of e-mail and web software can facilitate recruitment, enrollment, and data acquisition through direct contact with study participants. This experience yields a series of lessons learned for using e-mail and the Internet to support multi-site trials.     

Challenges in telemedicine equivalence studies

As the number of telemedicine programs continues to grow, the accuracy of diagnosis over interactive televideo is a central concern. Although investigators have begun to address diagnostic equivalency in telehealth clinics, few published studies reflect strong research design. The two presented telehealth programs completed randomized controlled trials in real-world clinical settings that addressed some of the methodologic shortcomings of prior studies. Diagnostic equivalency studies were completed across five telehealth specialty clinics: physical therapy, speech therapy, ambulatory pediatrics, child psychiatry, and developmental disabilities services. The two research teams encountered similar decision points in designing and implementing the equivalency protocols. This article addresses methodologic issues in choosing design, participants, technology, and evaluation measures. Although the paper focuses on interactive televideo, the issues raised are pertinent across telehealth technologies.     

Tailored program evaluation: Past, present, future

This paper discusses measurement issues related to the evaluation of computer-tailored health behavior change programs. As the first generation of commercially available tailored products is utilized in health promotion programming, programmers and researchers are becoming aware of the unique challenges that the evaluation of these programs presents. A project is presented that used an online tailored health behavior assessment (HBA) in a worksite setting. Process and outcome evaluation methods are described and include the challenges faced, and strategies proposed and implemented, for meeting them. Implications for future research in tailored program development, implementation, and evaluation are also discussed.     

Multidisciplinary eHealth survey evaluation methods

This paper describes the development process of an evaluation framework for describing and comparing web survey tools. We believe that this approach will help shape the design, development, deployment, and evaluation of population-based health interventions. A conceptual framework for describing and evaluating web survey systems will enable the developers of health interventions to identify, select, and use the most appropriate and relevant survey tools. The primary contributors included survey methodologists, computer scientists, informaticians, technical communications specialists, educators, evaluators, clinical practitioners, and health services researchers, who together developed a comprehensive ontology to describe the electronic survey domain. This paper will describe the survey ontology development process and detail lessons learned in the creation of the web survey system evaluation framework.