A framework for evaluating eHealth research

Health care is in the midst of a consumer-oriented technology explosion. Individuals of all ages and backgrounds have discovered eHealth. But the challenges of implementing and evaluating eHealth are just beginning to surface, and, as technology changes, new challenges emerge. Evaluation is critical to the future of eHealth. This article addresses four dimensions of eHealth evaluation: (1) design and methodology issues; (2) challenges related to the technology itself; (3) environmental issues that are not specific to eHealth but pose special problems for eHealth researchers; and (4) logistic or administrative concerns of the evaluation methodology selected. We suggest that these four dimensions must be integrated to provide a holistic framework for designing and implementing eHealth research projects, as well as for understanding the totality of the eHealth intervention. The framework must be flexible enough to adapt to a variety of end users, regardless of whether the end user is a healthcare organization, a for-profit business, a community organization, or an individual. The framework is depicted as a puzzle with four interlocking pieces.     

eHealth recruitment challenges

Little is known about effective eHealth recruitment methods. This paper presents recruitment challenges associated with enrolling African-American girls aged 8–10 years in an eHealth obesity prevention program, their effect on the recruitment plan, and potential implications for eHealth research.Although the initial recruitment strategy was literature-informed, it failed to enroll the desired number of girls within a reasonable time period. Therefore, the recruitment strategy was reformulated to incorporate principles of social marketing and traditional marketing techniques. The resulting plan included both targeted, highly specific strategies (e.g., selected churches), and more broad-based approaches (e.g., media exposure, mass mailings, radio advertisements).The revised plan enabled recruitment goals to be attained. Media appeared to be particularly effective at reaching the intended audience. Future research should identify the most effective recruitment strategies for reaching potential eHealth audiences.     

Stakeholder driven indicators for eHealth performance management

The goal of the present article is to compile a corpus of indicators of eHealth development evaluation that would essentially reflect stakeholder approaches and complement technical indicators of assessment of an eHealth system. Consequently, the assessment of the development of an eHealth system would reflect stakeholder approaches and become an innovative solution in attempting to improve productivity of IT projects in the field of health care. The compiled minimum set of indicators will be designed to monitor implementation of the national eHealth information system. To ensure reliability of the quality research, the respondents were grouped in accordance to the geographical distribution and diversity of the levels and types of the represented jobs and institutions. The applied analysis implies several managerial insights on the hierarchy of eHealth indicators. These insights may be helpful in recommending priority activities in implementation of an eHealth data system on the national or international level. The research is practically useful as it is the first to deal with the topic in Lithuania and its theoretical and practical aspect are particularly relevant in implementation of an eHealth data system in Lithuania. The eHealth assessment indicators presented in the article may be practically useful in two aspects: (1) as key implementation guidelines facilitating the general course of eHealth system development and (2) as a means to evaluate eHealth outcomes.     

Planning and implementation of a participatory evaluation strategy: A viable approach in the evaluation of community-based participatory programs addressing cancer disparities

Community-based participatory research (CBPR) has been posited as a promising methodology to address health concerns at the community level, including cancer disparities. However, the major criticism to this approach is the lack of scientific grounded evaluation methods to assess development and implementation of this type of research. This paper describes the process of development and implementation of a participatory evaluation framework within a CBPR program to reduce breast, cervical, and colorectal cancer disparities between African Americans and whites in Alabama and Mississippi as well as lessons learned. The participatory process involved community partners and academicians in a fluid process to identify common ground activities and outcomes. The logic model, a lay friendly approach, was used as the template and clearly outlined the steps to be taken in the evaluation process without sacrificing the rigorousness of the evaluation process. We have learned three major lessons in this process: (1) the importance of constant and open dialogue among partners; (2) flexibility to make changes in the evaluation plan and implementation; and (3) importance of evaluators playing the role of facilitators between the community and academicians. Despite the challenges, we offer a viable approach to evaluation of CBPR programs focusing on cancer disparities.     

Evaluation capacity building in the nonformal education context: Challenges and strategies

Policymakers’ demand for increased accountability has compelled organizations to pay more attention to internal evaluation capacity building (ECB). The existing literature about ECB has focused on capacity building experiences and organizational research, with limited attention on challenges that internal evaluation specialists face in building organizational evaluative capacity. To address this knowledge gap, we conducted a Delphi study with evaluation specialists in the United States’ Cooperative Extension Service and developed a consensus on the most pervasive ECB challenges as well as the most useful strategies for overcoming ECB challenges. Challenges identified in this study include limited time and resources, limited understanding of the value of evaluation, evaluation considered as an afterthought, and limited support and buy-in from administrators. Alternatively, strategies found in the study include a shift in an organizational culture where evaluation is appreciated, buy-in and support from administration, clarifying the importance of quality than quantity of evaluations, and a strategic approach to ECB. The challenges identified in this study have persisted for decades, meaning administrators must understand the persistence of these issues and make an earnest investment (financial and human resource) to make noticeable progress. The Delphi approach can be used more often to prioritize ECB efforts.     

The Journey Around My PhD: Pitfalls,Insights and Diamonds

Embarking on postgraduate research involves making a transition from practitioner to researcher. While there is a well established body of knowledge to address research methodology, it is presented in a fashion that suggests the research process is straightforward and linear and fails to offer insights into the ways postgraduate students construct their researcher identities. This paper will draw on the authors' experiences as PhD students and highlight what was learned about the research process from involvement in a research project. The paper will highlight some of the challenges of undertaking postgraduate research as well as some of the insights gained.     

An evaluation capacity building toolkit for principal investigators of undergraduate research experiences: A demonstration of transforming theory into practice

This paper describes the approach and process undertaken to develop evaluation capacity among the leaders of a federally funded undergraduate research program. An evaluation toolkit was developed for Computer and Information Sciences and Engineering¹ Research Experiences for Undergraduates² (CISE REU) programs to address the ongoing need for evaluation capacity among principal investigators who manage program evaluation. The toolkit was the result of collaboration within the CISE REU community with the purpose being to provide targeted instructional resources and tools for quality program evaluation. Challenges were to balance the desire for standardized assessment with the responsibility to account for individual program contexts. Toolkit contents included instructional materials about evaluation practice, a standardized applicant management tool, and a modulated outcomes measure. Resulting benefits from toolkit deployment were having cost effective, sustainable evaluation tools, a community evaluation forum, and aggregate measurement of key program outcomes for the national program. Lessons learned included the imperative of understanding the evaluation context, engaging stakeholders, and building stakeholder trust. Results from project measures are presented along with a discussion of guidelines for facilitating evaluation capacity building that will serve a variety of contexts.     

Challenges related to the analytical process in realist evaluation and latest developments on the use of NVivo from a realist perspective

ABSTRACT

Realist evaluation (RE) is a research design increasingly used in program evaluation, that aims to explore and understand the influence of context and underlying mechanisms on intervention or program outcomes. Several methodological challenges, however, are associated with this approach. This article summarizes RE key principles and examines some documented challenges and solutions when analyzing RE data, including the development of Context-Mechanism-Outcome configurations. An analytic method using NVivo features is also presented. This method makes it possible to respond to certain analytic difficulties associated with RE by facilitating the identification of patterns and ensuring transparency in the analytical process.     

Collaborative research in child welfare: a rationale for rigorous participatory evaluation designs to promote sustained systems change

Expansion of the child welfare evidence base is a major challenge. The field must establish how organizational systems and practice techniques yield outcomes for children and families. Needed research must be grounded in practice and must engage practitioners and administrators via participatory evaluation. The extent to which successful practices are transferable is also challenged by the diversity of child welfare systems. The U.S. Department of Health and Human Services' Children's Bureau funded Quality Improvement Centers (QICs) that were designed to promote collaborative, multisite research that would address these evaluation needs. This article, based on the findings of a regional and a national QIC, describes the challenges facing research collaboration and the strategies for achieving success.     

Experiences with systematic triangulation at the Global Environment Facility

Systematic triangulation may address common challenges in evaluation, such as the scarcity or unreliability of data, or the complexities of comparing and cross-checking evidence from diverse disciplines. Used to identify key evaluation findings, its application has proven to be effective in addressing the limitations encountered in country-level evaluation analysis conducted by the Independent Evaluation Office of the Global Environment Facility (GEF). These include the scarcity or unreliability of national statistics on environmental indicators and data series, especially in Least Developed Countries; challenges in evaluating the impacts of GEF projects; and inherent difficulties in defining the GEF portfolio of projects prior to the undertaking of the evaluation. In addition to responding to the need for further developing triangulation protocols, procedures and/or methodologies advocated by some authors, the approach offers a contribution to evaluation practice. This applies particularly to those evaluation units tasked with country-level evaluations in international organizations, facing similar constraints.     

From plan to action: Adapting evaluation to serve the developmental needs of a newly-funded multidisciplinary research center

The Southeastern Coastal Center for Agricultural Health and Safety (SCCAHS) is one of many newly-funded federal research centers, housing five multidisciplinary research projects and seven pilot projects, and serving a multi-state region. In the early stages of such a complex project, with multiple teams separated by geography and disciplines, the evaluation program has been integral in connecting internal and external stakeholders at the center and project levels. We used a developmental evaluation (DE) framework to respond to the complex political environment surrounding agricultural health and safety in the southeast; to engage external stakeholders in guiding the center’s research and outreach trajectories; to support center research teams in a co-creation process to develop logic models and tailored indicators; and to provide timely and feedback within the center to address communications gaps identified by the evaluation program. By using DE principles to shape monitoring and evaluation approaches, our evaluation program has adapted to the dynamic circumstances presented as our center’s progress has been translated from a plan in a grant proposal to implementation.     

Exploring challenges and lessons learned in cross-cultural environmental education research

The field of environmental education (EE) has been criticized for scarcity of research findings that can be used to compare the approaches and outcomes of various interventions and contexts. This is particularly problematic for EE programs that are implemented across multi-cultural settings because very little academic attention has been devoted to cross-cultural EE research methods. Intending to address this gap, we set out to develop and pilot a pre/post survey in Costa Rica that could help us investigate the impacts of a bird-focused curriculum on Latin American children’s knowledge, attitudes, behavioral intentions, and behaviors toward birds and the natural world. This article describes the challenges encountered and subsequent adaptations made throughout our ongoing, iterative survey development process, in order to appropriately address language, sociocultural context, audience, and research-to-practice tensions. We present key lessons learned, including the importance of having strong local partners, the need to create a realistic research timetable that accounts for unique challenges involved in undertaking cross-cultural EE research, and the limitations of quantitative methods in this sort of research.     

A senior manager with a knowledge management portfolio: the Santa Clara County experience

The agency director sought to create a systematically coordinated department that utilizes knowledge management strategies to promote evidence-informed practice. In his view, the organization was not providing needed information or organizational supports for practitioners to use knowledge effectively. To address this issue, he created a Director of Development and Operational Planning (DDOP) position with the responsibility to build structures and facilitate processes that support knowledge management. The DDOP oversees research and planning, government relations, legislative development and support, Board of Supervisors communications, staff development and training, community contracts, public information and in-house communication. The DDOP is reorganizing units under her supervision to create a knowledge management matrix that will implement new knowledge sharing strategies related to evaluation, contracts, legislation, organizational development, policy and planning, and staff development. The case study describes challenges and strategies related to: government regulations, size and complexity of the agency, staff resistance, and the developmental nature of the process.     

Fostering CHANGE: Lessons from implementing a multi-country,multi-sector nutrition-sensitive agriculture project

Using agriculture to improve nutrition is an approach growing in popularity, with programs becoming increasingly complex and multisectoral. While there is an active line of research assessing the impacts of such programs, little has been written about the process of successfully implementing them. As such, this paper uses a multisectoral nutrition-sensitive agriculture program implemented in four African countries as a case study to address key challenges in and lessons learned from implementation. We highlight the overall flexibility of nutrition-sensitive agriculture but also the need to adapt certain aspects to the particular context, as well as the opportunities for cross-context learning (and the limits to this). Integrating rigorous evaluation into such complex programs and forging diverse cross-sectoral partnerships offer both rewards and challenges, upon which we reflect. Main lessons learned from the program include the importance of carefully sequencing interventions, retaining flexibility in implementation, allowing for considerable time for cross-sector integration and coordination, and considering community impacts when designing research.     

Learning by doing: an experiential approach to program evaluation

Despite the increased importance of producing social workers who are prepared to evaluate practice, there is a paucity of literature addressing the pedagogical challenges of program evaluation courses. The present study evaluates the structure and pedagogical approach of a newly implemented program evaluation course. The course took a team-based, experiential learning approach to designing a program evaluation plan for students’ field placement agencies. Five student-led focus groups (N = 44) were conducted at the conclusion of the course to address two aims: to assess student perceptions of the content, structure, and delivery of the course; and to assess student attitudes toward applied research. Data were collected between November and December 2014 in a public university’s graduate social work program in the northeastern United States. The steps of grounded theory data analysis were implemented to elicit major themes from student feedback. Results suggest that perceptions of applicability play a central role in shaping student attitudes toward program evaluation. Experiential and team-based pedagogical approaches appear to increase accessibility of course content for students. Suggestions for enhancing student learning in program evaluation courses are discussed.     

Workforce training in family therapy and systemic practices: An evaluation framework and case study

Internationally, while the interdisciplinary field of family therapy and systemic practices is well established and evidence for therapeutic impact is advanced, evaluation of training and measurement of its impact lags behind. This paper addresses this gap, utilising a single case study at the Bouverie Centre, the largest family therapy and workforce development service in Australia. We describe (a) the scope of workforce training and implementation at the Centre, (b) the development of our training research and evaluation framework, and (c) standardised data collection methodology, including a pre-post training reflective survey. Collectively, this work comprises our current evaluation framework through which we will subsequently examine training impact. Future research and practice recommendations are offered to address challenges to workforce training evaluation and progress efficacy of family therapy training.     

Affirmative Practice and Alternative Sexual Orientations: Helping Clients Navigate the Coming Out Process

Those who differ from the dominant heterosexual ideal of exclusively other-sex attraction and intimacy encounter unique challenges, such as the coming out process, during which individuals with alternative sexual orientations must explore, define, and disclose their orientations in a way straight individuals need not. This article focuses on how clinicians can aid clients throughout the coming out process in a way that affirms the full range of sexual orientations. Following an overview of alternative sexual orientations and models of the coming out process, a case example is used to illustrate how clinicians can help clients address three challenges of coming out: overcoming internalized biases; clarifying their sexual orientation and identity; and making decisions about disclosure.     

Gene–Environment Interplay: Where We Are,Where We Are Going

The idea that both genetic and environmental influences contribute to behavioral outcomes is widely accepted. However, the practice of examining candidate Gene × Environment interaction (cGxE) is controversial. In this article, we summarize some of the key issues involved in cGxE research and provide recommendations for work in this area. Highlighted challenges include the selection of the gene, the development of the cGxE hypothesis, and the coding of the genotype. To address these challenges and gain confidence in cGxE findings, we recommend using empirical data to select and code genes/variants, using theory to develop cGxE hypotheses and a rigorous and transparent approach to hypothesis testing. Family researchers have much to offer to the study of Gene × Environment research in view of their process‐oriented theories that are grounded in decades of nuanced measurement of the environment; implementing these best practices will help deliver on that promise.     

The problems and prospects in the teaching of mixed methods research

There are pedagogical challenges USA students and instructors face within mixed methods classrooms. Instructors of mixed methods are often self-taught, lacking adequate training in both qualitative and quantitative approaches to research. Students are not often trained in both research approaches. These dual training gaps can result in deep pedagogical issues compromising students’ ability to fully understand mixed methods research praxis and leaving teachers feeling ill equipped to address students’ learning concerns. To tackle the myriad of challenges confronted in the mixed methods classroom requires structural changes to the current way graduate training programs in social research methods are organized and taught. Developing a team-based teaching approach to mixed methods research that provides students with instructors who have the requisite qualitative and quantitative knowledge can serve as a pedagogical model that can begin at least to address the current methods and methodological skills gap in the teaching of mixed methods research.