An introduction to methodological challenges in the evaluation of eHealth research: Perspectives from the Health e-Technologies Initiative

In February 2002 the Health e-Technologies Initiative (HETI), a program office of the Robert Wood Johnson Foundation^®, was created to advance discovery of scientific knowledge regarding the effectiveness of interactive eHealth applications. This article is the introduction to a series of seven articles written by grantees of HETI which address challenges, lessons learned, and proposed solutions as researchers implement eHealth projects. From this body of work it is clear that the overall process of conducting evaluation research in eHealth requires careful and detailed planning, recognition of the heightened sensitivity of IRBs, and institutions around the electronic collection and communication of personal health information, and a combination of tenacity and creativity to address the inevitable thorny methodological challenges to eHealth research. Use of established guidelines to help standardize the evaluation process, where feasible, is recommended.     

A strategy for monitoring and evaluating massive open online courses

We argue that the complex, innovative and adaptive nature of Massive Open Online Course (MOOC) initiatives poses particular challenges to monitoring and evaluation, in that any evaluation strategy will need to follow a systems approach. This article aims to guide organizations implementing MOOCs through a series of steps to assist them in developing a strategy to monitor, improve, and judge the merit of their initiatives. We describe how we operationalise our strategy by first defining the different layers of interacting agents in a given MOOC system. We then tailor our approach to these different layers. Specifically, a two-pronged approach was developed, where we suggest that individual projects be assessed through performance monitoring; assessment criteria for which would be defined at the outset to include coverage, participation, quality and student achievement. In contrast, the success of an overall initiative should be considered within a more adaptive, emergent evaluation inquiry framework. We present the inquiry framework we developed for MOOC initiatives, and show how this framework might be used to develop evaluation questions and an assessment methodology. We also define the more fixed indicators and measures for project performance monitoring. Our strategy is described as it was developed to inform the evaluation of a MOOC initiative at the University of Cape Town (UCT), South Africa.     

Issues and Challenges in Gatekeeping: A Framework for Implementation

Social work educators' role in gatekeeping within BSW and MSW academic programs continues to be a concern and challenge for the profession. Social work programs are the entry point in which students develop the required competencies to practice within the field. The social work literature on gatekeeping is extensive and expansive. The literature reviews and addresses the importance and need for gatekeeping as well as challenges in implementing gatekeeping within social work programs. This paper reviews the literature on gatekeeping in social work programs, discusses the legal issues pertinent to gatekeeping, and identifies the challenges faced by social work programs in implementing gatekeeping procedures. The paper suggests an integrated conceptual framework using the Council on Social Work Education's Educational Policy and Accreditation Standards (EPAS) for competencies in gatekeeping in social work programs.     

Towards a philosophy of interdisciplinarity

This paper aims to contribute to the expanding discourse on inter- and transdisciplinarity. Referring to well-established distinctions in philosophy of science, the paper argues in favor of a plurality of four different dimensions: Interdisciplinarity with regard to (a) objects (“ontology”), (b) knowledge/theories (epistemology), (c) methods/practices (methodology), and further, (d) problem perception/problem solving. Different philosophical thought traditions can be related to these distinguishable meanings. The philosophical framework of the four different dimensions will be illustrated by some of the most popular examples of research programs that are labeled “interdisciplinary”: nanoresearch/nanoscience/nanotechnology, complex systems theory/chaos theory, biomimicry/bionics, and technology assessment/sustainability research. Thus, a minimal philosophy of science is required to understand and foster inter- and transdisciplinarity.     

Casa de Esperanza [Part B]

When nonprofit organizations make significant changes in mission, there are many issues of organizational structure and culture that must be re‐examined and re‐aligned. As the second segment in the case of Casa de Esperanza illustrates, there are many human issues that must be navigated by leaders making such changes. Case B illustrates the challenges that must be confronted as leaders embrace the agency's identity as a community‐based Latina organization, rather than a government‐funded domestic violence organization. The case describes the management planning and implementation processes, including changes in leadership, programming, and operations. Staff responses to these changes are also stressed, revealing the very human elements of organizational change. © 2004 Jodi Sandfort     

Stakeholder driven indicators for eHealth performance management

The goal of the present article is to compile a corpus of indicators of eHealth development evaluation that would essentially reflect stakeholder approaches and complement technical indicators of assessment of an eHealth system. Consequently, the assessment of the development of an eHealth system would reflect stakeholder approaches and become an innovative solution in attempting to improve productivity of IT projects in the field of health care. The compiled minimum set of indicators will be designed to monitor implementation of the national eHealth information system. To ensure reliability of the quality research, the respondents were grouped in accordance to the geographical distribution and diversity of the levels and types of the represented jobs and institutions. The applied analysis implies several managerial insights on the hierarchy of eHealth indicators. These insights may be helpful in recommending priority activities in implementation of an eHealth data system on the national or international level. The research is practically useful as it is the first to deal with the topic in Lithuania and its theoretical and practical aspect are particularly relevant in implementation of an eHealth data system in Lithuania. The eHealth assessment indicators presented in the article may be practically useful in two aspects: (1) as key implementation guidelines facilitating the general course of eHealth system development and (2) as a means to evaluate eHealth outcomes.     

Science as a service: understanding successful knowledge transfer in a New Zealand research institute

This paper reports on an exercise conducted within a state-owned body (Crown Research Institute) in New Zealand aimed at building greater understanding of the key factors in successful research programmes. Success was defined in this study as a high level of uptake of the emerging science, with commensurate benefits to both industry and the community. The methodology had three parts. A review of the knowledge and technology transfer literature; a series of 15 semi-structured interviews with science leaders; and a facilitated workshop. The purpose of the review was to generate a robust framework upon which to centre the interview dialogues, and two models were selected. The results varied, reflecting the diversity of research services provided by the organization, but the findings were predominantly new and valuable. The importance of the long term relationship with the end users was the strongest recurring theme. The methodology may have wider application in both research and consulting settings; for the benefits derived from the interactive process with staff, as well as for the specific findings.     

Towards an assessment for organizational participatory research health partnerships: A systematic mixed studies review with framework synthesis

Within the health sciences, organizational participatory research (OPR) is defined as a blend of research and action, in which academic researchers partner with health organization members. OPR is based on a sound partnership between all stakeholders to improve organizational practices. However, little research on the evaluation of OPR health partnership exists. This systematic mixed studies review sought to produce a new theoretical model that structures the evaluation of the OPR processes and related outcomes of OPR health partnerships. Six bibliographic databases were searched together with grey literature sources for OPR health partnership evaluation questionnaires. Six questionnaires were included, from which a pool of 95 OPR health partnership evaluation items were derived. The included questionnaires were appraised for the quality of their origin, development and measurement properties. A framework synthesis was performed using an existing OPR framework by organizing questionnaire items in a matrix using a hybrid thematic analysis. This led to our proposed Organizational Participatory Research Evaluation Model (OPREM) that includes three axes, Trust, Collective Learning and Sustainability (with specific dimensions) and 95 items. This model provides information to help stakeholders comprehensively structure the evaluation of their partnerships and subsequent improvement; thus, potentially helping to improve health organization practices.     

Service users’ involvement in social work education: focus on social change projects

ABSTRACT

This article presents and analyzes four projects focusing on diverse forms of service users’ involvement in social work training and research in different countries (Israel, Italy, Slovenia and UK). It highlights the value of service user involvement (SUI) to specific social change objectives and to social work education. The conceptual framework focused on the Standpoint Theory, while methodologically participatory action research was applied, and evaluation measures were developed. Key findings, facilitators and limitations to the involvement, students’ views of it and similarities and differences among the four projects are outlined. The challenges embedded in introducing and sustaining social change objectives in a co-production framework within social work education are identified, alongside the added value of meeting them. The differentiated impact the projects had on students is highlighted as well as their significance for health and social care providers were relevant. It is encouraging that in each project SUI was positively valued. The projects indicate the wide range of SUI in the content and format of social work education, as well as its applicability cross-culturally to a range of key issues pertaining to both training and research in social work.     

Spirituality-Based Social Work Values for Empowering Human Service Organizations

Abstract

This paper is based on three premises: (1) Spirituality is seen as an asset from which the client system draws its innate strengths to empower itself, (2) Just as it is important for social work clients to build on individual spiritual strengths to overcome personal challenges, it is equally crucial for human service organizations to implement spirituality-based professional values in organizational contexts, and (3) Spirituality issues need to be examined from the perspective of macro social work education. Based on these premises, the goal of this paper is to construct a conceptual framework that links spirituality-based social work values to empowerment strategies that can be employed in macro social work practice. Adding to the existing dimensions of social work knowledge relating to spirituality, this paper examines spirituality as empowerment issues. Also, the author explores spirituality-centered social work skills, focusing on organizational management and leadership skills.     

eHealth recruitment challenges

Little is known about effective eHealth recruitment methods. This paper presents recruitment challenges associated with enrolling African-American girls aged 8–10 years in an eHealth obesity prevention program, their effect on the recruitment plan, and potential implications for eHealth research.Although the initial recruitment strategy was literature-informed, it failed to enroll the desired number of girls within a reasonable time period. Therefore, the recruitment strategy was reformulated to incorporate principles of social marketing and traditional marketing techniques. The resulting plan included both targeted, highly specific strategies (e.g., selected churches), and more broad-based approaches (e.g., media exposure, mass mailings, radio advertisements).The revised plan enabled recruitment goals to be attained. Media appeared to be particularly effective at reaching the intended audience. Future research should identify the most effective recruitment strategies for reaching potential eHealth audiences.     

Striking a Balance between Program-Specific and Portfolio-Level Evaluation: Lessons Learned from a Multi-Site Evaluation on the Texas-Mexico Border

Funders are increasingly making strategic investments across multiple grantees, aiming for their portfolio to improve targeted outcomes in a specific issue area. To this end, funders might use multi-site evaluation (MSE) approaches to examine the impact of their collective investments. However, it is important to recognize that each program—and its own program evaluation—must be tailored to its setting, population, and local context to best meet the needs of its target population. Therefore, multi-site evaluations need to account for this complexity. This paper describes the Sí Texas project, a large initiative of eight grantees implementing different integrated behavioral health models to improve physical and mental health outcomes along the Texas-Mexico border. With over 4,200 MSE study participants, the evaluation for Sí Texas used a partnership-centered approach to both enhance the evidence base and build local organizational capacity. This paper describes this approach, the process of tailoring evaluation practices to the grantees’ context, and the challenge of balancing consistency at the grantee-level for the portfolio multi-site evaluation. Successes, challenges, and lessons learned related to study design, data collection, grantee partnership, and capacity building are discussed.     

A case for using adaptive platforms in the development and implementation of urban-centred adaptive management plans

The recurring challenges in developing and coordinating social frameworks for the effective adaptive management of environmental resources in urban environments may be addressed by developing platforms, similar to those promoted by collective action in the management of common pool resources. This paper compares how these two frameworks perceive and foster social organization vis-à-vis management action, particularly at the local scale of resource use. The significant potential for incorporation of platforms into the adaptive management framework is discussed, based on a set of issues identified in collective action research as important in the development of platforms. An urban-centred case study, the Las Vegas Wash Comprehensive Adaptive Management Plan, illustrates how an adaptive management framework, when its social components come together in a mutually supportive, heuristic way, begins to generate what in collective action is called a platform. The paper further explores how platforms may be more explicitly incorporated into the adaptive management framework to address some of the more persistent social organization difficulties for which adaptive management has been faulted, especially regarding scale issues and the urban context. A revised adaptive management model including a platform is provided, and suggestions made about issues that may be studied using this model.     

Workforce training in family therapy and systemic practices: An evaluation framework and case study

Internationally, while the interdisciplinary field of family therapy and systemic practices is well established and evidence for therapeutic impact is advanced, evaluation of training and measurement of its impact lags behind. This paper addresses this gap, utilising a single case study at the Bouverie Centre, the largest family therapy and workforce development service in Australia. We describe (a) the scope of workforce training and implementation at the Centre, (b) the development of our training research and evaluation framework, and (c) standardised data collection methodology, including a pre-post training reflective survey. Collectively, this work comprises our current evaluation framework through which we will subsequently examine training impact. Future research and practice recommendations are offered to address challenges to workforce training evaluation and progress efficacy of family therapy training.     

Analyzing mental health evaluation: moral and ethical dimensions

Now that evaluators have been sensitized to the importance of moral and ethical issues in their work, it is time to move beyond generalities and examine the moral and ethical implications of specific evaluation models in specific settings. This paper proposes a framework that can be used to examine moral and ethical dimensions of evaluation and illustrates it by analyzing a selected model of mental health evaluation. Such a systematic identification of moral issues can improve evaluation practice both proactively, by shaping the training of evaluators, and retrospectively, by contributing to meta-evaluation.     

Evaluation capacity building in the nonformal education context: Challenges and strategies

Policymakers’ demand for increased accountability has compelled organizations to pay more attention to internal evaluation capacity building (ECB). The existing literature about ECB has focused on capacity building experiences and organizational research, with limited attention on challenges that internal evaluation specialists face in building organizational evaluative capacity. To address this knowledge gap, we conducted a Delphi study with evaluation specialists in the United States’ Cooperative Extension Service and developed a consensus on the most pervasive ECB challenges as well as the most useful strategies for overcoming ECB challenges. Challenges identified in this study include limited time and resources, limited understanding of the value of evaluation, evaluation considered as an afterthought, and limited support and buy-in from administrators. Alternatively, strategies found in the study include a shift in an organizational culture where evaluation is appreciated, buy-in and support from administration, clarifying the importance of quality than quantity of evaluations, and a strategic approach to ECB. The challenges identified in this study have persisted for decades, meaning administrators must understand the persistence of these issues and make an earnest investment (financial and human resource) to make noticeable progress. The Delphi approach can be used more often to prioritize ECB efforts.     

An assessment scale of ECB: perspectives from the faculty and staff of technical colleges in Taiwan

Faculty and staff in Taiwanese technical colleges are required to perform school self-evaluations. Most members of technical colleges come from postsecondary schools and are thus unfamiliar with school self-evaluation procedures. An effective school self-evaluation involves a complete ECB,(1) making it necessary to build faculty and staff evaluation capacity. In this research, we analyzed previous studies, interviewed experts, conducted expert reviews, constructed an ECB assessment scale, and collected the opinions of 1054 faculty and staff, to confirm the validity of the proposed assessment scale. This scale included 21 items summarized in four dimensions: understanding the evaluation guidelines, familiarizing with evaluation requirements, participating in the evaluation activity, and implementing the evaluation connotation. We explain the four dimensions behind the ECB model, and propose three lessons learned from it.     

“Compassion,pleasantry, and hope”: A process evaluation of a volunteer-based nonprofit

As funders continue to emphasize the importance of documented results, nonprofit organizations must work to complete program evaluations that are both valuable and feasible. The purpose of this paper is to document a practical process evaluation of a southeastern nonprofit, a local Meals on Wheels. Using a mixed methods approach, we sought to answer four evaluation questions: (1) What are the essential program components, as identified by key stakeholders; (2) To what extent are volunteers implementing the identified essential components as intended; (3) What is the level of volunteer satisfaction with the program; and (4) What suggestions do stakeholders have for improving the program? Our findings indicate that most aspects of the program were implemented as intended, but inconsistencies occurred when volunteers were unsure of their assigned duties. In addition, volunteers had high levels of satisfaction and specific suggestions for improvement. From these results, we developed a conceptual model of factors contributing to quality of implementation and volunteer satisfaction that may be generalizable to other volunteer-based nonprofits. Specifically, we identified three factors that helped to facilitate satisfaction and performance: leadership, social contact, and fulfillment. Finally, this process evaluation demonstrates the feasibility of developing and implementing evaluation tools in similar organizations.     

Chains and networks,territories and scales: towards a relational framework for analysing the global economy

A vast and continually expanding literature on economic globalization continues to generate a miasma of conflicting viewpoints and alternative discourses. This article argues that any understanding of the global economy must be sensitive to four considerations: (a) conceptual categories and labels carry with them the discursive power to shape material processes; (b) multiple scales of analysis must be incorporated in recognition of the contemporary ‘relativization of scale’; (c) no single institutional or organizational locus of analysis should be privileged; and (d) extrapolations from specific case studies and instances must be treated with caution, but this should not preclude the option of discussing the global economy, and power relations within it, as a structural whole. This paper advocates a network methodology as a potential framework to incorporate these concerns. Such a methodology requires us to identify actors in networks, their ongoing relations and the structural outcomes of these relations. Networks thus become the foundational unit of analysis for our understanding of the global economy, rather than individuals, firms or nation states. In presenting this argument we critically examine two examples of network methodology that have been used to provide frameworks for analysing the global economy: global commodity chains and actor‐network theory. We suggest that while they fall short of fulfilling the promise of a network methodology in some respects, they do provide indications of the utility of such a methodology as a basis for understanding the global economy.