A framework for evaluating eHealth research

Health care is in the midst of a consumer-oriented technology explosion. Individuals of all ages and backgrounds have discovered eHealth. But the challenges of implementing and evaluating eHealth are just beginning to surface, and, as technology changes, new challenges emerge. Evaluation is critical to the future of eHealth. This article addresses four dimensions of eHealth evaluation: (1) design and methodology issues; (2) challenges related to the technology itself; (3) environmental issues that are not specific to eHealth but pose special problems for eHealth researchers; and (4) logistic or administrative concerns of the evaluation methodology selected. We suggest that these four dimensions must be integrated to provide a holistic framework for designing and implementing eHealth research projects, as well as for understanding the totality of the eHealth intervention. The framework must be flexible enough to adapt to a variety of end users, regardless of whether the end user is a healthcare organization, a for-profit business, a community organization, or an individual. The framework is depicted as a puzzle with four interlocking pieces.     

A use-and-transformation model for evaluating public R&D: illustrations from polycystic ovarian syndrome (PCOS) research

Evaluating federally funded research and development (R&D) presents unique challenges to both federal science agencies and evaluators. Often focusing only on outcome evaluative measures (such as productivity or economic value) can shortchange the true value of the federal investment. For example, program directors at the National Science Foundation (NSF) and National Institutes of Health (NIH) talk about the "value added" of the new interdisciplinary science centers that they have funded-and they hope to be able to capture how funding can generate increased capacity for new cutting-edge research in the future. The purpose of this paper is to present a use-and-transformation model for evaluating public R&D, which explicitly focuses on measuring capacity-based metrics for evaluation instead of outcome-based metrics. The theory for the model presented here explicitly uses the concept of a Knowledge Value Collective that was introduced by Bozeman and Rogers [Bozeman, B., & Rogers, J. D. (2002). A churn model of scientific knowledge value: Internet researchers as a knowledge value collective. Research Policy, 31(5), 769-794; Rogers, J. D., & Bozeman, B. (2001). "Knowledge value alliances": An alternative to the R&D project focus in evaluation. Science Technology & Human Values, 26(1), 23-55].     

A method for evaluating the behavior of power indices in weighted plurality games

In this paper, a systematic method to facilitate the comparison of a priori measures of power in an n-player r-candidate (n, r) weighted plurality game is proposed. This method, which exploits the notion of a structure of embedded winning coalitions (SEWC), enables the listing of all power profiles relevant to an (n, r) game under a given index and permits the computation of the probability of occurrence of each of these profiles. The vulnerability of an index to different paradoxes of power can also be systematically studied. For the purpose of illustration, we apply this method to the analysis of four well-known 2-candidate power indices namely the Shapley-Shubik index, the Banzhaf index, the Johnston index and the Deegan-Packel index. In each case, the set of power profiles and the likelihood of occurrence of each of these profiles are enumerated. The superadditivity property of these indices is also studied. Received: 20 October 1999/Accepted: 25 April 2001     

A strategy for monitoring and evaluating massive open online courses

We argue that the complex, innovative and adaptive nature of Massive Open Online Course (MOOC) initiatives poses particular challenges to monitoring and evaluation, in that any evaluation strategy will need to follow a systems approach. This article aims to guide organizations implementing MOOCs through a series of steps to assist them in developing a strategy to monitor, improve, and judge the merit of their initiatives. We describe how we operationalise our strategy by first defining the different layers of interacting agents in a given MOOC system. We then tailor our approach to these different layers. Specifically, a two-pronged approach was developed, where we suggest that individual projects be assessed through performance monitoring; assessment criteria for which would be defined at the outset to include coverage, participation, quality and student achievement. In contrast, the success of an overall initiative should be considered within a more adaptive, emergent evaluation inquiry framework. We present the inquiry framework we developed for MOOC initiatives, and show how this framework might be used to develop evaluation questions and an assessment methodology. We also define the more fixed indicators and measures for project performance monitoring. Our strategy is described as it was developed to inform the evaluation of a MOOC initiative at the University of Cape Town (UCT), South Africa.     

A method for evaluating and improving goal consensus for social action programs

A method is presented for investigating goal priority disagreements in social action programs. This method involves extracting the underlying themes of these disagreements via multidimensional scaling and correlational techniques. The primary application of the method is to enable the evaluator to determine which aspects of the program are assessable in goal-attainment terms and which are not. Further, the resultant multidimensional maps of goal disagreements may be used as cognitive aids in facilitating discussion among program participants about their disagreements. Throughout the exposition, an example of the application of the method to a youth outreach program is used for illustrative purposes.     

A flexible framework for planning and evaluating early-stage health interventions: FRAME-IT

Health interventions exhibit three stages of maturity: early-, mid-, and late-stages. Early-stage interventions have innovative content necessitating evaluation; however existing evaluation frameworks omit constructs and guidelines relevant to this evaluation. Early-stage interventions require planning and evaluation that supports creating, testing, and exploring content to establish general feasibility and enable refinement for further testing, prior to randomised controlled trialling and wider dissemination. Feasibility, Reach-out, Acceptability, Maintenance, Efficacy, Implementation, Tailorability (FRAME-IT) was developed for a mixed methods feasibility study of a novel well-being intervention. FRAME-IT was conceived as a complementary framework to Reach, Efficacy, Adoption, Implementation, Maintenance (RE-AIM; Glasgow et al., 1999) which is better suited for mid- and late- stage interventions. FRAME-IT is proposed to support: (1) early-stage intervention planning and design, by guiding research focus and data sourcing strategy with relevant constructs; (2) comprehensive evaluation, by including constructs appropriate for early-stage interventions, i.e. feasibility, acceptability, and tailorability; (3) future intervention scalability, by including and adapting some of RE-AIM’s constructs to encourage a smoother translation of research into practice as interventions are scaled-up.     

A usable body: coaxing the body into and out of captivity at the museum

The visual logic of ‘black’ identity as a social and historical fact is presented at museums depicting the transformation of enslaved Africans to racial citizens. I focus on the work of the body in narrating this transition at a black history museum: the historical object on display – the captive African and later the US citizen – and the imagined visitor moving through the physical space of the museum. I consider the visual treatment of the enslaved body in captivity as broken, bowed, beaten and often nude in contrast to the post-abolition period when the body is positioned as erect, individuated, self-possessed and always clothed. I also discuss how the imagined visitor’s body is oriented both to the narratives on display and to the built environment of the museum itself.     

A study on the consistency of stakeholder preferences for different types of information in evaluating police services

A questionnaire is designed to allow assessment of a simple additive value function for testing respondent preferences for different types of information used in evaluating police services. The questionnaire was distributed to 57 respondents representing five different stakeholder groups within the metropolitan Pittsburgh Criminal Justice System. The responses are analyzed to determine what types of information different stakeholder groups consider useful in evaluating police performance. Statistical analysis of results suggest that different stakeholders have a high degree of consistency in the importance attached to different types of information.     

Laptops for students: Understanding and evaluating the drivers for change

Abstract

This paper describes the development of the student laptop initiative at the University of Strathclyde, examines the process by which this project became an embedded service and assesses the impact that this technology-rich, pedagogically flexible initiative has had on institutional structures, roles and processes. The authors use the MIT90s Model for Institutional Change to explore the way in which technology influences organisational development. A wealth of quantitative and qualitative evidence has been gathered over the last 7 years to support the examination of whether the benefits of the initiative justify the cost to the student and to the institution.     

Measure for measure: evaluating performance and the arts organisation

This paper discusses the cultural policy and impacts of the application of performance indicators to subsidised arts organisations in Britain. The context of this study is the increased accountability and measurement of the outputs of arts organisations by government funding agencies, and the related drive for improved productivity and efficiency. In an extension of performance measurement regimes in ‘public good’ provision, and the arguments concerning economic importance of the cultural industries, the performing arts occupy an uncomfortable place as a pre-industrial activity being subjected to notions of economic efficiency. This discussion concerns the focus for analysis of the measurements used – the three-Es of economy, efficiency and effectiveness - and the rationale underlying their imposition. The outputs and performance indicators of the major subsidised arts organisations are then presented, based on a ten year time series, revealing declining audiences and production, but increasing ticket prices and expenditure, which arguably undermines the performing arts ‘merit good’ status.     

Case study for evaluating campus sustainability: nitrogen balance for the University of Minnesota

American ecologists recently began to address the ecology of cities. Concurrently, higher education institutions have developed sustainability education programs and instituted sustainability policies and projects. This paper draws together these two disparate areas by examining the nitrogen (N) budget of the Twin Cities campus of the University of Minnesota. We addressed the question: what were the annual inputs, outputs and internal cycles of N on the University of Minnesota Twin Cities campus? We found that 508 Mg N yr-1 were brought into the campus and 494 Mg N yr-1 were emitted from the University of Minnesota campus. The largest N inputs were abiotic fixation (conversion of N₂ to NO_x by combustion) and food for humans and animals. The largest N output was NO_x, followed by wastewater. Our results expand the understanding of the ecology of institutions within an urban area and provide an opportunity for improving urban ecology education and environmental policy at educational institutions.     

Quantitative posture analysis of 2D, 3D, and optical microscope visualization methods for microsurgery tasks

The purpose of this paper is to present a quantitative posture analysis of microsurgery tasks performed with different visualization methods. Microsurgery is traditionally performed using a binocular microscope; however surgeons are constrained by the optical eyepieces and are forced to assume joint angles that deviate away from neutral postures. This may be especially problematic for the neck and can increase surgeon discomfort and fatigue. Alternative visualization methods may improve surgeon posture by eliminating the constraints imposed by the microscope. This study examines both 2D and 3D heads-up displays as possible alternatives. Six subjects performed microsurgical tasks with each visualization methods for four hours. Quantitative posture analysis was done using Maxtraq software that tracks reflective markers on the subjects. The initial analysis of neck, upper arm, and elbow angles found significant differences between each display. A biomechanical analysis found that the differences in angles can result in loads on the neck joint that are twice as high in the microscope than the headsup displays. Although the alternative displays can result in better postures, improvements the display technology is needed to improve microsurgical task performance.     

Viewpoint The body of intellectuals/the intellectuals’ body (remarks for Roland)

Apparently Arab scholars, when speaking of the text, use this admirable expression: the certain body. What body? We have several of them; the body of anatomists and physiologists, the one science sees or discusses: this is the text of grammarians, critics, commentators, philologists (the pheno-text). But we also have a body of bliss consisting solely of erotic relations, utterly distinct from the first body: it is another contour, another nomination; thus with the text: it is no more than the open list of the fires of language… R. Barthes, 1973:16–17 The corpus: what a splendid idea! Providing one is willing to read the body in the corpus: either because in the group of texts reserved for study (and which form the corpus) the pursuit is no longer of structure alone but of the figures of the utterance; or because one has a certain erotic relation with this group of texts (without which the corpus is merely a scientific image-repertoire). R. Barthes, 1975:161     

Youth evaluating programs for youth: stories of Youth IMPACT

Youth evaluators describe their findings from an extensive evaluation of forty youth programs in San Francisco. Interviews with current youth and the former program director provide insight into the promise and challenge of youth participation.     

Jargons for deafness as institutional constructions of the deaf body

Jargons for deafness are seen here as social institutional constructions of the deaf body. Social institutions develop agendas commensurate with their view of the place of deaf people in society, create jargons to define its deaf clientele base and proffer programmes to construct them. This study examines current jargons developed by constellations of 'healing', 'helping' and 'agencist' social institutions developed in the USA since 1990 in light of their purposes and practices.     

A novel approach for evaluating programs designed to serve military veterans: Using an adapted common components analysis

Evaluators are challenged to keep pace with the vast array of Veteran support programs operating in the United States, resulting in a situation in which many programs lack any evidence of impact. Due to this lack of evidence, there is no efficient way to suggest which programs are most effective in helping Veterans in need of support. One potential solution to this dilemma is to reconceptualize program evaluation, by moving away from evaluating programs individually to evaluating what is common across programs. The Common Components Analysis (CCA) is one such technique that aggregates findings from programs that have undergone rigorous evaluation at the level of program components (e.g., content, process, barrier reduction). Given that many Veteran programs lack outcome evidence from rigorous studies, an adaptation to CCA is needed. This report examines cross-sectional data from a pilot study using an adapted CCA across five domains of well-being (i.e., employment, education, legal/financial/housing, mental/physical health, and social/personal relationships). The purpose of this preliminary study is to determine the feasibility of eliciting program nominations and program components from Veterans via an online survey. When coupled with a longitudinal research design, this adaptation to CCA will allow for stronger causal claims about the expected impact of different program components within and across a variety of domains.