Evaluating the iterative development of VR/AR human factors tools for manual work

This paper outlines the approach taken to iteratively evaluate a set of VR/AR (virtual reality / augmented reality) applications for five different manual-work applications - terrestrial spacecraft assembly, assembly-line design, remote maintenance of trains, maintenance of nuclear reactors, and large-machine assembly process design - and examines the evaluation data for evidence of the effectiveness of the evaluation framework as well as the benefits to the development process of feedback from iterative evaluation. ManuVAR is an EU-funded research project that is working to develop an innovative technology platform and a framework to support high-value, high-knowledge manual work throughout the product lifecycle. The results of this study demonstrate the iterative improvements reached throughout the design cycles, observable through the trending of the quantitative results from three successive trials of the applications and the investigation of the qualitative interview findings. The paper discusses the limitations of evaluation in complex, multi-disciplinary development projects and finds evidence of the effectiveness of the use of the particular set of complementary evaluation methods incorporating a common inquiry structure used for the evaluation - particularly in facilitating triangulation of the data.     

Multidisciplinary eHealth survey evaluation methods

This paper describes the development process of an evaluation framework for describing and comparing web survey tools. We believe that this approach will help shape the design, development, deployment, and evaluation of population-based health interventions. A conceptual framework for describing and evaluating web survey systems will enable the developers of health interventions to identify, select, and use the most appropriate and relevant survey tools. The primary contributors included survey methodologists, computer scientists, informaticians, technical communications specialists, educators, evaluators, clinical practitioners, and health services researchers, who together developed a comprehensive ontology to describe the electronic survey domain. This paper will describe the survey ontology development process and detail lessons learned in the creation of the web survey system evaluation framework.     

The results framework--an innovative tool for program planning and evaluation.

This article constitutes a case study of the development and implementation of the "results framework," an innovative planning and evaluation tool that is rapidly becoming a standard requirement for United States Agency for International Development (USAID) projects. The framework is used in a USAID-funded regional initiative for HIV/AIDS prevention in Central America. This new program evaluation and monitoring tool provides many advantages over traditional evaluation approaches that use outside consultants to provide midterm and end-of-project evaluations. The results-framework process, which spans the life of the project, provides an opportunity for program staff, donors, partners, and evaluators to work as a team to collect and use rich, longitudinal data for project planning, implementation, and evaluation purposes.     

An application of the Impact Evaluation Process for designing a performance measurement and evaluation framework in K-12 environments

This article illustrates the application of the Impact Evaluation Process for the design of a performance measurement and evaluation framework for an urban high school. One of the key aims of this framework is to enhance decision-making by providing timely feedback about the effectiveness of various performance improvement interventions. The framework design process is guided by the Impact Evaluation Process, and included the participation of key stakeholders including administrative and teaching staff who all contributed to the performance measurement and evaluation framework design process. Key performance indicators at the strategic, tactical, and operational levels were derived from the school vision, and linked to specific interventions to facilitate the continuous evaluation and improvement process.     

An application of the Impact Evaluation Process for designing a performance measurement and evaluation framework in K-12 environments

This article illustrates the application of the Impact Evaluation Process for the design of a performance measurement and evaluation framework for an urban high school. One of the key aims of this framework is to enhance decision-making by providing timely feedback about the effectiveness of various performance improvement interventions. The framework design process is guided by the Impact Evaluation Process, and included the participation of key stakeholders including administrative and teaching staff who all contributed to the performance measurement and evaluation framework design process. Key performance indicators at the strategic, tactical, and operational levels were derived from the school vision, and linked to specific interventions to facilitate the continuous evaluation and improvement process.     

A comprehensive health service evaluation and monitoring framework

ObjectiveTo develop a framework for evaluating and monitoring a primary health care service, integrating hospital and community services.MethodA targeted literature review of primary health service evaluation frameworks was performed to inform the development of the framework specifically for remote communities. Key principles underlying primary health care evaluation were determined and sentinel indicators developed to operationalise the evaluation framework. This framework was then validated with key stakeholders.ResultsThe framework includes Donabedian's three seminal domains of structure, process and outcomes to determine health service performance. These in turn are dependent on sustainability, quality of patient care and the determinants of health to provide a comprehensive health service evaluation framework. The principles underpinning primary health service evaluation were pertinent to health services in remote contexts. Sentinel indicators were developed to fit the demographic characteristics and health needs of the population. Consultation with key stakeholders confirmed that the evaluation framework was applicable.ConclusionData collected routinely by health services can be used to operationalise the proposed health service evaluation framework. Use of an evaluation framework which links policy and health service performance to health outcomes will assist health services to improve performance as part of a continuous quality improvement cycle.     

A strategy for monitoring and evaluating massive open online courses

We argue that the complex, innovative and adaptive nature of Massive Open Online Course (MOOC) initiatives poses particular challenges to monitoring and evaluation, in that any evaluation strategy will need to follow a systems approach. This article aims to guide organizations implementing MOOCs through a series of steps to assist them in developing a strategy to monitor, improve, and judge the merit of their initiatives. We describe how we operationalise our strategy by first defining the different layers of interacting agents in a given MOOC system. We then tailor our approach to these different layers. Specifically, a two-pronged approach was developed, where we suggest that individual projects be assessed through performance monitoring; assessment criteria for which would be defined at the outset to include coverage, participation, quality and student achievement. In contrast, the success of an overall initiative should be considered within a more adaptive, emergent evaluation inquiry framework. We present the inquiry framework we developed for MOOC initiatives, and show how this framework might be used to develop evaluation questions and an assessment methodology. We also define the more fixed indicators and measures for project performance monitoring. Our strategy is described as it was developed to inform the evaluation of a MOOC initiative at the University of Cape Town (UCT), South Africa.     

Using a participatory evaluation design to create an online data collection and monitoring system for New Mexico's Community Health Councils

We present the collaborative development of a web-based data collection and monitoring plan for thirty-two county councils within New Mexico's health council system. The monitoring plan, a key component in our multiyear participatory statewide evaluation process, was co-developed with the end users: representatives of the health councils. Guided by the Institute of Medicine's Community, Health Improvement Process framework, we first developed a logic model that delineated processes and intermediate systems-level outcomes in council development, planning, and community action. Through the online system, health councils reported data on intermediate outcomes, including policy changes and funds leveraged. The system captured data that were common across the health council system, yet was also flexible so that councils could report their unique accomplishments at the county level. A main benefit of the online system was that it provided the ability to assess intermediate, outcomes across the health council system. Developing the system was not without challenges, including creating processes to ensure participation across a large rural state; creating shared understanding of intermediate outcomes and indicators; and overcoming technological issues. Even through the challenges, however, the benefits of committing to using participatory processes far outweighed the challenges.     

Design of a collaborative monitoring and evaluation system for a community-based nutrition project in rural Bangladesh

Poor diet and undernutrition are common among children living in Bangladesh. To promote appropriate complementary feeding of young children, an economic development (ED) program involving income-generating asset transfer was implemented alongside a social and behavior change (SBC) program. This paper introduces a collaborative monitoring and evaluation (M&E) system in which diverse collaborators (“research group”, “implementation team”, and “coordinators”) facilitate M&E data acquisition by leveraging their comparative advantages. The implementation team built a monitoring system to track the ED (n = 2960) and SBC participants (n=∼10,000) over 12 months. Based on the baseline design and the monitoring records, the collaborators planned an impact evaluation introducing a quasi-experimental design using two cross-sectional surveys and a prospective cohort survey of child feeding and nutritional status. Using various data sources generated from the M&E system, the collaborators will also reveal the program impact pathway through which each intervention component is delivered, received, and utilized alongside the context-specific facilitators and barriers, including the programs’ uptake. The collaborative M&E system enables the sharing of program goals, strengthens collaborators' commitment to the program, and extends the understanding of the program's progress and evaluation activities.     

Staff Development as a School Climate Intervention to Support Transgender and Gender Nonconforming Students: An Integrated Research Partnership Model for School Counselors and Counselor Educators

ABSTRACT

This article posits that an integrative research partnership model can assist school counselors in addressing the marginalization of transgender students in schools through staff development. A methodological integration of action research and program evaluation through quantitative and qualitative data collection is explained, and a collaborative process is highlighted. A case example is provided to demonstrate the use of the design.     

Mixed-methods library evaluation integrating the patron,library, and external perspectives: The case of Namibia regional libraries

The article presents a mixed-methods evaluation of regional libraries in Namibia, which incorporates three perspectives: the patron perspective (library users), the library perspective (library staff, management, and related officials), and the external perspective (including evaluators and monitoring data). Seven data collection methods were used: patron surveys, patron panel studies, focus group discussions, key informant and staff interviews, secondary data analysis, media analysis, and observations. The goal of the evaluation was to assess library performance for both formative and summative purposes by addressing evaluation questions on areas such as library services, use, and operations. Building upon the literature review of how mixed-methods approaches can contribute to library evaluation, the aim of this article is to show how a mixed-methods evaluation can be designed to examine multi-faceted library performance and to illustrate how the evaluation design allows information complementarity and can be utilized to present diverse viewpoints of the above three perspectives. The evaluation design, analysis process, and lessons learned from this study may be useful to evaluators engaged in evaluation of public services or programs (including public libraries) that examine multiple aspects of service performance and involve a variety of stakeholders.     

Using intervention-oriented evaluation to diagnose and correct students’ persistent climate change misconceptions: A Singapore case study

The evaluation of classroom-based educational interventions is fraught with tensions, the most critical of which is choosing between focusing the inquiry on measuring the effects of treatment or in proximately utilizing the data to improve practice. This paper attempted to achieve both goals through the use of intervention-oriented evaluation of a professional development program intended to diagnose and correct students’ misconceptions of climate change. Data was gathered, monitored and analyzed in three stages of a time-series design: the baseline, treatment and follow-up stages. The evaluation itself was the ‘intervention’ such that the data was allowed to ‘contaminate’ the treatment. This was achieved through giving the teacher unimpeded access to the collected information and to introduce midcourse corrections as she saw fit to her instruction. Results showed a significant development in students’ conceptual understanding only after the teacher's decision to use direct and explicit refutation of misconceptions. Due to the accessibility of feedback, it was possible to locate specifically at which point in the process that the intervention was most effective. The efficacy of the intervention was then measured through comparing the scores across the three research stages. The inclusion of a comparison group to the design is recommended for future studies.     

The American Stop Smoking Intervention Study. Conceptual framework and evaluation design.

Reducing tobacco use, especially cigarette smoking, is a public health priority. The American Stop Smoking Intervention Study (ASSIST) was initiated in 1991 to prevent and reduce tobacco use primarily through policy-based approaches to alter the social-political environment. This article describes the conceptual design, research framework, evaluation components, and analytic strategies that are guiding the evaluation of this demonstration research endeavor. The ASSIST evaluation is a unique analysis of the complex relationships between the social context, public health activity at the state level, tobacco use, and individual behavior. The measures of tobacco control activity developed for this evaluation may be useful in ongoing national cancer control surveillance efforts, and the lessons learned will enhance the development of tobacco control programs.     

Participatory analytic hierarchy process for resource allocation in agricultural development projects

In the framework of 2030 Agenda sustainable agriculture plays a central role. In the field of international aid the participatory approaches to assessment, research, management and budgeting have been widely studied and applied in the last decades, mostly because international aid initiatives, in all their relevant phases from planning through implementation, monitoring and evaluation, are faced with the problem of identifying initiatives that could be successfully and sustainably implemented. The present paper investigates the use of a modified form of analytic hierarchy process, namely the participatory analytic hierarchy process, as a tool for choice criteria elicitation and resource allocation in the framework of an operational planning for agricultural development projects in the Dioceses of Goma, Nord Kivu, Democratic Republic of Congo. During our research, we were able to make use of most frequent inconsistencies in pairwise comparison matrixes in order to stimulate the debate, to adjust local preferences and to build consensus across the group. From an operational point of view, the PAHP methodology was also suited for training the project team and for identifying a shared resource allocation pattern, which matches the existing international guidelines for agricultural development in the region.     

Learning for the Twenty-First Century: Maximising Learning Transfer from Learning and Development Activity

This article builds on the learning gathered from the evaluation of a Scotland-wide child protection training programme. Based on Kirkpatrick's four-level framework, the study aimed to evaluate the impact of the learning programme on the practice of participants in the workplace. The study prompted the authors to examine the literature on maximising learning transfer across a range of occupations. This literature is discussed in the light of the findings from the evaluation and the experience of the authors in designing and implementing continuous professional development programmes in social services.

The practice of inspection, monitoring and audit seems to have impacted negatively on staff's willingness to participate in data collection, and coupled with what appeared to be confusion about data protection legislation, the perceived burden of audit and inspection has significant implications for research and evaluation activity.

Suggestions are made for the future design, delivery and evaluation of learning and development activity, and the role of managers is explored in relation to their responsibility for preparation, support and follow-up of workplace learning.     

Lessons learned about collaborative evaluation using the Capacity for Applying Project Evaluation (CAPE) framework with school and district leaders

Schools, districts, and state-level educational organizations are experiencing a great shift in the way they do the business of education. This shift focuses on accountability, specifically through the expectation of the effective utilization of evaluative-focused efforts to guide and support decisions about educational program implementation. In as much, education leaders need specific guidance and training on how to plan, implement, and use evaluation to critically examine district and school-level initiatives. One specific effort intended to address this need is through the Capacity for Applying Project Evaluation (CAPE) framework. The CAPE framework is composed of three crucial components: a collection of evaluation resources; a professional development model; and a conceptual framework that guides the work to support evaluation planning and implementation in schools and districts. School and district teams serve as active participants in the professional development and ultimately as formative evaluators of their own school or district-level programs by working collaboratively with evaluation experts.The CAPE framework involves the school and district staff in planning and implementing their evaluation. They are the ones deciding what evaluation questions to ask, which instruments to use, what data to collect, and how and to whom results should be reported. Initially this work is done through careful scaffolding by evaluation experts, where supports are slowly pulled away as the educators gain experience and confidence in their knowledge and skills as evaluators. Since CAPE engages all stakeholders in all stages of the evaluation, the philosophical intentions of these efforts to build capacity for formative evaluation strictly aligns with the collaborative evaluation approach.     

Lessons learned about collaborative evaluation using the Capacity for Applying Project Evaluation (CAPE) framework with school and district leaders

Schools, districts, and state-level educational organizations are experiencing a great shift in the way they do the business of education. This shift focuses on accountability, specifically through the expectation of the effective utilization of evaluative-focused efforts to guide and support decisions about educational program implementation. In as much, education leaders need specific guidance and training on how to plan, implement, and use evaluation to critically examine district and school-level initiatives. One specific effort intended to address this need is through the Capacity for Applying Project Evaluation (CAPE) framework. The CAPE framework is composed of three crucial components: a collection of evaluation resources; a professional development model; and a conceptual framework that guides the work to support evaluation planning and implementation in schools and districts. School and district teams serve as active participants in the professional development and ultimately as formative evaluators of their own school or district-level programs by working collaboratively with evaluation experts. The CAPE framework involves the school and district staff in planning and implementing their evaluation. They are the ones deciding what evaluation questions to ask, which instruments to use, what data to collect, and how and to whom results should be reported. Initially this work is done through careful scaffolding by evaluation experts, where supports are slowly pulled away as the educators gain experience and confidence in their knowledge and skills as evaluators. Since CAPE engages all stakeholders in all stages of the evaluation, the philosophical intentions of these efforts to build capacity for formative evaluation strictly aligns with the collaborative evaluation approach.     

Developing a conceptual evaluation framework for gender equality interventions in research and innovation

In this article, we discuss the development of a conceptual evaluation framework to design and assess gender equality interventions and their effects in research and innovation. The conceptual framework presented herewith embraces the complexity, gender-sensitive and theory-based evaluation approaches ensuring that design and evaluation of gender equality interventions consider the complex systems that constitute the context in which the interventions operate. The evaluation framework offers a non-linear concept, where the notion of contribution - not attribution - to achieve impact is central to the integration of team, organizational and system factors in policy design and evaluation. The paper opens the “black box” to address the question of how and why a policy intervention works and in which context and discusses a systematic process on how to approach the interwoven linkages between input, implementation and effects in gender equality interventions in research and innovation, accounting for context sensitivity and methodological pluralism. The evaluation framework may serve as reference for researchers, evaluators, policymakers and other stakeholders in designing and assessing gender equality interventions, and in further developing their evidence, and theoretical and methodological base.     

Design of an educational strategy based on Intervention Mapping for nutritional health promotion in Child Care Centers

Early childhood is a critical period for instilling healthy habits to prevent overweight and obesity. This paper describes the development of an educational intervention for the promotion of healthy eating and physical activity among two-to-four-year-old children in public child care centers (CCCs) in Mexico City. Following the Intervention Mapping (IM) protocol, we developed the Bright Futures multidisciplinary intervention. First, a formative research process to identify the personal and environmental determinants of childhood overweight and obesity, behavioral outcomes and performance objectives was conducted. Then, a matrix of change objectives by intersecting the performance objectives with the determinants was integrated. Bright Futures lasted six months and included 24 weekly sessions, each composed of five phases: warm-up, theory, active movements, relaxation, and hydration. Ad hoc interactive teaching materials focused on recreational activities, and formulated plans for the adoption, implementation and process/impact evaluation of the intervention was developed. IM successfully guided the design of a theory-driven and evidence-based intervention for children in CCCs within a socio-ecological and participatory planning framework. This is one of the first studies in Mexico to use IM in the context of CCCs.