Developing a conceptual evaluation framework for gender equality interventions in research and innovation

In this article, we discuss the development of a conceptual evaluation framework to design and assess gender equality interventions and their effects in research and innovation. The conceptual framework presented herewith embraces the complexity, gender-sensitive and theory-based evaluation approaches ensuring that design and evaluation of gender equality interventions consider the complex systems that constitute the context in which the interventions operate. The evaluation framework offers a non-linear concept, where the notion of contribution - not attribution - to achieve impact is central to the integration of team, organizational and system factors in policy design and evaluation. The paper opens the “black box” to address the question of how and why a policy intervention works and in which context and discusses a systematic process on how to approach the interwoven linkages between input, implementation and effects in gender equality interventions in research and innovation, accounting for context sensitivity and methodological pluralism. The evaluation framework may serve as reference for researchers, evaluators, policymakers and other stakeholders in designing and assessing gender equality interventions, and in further developing their evidence, and theoretical and methodological base.     

A flexible framework for planning and evaluating early-stage health interventions: FRAME-IT

Health interventions exhibit three stages of maturity: early-, mid-, and late-stages. Early-stage interventions have innovative content necessitating evaluation; however existing evaluation frameworks omit constructs and guidelines relevant to this evaluation. Early-stage interventions require planning and evaluation that supports creating, testing, and exploring content to establish general feasibility and enable refinement for further testing, prior to randomised controlled trialling and wider dissemination. Feasibility, Reach-out, Acceptability, Maintenance, Efficacy, Implementation, Tailorability (FRAME-IT) was developed for a mixed methods feasibility study of a novel well-being intervention. FRAME-IT was conceived as a complementary framework to Reach, Efficacy, Adoption, Implementation, Maintenance (RE-AIM; Glasgow et al., 1999) which is better suited for mid- and late- stage interventions. FRAME-IT is proposed to support: (1) early-stage intervention planning and design, by guiding research focus and data sourcing strategy with relevant constructs; (2) comprehensive evaluation, by including constructs appropriate for early-stage interventions, i.e. feasibility, acceptability, and tailorability; (3) future intervention scalability, by including and adapting some of RE-AIM’s constructs to encourage a smoother translation of research into practice as interventions are scaled-up.     

The evidence base for social support interventions in child and family practice: Emerging issues for research and practice

This paper examines the evidence base for interventions which seek to enhance social support resources for families with children at risk of adverse developmental outcomes. Substantive and methodological issues related to the implementation of social support interventions are discussed and a research agenda for the development, testing and evaluation of such interventions is presented.     

Facilitating evaluations of innovative,competence-based assessments: Creating understanding and involving multiple stakeholders

Schools are held more responsible for evaluating, quality assuring and improving their student assessments. Teachers’ lack of understanding of new, competence-based assessments as well as the lack of key stakeholders’ involvement, hamper effective and efficient self-evaluations by teachers of innovative, competence-based assessments (CBAs). While evaluating two CBAs in Agricultural Vocational Education and Training institutions, two interventions in the evaluation process aimed to tackle these problems were examined: (1) starting with explicating the CBA in the teacher team using a concrete explication format and (2) qualitatively involving key stakeholders (i.e., teachers, students and employers) in the evaluation of the CBA through mixed-group interviews. Quantitative and qualitative analysis, as well as stakeholders’ perceptions are used to find indications for the added value of these interventions for evaluation and further improvement of the CBAs. Results show that external facilitation is needed to make both interventions work.However, under this condition, explicating the CBA led to more complete, concrete and shared understandings of the actual CBA among teachers and mixed-groups interviews resulted in more concrete and elaborate evaluations of the CBAs’ quality and more ideas for improvement. Both interventions can facilitate building up elaborate, more valid and concrete arguments for CBA quality in self-evaluations, certainly in the case of evaluating innovative assessments. Lessons learned will provide guidelines for incorporating the interventions into other evaluations of innovative programs.     

Evaluation considerations for community-based gender-informed health interventions

Evaluations of gender-based interventions have been consistently criticized for their lack of methodological rigor. This is largely due to the complex design of many of the interventions, coupled with difficulties in measuring the outcome and impact of these interventions. This article proposes a number of ways to improve these evaluations both at the community and individual level. We recommend use of organizational theory and narrative inquiry methods, such as the appreciative inquiry technique, to examine how communities design gender-based interventions. In addition, we suggest a variety of methods to measure the effects of these interventions on gender norms in the community for example, policy analysis, multilevel modeling, and social conversations. With respect to measuring outcomes at the individual level, we argue for more rigorous evaluation designs in order to improve internal and external validity claims. Additionally, we suggest that evaluations should incorporate different methodologies, for example autobiographical narratives, which allows one to give saliency to the subjective voices of participants. Finally, we emphasize that evaluation designs need to document the long term effects of intervention programs and define the expected outcomes with greater specificity.     

Evaluating social exclusion interventions in university-community partnerships

Most university-community partnerships (UCPs) involve elements of community-level social exclusion interventions. As such, they face substantial challenges in management and evaluation. This paper highlights the central challenges associated with evaluation of UCP and other social exclusion interventions at the community level, and suggests methods to overcome them. The main body of the paper presents a case study based on a four-year action research involving evaluation of a social exclusion intervention initiated and implemented by a UCP in Israel. The case study highlights the challenges faced by the evaluation team, the solutions provided, and the contribution of the evaluation to improvement and accountability.     

Evaluating theory-based evaluation: information, norms, and adherence

Programmatic social interventions attempt to produce appropriate social-norm-guided behavior in an open environment. A marriage of applicable psychological theory, appropriate program evaluation theory, and outcome of evaluations of specific social interventions assures the acquisition of cumulative theory and the production of successful social interventions - the marriage permits us to advance knowledge by making use of both success and failures. We briefly review well-established principles within the field of program evaluation, well-established processes involved in changing social norms and social-norm adherence, the outcome of several program evaluations focusing on smoking prevention, pro-environmental behavior, and rape prevention and, using the principle of learning from our failures, examine why these programs often do not perform as expected. Finally, we discuss the promise of learning from our collective experiences to develop a cumulative science of program evaluation and to improve the performance of extant and future interventions.     

Making sense of the emerging conversation in evaluation about systems thinking and complexity science

In the last twenty years, a conversation has emerged in the evaluation field about the potential of systems thinking and complexity science (STCS) to transform the practice of evaluating social interventions. Documenting and interpreting this conversation are necessary to advance our understanding of the significance of using STCS in planning, implementing, and evaluating social interventions. Guided by a generic framework for evaluation practice, this paper reports on an inter-disciplinary literature review and argues that STCS raises some new ways of thinking about and carrying out the following six activities: 1) supporting social problem solving; 2) framing interventions and contexts; 3) selecting and using methods; 4) engaging in valuing; 5) producing and justifying knowledge; and 6) facilitating use. Following a discussion of these issues, future directions for research and practice are suggested.     

Enhancing the intercultural effectiveness of exchange programmes: formal and non-formal educational interventions

This paper examines how the addition of intercultural interventions carried out throughout European credit-bearing exchange programmes can enhance sojourners’ development of intercultural competencies, and it explores how both formal and non-formal pedagogical interventions may be designed and implemented. Such interventions were conducted at a Portuguese university with 31 sojourners throughout one academic year, and their impact was assessed using a mixed methods research design. Sojourners included incoming students of the exchange programmes Campus Europae and Erasmus, as well as highly skilled immigrants. Findings confirm the positive impact of interventions on the development of intercultural competencies and, in turn, their contribution to internationalisation efforts. Implications for further research suggest a need to increase interventions and to develop a systematic approach for fostering intercultural competencies throughout the study abroad cycle.     

An application of the Impact Evaluation Process for designing a performance measurement and evaluation framework in K-12 environments

This article illustrates the application of the Impact Evaluation Process for the design of a performance measurement and evaluation framework for an urban high school. One of the key aims of this framework is to enhance decision-making by providing timely feedback about the effectiveness of various performance improvement interventions. The framework design process is guided by the Impact Evaluation Process, and included the participation of key stakeholders including administrative and teaching staff who all contributed to the performance measurement and evaluation framework design process. Key performance indicators at the strategic, tactical, and operational levels were derived from the school vision, and linked to specific interventions to facilitate the continuous evaluation and improvement process.     

An application of the Impact Evaluation Process for designing a performance measurement and evaluation framework in K-12 environments

This article illustrates the application of the Impact Evaluation Process for the design of a performance measurement and evaluation framework for an urban high school. One of the key aims of this framework is to enhance decision-making by providing timely feedback about the effectiveness of various performance improvement interventions. The framework design process is guided by the Impact Evaluation Process, and included the participation of key stakeholders including administrative and teaching staff who all contributed to the performance measurement and evaluation framework design process. Key performance indicators at the strategic, tactical, and operational levels were derived from the school vision, and linked to specific interventions to facilitate the continuous evaluation and improvement process.     

Analysing facilitating and hindering factors for implementing gender equality interventions in R&I: Structures and processes

This article analyses the facilitating and hindering factors that have affected the implementation of gender equality interventions in research and innovation in Europe. It applies the evaluation framework developed in the EFFORTI project that recognizes the complexity of evaluating gender equality interventions in R&I, the importance of factoring in context to any sound evaluation as well as the need to distinguish between the design and implementation of interventions in evaluations. It is based on the analysis of 19 empirical case studies carried out throughout Europe and focuses on those structural and procedural factors that have either facilitated or hindered the implementation process of these interventions. Findings include how the governance framework; top-management commitment; bottom-up participation; framing synergies with other initiatives, strategies for tackling resistance; resources; sustainability of actions; gender competence, experience and knowledge and transparency, targets, standards and monitoring; and accessible data and information all contributed to the successful implementation of the interventions.     

Gender equality regimes and evaluation regimes in Europe and their implications for policy design and evaluation

PurposeThis article starts from the assumption that it is important for the evaluation of gender equality interventions in RTDI (Research, Technology Development and Innovation) to consider its context regarding gender equality regime and evaluation culture because this context does influence effects and long term impacts of such activities. It aims to provide key characteristics for the differentiation of gender equality regimes and evaluation regimes to be considered when designing policy interventions or evaluating specific gender equality interventions in RTDI.Design/methodology/approachAfter a literature review of relevant typologies for welfare state and gender equality regimes, it was analysed to which extent the seven EFFORTI countries correspond to certain typologies based on the data basis of the context research performed in the EFFORTI project. For this context research, international/national and qualitative/quantitative data regarding the relevant framework conditions were firstly collected for each of the EFFORTI countries and secondly compared in a cross-country analysis.FindingsThe research showed that when it comes to gender equality policies, most EFFORTI countries can either be assigned to the Social-Democratic category or Conservative Equal Employment Regime category in the typology of von Wahl (2005), with the latter type tending to provide less favourable conditions for women in the general labour market as well as women in RTDI (e.g. overtime culture). In how far these different types of context can have an impact on the evaluation of interventions can be exemplarily illustrated using case studies carried out as part of EFFORTI.Connecting the typologies for gender equality policy with evaluation regimes has not proved fruitful, as the two discourses have only begun to converge in recent years. The evaluation regimes and cultures of the respective countries have therefore been described independently. However, it was shown that countries with more expertise in certain areas (e.g. gender, evaluations) developed more routinized and institutionalised procedures in the respective field.Originality/valueConsidering the different framework conditions is relevant when it comes to evaluation as different national contexts might require different policy and designs of activities, but might also shape the interventions’ effects. This article therefore aims to provide support in this regard for future evaluations of gender policies.     

Influencing alcohol control policies and practices at community festivals

The goal of this study was to assess the feasibility and effectiveness of two interventions aimed at reducing alcohol-related risks at community festivals--a training program for festival planners and a community organizing campaign. We randomly selected four festivals for each intervention and had 24 comparison festivals. Our assessment included process evaluation to track and evaluate types of alcohol policies resulting from each of the interventions, pre and post telephone surveys of key festival planners and law enforcement agencies, and pre and post pseudo-underage and pseudo-intoxicated purchase attempts. Analyses showed that both interventions were feasible and were successful in influencing adoption of written policies and improving alcohol-related practices. However, neither intervention appeared to decrease propensity for illegal alcohol sales at these events, likely due, in part, to the short time frame of the interventions. Future research should assess effects of the interventions on alcohol-related problems and effects of enforcement interventions.     

Multidisciplinary eHealth survey evaluation methods

This paper describes the development process of an evaluation framework for describing and comparing web survey tools. We believe that this approach will help shape the design, development, deployment, and evaluation of population-based health interventions. A conceptual framework for describing and evaluating web survey systems will enable the developers of health interventions to identify, select, and use the most appropriate and relevant survey tools. The primary contributors included survey methodologists, computer scientists, informaticians, technical communications specialists, educators, evaluators, clinical practitioners, and health services researchers, who together developed a comprehensive ontology to describe the electronic survey domain. This paper will describe the survey ontology development process and detail lessons learned in the creation of the web survey system evaluation framework.     

The problem of replicability in program evaluation. The component solution using the example of case management

Specificity and replicability of a program are crucial in order to rigorously evaluate the program's effectiveness. However, the traditional approaches to evaluation have ignored the problem that most social and mental health programs consist of vaguely specified interventions and cannot meet minimal standards of replicability. Consequently, the results of effectiveness evaluations of poorly defined programs have little general generalizability. Nevertheless, effectiveness information concerning su ch programs is much needed. One solution to this problem is to focus on program components, examining interventions at a level where operational definitions can be employed and replicability is possible. In this paper, this approach is expanded using the example of case management programs. A component-oriented approach ensures a systematic evaluation of program effectiveness for which the first question for evaluators is whether a program consists of interventions that reliably produce specific immediate outcomes. Then, and only then, is it possible to address questions concerning whether a program will reliably produce those outcomes that are the overall objectives of the program.     

Analysis approaches to community evaluation

Analysis approaches to the evaluation of community interventions must be sensitive to a wide variety of analytic contaminants that may bias the statistical assessment of changes in outcome measures. These contaminants include model misspecifications related to failures to control for community-specific time trends, temporal autocorrelated errors in equations, spatial autocorrelated errors among geographic units, and other failures of unit independence otherwise indexed by estimated intraclass correlations. Although an enormous amount of progress has been made toward the solution of many of these analytic problems over the past years, the contemporary evaluator of community interventions is left with a number of unenviable design and analysis choices; choices that inevitably force an assessment of the relative threats of different sources of error to the internal and external validity of the evaluation. This article describes the choices made for the evaluation of the Community Trial Project outcome data.     

Grounding Online Prevention Interventions in Theory: Guidelines from a Review of Selected Theories and Research

ABSTRACT

Literature suggests that grounding online interventions in theory improves their success. This article reviews 10 theories selected from the fields of psychotherapy, social work, health promotion, gaming, and innovation dissemination that can be used for grounding the design, development, implementation, and evaluation of online interventions. Selected studies are reviewed to illustrate how these theories have been used in online human service interventions. After this review, an integrated theoretical approach for grounding online interventions is proposed along with guidelines suggested by the review.     

Variables Affecting Readiness to Benefit From Career Interventions

This article identifies and briefly describes the broad range of variables that may influence clients’ readiness to benefit from career interventions. The article also discusses consequences of low readiness for effective use of career interventions and addresses implications for practice as well as for future research. Variables contributing to low readiness for effective use of career interventions include personal characteristics and circumstances; knowledge of self, options, and decision making; and prior experience with career interventions. Consequences of low readiness for using career interventions include premature disengagement, negative perception of skills and interests, selective acquisition of incomplete information, premature choice foreclosure, protracted exploration, dependent decision‐making style, and poor evaluation of options.     

Usability testing,initial implementation,and formative evaluation of an evidence-based intervention: Lessons from a demonstration project to reduce long-term foster care

The field of child welfare faces an undersupply of evidence-based interventions to address long-term foster care. The Permanency Innovations Initiative is a five-year federal demonstration project intended to generate evidence to reduce long stays in foster care for those youth who encounter the most substantial barriers to permanency. This article describes a systematic and staged approach to implementation and evaluation of a PII project that included usability testing as one of its key activities. Usability testing is an industry-derived practice which analyzes early implementation processes and evaluation procedures before they are finalized. This article describes the iterative selection, testing, and analysis of nine usability metrics that were designed to assess three important constructs of the project's initial implementation and evaluation: intervening early, obtaining consent, and engaging parents. Results showed that seven of nine metrics met a predetermined target. This study demonstrates how findings from usability testing influenced the initial implementation and formative evaluation of an evidence-supported intervention. Implications are discussed for usability testing as a quality improvement cycle that may contribute to better operationalized interventions and more reliable, valid, and replicable evidence.