Developing a conceptual evaluation framework for gender equality interventions in research and innovation

In this article, we discuss the development of a conceptual evaluation framework to design and assess gender equality interventions and their effects in research and innovation. The conceptual framework presented herewith embraces the complexity, gender-sensitive and theory-based evaluation approaches ensuring that design and evaluation of gender equality interventions consider the complex systems that constitute the context in which the interventions operate. The evaluation framework offers a non-linear concept, where the notion of contribution - not attribution - to achieve impact is central to the integration of team, organizational and system factors in policy design and evaluation. The paper opens the “black box” to address the question of how and why a policy intervention works and in which context and discusses a systematic process on how to approach the interwoven linkages between input, implementation and effects in gender equality interventions in research and innovation, accounting for context sensitivity and methodological pluralism. The evaluation framework may serve as reference for researchers, evaluators, policymakers and other stakeholders in designing and assessing gender equality interventions, and in further developing their evidence, and theoretical and methodological base.     

A conceptual tool: Making social workers' assumptions explicit

This paper emphasizes the importance for social work practitioners of making their assumptions about human nature explicit and of taking responsibility for the clinical consequences of using these assumptions. More specifically, it proposes a conceptual tool to help practitioners examine both the assumptions that guide their deliberations and the practical implications of these assumptions. This conceptual tool is based on nine fundamental assumptions about the human condition pertaining to matters such as freedom-determination, subjectivity-objectivity, proactivity-reactivity, and so forth. Each assumption is presented on a continuum along which practitioners can locate themselves. By deliberately placing themselves on each of the nine continua, the practitioners are encouraged to articulate their clinical concepts. This articulation enables them to maintain a critical dialogue with themselves about the ideas they use to understand and help their clients and makes the underlying concepts, and the decisions that stem from them, more amenable to the practitioner's conscious evaluation and control. The framework can also be used to compare the assumptions of practitioners to different theoretical approaches and can thereby help practitioners to better understand their own theoretical predilections. Finally, I propose that this framework be employed as an educational aid for the professional development of students and practitioners in social work education.     

Pragmatic ex-ante evaluation using an innovative conceptual framework: The case of a high-tech entrepreneurship program for women

Program designers at local project level usually design interventions under conditions of scarce resources and bounded rationality lacking sufficient information about the process and impact mechanisms of the complex social programs they design and implement. This paper proposes a model for pragmatic ex ante evaluation using an innovative conceptual framework for practitioners working in the field to improve gender equality in research and innovation.The presented and applied model is pertinent to the theory-oriented evaluation tradition using a logic frame and a theory of change approach adapted to the special requirements of ex-ante evaluation. The model is illustrated by the case study of a program for the promotion of women entrepreneurship in the STEM fields.It is concluded that in a setting with limited resources and information, ex-ante evaluations should aim at defining clear conditions of proportionality and need to take into account the extent to which contextual factors hinder or facilitate the implementation of a program, and the specific institutional and system mechanisms at play, in order to enable proactive risk management from the beginning.     

The Taping Project

This article describes and evaluates the effectiveness of a “Taping Project,” a teaching method for student self-evaluation in racism courses. The project offers students the opportunity (a) to hear in private the assumptions about themselves and others that they have brought into the course, (b) to reexamine these assumptions in the context of what they learn in the course, and (c) to consider if and how these assumptions have changed. Formal evaluation of the project’s effectiveness in helping students observe and reflect critically on their own development is reported in two ways: content analysis of 53 student essays using grounded theory analysis, and student and faculty evaluations of the assignment.     

CQI approaches for evaluating management development programs: A case study with health service managers from geographically remote settings

Health systems are known for being complex. Yet, there is a paucity of evidence about programs that successfully develop competent frontline managers to navigate these complex systems. There is even less evidence about developing frontline managers in areas of contextual complexity such as geographically remote and isolated health services. This study used a customised management development program containing continuous quality improvement (CQI) approaches to determine whether additional levels of evaluation could provide evidence for program impact. Generalisability is limited by the small sample size; however, the findings suggest that continuous improvement approaches, such as action learning workplace-based CQI projects not only provide for real-world application of the manager’s learning; they can potentially produce the type of data needed to conduct evaluations for organisational impact and cost-benefits. The case study contributes to the literature in an area where there is a scarcity of empirical research. Further, this study proposes a pragmatic method for using CQI approaches with existing management development programs to generate the type of data needed for multi-level evaluation.     

Socioeconomic status and cumulative disadvantage processes across the life course: implications for health outcomes

Given the complexity surrounding various interactions among health determinants and the challenge of being able to adequately describe the dynamic processes through which health determinants have their effects, the purpose of this paper is to provide a conceptual overview demonstrating the effects of socioeconomic status and cumulative disadvantage on producing health disparities across the life course. The idea underlying cumulative disadvantage is that socioeconomic-based health inequalities will increase across the life course, mostly because of differential exposure to risk factors and access to protective resources. The advantage of life course sociology is its consideration of early life experiences, and the social and historical context of their occurrences, as important contingencies in producing these systematic socioeconomic differences in health gradients.     

Mission,money, and merit: Strategic decision making by nonprofit managers

Public and nonprofit organizations need to make strategic choices about where to invest their resources. They also need to expose hidden managerial assumptions and lack of adequate knowledge that prevent the attainment of consensus in strategic decision making. The approach we developed and tested in the field used a dynamic, three‐dimensional model that tracks individual programs in an organization's portfolio on their contribution to mission, money, and merit. The first dimension measures whether the organization is doing the right things; the second, whether it is doing things right financially; and the third, whether it doing things right in terms of quality. Senior managers provide their own evaluations of the organization's programs. Both the consensus view and the variation in individual assessments contribute to an improved managerial understanding of the organization's current situation and to richer discussions in strategic decision making. In field tests, this visual model proved to be a useful and powerful tool for illuminating underlying assumptions and variations in knowledge among managers facing the complex, multidimensional tradeoffs needed in strategic decision making.     

Analysing facilitating and hindering factors for implementing gender equality interventions in R&I: Structures and processes

This article analyses the facilitating and hindering factors that have affected the implementation of gender equality interventions in research and innovation in Europe. It applies the evaluation framework developed in the EFFORTI project that recognizes the complexity of evaluating gender equality interventions in R&I, the importance of factoring in context to any sound evaluation as well as the need to distinguish between the design and implementation of interventions in evaluations. It is based on the analysis of 19 empirical case studies carried out throughout Europe and focuses on those structural and procedural factors that have either facilitated or hindered the implementation process of these interventions. Findings include how the governance framework; top-management commitment; bottom-up participation; framing synergies with other initiatives, strategies for tackling resistance; resources; sustainability of actions; gender competence, experience and knowledge and transparency, targets, standards and monitoring; and accessible data and information all contributed to the successful implementation of the interventions.     

The results framework--an innovative tool for program planning and evaluation.

This article constitutes a case study of the development and implementation of the "results framework," an innovative planning and evaluation tool that is rapidly becoming a standard requirement for United States Agency for International Development (USAID) projects. The framework is used in a USAID-funded regional initiative for HIV/AIDS prevention in Central America. This new program evaluation and monitoring tool provides many advantages over traditional evaluation approaches that use outside consultants to provide midterm and end-of-project evaluations. The results-framework process, which spans the life of the project, provides an opportunity for program staff, donors, partners, and evaluators to work as a team to collect and use rich, longitudinal data for project planning, implementation, and evaluation purposes.     

The personal factor in evaluation use : A case study of a steering committee's use of a conservation tillage survey

In 1985 a statewide farmer survey about conservation tillage was conducted. It provided baseline data for a major energy conservation project. The data were used for 2 years, at which time a case study was undertaken to identify factors which influenced the tillage survey's use. The influence of the personal factor on evaluation use had been identified prior to planning the tillage survey. Therefore, the process used for conducting the tillage survey capitalized on the personal factor by using a team approach. The case study identified ways intended users actually used the tillage survey results, associated these findings with the team strategy, and generated important factors contributing to evaluation use when the personal factor was accounted for. Factors supporting the personal factor which appear to have influenced use were timeliness, intended users' ownership of the information, interaction among decision makers and the evaluator, methodological appropriateness and quality, and planning for use of the results.     

Work functioning: a conceptual framework

Adequate and reliable methods for evaluation of clients with work disabilities are crucial for both the individuals who are assessed and for society as a whole. Sound and precise work assessments are needed to guide clients to suitable interventions using a minimum of rehabilitation resources. Occupational therapy literature contains evidence that work function assessments are complex and that there is confusion over work assessment concepts. Therefore, further development and evaluation of adequate concepts and theoretical models within the area of work assessment is needed. This article proposes a conceptual framework for different dimensions of work functioning and points out important factors for work assessment. The concepts proposed and defined in the article are: work functioning, work participation, work performance, and individual capacity.     

"The way to do is to be": exploring the interface between values and research

The Science, Religion, and Development (SRD) project of the International Development Research Centre aims to determine the conceptual and practical linkage between science, religion and development. Additionally, the center aims to mold ?being oriented? approaches, where values based in religion and spirituality inform the perspective and choices of methods of development researchers and practitioners. The SRD project has three phases. Phase one brought together four experts in science, international development, and theology who reflected on the relationship between the moral/spiritual realms of life and the technological/scientific models of development. The second and third phases will involve the publication of these reflections in book form. There are certain similarities between ?being oriented? approaches and other methodologies. As with other approaches, the SRD project has concern for inclusion and authentic dialogue as well as a commitment to transparency about the assumptions and biases that are present in research. Values specific to ?being oriented? approaches include its ability to provide a deeper and larger awareness which can offer new possibilities for response, changing the researchers themselves, as well as their understanding of the communities with whom they work.     

The ethics-economics-policy paradigm: The foundation for an integrated land trust conservation decision-support model

This paper sets forth a transdisciplinary approach, based upon current trans-Atlantic research carried out in Britain and the U.S., on the complex and interdisciplinary relationship of the ethics-economics-policy paradigm. To provide the proper associations within the land trust framework, the paradigm is viewed from the little explored private lands perspective.The results of interviews carried out with conservation professionals in Britain and the U.S. indicate a strong desire to use a more integrated framework to facilitate the decision-making process. Approximately 96% of the 139 land trusts interviewed felt that their organization's conservation efforts would ultimately be enhanced through the use of a decision-support tool that extended beyond traditional ecological criteria and constraints and incorporated both qualitative social and quantitative economic information. To this end, this paper will both examine the current status of land trust decisions making and frame the fundamentals and dimensions (e.g., land resources, sustainability, community, environment, etc.) of the ethics-economics-policy paradigm.The conceptual integrated land conservation decision-support (ILCDS) model is introduced. Aims of the model, as prescribed by the ethics-economics-policy paradigm, are to integrate concerns of economic efficiency, equity (between and within generations), behavioral models of resource use and ecological integrity, and other patterns of human and economic development within a private land–land trust context.     

Intimate partner violence and child abuse: a child‐centred perspective

A substantial body of literature assessing the impact of intimate partner violence on children and associations between intimate partner violence and child abuse now exists. Central to knowledge about these areas of practice and research is a robust, child‐centred consideration of some of the more challenging conceptual issues they give rise to. In this paper, we aim to stimulate debate by presenting a critical, child‐centred perspective on the intersection between intimate partner violence and child abuse. Initially, we provide the context for the review by presenting a brief overview of the literature. We then consider three central issues from a child‐centred perspective. We discuss the language used to describe children forced to live with intimate partner violence, and the importance of hearing children's perspectives on their experiences of such conflict. Finally, we consider similarities and differences between the intimate partner violence and child abuse fields from a child‐centred viewpoint. We conclude that increasing conceptual clarity around these issues will enhance research in the field, and ultimately improve interventions designed to protect children forced to live with intimate partner violence. Copyright © 2009 John Wiley & Sons, Ltd.     

A comparison of linear and systems thinking approaches for program evaluation illustrated using the Indiana Interdisciplinary GK-12

Logic models are based on linear relationships between program resources, activities, and outcomes, and have been used widely to support both program development and evaluation. While useful in describing some programs, the linear nature of the logic model makes it difficult to capture the complex relationships within larger, multifaceted programs. Causal loop diagrams based on a systems thinking approach can better capture a multidimensional, layered program model while providing a more complete understanding of the relationship between program elements, which enables evaluators to examine influences and dependencies between and within program components. Few studies describe how to conceptualize and apply systems models for educational program evaluation. The goal of this paper is to use our NSF-funded, Interdisciplinary GK-12 project: Bringing Authentic Problem Solving in STEM to Rural Middle Schools to illustrate a systems thinking approach to model a complex educational program to aid in evaluation. GK-12 pairs eight teachers with eight STEM doctoral fellows per program year to implement curricula in middle schools. We demonstrate how systems thinking provides added value by modeling the participant groups, instruments, outcomes, and other factors in ways that enhance the interpretation of quantitative and qualitative data. Limitations of the model include added complexity. Implications include better understanding of interactions and outcomes and analyses reflecting interacting or conflicting variables.     

Assumptions,conjectures, and other miracles: The application of evaluative thinking to theory of change models in community development

Unexamined and unjustified assumptions are the Achilles’ heel of development programs. In this paper, we describe an evaluation capacity building (ECB) approach designed to help community development practitioners work more effectively with assumptions through the intentional infusion of evaluative thinking (ET) into the program planning, monitoring, and evaluation process. We focus specifically on one component of our ET promotion approach involving the creation and analysis of theory of change (ToC) models. We describe our recent efforts to pilot this ET ECB approach with Catholic Relief Services (CRS) in Ethiopia and Zambia. The use of ToC models, plus the addition of ET, is a way to encourage individual and organizational learning and adaptive management that supports more reflective and responsive programming.     

Making sense of the emerging conversation in evaluation about systems thinking and complexity science

In the last twenty years, a conversation has emerged in the evaluation field about the potential of systems thinking and complexity science (STCS) to transform the practice of evaluating social interventions. Documenting and interpreting this conversation are necessary to advance our understanding of the significance of using STCS in planning, implementing, and evaluating social interventions. Guided by a generic framework for evaluation practice, this paper reports on an inter-disciplinary literature review and argues that STCS raises some new ways of thinking about and carrying out the following six activities: 1) supporting social problem solving; 2) framing interventions and contexts; 3) selecting and using methods; 4) engaging in valuing; 5) producing and justifying knowledge; and 6) facilitating use. Following a discussion of these issues, future directions for research and practice are suggested.     

Assembling the pieces: a framework for the integration of multi-functional ecological main structure in the emerging urban region of Bogotá, Colombia

Bogotá, the capital of the Republic of Colombia, is a tropical highland city located 2,650 m above sea level. It is the 25th largest city in the world and, among large cities, is also one of the highest. In common with other cities in Latin America, a large part of its urban growth during recent centuries has been unplanned and informal. The introduction of green spaces into urban planning in Bogotá began in the mid-20th century, but was first included in official legislation during the 1990s through the concept of Ecological Main Structure (EMS). Initially developed by Dutch scholars, EMS was brought to Colombia via biological conservation practitioners as a means of enhancing biological connectivity in rural and natural landscapes, extended in this case to urban landscapes as a top-level planning instrument. EMS originally included a variety of components, from protected areas and biological conservation tools to environmental urban elements - the emphasis being on biodiversity conservation, without sufficient recognition of specific urban structures and functions. This process led to conceptual disciplinary-based divergence and conflicting political interpretations. The current emergence of EMS as a planning tool for urban regions represents an opportunity for integration, although the risk of divergent interpretations remains, as no integrative conceptual framework has yet been developed. In this paper we review the concepts underlying EMS that have been incorporated within urban and regional planning, especially those of ecological networks and green infrastructure, and also diagnose conceptual and institutional barriers to its current integration, challenges and opportunities which are set in the context of an emerging urban region. We propose a trans-disciplinary framework for multi-level integration of EMS, along a gradient from wild environments to built structures that incorporates emerging concepts such as urban biodiversity, ecosystem services and design in the urban landscape, with the aim of contributing to the creation of an urban landscape that is resilient to environmental change and suitable for human well-being and adaptation.     

Planning an integrated agriculture and health program and designing its evaluation: Experience from Western Kenya

Multi-sectoral programs that involve stakeholders in agriculture, nutrition and health care are essential for responding to nutrition problems such as vitamin A deficiency among pregnant and lactating women and their infants in many poor areas of lower income countries. Yet planning such multi-sectoral programs and designing appropriate evaluations, to respond to different disciplinary cultures of evidence, remain a challenge. We describe the context, program development process, and evaluation design of the Mama SASHA project (Sweetpotato Action for Security and Health in Africa) which promoted production and consumption of a bio-fortified, orange-fleshed sweetpotato (OFSP). In planning the program we drew upon information from needs assessments, stakeholder consultations, and a first round of the implementation evaluation of a pilot project. The multi-disciplinary team worked with partner organizations to develop a program theory of change and an impact pathway which identified aspects of the program that would be monitored and established evaluation methods. Responding to the growing demand for greater rigour in impact evaluations, we carried out quasi-experimental allocation by health facility catchment area, repeat village surveys for assessment of change in intervention and control areas, and longitudinal tracking of individual mother-child pairs. Mid-course corrections in program implementation were informed by program monitoring, regular feedback from implementers and partners’ meetings. To assess economic efficiency and provide evidence for scaling we collected data on resources used and project expenses. Managing the multi-sectoral program and the mixed methods evaluation involved bargaining and trade-offs that were deemed essential to respond to the array of stakeholders, program funders and disciplines involved.