Innovating Practices to Prepare Students for Graduate School: Lessons From a Social Work MOOC

This article describes the implementation of a massive open online course (MOOC) developed by a school of social work and piloted in the incoming 2016 MSW class (N=397). Using implementation science as a framework, we describe the MOOC as an intervention for preparing students for graduate school and share findings from the pilot evaluation. We argue that MOOCs can represent an innovative approach for other schools of social work, but our lessons learned suggest that they also require resources, time investment, organizational support, and understanding of the implications for students and the school. Future questions for research include examining the long-term benefits of MOOCs to students, their broader application, and the cost-benefit analysis for schools interested in developing MOOCs.     

Using developmental evaluation as a system of organizational learning: An example from San Francisco

In the last 20 years, developmental evaluation has emerged as a promising approach to support organizational learning in emergent social programs. Through a continuous system of inquiry, reflection, and application of knowledge, developmental evaluation serves as a system of tools, methods, and guiding principles intended to support constructive organizational learning. However, missing from the developmental evaluation literature is a nuanced framework to guide evaluators in how to elevate the organizational practices and concepts most relevant for emergent programs. In this article, we describe and reflect on work we did to develop, pilot, and refine an integrated pilot framework. Drawing on established developmental evaluation inquiry frameworks and incorporating lessons learned from applying the pilot framework, we put forward the Evaluation-led Learning framework to help fill that gap and encourage others to implement and refine it. We posit that without explicitly incorporating the assessments at the foundation of the Evaluation-led Learning framework, developmental evaluation’s ability to affect organizational learning in productive ways will likely be haphazard and limited.     

Theory of change models deepen online learning evaluation

Massive Open Online Courses (MOOCs) have gained traction as resources for professional development. This article presents the method that we used to evaluate a professional development MOOC for postdoctoral trainees that was created by a university consortium in the US. Most approaches to evaluating MOOCs focus on analysis of participation, outcomes from course assignments, self-reported learning outcomes, course completion and user pathways through the online content or clickstream data. Few published evaluations describe in detail how learning happens within online courses and the anticipated medium and longer term cognitive and behavioral outcomes on participants. This work aims to guide those who are designing, implementing and evaluating MOOCs through applying theories of change to focus evaluation on the process of learning. This approach can be used as a complement to traditional approaches for evaluating MOOCs. We described how we worked with the MOOC team building the content to develop a theory of change for each module (or lesson) within the MOOC and used the theory of change to guide evaluation of short and medium term participant cognitive, affective and attitudinal, and behavioral outcomes. Finally, we share lessons learned and suggestions for implementing theories of change in both the design and evaluation phases of MOOC development.     

Participatory analytic hierarchy process for resource allocation in agricultural development projects

In the framework of 2030 Agenda sustainable agriculture plays a central role. In the field of international aid the participatory approaches to assessment, research, management and budgeting have been widely studied and applied in the last decades, mostly because international aid initiatives, in all their relevant phases from planning through implementation, monitoring and evaluation, are faced with the problem of identifying initiatives that could be successfully and sustainably implemented. The present paper investigates the use of a modified form of analytic hierarchy process, namely the participatory analytic hierarchy process, as a tool for choice criteria elicitation and resource allocation in the framework of an operational planning for agricultural development projects in the Dioceses of Goma, Nord Kivu, Democratic Republic of Congo. During our research, we were able to make use of most frequent inconsistencies in pairwise comparison matrixes in order to stimulate the debate, to adjust local preferences and to build consensus across the group. From an operational point of view, the PAHP methodology was also suited for training the project team and for identifying a shared resource allocation pattern, which matches the existing international guidelines for agricultural development in the region.     

Transactional activism without transactions: network perspective on anti-corruption activism in the Czech Republic

In 2007, Petrova and Tarrow coined the term transactional activism, arguing that, despite weak individual-level political participation, civil societies in Central and Eastern Europe were surprisingly strong due to their capacity to establish transactional links. Yet the research which followed, relying mostly on quantitative data, has not uncovered much evidence of how transactional activism works. To make advances here, we take a more qualitative approach and focus on transactional ties among NGOs, developing a more fine-grained conceptualization. Specifically, we distinguish associative ties of loosely defined cooperation and interlocking ties based on the division of labour. We utilize this new conceptualization through a successful anti-corruption initiative known as ‘Reconstruction of the State’. The initiative presented itself largely in transactional terms (plurality of participating actors) and was extremely successful in the run-up to the Parliamentary elections in 2013 in the Czech Republic, inspiring similar initiatives abroad. We use social network analysis and qualitative interviews to test our concept of interlocking transactional ties. In contrast to our expectations, we find limited division of labour in the initiative, with just one organization, in the main, shouldering most of the important tasks. We use our findings to question the previously claimed importance of transactional activism in Central and Eastern Europe. Specifically, we call for more robust evidence describing how cooperation among organizations empowers civil society.     

Public–private partnerships for e-government in developing countries: An early stage assessment framework

Although public–private partnerships (PPP) and electronic government (e-government) have proven to be fruitful mechanisms for economic development and emerging economies seem to recognize their importance, consistent methods for analyzing the early planning stages of e-government portfolios are lacking. The present work utilized a comprehensive literature review to understand the evaluation criteria for PPP projects throughout the early-stage planning process. A qualitative meta-synthesis was employed to identify critical factors for PPP and e-government, with a particular focus on developing countries, PPP, and e-government criteria. Our research presents a framework named “PPP4e-Gov” (public–private partnerships for e-government). The framework compares risk and value factors of e-government PPP projects and adopts a weighted scoring model that estimates the risks that should be considered in a project and how much value a given e-government initiative may generate if the PPP option is chosen. As an illustration of how the framework may be used, the paper interviewed ten practitioners in Costa Rica who tried out PPP4e-Gov and showcased how to plan their e-government initiatives.     

Reflections on facilitating a participatory community self-evaluation

This paper is a reflective essay about our experience in conducting a participatory, community self-evaluation of a neighborhood revitalization effort in Milwaukee, Wisconsin. We took a controversial approach to evaluation by focusing on learning community dialogues as a vehicle for program evaluation. As we implemented our strategy, two sets of contradictions surfaced. The first points to the fundamental differences between conventional and constructivist research paradigms. The second centers around our focus on dialogue as a strategy for evaluation. In response to these contradictions we performed three complementary roles: program evaluator, process consultant, and learning facilitator. We are not implying that we performed an outstanding job in these roles. Rather, this article is our way of taking stock of what we learned about our practice as evaluators and how we may expand our repertoire of skills in the future.     

New materialist social inquiry: designs,methods and the research-assemblage

This paper discusses issues of research design and methods in new materialist social inquiry, an approach that is attracting increasing interest across the social sciences as an alternative to either realist or constructionist ontologies. New materialism de-privileges human agency, focusing instead upon how assemblages of the animate and inanimate together produce the world, with fundamental implications for social inquiry methodology and methods. Key to our exploration is the materialist notion of a ‘research-assemblage’ comprising researcher, data, methods and contexts. We use this understanding first to explore the micropolitics of the research process, and then – along with a review of 30 recent empirical studies – to establish a framework for materialist social inquiry methodology and methods. We discuss the epistemological consequences of adopting a materialist ontology.     

Assumptions,conjectures, and other miracles: The application of evaluative thinking to theory of change models in community development

Unexamined and unjustified assumptions are the Achilles’ heel of development programs. In this paper, we describe an evaluation capacity building (ECB) approach designed to help community development practitioners work more effectively with assumptions through the intentional infusion of evaluative thinking (ET) into the program planning, monitoring, and evaluation process. We focus specifically on one component of our ET promotion approach involving the creation and analysis of theory of change (ToC) models. We describe our recent efforts to pilot this ET ECB approach with Catholic Relief Services (CRS) in Ethiopia and Zambia. The use of ToC models, plus the addition of ET, is a way to encourage individual and organizational learning and adaptive management that supports more reflective and responsive programming.     

The situational logic of entrepreneurship: a realist approach to national policy

National policy initiatives to encourage entrepreneurial behaviour are often developed with limited knowledge of the contextual constraints and causal factors. In this paper, we adopt a social realist approach, generally associated with critical realism, as a framework for analysing levels of entrepreneurial behaviour. The trigger for our study is the apparent anomaly of low levels of German entrepreneurship. We apply the morphogenetic framework of the sociologist Margaret Archer to Germany and then compare this to the United States. We conclude that particular so-called situational logics at the intersection of structure and culture constitute mechanisms which provide strategic direction for the decisions of potential entrepreneurs. We argue that a social realist stance based upon Archer’s sociology provides a practical grounding for research and policy development.     

An Initiators' Perspective on the Continuity of Citizens' Initiatives in Rural Areas

Against the backdrop of depopulation and the shift toward Big Society, citizens' initiatives in rural areas are believed to be able to mitigate the decline of service provision in rural regions. Consequently, this mitigation requires the continuity of such initiatives. However, so far, we lack an initiators' perspective on the relevance of this continuity and the factors influencing it. From a theoretical standpoint, continuity of citizens' initiatives can be understood at three levels: the participant, group, and initiative levels. Based on empirical data obtained from 157 questionnaires distributed to a variety of initiatives, and using regression analysis, the focus of this paper is twofold. First, how and at which level the initiators understand continuity of an initiative is considered. Second, factors influencing the expected continuity of an initiative are researched. The results reveal that continuity differs from merely being successful and is influenced by other factors as well. Furthermore, continuity on the initiative level—the realization of a certain goal—is most prominent in analyzing expected continuity. In conclusion, we explore the roles that citizens' initiatives can be expected to play in service provision and which levels of continuity align with these expectations.     

Analysing facilitating and hindering factors for implementing gender equality interventions in R&I: Structures and processes

This article analyses the facilitating and hindering factors that have affected the implementation of gender equality interventions in research and innovation in Europe. It applies the evaluation framework developed in the EFFORTI project that recognizes the complexity of evaluating gender equality interventions in R&I, the importance of factoring in context to any sound evaluation as well as the need to distinguish between the design and implementation of interventions in evaluations. It is based on the analysis of 19 empirical case studies carried out throughout Europe and focuses on those structural and procedural factors that have either facilitated or hindered the implementation process of these interventions. Findings include how the governance framework; top-management commitment; bottom-up participation; framing synergies with other initiatives, strategies for tackling resistance; resources; sustainability of actions; gender competence, experience and knowledge and transparency, targets, standards and monitoring; and accessible data and information all contributed to the successful implementation of the interventions.     

Trustworthiness in evaluation practice: an emphasis on the relational

As canons for trustworthiness developed explicitly in the discourse of qualitative inquiry, the emphasis was on procedural matters rather than fundamentally relational ones. A nod was made to the relational in such strategies as “member checks” but the issues of how the evaluator actually relates to participants and to the larger communities of practice and discourse—matters subsumed under moral principles and ethical standards—were often marginalized. This chapter posits that the first consideration in designing and conducting rigorous evaluation inquiry, and in critiquing the results of any research, should be the study's trustworthiness. Judging a study's trustworthiness, however, should focus on much more than procedural matters; it should also rely on moral principles and ethical standards which specifically address how we relate to an evaluation's participants. Central to these considerations are cross-cultural sensibilities about the nuanced meanings associated with any principles and standards. The chapter provides a theoretical framework for this position and offers two illustrative examples in the form of dialogues.     

Corporate responsibility in U.S. corporate websites: A pilot study

Corporate social responsibility (CSR) has become an essential component in the management of companies around the globe. Any key strategy used for CSR administration must reflect strong and effective communication channels to show CSR efforts. This study analyzes how the top 50 American profitable corporations from the Fortune 500 of 2009 communicate their CSR initiatives to different stakeholders through their corporate websites. To do this, an evaluation framework has been designed to examine features presented in CSR corporate websites.     

An approach to design long-term monitoring and evaluation frameworks in multi-actor systems—A case in water management

Learning-by-doing and adaptive management require careful monitoring and evaluation of the outcomes of environmental policies and programs under implementation. Selecting relevant indicators is difficult, especially when monitoring over a longer period of time. Further challenges arise when policies are developed as a collaborative effort among multiple actors.This paper discusses an approach to design frameworks for long-term monitoring and evaluation in multi-actor systems. It uses Dynamic Actor Network Analysis (DANA) as an actor-sensitive method to reconstruct program theories. This is combined with elements of assumption-based planning to identify critical assumptions and associated indicators to incorporate the dynamic aspects related to long-term monitoring.An application of this approach is described for a case of water management in the Netherlands. Here, mapping multiple perspectives and identifying critical assumptions helped to broaden the scope of monitoring in important ways. Identifying associated indicators and expectations on their development in response to policy implementation proved more difficult.From this case, it can be concluded that the approach is feasible, useful, but also demanding. However, with continuing trends of networked governance and adaptive management, additional efforts to reflect these trends in monitoring and evaluation, through this and similar approaches, are needed.     

An approach to design long-term monitoring and evaluation frameworks in multi-actor systems--a case in water management

Learning-by-doing and adaptive management require careful monitoring and evaluation of the outcomes of environmental policies and programs under implementation. Selecting relevant indicators is difficult, especially when monitoring over a longer period of time. Further challenges arise when policies are developed as a collaborative effort among multiple actors. This paper discusses an approach to design frameworks for long-term monitoring and evaluation in multi-actor systems. It uses Dynamic Actor Network Analysis (DANA) as an actor-sensitive method to reconstruct program theories. This is combined with elements of assumption-based planning to identify critical assumptions and associated indicators to incorporate the dynamic aspects related to long-term monitoring. An application of this approach is described for a case of water management in the Netherlands. Here, mapping multiple perspectives and identifying critical assumptions helped to broaden the scope of monitoring in important ways. Identifying associated indicators and expectations on their development in response to policy implementation proved more difficult. From this case, it can be concluded that the approach is feasible, useful, but also demanding. However, with continuing trends of networked governance and adaptive management, additional efforts to reflect these trends in monitoring and evaluation, through this and similar approaches, are needed.     

The Families and Democracy and Citizen Health Care Project

This article describes and updates the work of the Families and Democracy and Citizen Health Care Project, which engages with community concerns in order to effect change on many system levels. The project draws on family therapy's tradition of interest in larger social issues and adds democratic public theory and community organizing strategies. Since 1999 we have developed 14 citizen initiatives with a wide range of groups on a diverse set of problem areas. We describe the overall project and several of its initiatives, we address research and evaluation issues, and we outline opportunities for marriage and family therapists to learn how to do this work in their own communities.     

Evaluating Area‐based Interventions: The Case of ‘Communities for Children’

Increasingly, governments in wealthy countries are designing early intervention initiatives around principles of ‘community regeneration’ or ‘place management’. Because these initiatives are multi‐site, aimed at long‐term systemic change, and implemented amidst a range of other initiatives, assessing their quality and outcomes demands departure from conventional programme evaluation approaches. This article analyses the challenges of evaluating area‐based interventions in the child welfare field, and shows how the National Evaluation of Australia's ‘Communities for Children’ initiative seeks to overcome these through its mixed method design and the longitudinal Stronger Families in Australia study.