Evaluating systems change in a juvenile justice reform initiative

Evaluating comprehensive, interagency initiatives to reform human services systems presents substantial challenges to traditional research models. Outcomes are observed at the system level rather than the individual level, and the validity of study results may be challenged on a variety of dimensions, particularly small sample sizes and measurement error. We report the results of a cross-site evaluation of the first phase of Reclaiming Futures, a five-year effort to improve services and interventions for justice-involved youth in 10 communities across the United States. We present the evaluation findings and discuss implications for similar research endeavors. The methods used in this study may be useful for other system-level evaluation efforts. A social network analysis analyzes changes in the size, density, and cohesiveness of stakeholder networks. Bivariate models test for associations between Reclaiming Futures implementation and key stakeholder perceptions of effective system change. Multivariate models are used to explore selection effects on item response. In general, we find positive correlations between successful implementation of Reclaiming Futures and indicators of effective and efficient system change.     

Implementing Multidisciplinary Wellness Programs in Subsidized Housing: University–Community Partnerships

University–community partnerships (UCPs) are an important avenue for providing wellness services to seniors in subsidized housing, but there is sparse literature on this topic. The goals of this article are to (a) describe our UCP, which was unique in involving ethnically diverse older adults, and (b) discuss findings of an integrative literature review of UCP examples in subsidized housing. Results showed that UCPs typically received funding for program development and implementation; used multidisciplinary approaches to needs assessment and programming; and reported positive outcomes such as improvement in self-rated health or participant endorsements. This article provides insights and guidance for future UCPs that are much needed in subsidized housing.     

How to make good choices? Ethical perspectives guiding social workers moral reasoning

Framing and distinguishing mainstream social work ethical perspectives could be useful for social work students and practitioners facing moral choices in daily practice. The article presents a summary and classification of a range of ethical theories and provides some insights into the relationship between ethical thinking and practice issues. It distinguishes between classical approaches and ‘situated’ positions. The former – which includes teleological, deontological, and rights-based approaches – refer to universal and general principles. ‘Situated’ ethics are linked to specific and unique situations of professional practice. Finally, the article suggests some useful questions derived from each approach in order to help students to reflect on complex ethical choices occurring in daily practice.     

An approach to design long-term monitoring and evaluation frameworks in multi-actor systems—A case in water management

Learning-by-doing and adaptive management require careful monitoring and evaluation of the outcomes of environmental policies and programs under implementation. Selecting relevant indicators is difficult, especially when monitoring over a longer period of time. Further challenges arise when policies are developed as a collaborative effort among multiple actors.This paper discusses an approach to design frameworks for long-term monitoring and evaluation in multi-actor systems. It uses Dynamic Actor Network Analysis (DANA) as an actor-sensitive method to reconstruct program theories. This is combined with elements of assumption-based planning to identify critical assumptions and associated indicators to incorporate the dynamic aspects related to long-term monitoring.An application of this approach is described for a case of water management in the Netherlands. Here, mapping multiple perspectives and identifying critical assumptions helped to broaden the scope of monitoring in important ways. Identifying associated indicators and expectations on their development in response to policy implementation proved more difficult.From this case, it can be concluded that the approach is feasible, useful, but also demanding. However, with continuing trends of networked governance and adaptive management, additional efforts to reflect these trends in monitoring and evaluation, through this and similar approaches, are needed.     

An approach to design long-term monitoring and evaluation frameworks in multi-actor systems--a case in water management

Learning-by-doing and adaptive management require careful monitoring and evaluation of the outcomes of environmental policies and programs under implementation. Selecting relevant indicators is difficult, especially when monitoring over a longer period of time. Further challenges arise when policies are developed as a collaborative effort among multiple actors. This paper discusses an approach to design frameworks for long-term monitoring and evaluation in multi-actor systems. It uses Dynamic Actor Network Analysis (DANA) as an actor-sensitive method to reconstruct program theories. This is combined with elements of assumption-based planning to identify critical assumptions and associated indicators to incorporate the dynamic aspects related to long-term monitoring. An application of this approach is described for a case of water management in the Netherlands. Here, mapping multiple perspectives and identifying critical assumptions helped to broaden the scope of monitoring in important ways. Identifying associated indicators and expectations on their development in response to policy implementation proved more difficult. From this case, it can be concluded that the approach is feasible, useful, but also demanding. However, with continuing trends of networked governance and adaptive management, additional efforts to reflect these trends in monitoring and evaluation, through this and similar approaches, are needed.     

eHealth recruitment challenges

Little is known about effective eHealth recruitment methods. This paper presents recruitment challenges associated with enrolling African-American girls aged 8–10 years in an eHealth obesity prevention program, their effect on the recruitment plan, and potential implications for eHealth research.Although the initial recruitment strategy was literature-informed, it failed to enroll the desired number of girls within a reasonable time period. Therefore, the recruitment strategy was reformulated to incorporate principles of social marketing and traditional marketing techniques. The resulting plan included both targeted, highly specific strategies (e.g., selected churches), and more broad-based approaches (e.g., media exposure, mass mailings, radio advertisements).The revised plan enabled recruitment goals to be attained. Media appeared to be particularly effective at reaching the intended audience. Future research should identify the most effective recruitment strategies for reaching potential eHealth audiences.     

What is mixed in welfare mix? Welfare ideologies at stake in the Lithuanian case of social service delivery

The article deals with the development of social welfare and social services in Lithuania by describing cultural contexts and disclosing evaluations of social service providers and recipients and needs of social service delivery. The authors of the article discuss the changing role of the state, pose a question as to what the institutions and the actors are and to what extent should they participate in the creation of social welfare. One of the aims of this paper is to describe the development of the welfare mix in Lithuania, by showing that welfare is inevitably woven into the historical, economic, political and social context; the distinctive cultural configuration of the welfare mix in social services delivery in Lithuania is revealed. Empirical research (survey of social services recipients and providers) presented in the article discloses that actors of social services delivery experience tension and ambiguity between the demand of neoliberal policy to choose and be responsible and the expectation that the state will participate in the social service market. At the same time social services recipients and providers express a need for different actors to take an active part in the welfare system.     

Taking and giving: the case of the JDC and OZE in Lithuania 1919–26

This article examines the conditions that facilitated the activity of the American Joint Distribution Committee (JDC) in Lithuania during the early 1920s. In particular, it discusses the interaction between the JDC and the Society for the Protection of the Health of the Jews (OZE). It demonstrates that, unlike in other countries, the JDC's involvement in Lithuania went beyond the conventional development of policies and controlling budgets. The JDC practically took over the health section of the Lithuanian Jewish National Council and implemented its own policies. This step was possible due to the unique political situation of Jews in Lithuania and the fact that the OZE centre in Vilna was detached from the rest of that organisation's activists. This study demonstrates the importance of close analysis of specific localities when dealing with the history of philanthropy and the politics of taking and giving in Eastern Europe during the interwar era.     

A framework for evaluating eHealth research

Health care is in the midst of a consumer-oriented technology explosion. Individuals of all ages and backgrounds have discovered eHealth. But the challenges of implementing and evaluating eHealth are just beginning to surface, and, as technology changes, new challenges emerge. Evaluation is critical to the future of eHealth. This article addresses four dimensions of eHealth evaluation: (1) design and methodology issues; (2) challenges related to the technology itself; (3) environmental issues that are not specific to eHealth but pose special problems for eHealth researchers; and (4) logistic or administrative concerns of the evaluation methodology selected. We suggest that these four dimensions must be integrated to provide a holistic framework for designing and implementing eHealth research projects, as well as for understanding the totality of the eHealth intervention. The framework must be flexible enough to adapt to a variety of end users, regardless of whether the end user is a healthcare organization, a for-profit business, a community organization, or an individual. The framework is depicted as a puzzle with four interlocking pieces.     

Engaging communities: Ostrom's economic commons,social capital and public relations

This article discusses the research of economics Nobelist Elinor Ostrom. It argues her analysis of long term community co-operation contains important insights for public relations, especially in the area of community engagement. Informed by Ostrom's focus on common pool resource management, the article highlights the importance of organisations adopting an attitude of mutuality to the external stakeholder environment. It also associates public relations practice with the promotion of a social commons. It then examines Ostrom's insights on the role that face-to-face communication plays in the generation of social capital and the norms of behaviour that promote co-operation. The article ends with a discussion of the implications of these findings for the practice of public relations in organisations and suggests areas for further research.     

Pathways to poverty reduction

Of the myriad approaches to reducing poverty, which have proved effective on a national scale? This article analyses 15 systematically selected national cases of demonstrated rapid poverty reduction, seeking insights into effective approaches to reducing poverty. From these 15 economies, in which the bottom quintile experienced an annual increase in income of at least 6% over at least a decade, emerge four poverty‐reduction pathways: (1) industrialisation, (2) rural development, (3) social welfare and (4) petroleum‐generated employment. In addition to helping us understand what policy approaches have actually helped reduce poverty, this article has implications for the understanding of economic growth, the impact of pro‐growth policies, the relationship between state and market, and the roles of non‐government organisations (NGOs) and civil society.     

Hybrid networks and the global politics of the digital revolution – a practice‐oriented,relational and agnostic approach

The rapid growth of internet users and the importance of networked technologies for most spheres of life raise questions about how to foster and govern the digital revolution on a global scale. Focusing on internet governance and the use of ICTs for development purposes, I provide a multi‐sited, ethnographic exploration of two UN‐based multi‐stakeholder arrangements – comprising governments, business and civil society groups – that have contributed to the construction of the digital revolution as an object of global governance. In this article I show how analytical insights from governmentality studies and actor‐network theory can be used to capture how objects of governance and organizational arrangements are constructed and consolidated. Conventional approaches to networks and governance tend to treat organizational arrangements and issue areas as bounded, separate and fixed. By contrast, I demonstrate the merits of a practice‐oriented, relational and agnostic research strategy, which foregrounds the governmental techniques and moments of translation involved when new objects and modes of governance are assembled and negotiated.     

The participatory design of a performance oriented monitoring and evaluation system in an international development environment

This article illustrates the application of the impact monitoring and evaluation process for the design and development of a performance monitoring and evaluation framework in the context of human and institutional capacity development. This participative process facilitated stakeholder ownership in several areas including the design, development, and use of a new monitoring and evaluation system, as well their targeted results and accomplishments through the use of timely performance data gathered through ongoing monitoring and evaluation. The process produced a performance indicator map, a comprehensive monitoring and evaluation framework, and data collection templates to promote the development, implementation, and sustainability of the monitoring and evaluation system of a farmer's trade union in an African country.     

An evaluation of stakeholder capacity in the implementation of millenium village primary school meal project

The paper focused on evaluation of stakeholder capacity to implement the millennium village primary school meal project in Kenya. The study which was anchored on the stakeholder theory as well as the Context Input Process Product (CIPP) model of evaluation used mixed methods research design with ex-post facto and case study as its two research elements. The mixed methods was adopted because when both qualitative and quantitative approaches are used in a single study, the strengths of both produce a research synergy in which the collective benefits are greater than what is obtained from either approach when used alone. A sample size of 186 participants was selected from seven (7) primary schools using stratified random sampling. The participants selected purposively included pupils, teachers, head teachers, parents, Area Education Officer (AEO) and project staff. Data was collected using questionnaires, face to face interview guide, Focus Group Discussion (FGD) and document analysis guide. The study findings indicated that the project was implemented with great success as indicated by stakeholders’ capacity to understand project objectives, their involvement in project planning and implementation and their positive attitude towards the project. This has contribute to poverty reduction in the community through the project and sustainable approaches for project implementation.     

Application of logic models in a large scientific research program

It is the purpose of this article to discuss the development and application of a logic model in the context of a large scientific research program within the Commonwealth Scientific and Industrial Research Organisation (CSIRO). CSIRO is Australia's national science agency and is a publicly funded part of Australia's innovation system. It conducts mission-driven scientific research focussed on delivering results with relevance and impact for Australia, where impact is defined and measured in economic, environmental and social terms at the national level. The Australian Government has recently signalled an increasing emphasis on performance assessment and evaluation, which in the CSIRO context implies an increasing emphasis on ensuring and demonstrating the impact of its research programs. CSIRO continues to develop and improve its approaches to impact planning and evaluation, including conducting a trial of a program logic approach in the CSIRO Preventative Health National Research Flagship. During the trial, improvements were observed in clarity of the research goals and path to impact, as well as in alignment of science and support function activities with national challenge goals. Further benefits were observed in terms of communication of the goals and expected impact of CSIRO's research programs both within CSIRO and externally. The key lesson learned was that significant value was achieved through the process itself, as well as the outcome. Recommendations based on the CSIRO trial may be of interest to managers of scientific research considering developing similar logic models for their research projects. The CSIRO experience has shown that there are significant benefits to be gained, especially if the project participants have a major role in the process of developing the logic model.     

An introduction to methodological challenges in the evaluation of eHealth research: Perspectives from the Health e-Technologies Initiative

In February 2002 the Health e-Technologies Initiative (HETI), a program office of the Robert Wood Johnson Foundation^®, was created to advance discovery of scientific knowledge regarding the effectiveness of interactive eHealth applications. This article is the introduction to a series of seven articles written by grantees of HETI which address challenges, lessons learned, and proposed solutions as researchers implement eHealth projects. From this body of work it is clear that the overall process of conducting evaluation research in eHealth requires careful and detailed planning, recognition of the heightened sensitivity of IRBs, and institutions around the electronic collection and communication of personal health information, and a combination of tenacity and creativity to address the inevitable thorny methodological challenges to eHealth research. Use of established guidelines to help standardize the evaluation process, where feasible, is recommended.