Looking beneath the surface of a preceptor-training programme through a realist evaluation

South Africa’s department of health devoted themselves to the clinical nursing education and training model, which indorses preceptors as essential stakeholders to promote competence in students. A preceptor-training programme was developed that build on this model and implemented through an intervention. The initial programme theory hypothesised that trained preceptors would promote support and develop competence in students in comparison to untrained preceptors. Results showed a poor outcome because few preceptors participated and a high student attrition rates. The authors then posed the question “Why did the preceptorship training programme not work?” This article offers insight into the context, mechanism, and outcome of the preceptor-training programme through a realist evaluation. Reflective field notes, gathered during the intervention, were used and gave insight regarding how, for whom, and under which circumstances the programme could work and offered a refined programme theory for preceptorship. The main conclusion drawn showed that a preceptor-training programme alone is not the sole determining factor to ensure preceptors’ motivation to transfer their learning; a systems approach, ensuring the effectiveness of the programme, should be followed. Reflection on findings indicated the implementation context as a major determining factor of the training programme.     

“We believe in you,like really believe in you”: Initiating a realist study of (re)engagement initiatives for youth not in employment,education or training with experiences from northern Sweden

Due to a scarcity of rigorous evaluations and to commence a realist study addressing the lack of knowledge about the workings of interventions directed towards "NEET" youth, this research aimed to understand how and under what circumstances (re)engagement initiatives are expected to facilitate the social integration of young people who are in a situation that prevents them from entering into studies or work. By conducting the first phase in realist evaluation, qualitative interviews with five managerial stakeholders from two northern Swedish initiatives and reviews of documents were carried out for data collection. Using thematic analysis and retroductive reasoning, an intervention-context-actors-mechanisms-outcomes configuration was developed to elicit an initial programme theory that explained how the initiatives were presumed to operate and under what contextual contingencies. The results indicate that the intervention is expected to improve the youths’ wellbeing and engage them in work or studies by strengthening their competence and confidence in a caring and collaborative context. To incorporate the diverse voices and heterogeneous experiences of youth themselves, and ascertain whether the intervention works as intended, for whom, in what conditions and why, the results now need to be tested in selected cases and refined in subsequent phases of evaluation research.     

Lessons from Nigeria for improved thinking and working politically in the extractives sector

Despite the wealth that comes from being the biggest oil producer in Africa, Nigeria has some of the worst development indicators in the world. From 2011 to mid‐2016, the DFID‐funded Facility for Oil Sector Transparency and Reform (FOSTER) programme's unique design aimed to reduce incentives for the capture of oil revenue by elites and international oil companies, restoring the potential of those revenues to accelerate economic and social development. This article asks what lessons FOSTER's successes and failures offer for improved “thinking and working politically” (TWP). It examines the outcomes from five “clusters” of interventions: three viewed by the FOSTER team as successes and two as failures. The article identifies factors for successful TWP‐based programming, including the need for local ownership rooted in staff with a combination of technical expertise, a deep knowledge of the local political context and excellent networking abilities. The research used a qualitative and inductive approach. Field research was undertaken with 44 semi‐structured qualitative interviews during one month of fieldwork in Abuja and Lagos. The research also included reviews of FOSTER's internal documentation and evaluation frameworks, as well as analysis of newspaper articles and grey literature on the oil sector in Nigeria. The project offers important lessons for politically informed programming about how interventions were implemented (process), what was actually done (content) and how the project responded to changes in context that created or blocked opportunities for reform (responsiveness). The article identifies factors for successful TWP‐based programming, including the need for local ownership rooted in staff with a combination of technical expertise, a deep knowledge of the local political context and excellent networking abilities. The findings have important implications for programme design. They demonstrate the value of built‐in flexibility that allows staff to choose and switch the partners they work with and the channels they work through. They also show that a key aspect of TWP‐based programming is implicit acceptance that some failure may be unavoidable, since this permits staff to balance risk against opportunities. Finally, a better understanding of FOSTER's failures reveals the challenges of a TWP‐based approach and the trade‐offs it demands.     

Using a logic model to evaluate the Kids Together early education inclusion program for children with disabilities and additional needs

Despite clear evidence that learning and social opportunities for children with disabilities and special needs are more effective in inclusive not segregated settings, there are few known effective inclusion programs available to children with disabilities, their families or teachers in the early years within Australia. The Kids Together program was developed to support children with disabilities/additional needs aged 0–8 years attending mainstream early learning environments. Using a key worker transdisciplinary team model, the program aligns with the individualised package approach of the National Disability Insurance Scheme (NDIS).AimThis paper reports on the use of a logic model to underpin the process, outcomes and impact evaluation of the Kids Together program.MethodsThe research team worked across 15 Early Childhood Education and Care (ECEC) centres and in home and community settings. A realist evaluation using mixed methods was undertaken to understand what works, for whom and in what contexts. The development of a logic model provided a structured way to explore how the program was implemented and achieved short, medium and long term outcomes within a complex community setting.Discussion and conclusionKids Together was shown to be a highly effective and innovative model for supporting the inclusion of children with disabilities/additional needs in a range of environments central for early childhood learning and development. The use of a logic model provided a visual representation of the Kids Together model and its component parts and enabled a theory of change to be inferred, showing how a coordinated and collaborative approached can work across multiple environments.     

DFID governance programming in Nigeria: What difference has thinking and working politically made in practice?

The UK Department for International Development's country office in Nigeria (DFID Nigeria) has gradually adopted a thinking and working politically (TWP) approach in its governance programming. The initial focus on strengthening analysis has progressively been linked to discussion about the country and programme strategy, as well as programme‐management practices. Important lessons can be learned from this experience on how the TWP approach can be applied in practice. The article addresses three research questions: (1) To what extent have DFID Nigeria and its governance programmes adopted a TWP approach? (2) How has the TWP approach influenced the design and delivery of programming? (3) Has the application of the TWP approach enhanced the results of the DFID governance programming? Using project documentation and wider theoretical literature, but mainly relying on extensive participant observation within DFID Nigeria programmes, the article uses a historical perspective to outline how the TWP approach has been applied in Nigeria over 15 years of programme design, delivery, lesson‐learning and refinement . Published evaluation reports are used to provide evidence of programme results. DFID Nigeria and its programmes have progressively adopted TWP principles. This has led to clear changes in country strategy and programme design, as well as programme‐management practices. There is some evidence that the adoption of these principles has enhanced the results of DFID Nigeria programmes. By focusing on experimentation and “small bets,” TWP has proven relatively successful in generating and supporting ‘islands of effectiveness,’ but has had more limited impact in terms of generating more systemic, transformational change. The results obtained in Nigeria using the TWP approach have depended not only on Nigeria's political economy but also on the political economy of the development agency and donor country. DFID Nigeria's ability to engage in critical self‐reflection and to create an authorizing environment for risk‐taking have been vital to create enabling conditions for the TWP approach. However, other aspects of DFID and the UK's political economy are creating constraints that limit the prospects to go further and to go deeper in adopting the principles of TWP.     

Using a systems orientation and foundational theory to enhance theory-driven human service program evaluations

This paper offers a framework for using a systems orientation and “foundational theory” to enhance theory-driven evaluations and logic models. The framework guides the process of identifying and explaining operative relationships and perspectives within human service program systems. Self-Determination Theory exemplifies how a foundational theory can be used to support the framework in a wide range of program evaluations. Two examples illustrate how applications of the framework have improved the evaluators’ abilities to observe and explain program effect. In both exemplars improvements involved addressing and organizing into a single logic model heretofore seemingly disparate evaluation issues regarding valuing (by whose values); the role of organizational and program context; and evaluation anxiety and utilization.     

Reflections on experiential learning in evaluation capacity building with a community organization,Dancing With Parkinson’s

This paper discusses what was learned about evaluation capacity building with community organizations who deliver services to individuals with neurological disorders. Evaluation specialists engaged by the Ontario Brain Institute Evaluation Support Program were paired with community organizations, such as Dancing With Parkinson’s. Some of the learning included: relationship building is key for this model of capacity building; community organizations often have had negative experiences with evaluation and the idea that evaluations can be friendly tools in implementing meaningful programs is one key mechanism by which such an initiative can work; community organizations often need evaluation most to be able to demonstrate their value; a strength of this initiative was that the focus was not just on creating products but mostly on developing a learning process in which capacities would remain; evaluation tools and skills that organizations found useful were developing a theory of change and the concept of heterogeneous mechanisms (informed by a realist evaluation lens).     

Application of logic models in a large scientific research program

It is the purpose of this article to discuss the development and application of a logic model in the context of a large scientific research program within the Commonwealth Scientific and Industrial Research Organisation (CSIRO). CSIRO is Australia's national science agency and is a publicly funded part of Australia's innovation system. It conducts mission-driven scientific research focussed on delivering results with relevance and impact for Australia, where impact is defined and measured in economic, environmental and social terms at the national level. The Australian Government has recently signalled an increasing emphasis on performance assessment and evaluation, which in the CSIRO context implies an increasing emphasis on ensuring and demonstrating the impact of its research programs. CSIRO continues to develop and improve its approaches to impact planning and evaluation, including conducting a trial of a program logic approach in the CSIRO Preventative Health National Research Flagship. During the trial, improvements were observed in clarity of the research goals and path to impact, as well as in alignment of science and support function activities with national challenge goals. Further benefits were observed in terms of communication of the goals and expected impact of CSIRO's research programs both within CSIRO and externally. The key lesson learned was that significant value was achieved through the process itself, as well as the outcome. Recommendations based on the CSIRO trial may be of interest to managers of scientific research considering developing similar logic models for their research projects. The CSIRO experience has shown that there are significant benefits to be gained, especially if the project participants have a major role in the process of developing the logic model.     

Institutionalising the evaluation function: A South African study of impartiality,use and cost

PurposeThis article explores the implications of outsourcing the evaluation function in South Africa, a context where there is a mismatch between evaluation supply and demand. It unpacks the tradeoffs between internal and external evaluation, and challenges some commonly held assumptions about both.ApproachBased on experiences as an internal evaluator, external evaluator, evaluation manager, and building evaluation capacity, the author explores how each role changes when evaluation is a scarce skill, and looks at implications outsourcing has for both the organization, and the evaluation.FindingsThe purpose of the evaluation must drive the decision to outsource. However, with changing models of collaboration, there may be hybrid options that allow organizations to build evaluation capacity.Practical implicationsOrganisations are faced with a trade-off between commissioning an evaluation, and building internal evaluation capacity. To better understand each approach, it is important to consider the purpose and context of the evaluation. This shifts some commonly held assumptions about internal and external evaluations. Re-examining these assumptions will help organizations make a more informed decision about an evaluation approach.Originality/valueThe field of evaluation is particularly concerned with evaluation use. Most of the literature on this has focused on the approach of individual evaluators, and insufficient attention has been paid to the institutional architecture of the evaluation. This article considers how some of the organisational structures around an evaluation contribute to evidence use, and the case study of South Africa also shifts the focus to the central but overlooked role of context in the debate.     

Clarifying the role of belief-motive explanations in multi-stakeholder realist evaluation

In realist evaluation, where researchers aim to make program theories explicit, they can encounter competing explanations as to how programs work. Managing explanatory tensions from different sources of evidence in multi-stakeholder projects can challenge external evaluators, especially when access to pertinent data, like client records, is mediated by program stakeholders. In this article, we consider two central questions: how can program stakeholder motives shape a realist evaluation project; and how might realist evaluators respond to stakeholders’ belief-motive explanations, including those about program effectiveness, based on factors such as supererogatory commitment or trying together in good faith? Drawing on our realist evaluation of a service reform initiative involving multiple agencies, we describe stakeholder motives at key phases, highlighting a need for tactics and skills that help to manage explanatory tensions. In conclusion, the relevance of stakeholders’ belief-motive explanations (‘we believe the program works’) in realist evaluation is clarified and discussed.     

Designing centralized waiting lists for attachment to a primary care provider: Considerations from a logic analysis

Access to a regular primary care provider is essential to quality care. In Canada, where 15 % of patients are unattached (i.e., without a regular provider), centralized waiting lists (CWLs) help attach patients to a primary care provider (family physician or nurse practitioner). Previous studies reveal mechanisms needed for CWLs to work, but focus mostly on CWLs for specialized health care. We aim to better understand how to design CWLs for unattached patients in primary care.In this study, a logic analysis compares empirical evidence from a qualitative case study of CWLs for unattached patients in seven Canadian provinces to programme theory derived from a realist review on CWLs. Data is analyzed using context-intervention-mechanism-outcome configurations.Results identify mechanisms involved in three components of CWL design: patient registration, patient prioritization, and patient assignment to a provider for attachment. CWL programme theory is revised to integrate mechanisms specific to primary care, where patients, rather than referring providers, are responsible for registering on the CWL, where prioritization must consider a broad range of conditions and characteristics, and where long-term acceptability of attachment is important. The study provides new insight into mechanisms that enable CWLs for unattached patients to work.     

How to use programme theory to evaluate the effectiveness of schemes designed to improve the work environment in small businesses

Due to the many constraints that small businesses (SBs) face in meeting legislative requirements, occupational health and safety (OHS) regulatory authorities and other OSH actors have developed programmes which can reach out to SBs and motivate and assist them in improving the work environment. A number of conceptual models help to enhance our understanding of OHS interventions in SBs and their effectiveness. However, they have mainly been evaluated on output rather than the process relating to the change theory underlying the intervention, and hence have seldom been rigorously evaluated. Thus little is known about how particular features of SBs can be taken into account when designing and implementing national programmes. This paper shows how realist analysis and programme theory may be used as a framework for evaluating, developing and improving national intervention programmes for the improvement of the work environment and reducing injuries in SBs. It illustrates this for a specific New Zealand intervention: the Workplace Safety Discount scheme and its implementation in the agriculture sector. In practice, realist analysis should be performed during the planning, implementation and management stages so that ongoing findings can be fed back to the participant social actors to help them make appropriate changes to enhance the likelihood of success.     

Using logic model and visualization to conduct portfolio evaluation

Portfolio evaluation is the evaluation of multiple projects with a common purpose. While logic models have been used in many ways to support evaluation, and data visualization has been used widely to present and communicate evaluation findings, adopting logic models for portfolio evaluation and using data visualization to share findings simultaneously is surprisingly limited in the literature. With the data from a sample portfolio of 209 projects which aims to improve the system of early care and education (ECE), this study illustrated how to use logic model and data visualization techniques to conduct a portfolio evaluation by answering two evaluation questions: “To what extent are the elements of a logic model (strategies, sub-strategies, activities, outcomes, and impacts) reflected in the sample portfolio?” and “Which dominant paths through the logic model were illuminated by the data visualization technique?” For the first question, the visualization technique illuminated several dominant strategies, sub-strategies, activities, and outcomes. For the second question, our visualization techniques made it convenient to identify critical paths through the logic model. Implications for both program evaluation and program planning were discussed.     

Developing a public health policy-research nexus: An evaluation of Nurse Practitioner models in aged care

A frustration often expressed by researchers and policy-makers in public health is an apparent mismatch between respective priorities and expectations for research. Academics bemoan an oversimplification of their work, a reticence for independent critique and the constant pressure to pursue evaluation funding. Meanwhile, policy-makers look for research reports written in plain language with clear application, which are attuned to current policy settings and produced quickly. In a context where there are calls in western nations for evidence based policy with stronger links to academic research, such a mismatch can present significant challenges to policy program evaluation. The purpose of this paper is to present one attempt to overcome these challenges. Specifically, the paper describes the development of a conceptual framework for a large-scale, multifaceted evaluation of an Australian Government health initiative to expand Nurse Practitioner models of practice in aged care service delivery. In doing so, the paper provides a brief review of key points for the facilitation of a strong research-policy nexus in public health evaluations, as well as describes how this particular evaluation embodies these key points. As such, the paper presents an evaluation approach which may be adopted and adapted by others undertaking public health policy program evaluations.     

Enhancing Service Evaluability: Lessons from a Programme for Disaffected Young People

The effectiveness of children's services is often limited by a series of problems that also impede meaningful evaluation. This article describes and assesses research strategies to enhance the evaluability of a programme for disaffected young people, arguing that they have the potential to improve services more widely. It explores methods for developing a logic model, setting target group criteria, tightening programme components, identifying sufficient suitable candidates and selecting appropriate measures. Examples of other preparatory work aimed at helping the evaluation are given, including firming‐up the programme and evaluation ethics and dealing with the politics of a fairly complex evaluation (a randomised controlled trial) that involves numerous stakeholders.     

Risky behaviour: a new framework for understanding why young people take risks

Theories of youth risk taking range from the realist to the sociocultural. Much of this theorising, particularly in the field of epidemiology, has been strongly influenced by the Health Belief Framework. More recently, attention has shifted to understanding how young people perceive risk and what makes some of them resilient to risk taking. In this article we develop a framework that brings together diverse theoretical perspectives on youth risk taking. We draw on lessons from across the social science disciplines to inform a conceptual framework incorporating the broad context and internal processes of young people's decisions to take risks. Our Youth Risk Interpretation Framework (Y-RIF) has been developed from insights gained during an ethnographic study conducted in South Africa (Graham, Lauren, 2012. Understanding risk in the everyday identity-work of young people on the East Rand of Johannesburg. Doctoral Thesis. University of Johannesburg, Johannesburg.). We argue that our framework is useful, as it offers new ways of understanding why some young people take risks while others are more cautious. It could be used to inform youth behaviour surveillance research and interventions. However, it will need to be rigorously tested.     

Working with Families Who Experience Parental Mental Health and/or Drug and Alcohol Problems in the Context of Child Protection Concerns: Recommendations for Service Improvement

Child abuse and neglect often occur within the context of multiple risk factors, in particular parental mental health (MH) and/or drug and alcohol (D&A) problems. Interventions aimed at improving parental MH and D&A issues can have a positive impact on children now, as well as in the future. However, implementing sustainable service models that facilitate positive change for families with multiple risk factors is challenging. The purpose of the present study was to gain feedback from key stakeholders on a service model targeted at families where there are parental D&A, MH and child protection concerns to identify possible strengths and limitations of the model. This identified possible strategies for service improvement from the perspective of discharged clients and clinical staff. Gaining feedback from key stakeholders on service models is increasingly recognised as central to service evaluation and development. Ten interviews were conducted with clinical staff and twenty interviews with discharged clients of a pilot service that works with families where the child or children are at risk of significant harm in the context of parental MH and/or D&A issues. The interviews with clinicians highlighted difficulties working with this complex client group and its impact on staff burnout. Clinicians suggested how the model could be changed to better support clinical staff from burnout. Interviews with discharged clients highlighted the importance of the relationship with the worker in establishing client engagement and facilitating change. The way in which these recommendations informed the design of the service model is discussed.     

Devil is in the details: Using logic models to investigate program process

Theory-based logic models are commonly developed as part of requirements for grant funding. As a tool to communicate complex social programs, theory based logic models are an effective visual communication. However, after initial development, theory based logic models are often abandoned and remain in their initial form despite changes in the program process. This paper examines the potential benefits of committing time and resources to revising the initial theory driven logic model and developing detailed logic models that describe key activities to accurately reflect the program and assist in effective program management. The authors use a funded special education teacher preparation program to exemplify the utility of drill down logic models. The paper concludes with lessons learned from the iterative revision process and suggests how the process can lead to more flexible and calibrated program management.     

Leadership in Social Work Education and the Social Change Model of Leadership

Leadership is a topic of international debate in both social work education and practice. Questions remain, however, regarding what specific models of leadership should be emphasized and how leadership content should be infused within the social work curriculum. This paper seeks to contribute to the ongoing discussion about the infusion of leadership within social work education. Specifically, this paper shares the design and development of a graduate-level social work course in program evaluation that infused education around the social change model of leadership (SCM) and then explores qualitatively how 39 graduate students interpreted and applied this model of leadership through an experiential community-based evaluation project. Findings of this study suggest that students utilized the values identified in the SCM in a range of ways to understand and interpret their leadership experience both in their collaborative groups and in their partnerships with community-based organizations. Community organizations also reported benefiting from the experience in partnership with the students. Implications are discussed in relation to the potential value and impact of this model for social work education.     

The Letterbox Club book gifting intervention: Findings from a qualitative evaluation accompanying a randomised controlled trial

It is now widely accepted that reporting the results of randomised controlled trials should encompass a complimentary focus on both outcome measures and process measures. Reflective of the current thinking, this article reports on the findings from a qualitative evaluation that accompanied a randomised controlled trial of the Letterbox Club; a book gifting intervention for children in foster care. Outcome measures used in the trial have recently been reported on (Mooney, Winter, & Connolly, 2016). Findings showed no significant effects in terms of improvements to children's literacy skills and/or enjoyment of reading. Through in-depth interviews with 20 foster children, their carers and the programme developer, the qualitative evaluation focused on how and why the intervention did not achieve greater impact. Findings illustrate differences between the ‘hoped for’ outcomes of the intervention, and carer/child levels of engagement with and experiences of the programme. In order to move the programme forward and begin to explore the findings, a logic model is suggested which accounts for the current findings regarding the use and lived experiences which carers had with the packs. Wider implications for the conduct of these types of randomised controlled trials are discussed.