Using an action research approach to involving service users in the assessment of professional competence

This paper reports on a piece of action research that has involved people who use mental health services in systematically providing feedback from a user perspective on participants’ assessed work completed for one module of a masters’ training programme in mental health. In an attempt to improve professional practice and include people who were accessing mental health services in so doing, it outlines how users were trained to provide feedback and the training methods employed. The findings summarise the kind of issues users raised in their feedback to participants about the evidence professionals provided to demonstrate their learning from the training programme. A focus group interview with professionals provides a contrasting insight into the participants’ experience of having their work commented upon from a user perspective. The paper draws on the experience of a five-year external evaluation of an interdisciplinary programme in community mental health at Birmingham University in the UK which has highlighted the involvement of people who use mental health services as a particular innovation in the design, delivery and evaluation of the curriculum.     

Dealing with context in logic model development: Reflections from a realist evaluation of a community health worker programme in Nigeria

Logic models (LMs) have been used in programme evaluation for over four decades. Current debate questions the ability of logic modelling techniques to incorporate contextual factors into logic models. We share experience of developing a logic model within an ongoing realist evaluation which assesses the extent to which, and under what circumstances a community health workers (CHW) programme promotes access to maternity services in Nigeria. The article contributes to logic modelling debate by: i) reflecting on how other scholars captured context during LM development in theory-driven evaluations; and ii) explaining how we explored context during logic model development for realist evaluation of the CHW programme in Nigeria. Data collection methods that informed our logic model development included documents review, email discussions and teleconferences with programme stakeholders and a technical workshop with researchers to clarify programme goals and untangle relationships among programme elements. One of the most important findings is that, rather than being an end in itself, logic model development is an essential step for identifying initial hypotheses for tentative relevant contexts, mechanisms and outcomes (CMOs) and CMO configurations of how programmes produce change. The logic model also informed development of a methodology handbook that is guiding verification and consolidation of underlying programme theories.     

Evaluating the Impact of Post‐Qualifying Social Work Education

Post‐qualifying awards in social work are well established within the continuing professional development agenda for qualified social workers in the UK. The evaluation of education and training should be an integral part of this agenda because it is important to ensure that programmes continue to meet standards of delivery, are successful in meeting their aims and objectives and are making an impact on practice. However, there is a limited amount of published work on the evaluation of post‐qualifying social work education, with studies often focusing on programme delivery rather than on their impact on practice.

This paper explores evaluative work within the current post‐qualifying social work framework and discusses the results of an evaluation of the Vulnerable Adults and Community Care Practice programme, a specialist post‐qualifying social work education programme run by a UK university, as an example of an evaluation of the impact on practice. The results indicate positive evidence of impact on practice and demonstrate examples of how the programme has had a direct effect on individuals, teams, organisations and on people who use services.     

Evaluating a community based mental health programme in West Bengal,India: Description of the methodology and lessons learned

Mental disorders impose an enormous burden on society. In developing countries like India, there is a lack of adequate number of trained mental health professionals to provide specialized care and 75–85 % of affected individuals do not have access to appropriate mental health services. The National Mental Health Programme (NMHP) is being implemented by the Government of India to support state governments in providing mental health services in the country. The Urban Mental Health Programme (UMHP) is a pilot initiative that has attempted the integration of mental health services in primary health care settings in two municipal wards in Kolkata, West Bengal, India. The overarching aim of this paper is to describe the methodology used for the evaluation of the community based mental health programme and to understand the processes of the programme in terms of barriers and facilitators. The current evaluation is based on a concurrent nested design, where qualitative and quantitative data are both collected at the same time but analysed separately and priority was given to qualitative data. This experience will contribute in helping other researchers to make some evaluations more effective, useful and manageable. Ethics approval was obtained from an institutional ethics committee of an organization (Ekjut) based in Ranchi, Jharkhand, India. The evaluation was undertaken by the George Institute for Global Health, New Delhi from February- June 2016.     

Early multiple intervention: a view from On Track

On Track started life in 2000 as part of the Crime Reduction Programme in the Home Office, but was shifted to the Children and Young People's Unit less than a year later. This paper describes tensions that emerged during the first phase of the evaluation of this innovative programme, designed to provide interventions for four to twelve year olds at risk of becoming involved in crime. Analysis of interventions and multiple intervention is presented, together with reflections on the need for a different kind of evaluation and a more positive approach to services for children and young people.     

Understanding Mothers’ Engagement with Antenatal Parent Education Services: A Critical Analysis of a local Sure Start Service

This article discusses the issue of engagement in antenatal parent education for women living in deprived areas. The article focuses on the findings of a local evaluation of a Sure Start parent education programme designed to improve parental engagement with antenatal services. The article discusses the re‐orientation of this service, the impact on service users and the implications for professionals. Evidence suggests that this service has been beneficial for those involved and resulted in improved levels of engagement, with 9 per cent of eligible women attending prenatal sessions prior to the introduction of the Sure Start local programme's parent education service and 34 per cent of eligible women attending after its introduction. Despite this, the service only reaches a small proportion of the eligible population. The article also discusses the issue of professional resistance from mainstream services as an unintended consequence of the introduction of the Sure Start local programme and considers the implications of this for effective delivery. Copyright © 2006 The Author(s).     

Making activism a participatory,inclusive and developmental process: a research programme involving mental health service users

Abstract

Understanding real-world experiences and issues is a priority for research funders, yet there are challenges in directly engaging with people for evaluation and investigation of the professional services they receive. Much professional work takes place within institutional restrictions, arising from legislation, policy and local practices. Collaborative research is therefore complex but relevant for improving services. This research programme started in 2002 with focus group research, led by occupational therapists based in a local National Health Service trust and a university, followed by eight further projects including a Photovoice project. This programme and our perspectives from reflections suggest we do not always know who we are, what we want or who cares about research outcomes, but it is important to embrace the potential of activism, to energise and focus people for positive action, wherever they are based.     

Working with perpetrators of domestic violence to protect women and children: a partnership between Cheshire Probation Service and the NSPCC

Domestic violence is increasingly becoming recognized as a serious social problem with an enormous cost to society. While historically services have been established for women survivors, it is only recently that programmes designed to address men's violence towards women have become established, led predominantly within the criminal justice sector. This paper describes one such programme established in partnership between Cheshire Probation Service and the NSPCC. It identifies a number of factors considered to be essential in its successful implementation, including the importance of working in partnership, a sound and shared theoretical basis, the provision of a parallel support package for women and children and ongoing programme evaluation. Copyright © 1999 John Wiley & Sons, Ltd.     

Steps in the Right Direction,Against the Odds,An Evaluation of a Community‐Based Programme Aiming to Reduce Inactivity and Improve Health and Morale in Overweight and Obese School‐Age Children

The study describes an evaluation of a 48‐week physical activity and nutritional education programme for overweight/obese school‐age children using quantitative and qualitative methods. The majority of participants were obese or severely obese when enrolled, and while some improvements in body mass index, self‐esteem and engagement in a range of physical activities were noted, difficulties in evaluating a complex community‐based intervention were highlighted. Children, parents and health professionals described a range of benefits from attending the programme but more intensive services are likely to be needed for the most seriously obese children and for those families resisting intervention.     

Learning for the Twenty-First Century: Maximising Learning Transfer from Learning and Development Activity

This article builds on the learning gathered from the evaluation of a Scotland-wide child protection training programme. Based on Kirkpatrick's four-level framework, the study aimed to evaluate the impact of the learning programme on the practice of participants in the workplace. The study prompted the authors to examine the literature on maximising learning transfer across a range of occupations. This literature is discussed in the light of the findings from the evaluation and the experience of the authors in designing and implementing continuous professional development programmes in social services.

The practice of inspection, monitoring and audit seems to have impacted negatively on staff's willingness to participate in data collection, and coupled with what appeared to be confusion about data protection legislation, the perceived burden of audit and inspection has significant implications for research and evaluation activity.

Suggestions are made for the future design, delivery and evaluation of learning and development activity, and the role of managers is explored in relation to their responsibility for preparation, support and follow-up of workplace learning.     

The Caring Experience: Learning about Community Care through Spending 24 Hours with People Who Use Services and Family Carers

This paper describes a teaching and learning initiative developed in a UK social work programme that has involved carers, service users, academic social work staff and practitioners. Post‐graduate student social workers spend 24 hours in agreed periods with family carers or people who use social work services. The paper explains the origins of this initiative, rooted in a model of service user and carer involvement established over a number of years within the social work programme at the University of Dundee, Scotland. The development of this teaching and learning method is outlined and an account is given by a family carer describing their experience of being involved in the programme. Findings from the evaluation of this first year of operation are reported.     

Hard-to-reach or out-of-reach? Reasons why women refuse to take part in early interventions

The aim of this study was to explore the reasons why vulnerable women refuse to take part in early interventions. In-depth interviews were conducted with 19 women who refused to take part in an evaluation of an intensive home visiting programme. A number of themes were identified including perceptions about vulnerability, misperceptions about the service, misgivings about the service, and lack of trust. It is concluded that women who refuse to take part in early interventions are a diverse group, and that service providers need to take this diversity into account, if they are to improve uptake of services by vulnerable women. Service providers may also need to consider new ways of providing services.     

A complex solution to a complicated problem? Early messages from the national evaluation of the Children's Fund Prevention Programme

This paper is concerned with exploring the research and policy base that underpinned the development of the Children's Fund Prevention Programme. It discusses how the initiative developed, how it is being evaluated and how it is now contributing to the new policy agenda around prevention services. Early findings are discussed highlighting some of the challenges local projects have to manage around partnership working, participation of children and young people in the programme, and the embedding of prevention strategies in local practice. Finally the conclusion sets the Children's Fund in forthcoming central government policy activity and examines how the evaluation can best inform the development of collaborative preventative services to children and their families.     

Training care managers in risk assessment: Outcomes of an in-service training programme

Despite a recognition that social service workers require training in risk assessment, it would appear that there currently exists minimal information regarding appropriate models for risk assessment to underpin care managers' practice and, as a result, minimal discussion to inform considerations regarding appropriate training in this area. To date, not one study has appeared in the literature specifically examining whether in-service training provided to care managers actually results in changes to their work practices. This study provides results from an evaluation of an in-service training programme in risk assessment and demonstrates that the training had minimal impact on trainee behaviours. A number of reasons are offered as to why the training failed to transfer to practice; chief amongst these being insufficient time to learn the content of the training and factors associated with the specific work environment. Importantly, the particular model of risk assessment taught on the training programme was considered by care managers as far too time consuming and complicated to use, given the demands on their time and heavy workloads. It is suggested that if this particular risk assessment model is to be adopted by care managers, social services departments must ensure that such training is of appropriate duration to enable skills to be learnt, and that workplace contingencies are addressed to facilitate rather than hinder the use of training. These findings are considered to have practical significance for the design of similar short duration training programmes within social services.     

Hiring subsidies for people with a disability: Evidence from a small‐scale social field experiment

The effectiveness of hiring subsidies for people with disabilities remains unclear due to potential free‐rider, substitution and signalling effects. The authors propose a novel evaluation approach wherein it is randomly decided whether or not job applications disclose the subsidy to potential employers. Based on call‐back rates for interviews, the subsidy is found to be ineffective or even counterproductive in a group of adolescents having completed their vocational training programme. However, the negative signalling effect seems to be much weaker in a group of clients of job‐coaching services who acquired their disability during their working life.     

A Transformation Programme for Children's Social Care Managers Using an Interactional and Reflective Supervision Model to Develop Supervision Skills

In this paper, I describe a programme to promote interactional and reflective supervision for managers in children's services. The programme works at whole system change to embed a culture of reflective practice and management.     

Listening to children: Evaluation of a positive parenting programme through art-based research

This paper presents data from a unique programme evaluation of the parenting programme titled ‘Learning together, growing as a family’ applied in 14 cities in Spain and targeting families at risk of neglectful behaviour. The programme evaluation is based on the voices of children using the qualitative methodology of art-based research. Eighty-six children 6–12 years of age were interviewed in groups. The outcomes of the evaluation reveal that children perceive improvements in the parental competencies of their parents and in themselves and that these changes serve as a catalyst for positive family change.     

Exploring the Potential of Shadow Controls in the Evaluation of Children’s Services

This article describes and presents a framework for an under‐used evaluation technique in the context of an evaluation of a programme for disaffected young people. Shadow controls—the use of expert judgement to estimate the success of a programme—are often dismissed in research design as an unreliable form of comparison, but can be useful in situations where there is limited scope for a control group or to enhance the causal inference attributable to non‐experimental evaluations. The exercise described uses a practice tool as a structure for making predictions about the situations of the young people on the programme, assuming they do not receive an intervention. These predictions (shadow controls) are then compared to outcome data for the young people at the end of the programme. The results of the exercise provide some important messages about the programme’s effectiveness and the potential for strengthening non‐experimental evaluation methods. The article also discusses how the method can usefully inform evaluations of social programmes and encourage agency and user collaboration.     

Evaluating the effectiveness of a programme for improving the participation and academic success of an underrepresented minority group in New Zealand

Pacific peoples are a minority under-represented ethnic group in higher education in New Zealand. This article explores the effectiveness of a specific programme, which sought to improve outcomes of Pacific students in the tertiary environment. The aim of the evaluation was to determine the effectiveness of an intervention programme (2013–2015) to increase the participation and academic success of Pacific students in the first year in Health Sciences. The study found the academic results of Pacific students who participated in the intervention programme were significantly better compared to those who did not. The findings inform future research, suggesting that, when assessing the effectiveness of a programme, it is useful to explore the performance of the whole cohort separately to those who declared intention or interest to attend the programme. Strategies to support participation of each of these groups are likely to be different. Having a standardised approach when comparing groups will adjust for any confounding factors or prior differences. This will allow a more accurate assessment of the effectiveness of the programme being evaluated. This paper presents the importance of a robust approach to the delivery and evaluation of intervention programmes for improving outcomes for underrepresented students in the tertiary environment.