Staff and client perspectives on the Journey Mapping online evaluation tool in a drug court program

The purpose of this study was to assess staff and client perspectives on the Internet-based Journey Mapping program evaluation tool. A drug court program was chosen for a case study research design. Six staff and 10 clients participated in interviews and observations, and also responded to a questionnaire. A staff survey provided additional data. The study revealed evidence that (1) Journey Mapping was user-friendly, saved time, promoted positive thinking, facilitated communication, increased learning, and created client voice; (2) Journey Mapping, as a program evaluation tool, supported both administrative and treatment functions of the drug court; and (3) Journey Mapping's positive orientation supplied momentum for clients’ treatment progress. Journey Mapping's distinctive technology offered a viable option for including narrative methods along with quantitative methods in program evaluation.     

Rationale Argument Mapping Software

This software review focuses on using Rationale Argument Mapping Software to teach critical thinking skills in social work. Overall, the program is a great tool for helping students to think critically and precisely about their arguments.     

The Student Watershed Research Project (SWRP): collecting watershed data for use by resource management agencies

From 1992 to 1994, Saturday Academy and the Oregon Graduate Institute of Science and Technology instituted the pilot phase of a long-term monitoring program called the Student Watershed Research Project (SWRP). The SWRP program was developed to create and maintain collaboration among eighth through twelfth grade teachers and students, scientists, businesses, governmental agencies, and community members which would couple environmental education with the collection of high quality, reproducible watershed health data.The original authors of this paper set out to evaluate the impacts of the SWRP program in terms of changes in educational attitudes of participants and collection of accurate data. Findings demonstrate that assuring accuracy of student collected data was challenging, that teachers and students benefited greatly from a hands-on research approach to science education, and that such an approach would be difficult without the support of a program like SWRP.The SWRP program is entering its seventh year in the Portland, Oregon and Vancouver, Washington metropolitan areas with approximately half of the pilot phase participants still involved. A Quality Assurance Project Plan has been developed to assure data accuracy. Evaluation student science education attitudes documents improved results on a yearly basis.     

Strategies for Creating MIS Technology to Improve Social Work Practice and Research

SUMMARY

This paper illustrates the potential for Management Information System (MIS) technology to integrate information collection, management and reporting within a single program or network of organizations. Properly devised and created, MIS applications can improve administration, service delivery and practice evaluation. Three strategies are offered to guide the design and development of MIS software. This paper is based on lessons from the production and implementation of MIS software that serves as a management and evaluation tool for a nationwide policy demonstration. Data from the MIS have helped to shape state and federal policy.     

Evaluating unintended program outcomes through Ripple Effects Mapping (REM): Application of REM using grounded theory

Several evaluation models exist for investigating unintended outcomes, including goal-free and systems evaluation. Yet methods for collecting and analyzing data on unintended outcomes remain under-utilized. Ripple Effects Mapping (REM) is a promising qualitative evaluation method with a wide range of program planning and evaluation applications. In situations where program results are likely to occur over time within complex settings, this method is useful for uncovering both intended and unintended outcomes. REM applies an Appreciative Inquiry facilitation technique to engage stakeholders in visually mapping sequences of program outcomes. Although it has been used to evaluate community development and health promotion initiatives, further methodological guidance for applying REM is still needed. The purpose of this paper is to contribute to the methodological development of evaluating unintended outcomes and extend the foundations of REM by describing steps for integrating it with grounded theory.     

Register data in the evaluation and program planning of addiction treatment programs: Using Sweden as an example

Data from large-scale registers is often underutilized when evaluating addiction treatment programs. Since many programs collect register data regarding clients and interventions, there is a potential to make greater use of such records for program evaluation. The purpose of this article is to discuss the value of using large-scale registers in the evaluation and program planning of addiction treatment systems and programs. Sweden is used as an example of a country where register data is both available and is starting to be used in national evaluation and program planning efforts.The article focuses on possibilities, limitations and practicalities when using large-scale register data to conduct evaluations and program planning of addiction treatment programs. Main conclusions are that using register data for evaluation provides large amounts of data at low cost, limitations associated to the use of register data may be handled statistically, register data can answer important questions in planning of addiction treatment programs, and more accurate measures are needed to account for the diversity of client populations.     

Using freedom of information requests to facilitate research

The Freedom of Information Act (FOI) is a powerful tool for social researchers. However, these researchers have yet to harness the full potential of FOIA. Drawing on a number of studies using FOIA as a method of obtaining data, this article identifies the potential uses of FOIA and the limitations in its use, offering important advice on research design for studies which use FOIA both in the UK and abroad. The section ‘The role of freedom of information in research’ outlines the provisions of the UK FOIA and the section ‘How to make a request for information’ outlines the costs and benefits of utilising FOIA as a data collection tool. In sections ‘The risks and rewards of FOIA’ and ‘Combining FOIA and other methodologies,’ the article considers the place of FOIA within mixed methodology and cross jurisdictional studies, demonstrating the central role that FOIA can play in research.     

Psychology in the Real World: Community Based Groupwork

Critical pedagogy is congruent with the social work discipline, which is engaged, people-focused, and centers on social justice, liberation and human rights. While there is growing recognition of the importance of better preparing social work students to engage in critical, anti-oppressive practice, students have limited opportunities for transformative learning experiences within community settings, outside of official practicums. Masters of Social Work students pursuing a specialization in international and community development (ICD) at a Canadian university were matched with community Journey Guides who provided mentorship and opportunities for students to become involved in community-based social justice initiatives. This article presents the eight-step experiential framework that was used as a pedagogical tool to support student learning, and the results from the program’s evaluation with the first student cohort. Using surveys and focus groups, the study found the guiding relationship was characterized by acceptance, friendliness, encouragement, and motivation. Students engaged in critical dialogues with Guides, gained community development experience and skills and enhanced their social justice knowledge.     

Engaging families through observation and reflection: a collaborative approach to making learning visible

Early childhood education is based on the assertion that young children are better prepared for school and life when educators and parents work together on their behalf. Family engagement is a central force in the education of young children, yet research suggests that parents and teachers often do not collaborate, and this can have an adverse impact on young children’s learning outcomes. This brief explores a new observation approach, referred to as Journey of Discoveries, to examine its influence on family engagement and its potential use as a collaborative tool for educators and parents to gauge children’s learning and development. Journey of Discoveries was tested with volunteer families and teachers in three select Early Head Start programs in California in 2014–2015. Responses from participants illustrated seven themes or potential pathways emerging from the approach that support family engagement, including: (a) teachers’ and parents’ celebration of children’s learning, (b) emotional responses to children’s learning, (c) importance of sharing learning as stories, (d) process of reflection, (e) holistic viewing of children’s learning, (f) the honoring of diverse voices, and (g) parents’ roles as leaders. The practical and policy implications of the Journey of Discoveries approach are also discussed.     

The use of node-link mapping in the chemical dependency treatment of adolescents.

The current study examined treatment perceptions of youth adjudicated to an eight-month residential treatment facility. Participants were randomly assigned to standard or mapping-enhanced counseling. They completed established questionnaires assessing group sessions, program features, treatment motivation, and number of program infractions. Mapping residents rated group sessions as deeper over time. Mapped sessions appeared to also maintain residents' arousal over time. Mapping residents rated themselves as more motivated to get along with staff, and also tended to be more motivated to engage in the treatment process. In addition, mapping residents had fewer program infractions thirty days after the beginning of the experiment compared to standard residents. These findings extend previous research on node-link mapping in adult treatment settings.     

A visual management tool for program planning,project management and evaluation in paediatric health care

This article describes the development and implementation of a custom-designed Excel-based visual management tool. The tool’s purpose was to support program planning and evaluation by our resource support team within a paediatric health care setting. Our aims in developing it were to 1) establish a streamlined process and supporting tools to efficiently plan and prioritize program directions and activities; 2) track progress; and 3) evaluate and report on our performance, outputs and outcomes. A collaborative approach based on the ADKAR (Awareness, Desire, Knowledge, Ability, Reinforcement) change management model and the LEADS (Lead self, Engage others, Achieve results, Develop coalitions, Systems transformation) leadership framework was used to guide the design and implementation processes. Team members reported high perceived effectiveness and efficiency with respect to the tool’s utility in supporting its proposed aims. A graded approach to building knowledge and skills in using the tool, to individual responsibility for data entry, and to accountability by team members facilitated its successful implementation. Administrative support is important for sustainability and continual improvement of the tool to address changing team needs over time.     

Addressing challenges to carrying out intervention programs with youth populations: Successes and strategies

We identify five challenges notoriously faced by researchers conducting youth intervention studies: access to the target population, successful recruitment, ensuring continued attendance, promoting engaged, enthused, and task-focused participation, and efficient data collection. To ensure research quality, we have devised strategies to address these obstacles. Successes and lessons are included from The Compass Project (TCP), a 9-week morality strengthening program designed to facilitate positive attitudinal and behavioral outcomes in young people. Despite four of the five identified challenges being overcome in TCP, the fifth challenge of data collection was insurmountable as many participants failed to complete questionnaire scales. We propose that researchers build on our success by building rapport and trust with participants and youth organizations and building a participant sense of community, and improve upon our design by scrutinizing the format, accessibility, and length of data measures. Ultimately, tests of whether intervention programs can result in positive outcomes in the lives of young people hinge on adequately overcoming the identified challenges. Implementation of the proposed strategies will be instrumental to allow for meaningful and powerful statistical analyses to more accurately gauge the positive impact of intervention programs on young people's lives.     

Challenges related to the analytical process in realist evaluation and latest developments on the use of NVivo from a realist perspective

ABSTRACT

Realist evaluation (RE) is a research design increasingly used in program evaluation, that aims to explore and understand the influence of context and underlying mechanisms on intervention or program outcomes. Several methodological challenges, however, are associated with this approach. This article summarizes RE key principles and examines some documented challenges and solutions when analyzing RE data, including the development of Context-Mechanism-Outcome configurations. An analytic method using NVivo features is also presented. This method makes it possible to respond to certain analytic difficulties associated with RE by facilitating the identification of patterns and ensuring transparency in the analytical process.     

Developing a new system to measure outcomes in a service coordination program for youth with severe emotional disturbance

This paper presents information on re-developing an outcome evaluation for a state-funded program providing service coordination utilizing wraparound to youth with severe emotional disturbance (SED) and their families. Originally funded by the Robert Wood Johnson Foundation, the Kentucky IMPACT program has existed statewide since 1990. Changing data needs and limitations of the original evaluation required revamping the program's data collection system. The new evaluation uses the extant knowledge base to improve: (1) design, (2) measures, and (3) utility. A pre-post design with multiple follow-ups provides the framework for data collection. An ecological framework provides a conceptual structure for selecting measures focusing on both the service recipients and their environment. Data collection via a personal digital assistant (PDA) ensures utility of the data for both consumers and researchers. Issues ranging from conceptualization to implementation of the project as well as lessons learned are discussed.     

Evaluation of a complex, multisite, multilevel grants initiative

The Safe Schools/Healthy Students (SS/HS) national evaluation seeks to assess both the implementation process and the results of the SS/HS initiative, exploring factors that have contributed to or detracted from grantee success. Each site is required to forge partnerships with representatives from education, mental health, juvenile justice, and law enforcement, coordinating and integrating their efforts and working together to contribute to comparable outcomes (e.g., reduced violence and alcohol and drug use, improved mental health services). The evaluation uses multiple data collection techniques (archival data, surveys, site visits, interviews, and focus groups) from a variety of sources (project directors, community partners, schools, and students) over several years. Certain characteristics of the SS/HS initiative represent unique challenges for the evaluation, including the absence of common metrics for baseline, outcome data, and lack of comparison group. A unifying program theory was required to address these challenges and synthesize the large amounts of qualitative and quantitative information collected. This article stresses the role of program theory in guiding the evaluation.     

A Microcomputer Management of Information System for Community-Based AIDS Prevention Ethnographic Team Research

Abstract

Ethnographers working on community-based intervention projects increasingly face the task of collecting and analyzing large amounts of qualitative data. In addition, these data sometimes must be simultaneously gathered across diverse populations and at multiple locations. A cost-effective approach that addresses the difficulties in conducting this type of qualitative research is the use of an ethnographic team. Yet, the team concept requires sophisticated strategies for managing large volumes of unstructured textbased data. This article documents how the Houston Nalional AlDS Demonstration Research (NADR) project configured specific basic microcomputer technologesto asslst and enhance the management of ethnographic team research. Specifically, this system was a valuable and affordable tool to increase the speed, efficiency and rigor of data collection and analysis.     

Evaluating development programs: building joint activity

There has been an increasing plea for a participatory approach to development programs. It is argued that for programs to be sustainable, they need to be conceptualized, negotiated and run by all stakeholders jointly. Concurrent with this trend, there has been an increasing demand for program evaluation strategies, which encompass an evaluation of the nature and level of participation of the different stakeholders in the program. This paper proposes a theoretical framework for such evaluation and subsequently provides a practical illustration. Engeström's notions of ‘activity system’ and ‘learning by expanding’, are proposed as a framework for the evaluation of participation and an action evaluation methodology is suggested as the practical tool within such framework. The paper presents a case study of a conservation education program in Africa, which operationalizes the proposed theoretical framework.     

Investigating call record data using sequence analysis to inform adaptive survey designs

Researchers have become increasingly interested in better understanding the survey data collection process in interviewer-administered surveys. However, tools for analysing paradata capturing information about field processes, also called call record data, are still not yet fully explored. This paper introduces sequence analysis as a simple tool for investigating such data with the aim of better understanding and improving survey processes. A novel approach is to use sequence analysis within interviewers, which allows the identification of unusual interviewer calling behaviours, and may provide guidance on interviewer performance. Combining the technique with clustering, optimal matching and multidimensional scaling, the method offers a way of visualising, displaying and summarising complex call record data. The method is introduced to inform survey management and survey monitoring. The method is hence informative for adaptive survey designs and will help to identify unusual behaviour and outliers and to improve survey processes. Sequence analysis is applied to call record data from the UK Understanding Society survey. The findings inform further modelling of call record data to increase efficiency in call scheduling.