Teaching Note—Keeping It Real: Program Evaluation Projects for an Undergraduate Research Class

ABSTRACT

This article describes a teaching innovation that focused on the redesign of an undergraduate social work research class. Students enrolled in the redesigned class had an opportunity to conduct program evaluation projects in community agencies. The projects included (a) pretest and posttest evaluation of reminiscence approach in improving the well-being of older adults in a long-term care facility, (b) evaluation of family support services at a Head Start center, and (c) pretest and posttest evaluation of an awareness program to promote community inclusion of individuals with an intellectual disability. Steps in course development, implementation, and evaluation as well as the positive student learning outcomes are discussed. The article concludes with limitations and potential areas for further improvement.     

Building Community Leaders in Underserved Communities: An Exploration of the Role of Seed-Funding for Community Projects by Program Graduates

Investing in community leadership development (CLD) remains a strategy with promise in creating vibrant communities. Although many low-income CLD programs teach effective leadership skills, these often lay fallow due to lacking opportunities for graduates to apply the skills. This study explores how providing competitive minigrants fuels the creation of value-adding community projects. In-depth interviews were conducted with 55 graduates of a low-income CLD program, 19 facilitators, and 12 community members knowledgeable about the program. The study observed that by providing seed money for graduate projects, CLD programs can help translate skills into practice thereby increasing community engagement and socio-economic wellbeing.     

CMHS/CSAT Collaborative Demonstration Program for Homeless Individuals

The Collaborative Demonstration Program for Homeless Individuals is providing an opportunity for several community agencies in the United States to explore innovative intervention and treatment strategies to address the needs of homeless people with co-occurring (mental illness and substance abuse) disorders. This cooperative agreement, funded jointly by the Center for Mental Health Services (CMHS) and the Center for Substance Abuse Treatment (CSAT), is a $3 million project supporting programs in various community agencies to develop, document, and evaluate programs for persons who are homeless with co-occurring disorders to be used as models in other communities. The program models include: Assertive Community Treatment, Psychosocial Rehabilitation, Therapeutic Community, and an integrated treatment model. The first phase of the two-phase project funded 16 programs in the U.S. to develop manuals within their primary treatment modalities for homeless people with co-occurring disorders. For Phase II of the project, six of the 16 model programs were competitively selected to demonstrate program impact through formal program evaluation efforts. The data derived from the various projects will be compiled into a compendium of information for state agencies and community service providers throughout the U.S.     

Psychology in the Real World: Community Based Groupwork

Critical pedagogy is congruent with the social work discipline, which is engaged, people-focused, and centers on social justice, liberation and human rights. While there is growing recognition of the importance of better preparing social work students to engage in critical, anti-oppressive practice, students have limited opportunities for transformative learning experiences within community settings, outside of official practicums. Masters of Social Work students pursuing a specialization in international and community development (ICD) at a Canadian university were matched with community Journey Guides who provided mentorship and opportunities for students to become involved in community-based social justice initiatives. This article presents the eight-step experiential framework that was used as a pedagogical tool to support student learning, and the results from the program’s evaluation with the first student cohort. Using surveys and focus groups, the study found the guiding relationship was characterized by acceptance, friendliness, encouragement, and motivation. Students engaged in critical dialogues with Guides, gained community development experience and skills and enhanced their social justice knowledge.     

The self-defined experience of secular foster-care services for ultra-religious women in Israel: Using phenomenology to create cultural sensitive services

This paper qualitatively explores how foster parents from a religious minority group – more specifically foster parents from the Jewish ultra-orthodox sector in Israel experience the act of fostering through secular welfare services. Central themes that the women raised include religious spiritual beliefs as enhancing positive meaning of the act of foster care: Ambivalence of other women in the community because of fostering through external state services: The husband and religious leader as a central support due to the spiritual meanings of foster care. The central themes in terms of the secular services were their role as a protective financial base for creating a clear contract concerning the foster care as compared to informal fostering within the community and a culturally sanctioned exposure to psychological concepts and secular childcare practices. Lack of understanding of the secular welfare services of the importance of each group within their community was also stated as a challenge. Overall, the findings reveal the complex ways that these ultra-religious women from a very closed community negotiate and integrate resources from within their community and from the secular foster services outside of their community. Implications for creating culturally adapted secular foster services for Ultra-orthodox Jewish women are discussed, as are the methodological implications of exploring phenomenological experience of minority groups of foster care services as a base for culturally sensitive understanding, is discussed.     

Enhancing participatory research with the community oriented primary care model: A case study in community mobilization

Refining Participatory Research with the methodology of Community Oriented Primary Care (COPC), sociologists, medical providers and other “professionals” were able to assist the people of a small, rural county in western Appalachian North Carolina create a “community health consortium.” The consortium grew out of a research process funded by the W.K. Kellogg Foundation, which included some 40 interviews with community groups that articulated health problems and barriers to health care. The consortium then undertook action projects, which in turn enhanced networking between community groups and agencies while reducing mistrust between them. The consortium empowered people to articulate their realities, needs and became a force for change able to make demands on political office holders and decision makers. Despite its democratic origins and ideology, the consortium needs to do more to extend its outreach to include the marginal and disenfranchised.     

Failing better: The stochastic art of evaluating community-led environmental action programs

This article provides insights into the evaluation of a government-funded action for climate change program. The UK-based program aimed to reduce CO2 emissions and encourage behavioral change through community-led environmental projects. It, thus, employed six community development facilitators, with expertise in environmental issues. These facilitators supported and learnt from 18 community groups over an 18-month period. The paper explores the narratives of the six professional facilitators. These facilitators discuss their experiences of supporting community groups. They also explain their contribution to the wider evaluation of the community-led projects. This paper reflects on the facilitator experience of the program’s outcome-led evaluation process. In turn, it also explores how the groups they supported experienced the process. The facilitator’s narratives reveal that often community-group objectives did not align with predefined outcomes established through theory of change or logic model methodologies, which had been devised in attempt to align to program funder aims. Assisting community action emerges in this inquiry as a stochastic art that requires funder and facilitator willingness to experiment and openness to the possibilities of learning from failure. Drawing on in-depth accounts, the article illustrates that a reflexive, interpretive evaluation approach can enhance learning opportunities and provides funders with more trustworthy representations of community-led initiatives. Yet, it also addresses why such an approach remains marginal within policy circles.     

The Liberal Education of a Medieval Guildsman

This article illustrates how community service-learning can be defined and implemented as a group inquiry project in which students apply academic knowledge and social problem-solving skills. The authors examined eleven different community service-learning projects put into practice by elementary and middle school teachers who participated in CiviConnection, a national civic education program. They also explored how the teachers' practices were related to their perspectives on citizenship education, as the national CiviConnections program was designed to align with the reflective inquiry perspective by integrating historical inquiry in community service-learning. From the reflective inquiry perspective, they identified four elements of service-learning as a group inquiry project and analyzed eleven class projects to see how they incorporated the four components: (1) identifying issues and defining problems, (2) studying issues in-depth while using disciplinary modes of investigation, (3) developing action plans through decision-making processes, and (4) developing personalized knowledge through reflection. The authors discuss the study results' implications and examine opportunities to consider how community service-learning should be developed and implemented as a group inquiry project in which students develop personalized knowledge about community issues through discipline-based inquiry skills and decision-making processes.     

Clarifying the role of belief-motive explanations in multi-stakeholder realist evaluation

In realist evaluation, where researchers aim to make program theories explicit, they can encounter competing explanations as to how programs work. Managing explanatory tensions from different sources of evidence in multi-stakeholder projects can challenge external evaluators, especially when access to pertinent data, like client records, is mediated by program stakeholders. In this article, we consider two central questions: how can program stakeholder motives shape a realist evaluation project; and how might realist evaluators respond to stakeholders’ belief-motive explanations, including those about program effectiveness, based on factors such as supererogatory commitment or trying together in good faith? Drawing on our realist evaluation of a service reform initiative involving multiple agencies, we describe stakeholder motives at key phases, highlighting a need for tactics and skills that help to manage explanatory tensions. In conclusion, the relevance of stakeholders’ belief-motive explanations (‘we believe the program works’) in realist evaluation is clarified and discussed.     

Community Partnerships

Abstract

Community challenges force human service agencies to collaborate in providing services. Such collaborations require practitioners to have skills not found in mainstream social work curricula. This paper explores how a new MSW program evolved through dialog with community leaders and resulted in a curriculum with a sole concentration of community partnerships.     

Meeting the child care needs of the female alcoholic

The agency partnership described in this article has been satisfying to both the clients involved and the staffs of both agencies. The foster parents reported greater satisfaction from these fostering relationships than many others because of the short-term nature of the placement and their feeling that they were an instrumental force in healing and reuniting a family. The staffs of both agencies have enjoyed the exchange of professional information provided by the direct case contact, as well as inservice training designed to educate both staffs about the other's area of expertise. The total number of clients served has been low by original estimates. It is difficult to determine how much this has to do with client need versus the need for improved marketing of the child care option: the female alcoholic may underutilize the resource in her effort to deny the need for treatment. Further evaluation is needed. The strength of the partnership between the two agencies lies in its program flexibility, facilitated by reduced financial investment for either agency. The initial proposal was designed as a new continuum of service. The program can be adapted to meet the child care needs of the female alcoholic client as the needs change and/or become more defined.     

The Transformative Power of Youth Grants: Sparks and Ripples of Change Affecting Marginalised Youth and their Communities

This study, based on research into a youth empowerment initiative in Canada, examines the transformational power of youth grants for marginalised youth and their communities. The positive changes on individual youth included increased confidence and skills, as well as strengthened social interactions between youth, and involved adults and organisations. To leverage grant impact, we identify the critical role of creating accountability at multiple levels, promoting sharing among grantees, and fostering allies and system thinkers. The evaluation points to the potential of grants for changing community's perception that youth are incapable of fostering community youth development.     

Using theory of change as a framework for examining community context and philanthropic impact

As part of the process of examining their theory of change (TOC) and reflecting on grant making activity, one Midwestern foundation employed a multipronged strategy to assess 209 community-based mental health grants across seven years of funding. This article details the evaluation approach, which comprised these areas of the TOC: grantees’ use of evidence-based interventions, cultural competency, quality improvement, community collaboration, and use of integrated care. Inductive analyses identified grantees’ use of innovative practices, trauma-informed care, and use of validated instruments. In an iterative exercise spanning multiple years, the foundation and researchers found ways to gather information about community capacity and impact while simultaneously minimizing grantee data collection/reporting burden. Findings revealed that community agencies were improving in their uptake of evidence-based interventions and use of validated instruments. Community grantee use of trauma informed care also improved over the study period. Grantee desire to collaborate and coordinate services within the community was also strong. The research also revealed that in multiple domains (such as technology and non-profit operating costs) grantees needed support in building agency capacity. These findings shed light on which aspects of the foundation’s TOC were being addressed through funding, and which areas of the community needed additional support.     

Participation in stakeholder-based evaluation: A case study

This paper reports on the evaluation of some of the elements of a multi-component program aimed at improving the accessibility of health and social services for the homeless. The program involved provincial, regional and municipal authorities and more than 20 non-profit and public agencies. In accordance with the overall approach of the program, the principle of stakeholder participation in the evaluation was emphasized by the funding agencies. However, even if the evaluation started under promising conditions, including administrative support, a strong commitment from the involved agencies and experience in formative evaluation within the research team, participation of stakeholders from charitable non-profit agencies met with many difficulties. Using a case study approach, this paper attempts to analyze and discuss how the organizational context of the evaluation, certain stakeholder characteristics and methodological features may have impacted on the evaluation process.     

Unequally Adrift: How Social Class and College Context Shape Students’ Mentorship Experiences

Faculty mentorship is a highly advantageous yet under-explored form of social capital which can grant access to co-curriculars (e.g., research assistantships), ensure strong letters of recommendation, and more. It is also typically informal and dependent on student initiative, requiring that students be skilled at engaging educational authority figures. Privileged students are most likely to have such skills as part of their dominant cultural capital, making faculty mentorship a site of social reproduction. To explore variations in this process, I compare two institution types: a small, teaching/undergraduate-focused regional university and a large, research-intensive flagship. In interviews with 68 working- and upper-middle-class students, I find that college context mediates the relationship of class background and faculty mentorship. Upper-middle-class students fostered advantageous faculty relationships at both universities, but working-class students diverged: at the flagship, they rarely approached professors in search of mentorship, while those at the regional university described close, beneficial connections with professors. I discuss working-class students’ dissimilar experiences in terms of each university’s structural and cultural characteristics (organizational habitus), particularly their institutional focus and size. I argue that through their particular organizational features, colleges can both reproduce and reduce inequalities, challenging the determinacy of precollege socialization in education.     

Strengthening child and youth programs: A look at inter-organizational mentoring strategies

Community-based non-profit organizations rarely have access to research or evaluation evidence to inform their programs and often lack the capacity to gather or use this information independently. In 2016, Wisdom2Action—a network of knowledge mobilization (KMb) experts, policy makers and service providers across Canada—launched an inter-organizational mentorship program to facilitate the implementation and sharing of best and promising practices within community-based programs for young people. This article outlines the findings from a developmental evaluation of eight mentoring relationships. Drawing on the Promoting Action on Research in Health Sciences (PARiHS) model of KMb, we look at mentoring as a type of facilitation that supports the increased use of evidence and evaluation information by non-profit organizations and identify key themes that support effective organizational mentorship in this sector. Findings reinforce the importance of establishing connected relationships and understanding context in mentoring relationships, creating adaptive and responsive work plans, ensuring consistent communication, and maintaining a focus on capacity-building if knowledge mobilization is to occur.     

Piloting online WellnessRx learning modules: Demonstration of developmental evaluation

WellnessRx is a health initiative focusing on healthy living through education, knowledge translation, and community engagement. Stakeholders of WellnessRx identified web-based education learning modules on nutrition and physical education as a priority to be integrated into existing health sciences curricula, as well as adapted for use by health professionals. Five learning modules were created with essential knowledge, skills, attitudes and resources or tools for health professional students and practitioners. As part of the ‘developmental evaluation framework’ for WellnessRx, two of these modules were piloted within two health professional student programs. This paper describes the pilot-evaluation experience involving student surveys, focus groups and interviews, and faculty perspectives. For both modules, student pre-post knowledge assessments indicated some improvements in post-module knowledge. Post module evaluations by students indicated benefits with the online delivery being flexible for access, self-health, case-based assessments and useful nutrition and physical activity guides. Challenges for students included their time to do the modules and the activity expectations. Instructors felt each module could be better targeted to different years within an undergraduate program. Through developmental evaluation, the pilot results along with recommendations and lessons learned provided the direction needed to further develop the WellnessRx logic model and proposed learning modules.     

Involving carers from minority ethnic backgrounds in carer-led research

This qualitative pilot study investigated the interest of minority ethnic informal carers, with no previous experience of research, in engaging in carer-led research. Individual face-to-face interviews and focus groups were conducted to explore the understanding and attitudes carers hold to research alongside the motivators and de-motivators to their involvement in carer-led projects. The findings reveal that carers are interested in initiatives with a practical beneficial outcome for carers or those they look after. Black and minority ethnic carers identified interpreter and translation resources, gender sensitivity and flexibility around their involvement as major considerations. Carers’ perceptions of their transferable skills from personal or professional experiences to research were also closely linked to their willingness to engage in carer-led studies.     

Building Community Capacity to Advocate for Policy Change: An Outcome Evaluation of the Neighborhoods Working in Partnership Project in Detroit

This article presents results of the outcome evaluation of Neighborhoods Working in Partnership (NWP), a community-based participatory research project of the Detroit Community–Academic Urban Research Center, designed to enhance policy advocacy skills through training and support, thereby increasing community capacity, engagement, and empowerment of Detroit residents to change policies. Focus groups and conversational interviews were conducted with NWP participants. Results indicate that the workshop empowered participants and enhanced dimensions of community capacity. Participants reported engaging in policy advocacy activities and various policy successes. Participants identified challenges and facilitating factors to their policy campaigns. Recommendations for similar trainings are provided.