Aligning collaborative and culturally responsive evaluation approaches

The authors, three African-American women trained as collaborative evaluators, offer a comparative analysis of collaborative evaluation (O'Sullivan, 2004) and culturally responsive evaluation approaches (Frierson et al., 2002, Kirkhart and Hopson, 2010). Collaborative evaluation techniques immerse evaluators in the cultural milieu of the program, systematically engage stakeholders and integrate their program expertise throughout the evaluation, build evaluation capacity, and facilitate the co-creation of a more complex understanding of programs. However, the authors note that without explicit attention to considerations raised in culturally responsive evaluation approaches (for example, issues of race, power, and privilege), the voices and concerns of marginalized and underserved populations may be acknowledged, but not explicitly or adequately addressed. The intentional application of collaborative evaluation techniques coupled with a culturally responsive stance enhances the responsiveness, validity and utility of evaluations, as well as the cultural competence of evaluators.     

Assumptions at the philosophical and programmatic levels in evaluation

Stakeholders and evaluators hold a variety of levels of assumptions at the philosophical, methodological, and programmatic levels. The use of a transformative philosophical framework is presented as a way for evaluators to become more aware of the implications of various assumptions made by themselves and program stakeholders. The argument is examined and demonstrated that evaluators who are aware of the assumptions that underlie their evaluation choices are able to provide useful support for stakeholders in the examination of the assumptions they hold with regard to the nature of the problem being addressed, the program designed to solve the problem, and the approach to evaluation that is appropriate in that context. Such an informed approach has the potential for development of more appropriate and culturally responsive programs being implemented in ways that lead to the desired impacts, as well as to lead to evaluation approaches that support effective solutions to intransigent social problems. These arguments are illustrated through examples of evaluations from multiple sectors; additional challenges are also identified.     

Collaborative evaluation within a framework of stakeholder-oriented evaluation approaches

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation planning and implementation. Unlike "distanced" evaluation approaches, which reject stakeholder participation as evaluation team members, Collaborative Evaluation assumes that active, on-going engagement between evaluators and program staff, result in stronger evaluation designs, enhanced data collection and analysis, and results that stakeholder understand and use. Among similar "participant-oriented" evaluation approaches (Fitzpatrick, Sanders, & Worthen, 2011), Collaborative Evaluation distinguishes itself in that it uses a sliding scale for levels of collaboration. This means that different program evaluations will experience different levels of collaborative activity. The sliding scale is applied as the evaluator considers each program's evaluation needs, readiness, and resources. While Collaborative Evaluation is a term widely used in evaluation, its meaning varies considerably. Often used interchangeably with participatory and/or empowerment evaluation, the terms can be used to mean different things, which can be confusing. The articles use a comparative Collaborative Evaluation Framework to highlight how from a theoretical perspective, Collaborative Evaluation distinguishes itself from the other participatory evaluation approaches.     

Building relationships and facilitating immigrant community integration: An evaluation of a Cultural Navigator Program

Despite the long history of immigration in the United States, communities around the country struggle to integrate newcomers into the economic, cultural, and political spheres of society. Utilizing results from the program evaluation of one public library's Cultural Navigator Program, the authors illustrate how communities and public institutions can promote integration and relationship-building between newly arrived immigrants and long-time residents. Existing social networks within receiving communities, conceptualized in this article as social capital, were leveraged to build capacity among newly arrived immigrants and foster inclusivity and integration at the community level. As a place of intervention, public libraries are suggested as a safe and shared space where community integration can be fostered. Insights derived from the evaluation inform a discussion on engaging approaches to immigrant integration. Lessons learned and recommendations for program evaluators and administrators are provided.     

Relational responsibilities in responsive evaluation

This article explores how we can enhance our understanding of the moral responsibilities in daily, plural practices of responsive evaluation. It introduces an interpretive framework for understanding the moral aspects of evaluation practice. The framework supports responsive evaluators to better understand and handle their moral responsibilities. A case is introduced to illustrate our argument.Responsive evaluation contributes to the design and implementation of policy by working with stakeholders and coordinating the evaluation process as a relationally responsible practice. Responsive evaluation entails a democratic process in which the evaluator fosters and enters a partnership with stakeholders. The responsibilities of an evaluator generally involve issues such as ‘confidentiality’, ‘accountability’ and ‘privacy’. The responsive evaluator has specific responsibilities, for example to include stakeholders and vulnerable groups and to foster an ongoing dialogue. In addition, responsive evaluation involves a relational responsibility, which becomes present in daily situations in which stakeholders express expectations and voice demands. In our everyday work as evaluators, it is difficult to respond to all these demands at the same time. In addition, this article demonstrates that novice evaluators experience challenges concerning over- and underidenfitication with stakeholders. Guidelines and quality criteria on how to act are helpful, but need interpretation and application to the unique situation at hand.     

Advances in collaborative evaluation

Collaborative evaluation is an approach that offers, among others, many advantages in terms of access to information, quality of information gathered, opportunities for creative problem-solving, and receptivity to findings. In the last decade, collaborative evaluation has grown in popularity along with similar participatory, empowerment, and utilization-focused evaluation approaches. Increasingly, it has brought together evaluators and stakeholders from different sectors, disciplines, and cultures to exchange knowledge on how collaboration can be used as a strategic tool for fostering and strengthening evaluation practice. O'Sullivan (2004) and Rodríguez-Campos, 2005, Rodríguez-Campos, 2008 brought forward collaborative evaluation conceptual frameworks that distinguished it from other approaches. This article updates those conceptual frameworks and discusses key collaborative evaluation advances in order to further clarify and facilitate engagement in sound practice.     

Creating Culturally Responsive Family Therapy Models and Research: Introducing the Use of Responsive Evaluation as a Method

Models of marriage and family therapy (MFT) typically reflect Western values and norms, and although cultural adaptations are made, many models/frameworks continue to be inappropriate or inadequate for use with non‐Western cultures. Worldwide, therapists are examining ways of using MFT models in a culturally sensitive manner, especially when working with clients who are seen as having minority status or perceived as “other” by the dominant group. This essay suggests the use of responsive evaluation as a theoretically consistent methodology for creating and evaluating culturally responsive therapies. This approach rigorously evaluates each unique client/therapist context, culture, power, needs, and beliefs. We describe responsive evaluation and discuss how each component addresses the research needs of examining culturally responsive family therapies. A case illustration is offered delineating the process of conducting culturally responsive therapy with a Cambodian sample using solution‐focused and narrative therapy.     

Culturally Responsive Teaching: What do selected United States elementary school teachers think?

In this paper culturally responsive teaching is viewed as one way of implementing the concept of multicultural education in schools. In recent years, scholars have devoted considerable attention to the importance of aligning classroom experiences with students' home culture as a way of enhancing social, academic and cultural enrichment of all children. This study was, therefore, designed to identify perceptions of teachers toward the importance of culturally responsive teaching in elementary schools in the United States. The responses of 33 teachers were analyzed by using both qualitative and quantitative techniques. The findings of this study, along with a summary and implications for future studies, are presented in this paper. The data analysis clearly indicated that over 96% of the respondents considered culturally responsive teaching to be an important part of working with culturally diverse students. It was significant to note that the way teachers reported their perception toward the importance of culturally responsive teaching matched the manner with which they responded to the characteristics of culturally relevant teaching.     

Advances in collaborative evaluation

Collaborative evaluation is an approach that offers, among others, many advantages in terms of access to information, quality of information gathered, opportunities for creative problem-solving, and receptivity to findings. In the last decade, collaborative evaluation has grown in popularity along with similar participatory, empowerment, and utilization-focused evaluation approaches. Increasingly, it has brought together evaluators and stakeholders from different sectors, disciplines, and cultures to exchange knowledge on how collaboration can be used as a strategic tool for fostering and strengthening evaluation practice. O'Sullivan (2004) and Rodríguez-Campos (2005, 2008) brought forward collaborative evaluation conceptual frameworks that distinguished it from other approaches. This article updates those conceptual frameworks and discusses key collaborative evaluation advances in order to further clarify and facilitate engagement in sound practice.     

Organizational characteristics that foster early adoption of cultural and linguistic competence in outpatient substance abuse treatment in the United States

Recent years have seen an increased interest in developing culturally and linguistically responsive systems of care in substance abuse treatment in the United States. This study examines the extent to which external and internal organizational pressures contributed to the degree of adoption of culturally and linguistically responsive practices in the nation's outpatient substance abuse treatment system early in the period of development of this system of care. Findings show that a higher degree of adoption of culturally competent practices was most likely in treatment programs with high dependence on external funding and regulation. Internally, programs with a larger number of professionals were associated with the lowest degree of adoption, while managers’ cultural sensitivity contributed significantly to a high degree of adoption of these responsive practices. Considering the passage of recent legislation enforcing the use of cultural and linguistic competence in health care, implications of these baseline findings on early adoption patterns are discussed for future research and health care policy evaluation.     

Avoiding evaluation cooptation. Lessons from a renal dialysis evaluation.

An evaluation of the impact of terminating contracts to transport low-income patients for renal dialysis demonstrates how evaluators may become apologists for management and conduct studies too narrow for effective decision making. The evaluation found that patients continued to receive treatment; their death rates had not increased. Changes in their economic and emotional status were not studied; consequently, the evaluators erroneously concluded that the clients had not been seriously harmed. The article reviews the process the evaluation used. The authors conclude that to avoid cooptation, evaluators need independent information available through conducting and analyzing pilot studies and meeting with program constituents. To avoid overly narrow studies, the next level of management should participate in their design. To use the information effectively, evaluators need to see themselves as advisors rather than technicians.     

“Thank You for Including Us!” – Introducing a Community‐Based Collaborative Approach to Translating Clinic Materials

When working with clients in another language, having culturally relevant and properly translated clinic materials is vital to effective clinical practice. Not having them presents a barrier for bilingual therapists and their clients. This paper reviews common translation methods and introduces a multi‐level, community‐based approach for translating clinic materials for non‐English speaking clients. Informed by decolonizing practices, this five‐tier method includes members from the target community as cultural brokers to verify the accuracy and nuances of language for their cultural group, as well as a constant comparative method to ensure methodological rigor in the process of inclusion. We present community member's feedback and recommendations, as well as discuss advantages and challenges of using this collaborative culturally responsive translation method.     

Agent-based modeling as a tool for program design and evaluation

Recently, systems thinking and systems science approaches have gained popularity in the field of evaluation; however, there has been relatively little exploration of how evaluators could use quantitative tools to assist in the implementation of systems approaches therein. The purpose of this paper is to explore potential uses of one such quantitative tool, agent-based modeling, in evaluation practice. To this end, we define agent-based modeling and offer potential uses for it in typical evaluation activities, including: engaging stakeholders, selecting an intervention, modeling program theory, setting performance targets, and interpreting evaluation results. We provide demonstrative examples from published agent-based modeling efforts both inside and outside the field of evaluation for each of the evaluative activities discussed. We further describe potential pitfalls of this tool and offer cautions for evaluators who may chose to implement it in their practice. Finally, the article concludes with a discussion of the future of agent-based modeling in evaluation practice and a call for more formal exploration of this tool as well as other approaches to simulation modeling in the field.     

Portfolio assessment in a collaborative program evaluation: the reliability and validity of a family literacy portfolio

Evaluations increasingly use open-ended assessments, such as portfolios, to gather information about program impact. Portfolios, however, present challenges in achieving acceptable levels of reliability and often lack evidence to support validity. This article describes the components of a portfolio assessment system developed in a collaborative evaluation of a family literacy program. Staff of a family literacy program and two evaluators collaborated to develop the assessment and the tools that frame the review of the portfolios. Rubrics, exemplar portfolios, portfolio logs, and lists of ‘look-fors’ were developed to guide raters in the scoring and to improve the reliability of scores and validity of decisions about the program. Inter-rater reliability for the portfolio assessment was 0.79 for holistic scores and 0.74 for analytic scores. With respect to validity, holistic and analytic scores correlated with criterion measures from 0.50 to 0.59, providing initial evidence of a common construct.     

Techniques for the analysis of geographic data in evaluation

Although geographic relationships are often an important aspect of a program or system under evaluation, these relationships have frequently been ignored by evaluators. Standard techniques do exist, however, for the analysis of geographic data. This paper presents three such techniques, Geocode Analysis, Trend Surface Analysis, and Social Area Analysis, and discusses their use in evaluation. With these techniques, evaluators can more adequately deal with the questions of geographic relationships which arise in many studies.     

Building effective working relationships across culturally and ethnically diverse communities

Amherst H. Wilder Foundation's Social Adjustment Program for Southeast Asians is implementing two collaborative, best practice, mental health and substance abuse prevention service models in Minnesota. It faced several issues in effectively bridging multiple cultural groups, including building a diverse collaborative team, involving families and youth, reconciling cultural variation in meeting styles, and making best practice models culturally appropriate. Researchers and program staff used multiple strategies to address these challenges and build successful partnerships. Through shared goals, flexibility, and a willingness to explore and address challenges, collaboratives can promote stronger relationships across cultural communities and improve their service delivery systems.     

The value of experiential approaches to evaluation training

The training of evaluators is an important task—important because of the good that well trained evaluators can do and because of the mischief that poorly trained evaluators can cause. There is broad agreement on the importance of producing evaluators who are methodologically versatile and organizationally savvy. To accomplish these educational ends it is necessary to supplement classroom based instruction with experiential approaches. This paper describes a theoretical model of an effective experiential educational program for training evaluators, assesses two operating programs against the requirements of the model, and reports the results of a quasi-experimental evaluation of the effectiveness of the programs in producing important educational outcomes.     

Developing Culturally Responsive Supervision Through Yarn Up Time and the CASE Supervision Model

ABSTRACT

This article explores a culturally focused supervision training program supporting Aboriginal and non-Aboriginal supervisors to provide cross-cultural supervision for Aboriginal staff within mainstream and Aboriginal community and human service organisations. The key features of the training program, Yarn Up Time and the CASE supervision model are a culturally respectful approach to cross-cultural professional supervision practice with the purpose of contributing to the development of culturally responsive supervision with Aboriginal staff and their clients. The article concludes with feedback from participants who attended the training and supports the future advancement of cross-cultural supervision practice and models in Australia.

IMPLICATIONS

• There continues to be a need for more culturally responsive supervision for Aboriginal staff and for non-Aboriginal social work practice with Aboriginal people.

• Yarn Up Time and the CASE model is a culturally appropriate approach for supervising Aboriginal staff and non-Aboriginal social workers’ supervision of social workers’ practice with Aboriginal communities.

• Social work supervisors need to be able to use an effective cultural supervision model to support Aboriginal staff in Yarn Up Time.

     

Evaluating from the outside: conducting cross-cultural evaluation research on an American Indian reservation

There is limited guidance for conducting competent and responsive cross-cultural evaluation research with American Indian communities. The authors draw on Fisher and Ball's Tribal Participatory Research Model to highlight ways in which this project is attempting to be culturally appropriate and sensitive as they partner with an American Indian community to implement and evaluate a youth-based initiative. Challenges encountered during the evaluation are shared, as well as the authors' collective responses to such challenges. Implications for future cross-cultural evaluation researchers are also discussed in light of these experiences.     

Pre- and post-planned evaluation: Which is preferable?

This paper compares the advantages and disadvantages of the “pre-planned” and “post-planned” approaches to evaluating program effectiveness. These evaluative approaches are compared along a number of dimensions which include: (a) Reliability of data and the cost of collecting it; (b) Internal validity; (c) External validity; (d) Evaluation obtrusiveness and threat; and (e) Program goal displacement and program direction. A model designed to help program managers decide when and under what conditions either of these two evaluative approaches should be employed is presented. One major theme throughout this discussion is that despite the growing interest in and use of pre-planned evaluation, the post-planned method has many advantages which often go unnoticed. This paper will help program administrators, planners and evaluators in selecting and implementing these two methods in light of their respective strengths and limitations.