Aligning collaborative and culturally responsive evaluation approaches

The authors, three African-American women trained as collaborative evaluators, offer a comparative analysis of collaborative evaluation (O'Sullivan, 2004) and culturally responsive evaluation approaches (Frierson, Hood, & Hughes, 2002; Kirkhart & Hopson, 2010). Collaborative evaluation techniques immerse evaluators in the cultural milieu of the program, systematically engage stakeholders and integrate their program expertise throughout the evaluation, build evaluation capacity, and facilitate the co-creation of a more complex understanding of programs. However, the authors note that without explicit attention to considerations raised in culturally responsive evaluation approaches (for example, issues of race, power, and privilege), the voices and concerns of marginalized and underserved populations may be acknowledged, but not explicitly or adequately addressed. The intentional application of collaborative evaluation techniques coupled with a culturally responsive stance enhances the responsiveness, validity and utility of evaluations, as well as the cultural competence of evaluators.     

Assumptions at the philosophical and programmatic levels in evaluation

Stakeholders and evaluators hold a variety of levels of assumptions at the philosophical, methodological, and programmatic levels. The use of a transformative philosophical framework is presented as a way for evaluators to become more aware of the implications of various assumptions made by themselves and program stakeholders. The argument is examined and demonstrated that evaluators who are aware of the assumptions that underlie their evaluation choices are able to provide useful support for stakeholders in the examination of the assumptions they hold with regard to the nature of the problem being addressed, the program designed to solve the problem, and the approach to evaluation that is appropriate in that context. Such an informed approach has the potential for development of more appropriate and culturally responsive programs being implemented in ways that lead to the desired impacts, as well as to lead to evaluation approaches that support effective solutions to intransigent social problems. These arguments are illustrated through examples of evaluations from multiple sectors; additional challenges are also identified.     

Collaborative evaluation within a framework of stakeholder-oriented evaluation approaches

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation planning and implementation. Unlike "distanced" evaluation approaches, which reject stakeholder participation as evaluation team members, Collaborative Evaluation assumes that active, on-going engagement between evaluators and program staff, result in stronger evaluation designs, enhanced data collection and analysis, and results that stakeholder understand and use. Among similar "participant-oriented" evaluation approaches (Fitzpatrick, Sanders, & Worthen, 2011), Collaborative Evaluation distinguishes itself in that it uses a sliding scale for levels of collaboration. This means that different program evaluations will experience different levels of collaborative activity. The sliding scale is applied as the evaluator considers each program's evaluation needs, readiness, and resources. While Collaborative Evaluation is a term widely used in evaluation, its meaning varies considerably. Often used interchangeably with participatory and/or empowerment evaluation, the terms can be used to mean different things, which can be confusing. The articles use a comparative Collaborative Evaluation Framework to highlight how from a theoretical perspective, Collaborative Evaluation distinguishes itself from the other participatory evaluation approaches.     

Building relationships and facilitating immigrant community integration: An evaluation of a Cultural Navigator Program

Despite the long history of immigration in the United States, communities around the country struggle to integrate newcomers into the economic, cultural, and political spheres of society. Utilizing results from the program evaluation of one public library's Cultural Navigator Program, the authors illustrate how communities and public institutions can promote integration and relationship-building between newly arrived immigrants and long-time residents. Existing social networks within receiving communities, conceptualized in this article as social capital, were leveraged to build capacity among newly arrived immigrants and foster inclusivity and integration at the community level. As a place of intervention, public libraries are suggested as a safe and shared space where community integration can be fostered. Insights derived from the evaluation inform a discussion on engaging approaches to immigrant integration. Lessons learned and recommendations for program evaluators and administrators are provided.     

Creating Culturally Responsive Family Therapy Models and Research: Introducing the Use of Responsive Evaluation as a Method

Models of marriage and family therapy (MFT) typically reflect Western values and norms, and although cultural adaptations are made, many models/frameworks continue to be inappropriate or inadequate for use with non‐Western cultures. Worldwide, therapists are examining ways of using MFT models in a culturally sensitive manner, especially when working with clients who are seen as having minority status or perceived as “other” by the dominant group. This essay suggests the use of responsive evaluation as a theoretically consistent methodology for creating and evaluating culturally responsive therapies. This approach rigorously evaluates each unique client/therapist context, culture, power, needs, and beliefs. We describe responsive evaluation and discuss how each component addresses the research needs of examining culturally responsive family therapies. A case illustration is offered delineating the process of conducting culturally responsive therapy with a Cambodian sample using solution‐focused and narrative therapy.     

Advances in collaborative evaluation

Collaborative evaluation is an approach that offers, among others, many advantages in terms of access to information, quality of information gathered, opportunities for creative problem-solving, and receptivity to findings. In the last decade, collaborative evaluation has grown in popularity along with similar participatory, empowerment, and utilization-focused evaluation approaches. Increasingly, it has brought together evaluators and stakeholders from different sectors, disciplines, and cultures to exchange knowledge on how collaboration can be used as a strategic tool for fostering and strengthening evaluation practice. O'Sullivan (2004) and Rodríguez-Campos (2005, 2008) brought forward collaborative evaluation conceptual frameworks that distinguished it from other approaches. This article updates those conceptual frameworks and discusses key collaborative evaluation advances in order to further clarify and facilitate engagement in sound practice.     

Advances in collaborative evaluation

Collaborative evaluation is an approach that offers, among others, many advantages in terms of access to information, quality of information gathered, opportunities for creative problem-solving, and receptivity to findings. In the last decade, collaborative evaluation has grown in popularity along with similar participatory, empowerment, and utilization-focused evaluation approaches. Increasingly, it has brought together evaluators and stakeholders from different sectors, disciplines, and cultures to exchange knowledge on how collaboration can be used as a strategic tool for fostering and strengthening evaluation practice. O'Sullivan (2004) and Rodríguez-Campos, 2005, Rodríguez-Campos, 2008 brought forward collaborative evaluation conceptual frameworks that distinguished it from other approaches. This article updates those conceptual frameworks and discusses key collaborative evaluation advances in order to further clarify and facilitate engagement in sound practice.     

Culturally Responsive Teaching: What do selected United States elementary school teachers think?

In this paper culturally responsive teaching is viewed as one way of implementing the concept of multicultural education in schools. In recent years, scholars have devoted considerable attention to the importance of aligning classroom experiences with students' home culture as a way of enhancing social, academic and cultural enrichment of all children. This study was, therefore, designed to identify perceptions of teachers toward the importance of culturally responsive teaching in elementary schools in the United States. The responses of 33 teachers were analyzed by using both qualitative and quantitative techniques. The findings of this study, along with a summary and implications for future studies, are presented in this paper. The data analysis clearly indicated that over 96% of the respondents considered culturally responsive teaching to be an important part of working with culturally diverse students. It was significant to note that the way teachers reported their perception toward the importance of culturally responsive teaching matched the manner with which they responded to the characteristics of culturally relevant teaching.     

Organizational characteristics that foster early adoption of cultural and linguistic competence in outpatient substance abuse treatment in the United States

Recent years have seen an increased interest in developing culturally and linguistically responsive systems of care in substance abuse treatment in the United States. This study examines the extent to which external and internal organizational pressures contributed to the degree of adoption of culturally and linguistically responsive practices in the nation's outpatient substance abuse treatment system early in the period of development of this system of care. Findings show that a higher degree of adoption of culturally competent practices was most likely in treatment programs with high dependence on external funding and regulation. Internally, programs with a larger number of professionals were associated with the lowest degree of adoption, while managers’ cultural sensitivity contributed significantly to a high degree of adoption of these responsive practices. Considering the passage of recent legislation enforcing the use of cultural and linguistic competence in health care, implications of these baseline findings on early adoption patterns are discussed for future research and health care policy evaluation.     

“Thank You for Including Us!” – Introducing a Community‐Based Collaborative Approach to Translating Clinic Materials

When working with clients in another language, having culturally relevant and properly translated clinic materials is vital to effective clinical practice. Not having them presents a barrier for bilingual therapists and their clients. This paper reviews common translation methods and introduces a multi‐level, community‐based approach for translating clinic materials for non‐English speaking clients. Informed by decolonizing practices, this five‐tier method includes members from the target community as cultural brokers to verify the accuracy and nuances of language for their cultural group, as well as a constant comparative method to ensure methodological rigor in the process of inclusion. We present community member's feedback and recommendations, as well as discuss advantages and challenges of using this collaborative culturally responsive translation method.     

Avoiding evaluation cooptation. Lessons from a renal dialysis evaluation.

An evaluation of the impact of terminating contracts to transport low-income patients for renal dialysis demonstrates how evaluators may become apologists for management and conduct studies too narrow for effective decision making. The evaluation found that patients continued to receive treatment; their death rates had not increased. Changes in their economic and emotional status were not studied; consequently, the evaluators erroneously concluded that the clients had not been seriously harmed. The article reviews the process the evaluation used. The authors conclude that to avoid cooptation, evaluators need independent information available through conducting and analyzing pilot studies and meeting with program constituents. To avoid overly narrow studies, the next level of management should participate in their design. To use the information effectively, evaluators need to see themselves as advisors rather than technicians.     

Techniques for the analysis of geographic data in evaluation

Although geographic relationships are often an important aspect of a program or system under evaluation, these relationships have frequently been ignored by evaluators. Standard techniques do exist, however, for the analysis of geographic data. This paper presents three such techniques, Geocode Analysis, Trend Surface Analysis, and Social Area Analysis, and discusses their use in evaluation. With these techniques, evaluators can more adequately deal with the questions of geographic relationships which arise in many studies.     

Building effective working relationships across culturally and ethnically diverse communities

Amherst H. Wilder Foundation's Social Adjustment Program for Southeast Asians is implementing two collaborative, best practice, mental health and substance abuse prevention service models in Minnesota. It faced several issues in effectively bridging multiple cultural groups, including building a diverse collaborative team, involving families and youth, reconciling cultural variation in meeting styles, and making best practice models culturally appropriate. Researchers and program staff used multiple strategies to address these challenges and build successful partnerships. Through shared goals, flexibility, and a willingness to explore and address challenges, collaboratives can promote stronger relationships across cultural communities and improve their service delivery systems.     

Agent-based modeling as a tool for program design and evaluation

Recently, systems thinking and systems science approaches have gained popularity in the field of evaluation; however, there has been relatively little exploration of how evaluators could use quantitative tools to assist in the implementation of systems approaches therein. The purpose of this paper is to explore potential uses of one such quantitative tool, agent-based modeling, in evaluation practice. To this end, we define agent-based modeling and offer potential uses for it in typical evaluation activities, including: engaging stakeholders, selecting an intervention, modeling program theory, setting performance targets, and interpreting evaluation results. We provide demonstrative examples from published agent-based modeling efforts both inside and outside the field of evaluation for each of the evaluative activities discussed. We further describe potential pitfalls of this tool and offer cautions for evaluators who may chose to implement it in their practice. Finally, the article concludes with a discussion of the future of agent-based modeling in evaluation practice and a call for more formal exploration of this tool as well as other approaches to simulation modeling in the field.     

The value of experiential approaches to evaluation training

The training of evaluators is an important task—important because of the good that well trained evaluators can do and because of the mischief that poorly trained evaluators can cause. There is broad agreement on the importance of producing evaluators who are methodologically versatile and organizationally savvy. To accomplish these educational ends it is necessary to supplement classroom based instruction with experiential approaches. This paper describes a theoretical model of an effective experiential educational program for training evaluators, assesses two operating programs against the requirements of the model, and reports the results of a quasi-experimental evaluation of the effectiveness of the programs in producing important educational outcomes.     

Relational responsibilities in responsive evaluation

This article explores how we can enhance our understanding of the moral responsibilities in daily, plural practices of responsive evaluation. It introduces an interpretive framework for understanding the moral aspects of evaluation practice. The framework supports responsive evaluators to better understand and handle their moral responsibilities. A case is introduced to illustrate our argument.Responsive evaluation contributes to the design and implementation of policy by working with stakeholders and coordinating the evaluation process as a relationally responsible practice. Responsive evaluation entails a democratic process in which the evaluator fosters and enters a partnership with stakeholders. The responsibilities of an evaluator generally involve issues such as ‘confidentiality’, ‘accountability’ and ‘privacy’. The responsive evaluator has specific responsibilities, for example to include stakeholders and vulnerable groups and to foster an ongoing dialogue. In addition, responsive evaluation involves a relational responsibility, which becomes present in daily situations in which stakeholders express expectations and voice demands. In our everyday work as evaluators, it is difficult to respond to all these demands at the same time. In addition, this article demonstrates that novice evaluators experience challenges concerning over- and underidenfitication with stakeholders. Guidelines and quality criteria on how to act are helpful, but need interpretation and application to the unique situation at hand.     

Evaluating from the outside: conducting cross-cultural evaluation research on an American Indian reservation

There is limited guidance for conducting competent and responsive cross-cultural evaluation research with American Indian communities. The authors draw on Fisher and Ball's Tribal Participatory Research Model to highlight ways in which this project is attempting to be culturally appropriate and sensitive as they partner with an American Indian community to implement and evaluate a youth-based initiative. Challenges encountered during the evaluation are shared, as well as the authors' collective responses to such challenges. Implications for future cross-cultural evaluation researchers are also discussed in light of these experiences.     

The results framework--an innovative tool for program planning and evaluation.

This article constitutes a case study of the development and implementation of the "results framework," an innovative planning and evaluation tool that is rapidly becoming a standard requirement for United States Agency for International Development (USAID) projects. The framework is used in a USAID-funded regional initiative for HIV/AIDS prevention in Central America. This new program evaluation and monitoring tool provides many advantages over traditional evaluation approaches that use outside consultants to provide midterm and end-of-project evaluations. The results-framework process, which spans the life of the project, provides an opportunity for program staff, donors, partners, and evaluators to work as a team to collect and use rich, longitudinal data for project planning, implementation, and evaluation purposes.     

Collaborative evaluation of a high school prevention curriculum: How methods of collaborative evaluation enhanced a randomized control trial to inform program improvement

Blending high-quality and rigorous research with pure evaluation practice can often be best accomplished through thoughtful collaboration. The evaluation of a high school drug prevention program (All Stars Senior) is an example of how perceived competing purposes and methodologies can coexist to investigate formative and summative outcome variables that can be used for program improvement. Throughout this project there were many examples of client learning from evaluator and evaluator learning from client. This article presents convincing evidence that collaborative evaluation can improve the design, implementation, and findings of the randomized control trial. Throughout this paper, we discuss many examples of good science, good evaluation, and other practical benefits of practicing collaborative evaluation. Ultimately, the authors created the term pre-formative evaluation to describe the period prior to data collection and before program implementation, when collaborative evaluation can inform program improvement.     

Assumption-aware tools and agency; an interrogation of the primary artifacts of the program evaluation and design profession in working with complex evaluands and complex contexts

Like artisans in a professional guild, we evaluators create tools to suit our ever evolving practice. The tools we use as evaluators are the primary artifacts of our profession, reflect our practice and embody an amalgamation of paradigms and assumptions. With the increasing shifts in evaluation purposes from judging program worth to understanding how programs work, the evaluator’s role is changing to that of facilitating stakeholders in a learning process. This involves clarifying purposes and choices, as well as unearthing critical assumptions. In such a role, evaluators become major tool-users and begin to innovate with small refinements or produce completely new tools to fit a specific challenge or context.We interrogate the form and function of 12 tools used by evaluators when working with complex evaluands and complex contexts. The form is described in terms of traditional qualitative techniques and particular characteristics of the elements, use and presentation of each tool. Then the function of each tool is analyzed with respect to articulating assumptions and affecting the agency of evaluators and stakeholders in complex contexts.