Assessing competency-based evaluation course impacts: A mixed methods case study

The contexts in which evaluators develop and apply their expertise are increasingly complex; evaluator education needs to provide robust opportunities to support and assess the progressive, lifelong development of relevant knowledge and skills. This mixed methods case study begins to address the dearth of empirical evidence assessing the impacts and learner experiences of competency-based approaches to evaluator education. A decade-in-the-making doctoral evaluation course based on the Canadian Evaluation Society’s Competencies for Canadian Evaluation Practice created an opportune study setting. We applied a systems perspective to Stufflebeam’s Context, Input, Process, and Product (CIPP) model to frame the case study analysis and presentation of the complex impacts generated by pre/post course competency self-assessments and reflections, integrated with an end-of-course focus group. Seven insights shed new light on the effective course design and implementation features for developing intended and unintended evaluator competencies. We conclude with a discussion of the theoretical, practical, and methodological implications for effective competency-based evaluator education.

• •Keywords: Case study; Competency-based approach; Evaluator education; Mixed methods research

     

One Organization's Experience in a Case Management/EAP Program:

Historically, Employee Assistance Programs (EAPs) have been viewed as an employee benefit even though there is documented evidence of their cost-effectiveness for the company. This article presents an integration of the Employee Assistance Program and Case Management as a way of providing a proactive method of containing health care costs. Such an integration will expand the focus of the Employee Assistance Program, strengthen our role as an employee advocate, and provide a method of demonstrating to corporations that EAPs do contribute to the "bottom line" profit margin of the company. This article outlines the components of an effective, comprehensive Employee Assistance Program and discusses the process of integrating a Case Management system that emphasizes local demographic control within the EAP.     

Expanding evaluator competency research: Exploring competencies for program evaluation using the context of non-formal education

The overlap of competencies between general program evaluation and specific contexts or content will always be reality because evaluators may need unique competencies to answer evaluation questions for particular contexts or content areas. Limited research exists that explores the essential competencies required by professionals who use evaluation as one part of their job portfolio, which leaves unanswered questions regarding the applicability of current evaluator competency models in such settings. We used a modified three-round Delphi technique to identify evaluator competencies for non-formal educators in Cooperative Extension (CE). Our panelists identified 36 competencies in the non-formal educational programming context for CE educators that they considered important to be included in evaluation capacity building efforts. We categorized our 36 identified competencies from the Delphi study into the five competency domains proposed by the American Evaluation Association. Our findings provide information to help guide professional development among non-formal educators related to program evaluation.     

The Tellado case study exercise: A metacritique

This paper provides an overview and critique of the papers in the Tellado Case Study Exercise. Two main themes are found in the critiques: the belief that Tellado lacked a clear audience for his evaluation and that he lacked an appropriate change orientation that would insure utilization. The sources of these criticisms are discussed from the perspective of evaluation as a linkage activity. The use of this perspective leads to the recognition that all parties to the case study may have focused on the wrong stakeholders in the utilization process. Thus, a major problem with utilization-focused and stakeholder based approaches to evaluation is revealed. Possible ways to avoid this problem are discussed. The paper concludes with a request for additional case studies to be contributed to the journal.     

The theory of change of the evaluation support program: Enhancing the role of community organizations in providing an ecology of care for neurological disorders

This paper discusses the Ontario Brain Institute’s theory of change for the Evaluation Support Program, a program designed to enhance the role of community organizations in providing care and services for people living with a brain disorder. This is done by helping community organizations build evaluation capacity and foster the use of evidence to inform their activities and services. Helping organizations to build capacities to track the ‘key ingredients’ of their successes will help ensure that successes are replicated and services can be improved to maximize the benefit that people receive from them. This paper describes the hypothesized outcomes and early impacts of the Evaluation Support Program, as well as how the program will contribute to the field of evaluation capacity building.     

Revisiting a theory of negotiation: the utility of Markiewicz (2005) proposed six principles

People invited to participate in an evaluation process will inevitably come from a variety of personal backgrounds and hold different views based on their own lived experience. However, evaluators are in a privileged position because they have access to information from a wide range of sources and can play an important role in helping stakeholders to hear and appreciate one another's opinions and ideas. Indeed, in some cases a difference in perspective can be utilised by an evaluator to engage key stakeholders in fruitful discussion that can add value to the evaluation outcome. In other instances the evaluator finds that the task of facilitating positive interaction between multiple stakeholders is just ‘an uphill battle’ and so conflict, rather than consensus, occurs as the evaluation findings emerge and are debated.As noted by Owen [(2006) Program evaluation: Forms and approaches (3rd ed.). St. Leonards, NSW: Allen & Unwin] and other eminent evaluators before him [Fetterman, D. M. (1996). Empowerment evaluation: An introduction to theory and practice. In D. M. Fetterman, S. J. Kaftarian, & A. Wandersman (Eds.), Empowerment evaluation: Knowledge and tools for self-assessment and accountability (pp. 3–46). Thousand Oaks, CA: Sage Publications; Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage Publications; Stake, R. A. (1983). Stakeholder influence in the evaluation of cities-in-schools. New Directions for Program Evaluation, 17, 15–30], conflict in an evaluation process is not unexpected. The challenge is for evaluators to facilitate dialogue between people who hold strongly opposing views, with the aim of helping them to achieve a common understanding of the best way forward. However, this does not imply that consensus will be reached [Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage]. What is essential is that the evaluator assists the various stakeholders to recognise and accept their differences and be willing to move on.But the problem is that evaluators are not necessarily equipped with the technical or personal skills required for effective negotiation. In addition, the time and effort that are required to undertake this mediating role are often not sufficiently understood by those who commission a review. With such issues in mind Markiewicz, A. [(2005). A balancing act: Resolving multiple stakeholder interests in program evaluation. Evaluation Journal of Australasia, 4(1–2), 13–21] has proposed six principles upon which to build a case for negotiation to be integrated into the evaluation process. This paper critiques each of these principles in the context of an evaluation undertaken of a youth program. In doing so it challenges the view that stakeholder consensus is always possible if program improvement is to be achieved. This has led to some refinement and further extension of the proposed theory of negotiation that is seen to be instrumental to the role of an evaluator.     

Lessons learned using a values-engaged approach to attend to culture,diversity, and equity in a STEM program evaluation

Evaluation must attend meaningfully and respectfully to issues of culture, race, diversity, power, and equity. This attention is especially critical within the evaluation of science, technology, engineering, and mathematics (STEM) educational programming, which has an explicit agenda of broadening participation. The purpose of this article is to report lessons learned from the implementation of a values-engaged, educative (Greene et al., 2006) evaluation within a multi-year STEM education program setting. This meta-evaluation employed a case study design using data from evaluator weekly systematic reflections, review of evaluation and program artifacts, stakeholder interviews, and peer review and assessment. The main findings from this study are (a) explicit attention to culture, diversity, and equity was initially challenged by organizational culture and under-developed evaluator–stakeholder professional relationship and (b) evidence of successful engagement of culture, diversity, and equity emerged in formal evaluation criteria and documents, and informal dialogue and discussion with stakeholders. The paper concludes with lessons learned and implications for practice.     

Collaborative evaluation within a framework of stakeholder-oriented evaluation approaches

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation planning and implementation. Unlike "distanced" evaluation approaches, which reject stakeholder participation as evaluation team members, Collaborative Evaluation assumes that active, on-going engagement between evaluators and program staff, result in stronger evaluation designs, enhanced data collection and analysis, and results that stakeholder understand and use. Among similar "participant-oriented" evaluation approaches (Fitzpatrick, Sanders, & Worthen, 2011), Collaborative Evaluation distinguishes itself in that it uses a sliding scale for levels of collaboration. This means that different program evaluations will experience different levels of collaborative activity. The sliding scale is applied as the evaluator considers each program's evaluation needs, readiness, and resources. While Collaborative Evaluation is a term widely used in evaluation, its meaning varies considerably. Often used interchangeably with participatory and/or empowerment evaluation, the terms can be used to mean different things, which can be confusing. The articles use a comparative Collaborative Evaluation Framework to highlight how from a theoretical perspective, Collaborative Evaluation distinguishes itself from the other participatory evaluation approaches.     

Experiments in evaluation capacity building: Enhancing brain disorders research impact in Ontario

This paper is the introductory paper on a forum on evaluation capacity building for enhancing impacts of research on brain disorders. It describes challenges and opportunities of building evaluation capacity among community-based organizations in Ontario involved in enhancing brain health and supporting people living with a brain disorder. Using an example of a capacity building program called the “Evaluation Support Program”, which is run by the Ontario Brain Institute, this forum discusses multiple themes including evaluation capacity building, evaluation culture and evaluation methodologies appropriate for evaluating complex community interventions. The goal of the Evaluation Support Program is to help community-based organizations build the capacity to demonstrate the value that they offer in order to improve, sustain, and spread their programs and activities. One of the features of this forum is that perspectives on the Evaluation Support Program are provided by multiple stakeholders, including the community-based organizations, evaluation team members involved in capacity building, thought leaders in the fields of evaluation capacity building and evaluation culture, and the funders.     

Pre- and post-planned evaluation: Which is preferable?

This paper compares the advantages and disadvantages of the “pre-planned” and “post-planned” approaches to evaluating program effectiveness. These evaluative approaches are compared along a number of dimensions which include: (a) Reliability of data and the cost of collecting it; (b) Internal validity; (c) External validity; (d) Evaluation obtrusiveness and threat; and (e) Program goal displacement and program direction. A model designed to help program managers decide when and under what conditions either of these two evaluative approaches should be employed is presented. One major theme throughout this discussion is that despite the growing interest in and use of pre-planned evaluation, the post-planned method has many advantages which often go unnoticed. This paper will help program administrators, planners and evaluators in selecting and implementing these two methods in light of their respective strengths and limitations.     

Cognitive complexity of evaluator competencies

A critical aspect of evaluator education and professional learning is to educate evaluators who know the major evaluation models and learn how to manage relationships and solve complex problems when conducting, critiquing, developing and interpreting evaluations. The American and Australian Evaluation Associations have specified desired evaluator competencies, although developing a core curriculum for evaluation still seems elusive. It is suggested that these various competencies can be considered in terms of their levels of cognitive complexity.A model of cognitive complexity is utilised to explore the tasks and thinking of evaluators, leading to an important distinction between ‘knowing that’ and ‘knowing how’ in relation to evaluation tasks. As an illustration of this posited relationship, the Australian ‘Evaluators Professional Learning Competencies’ were coded according to their cognitive complexity. Two-thirds of these competencies were classed as ‘knowing that’ or surface thinking, and one third were classified as ‘knowing how’ or deeper thinking. A taxonomy is offered as a method to understand models of learning necessary for evaluator education and training, as well as for further development of professional evaluator competencies.     

An evaluator of the future

For the evaluator who is part of the management of a human service organization, future expectations will be more demanding than at present. Operating staff, managers, and funders will already be familiar with the methods and the usefulness of capable program evaluation. Improved information tools will help the evaluator meet this challenge, by keeping the organization visible and manageable. A more fundamental change will be an improved “language of accountability”. Program standards, accreditation, and review now focus on structure and process characteristics. What is emerging is a growing consensus on definitions of client problems and program objectives in relation to these problems. This consensus spurs psychometric research on measures of problems and of the attainment of program objectives. Dependable, standard measures and meaningful comparative data will make the evaluation of program effectiveness possible.     

You got a problem with that? Exploring evaluators' disagreements about ethics

A random sample of American Evaluation Association (AEA) members were surveyed for their reactions to three case scenarios--informed consent, impartial reporting, and stakeholder involvement--in which an evaluator acts in a way that could be deemed ethically problematic. Significant disagreement among respondents was found for each of the scenarios, in terms of respondents' views of whether the evaluator had behaved unethically. Respondents' explanations of their judgments support the notion that general guidelines for professional behavior (such as AEA's Guiding Principles for Evaluators) can encompass sharply conflicting interpretations of how evaluators should behave in specific situations. Respondents employed in private business/consulting were less likely than those in other settings to believe that the scenarios portrayed unethical behavior by the evaluator, a finding that underscores the importance of taking contextual variables into account when analyzing evaluators' ethical perceptions. The need for increased dialogue among evaluators who represent varied perspectives on ethical issues is addressed.     

Reflections on experiential learning in evaluation capacity building with a community organization,Dancing With Parkinson’s

This paper discusses what was learned about evaluation capacity building with community organizations who deliver services to individuals with neurological disorders. Evaluation specialists engaged by the Ontario Brain Institute Evaluation Support Program were paired with community organizations, such as Dancing With Parkinson’s. Some of the learning included: relationship building is key for this model of capacity building; community organizations often have had negative experiences with evaluation and the idea that evaluations can be friendly tools in implementing meaningful programs is one key mechanism by which such an initiative can work; community organizations often need evaluation most to be able to demonstrate their value; a strength of this initiative was that the focus was not just on creating products but mostly on developing a learning process in which capacities would remain; evaluation tools and skills that organizations found useful were developing a theory of change and the concept of heterogeneous mechanisms (informed by a realist evaluation lens).     

Evaluating Dancing With Parkinson's: Reflections from the perspective of a community organization

In 2015, Dancing With Parkinson's (DWP), a Toronto-based community organization, participated in the Ontario Brain Institute's (OBI) newly launched Evaluation Support Program. This paper reflects on that experience. In particular, we identify the key lessons derived from the OBI initiative, discuss how these lessons have informed DWP practice going forward, and highlight what we consider to be the most valuable aspects of the Evaluation Support Program. While we now recognize the need to establish an evaluation culture within DWP, we find that there are significant challenges associated with both building and sustaining evaluation capacity in the context of a small community-based organization. Whereas DWP has built considerable strengths in terms of informal evaluation capacity, on its own, such capacity is insufficient to, for example, demonstrate DWP's impact to outside audiences or successfully scale up the program.     

The current state of evaluator education: A situation analysis and call to action

Education is critically important for the maintenance and growth of programme evaluation and its emergence as a profession. This position paper presents evaluator education as an initiative and provides a foundation for its evaluation. In it, the authors use Stufflebeam's Context, Input, Process, and Product (CIPP) model to frame a review of the literature and practice. We follow this analysis of the current state of evaluator education in formal settings, with discussion of its implications, and directions for future research and action. Our analysis suggests that there is much work to be done to understand the needs for evaluator education, to delineate standards for quality in both education and practice, to identify the inputs and processes most effective for addressing those needs, and to document its ultimate impacts. The paper provides a call to action for improving the quality, consistency, and integrity of this important work.     

Practice makes better? Testing a model for training program evaluators in situation awareness

Evaluation approaches should be appropriate for their contexts. Scholars and practitioners alike have widely acknowledged this view for at least four decades, and the ability to respond to context is clearly established as a core evaluator competency. Outside of evaluation, this knowledge of context, known as situation awareness, is seen as a critical feature of good decision making and as a factor that distinguishes experts from novices across a wide range of domains. Yet there are few opportunities for evaluators to explicitly build their skills in situation awareness. This study addressed that gap by examining the potential for one online training program, informed by research on deliberate practice, to accelerate evaluators’ progress towards expertise in situation awareness. Built to align with evidence on developing situation awareness skills, the EvalPractice portal combined (1) an extensive ‘case bank’ of real-world evaluation scenarios, (2) repetitive practice that allowed novice evaluators to practice interpreting these evaluation scenarios, and (3) immediate feedback on the accuracy of these efforts based on events from the real-life version of the scenario. Findings from a small EvalPractice pilot suggest that it may be possible to improve foundational situation awareness skills using deliberate practice, but that further research is required to understand strategies for building higher-level situation awareness skills in evaluation.     

Employment and the Risk of Domestic Abuse Among Low‐Income Women

This paper uses data from 2 randomized evaluations of welfare‐to‐work programs—the Minnesota Family Investment Program and the National Evaluation of Welfare‐to‐Work Strategies—to estimate the effect of employment on domestic abuse among low‐income single mothers. Unique to our analysis is the application of a 2‐stage least squares method, in which random assignment enables us to control for omitted characteristics that might otherwise confound the association between employment and domestic abuse. We find that increased maternal employment decreases subsequent reports of domestic abuse in both studies. In the Minnesota Family Investment Program—a program with an enhanced income disregard that allowed welfare mothers to keep a portion of their welfare income as earnings rose—an increase in household incomes appears to have contributed to reductions in reports of domestic abuse.