Expanding evaluator competency research: Exploring competencies for program evaluation using the context of non-formal education

The overlap of competencies between general program evaluation and specific contexts or content will always be reality because evaluators may need unique competencies to answer evaluation questions for particular contexts or content areas. Limited research exists that explores the essential competencies required by professionals who use evaluation as one part of their job portfolio, which leaves unanswered questions regarding the applicability of current evaluator competency models in such settings. We used a modified three-round Delphi technique to identify evaluator competencies for non-formal educators in Cooperative Extension (CE). Our panelists identified 36 competencies in the non-formal educational programming context for CE educators that they considered important to be included in evaluation capacity building efforts. We categorized our 36 identified competencies from the Delphi study into the five competency domains proposed by the American Evaluation Association. Our findings provide information to help guide professional development among non-formal educators related to program evaluation.     

Cognitive complexity of evaluator competencies

A critical aspect of evaluator education and professional learning is to educate evaluators who know the major evaluation models and learn how to manage relationships and solve complex problems when conducting, critiquing, developing and interpreting evaluations. The American and Australian Evaluation Associations have specified desired evaluator competencies, although developing a core curriculum for evaluation still seems elusive. It is suggested that these various competencies can be considered in terms of their levels of cognitive complexity.A model of cognitive complexity is utilised to explore the tasks and thinking of evaluators, leading to an important distinction between ‘knowing that’ and ‘knowing how’ in relation to evaluation tasks. As an illustration of this posited relationship, the Australian ‘Evaluators Professional Learning Competencies’ were coded according to their cognitive complexity. Two-thirds of these competencies were classed as ‘knowing that’ or surface thinking, and one third were classified as ‘knowing how’ or deeper thinking. A taxonomy is offered as a method to understand models of learning necessary for evaluator education and training, as well as for further development of professional evaluator competencies.     

Competencies and Their Assessment

This article explores competencies and methods for their assessment in higher education and in social work’s accreditation standards. Many contemporary policy and educational accreditation efforts employ the model of competency assessment. The current emphasis on accountability in higher education, including the Council on Social Work Education’s 2008 and draft 2015 Educational Policy and Accreditation Standards, is contextualized through a short history of this movement and its purposes. A multipart definition of competencies is offered based on McClelland’s pivotal conceptualization of competency assessment. Within this competency model, several methods of assessing competencies in social work education are described and critically examined. The importance of identifying an appropriate and complete list of competencies is also addressed. Several issues regarding how to assess professional competencies are identified for future professional discussion.     

Assessing competency-based evaluation course impacts: A mixed methods case study

The contexts in which evaluators develop and apply their expertise are increasingly complex; evaluator education needs to provide robust opportunities to support and assess the progressive, lifelong development of relevant knowledge and skills. This mixed methods case study begins to address the dearth of empirical evidence assessing the impacts and learner experiences of competency-based approaches to evaluator education. A decade-in-the-making doctoral evaluation course based on the Canadian Evaluation Society’s Competencies for Canadian Evaluation Practice created an opportune study setting. We applied a systems perspective to Stufflebeam’s Context, Input, Process, and Product (CIPP) model to frame the case study analysis and presentation of the complex impacts generated by pre/post course competency self-assessments and reflections, integrated with an end-of-course focus group. Seven insights shed new light on the effective course design and implementation features for developing intended and unintended evaluator competencies. We conclude with a discussion of the theoretical, practical, and methodological implications for effective competency-based evaluator education.

• •Keywords: Case study; Competency-based approach; Evaluator education; Mixed methods research

     

Demystifying reflective practice: Using the DATA model to enhance evaluators’ professional activities

Reflective practice (RP), one of six essential competency domains in evaluation identified by Stevahn, King, Ghere, and Minnema (2005), refers to thinking critically about one's evaluation practice, alone or with other people, and using critical insights to improve one's practice. Currently, evaluators have minimal guidance in navigating this essential professional competency, professed to be a necessary part of their practice. This article focuses on how RP can serve as a tool for evaluators through the use of the “DATA” integrated RP framework, developed by Peters, 1991, Peters, 2009. DATA is an acronym with each letter standing for a different step in the process of reflective practice. The “D” step of the acronym focuses on (D)escribing what is or has been happening in practice. The “A” step refers to (A)nalyzing the current state of practice—why is this happening the way it is? The “T” concentrates on a practice-oriented form of (T)heorizing, which comes from analysis and serves as a basis for the resulting (A)ct. The last “A” focuses on the specifics of an action plan to change one's evaluation practice in light of the practical theory developed through theorizing. This paper describes the DATA model and introduces the application of the framework in a practice context.     

Being practical, being safe: doing evaluations in contested spaces

This final article summarizes and synthesizes the full special edition. This volume questioned evaluation as philosophy, enterprise, and practice from the point of view of doing evaluation under conditions of social violence, disruption, and division. In this final article, we clarify the issues and problems which challenge the professional practice of evaluation and propose guiding questions for evaluators designing and doing evaluation in worlds like these. We attend to the consequences for evaluation and evaluator of choosing to inhabit these violent worlds as an evaluator.     

Further considerations of evaluation competencies in Taiwan

A list of evaluator competencies (Stevahn, King, Ghere, & Minnema, 2005) was adapted to fit the Taiwanese context by Lee, Altschuld, & Lee (2012). It was studied as to how it generalized to a large sample in Taiwan. Likert and Fuzzy surveys with needs assessment formats (importance and competence) were mailed via random assignment to two groups of participants. The questions for the study were: do the modified competencies relate country-wide to Taiwan, did the investigation uncover training needs for evaluators, and were there convergent rating patterns across the two forms of the instrument? The results supported a fit of the modified competencies to the context and convergent validity was observed but strong competency needs were not apparent. Reasons for the findings and implications for future research are discussed.     

Adoption Competent Clinical Practice: Defining Its Meaning and Development

The need for adoption competent mental health services has been well documented. However, the term “adoption-competent” has lacked a standardized, broadly accepted definition. This article reports findings from two related studies. The first examines how adoption competencies are demonstrated in practice by clinicians participating in an evidence-informed adoption competency training program. The second is an online survey designed to determine whether members of adoption kinship networks agree with a definition of an adoption competent mental health professional developed by experts. Both studies contribute to our understanding of what constitutes “adoption competent” clinical practice.     

Case study of an evaluation coaching model: Exploring the role of the evaluator

This study examined the role of the external evaluator as a coach. More specifically, using an evaluative inquiry framework (Preskill and Torres, 1999a, Preskill and Torres, 1999b), it explored the types of coaching that an evaluator employed to promote individual, team and organizational learning. The study demonstrated that evaluation coaching provided a viable means for an organization with a limited budget to conduct evaluations through support of a coach. It also demonstrated how the coaching processes supported the development of evaluation capacity within the organization. By examining coaching models outside of the field of evaluation, this study identified two forms of coaching — results coaching and developmental coaching — that promoted evaluation capacity building and have not been previously discussed in the evaluation literature.     

The whole elephant: Defining evaluation

Definitions help us understand the characteristics of an object or phenomenon and are a necessary precursor to understanding what a good version of it looks like. Evaluation as a field has resisted a common definition (Crane, 1988; Morell & Flaherty, 1978; M. F. Smith, 1999), which has implications for marketing, training, practice, and quality assurance. In this position paper, I describe the benefits and challenges of not having a clear, agreed-upon definition, then propose and explore the implications of two definitions for the evaluation profession based on values and valuation as the core of evaluation practice. The purpose is to describe a possible way forward through definition that would increase our professional profile, power, and contribution to social justice. The paper concludes with implications for evaluator competencies and evaluation education and questions for further research.     

The Internet's impact on policy evaluation: information compression and credibility

As with all media, the Internet structures and frames information, rewarding some information search and decision behaviors while punishing others and, thereby, strongly influences evaluation research results and possibilities. Now that the Internet is for many evaluators the information medium of choice, the impacts of the medium on evaluation deserve careful attention. The objective of this article is to lay groundwork for a theory of the impact of the Internet on evaluation and policy analysis. Questions addressed include the following: (a) What is the impact of the Internet on the evaluator's professional role, work norms, and work habits? (b) Does the use of the Internet affect who is an evaluator or the meaning of professional evaluation? and (c) How does evaluation via Internet affect the technical quality and credibility of evaluation? A key thesis is that the Internet compresses information in the sense that it is not always easy to distinguish among information resources and, especially, the authority of the information provider and the nature of the knowledge warrant. On one hand, the Internet's information compression seems to hold potential for the democratization of evaluation. On the other hand, the diminished ability to make quality distinctions about evaluation-relevant information may undercut the legitimacy of evaluation.     

Private thoughts and public actions:: A commentary on articles dealing with ethical issues in evaluation

This commentary represents a personal reaction to seven articles on ethical issues in evaluation. The author of the commentary begins with two assumptions: (1) Professional evaluators might not have the vocabulary and constructs to analyze their thoughts and actions in ethical terms, and (2) professional evaluators have often confronted troubling situations that probably contain ethical issues embedded within them. Consequently, she queries the seven articles individually and collectively in terms of the following questions: 1. What is the range of ethical problems I might encounter as an evaluator; and how do I recognize the ethical characteristics of professional dilemmas? 2. How do I describe troubling evaluation problems I have encountered that seem to have an ethical component? 3. What are some approaches I have used or might use in thinking about a proper course of action? 4. How do I know if, in a given situation, I have behaved ethically? 5. Where do we go from here?     

Core competencies in advanced training: what supervisors say about graduate training

In an attempt to identify needed mental health skills, many professional organizations have or are in the process of establishing core competency standards for their professions. The AAMFT identified 128 core competencies for the independent practice of MFT. The aim of this study was to learn the opinions of AAMFT Approved Supervisors as to how well prepared postgraduate trainees are when compared to the core competencies. One hundred thirty-five AAMFT Approved Supervisors provided their perspectives on (a) which competencies are most commonly learned in MFT graduate programs, (b) how well the graduates have mastered these competencies, and (c) the level to which the supervisors need the competencies to be mastered prior to entering advanced training. Results suggest that a gap exists between the level of mastery that the postgraduate trainees exhibit and the level desired by supervisors. Implications are suggested for closing this gap.     

Acquiring Start‐Up Funding for New Arts Organizations

This article addresses the challenge of assembling financial support for emergent arts organizations. In addition to organizational logistics and the demands of making art, such groups must garner financing from various sources, including governments/foundations, private sector companies, individual donations, and earned revenues. Drawing on the relevant literature, as well as insights from a case study of a recently formed arts organization—the Grand River Jazz Society (GRJS)—a framework is created proposing the key success factors and corresponding core competencies associated with each funding source. Governments and foundations require addressing mission as a key success factor, and navigating bureaucracy is an essential core competency in which early stage arts organizations need to be successful. Businesses, in contrast, look for action as a key success factor, and it is managerial core competencies that lead to sponsorship. Individuals respond to vision and require relational core competencies for successful involvement. Finally, the key success factor for earned revenue is a connection with audience members, necessitating core competencies with a services mindset. The article interweaves the conceptual frame with examples from the GRJS case study and also addresses the challenge of operating with a portfolio of income sources. Finally, results from a survey of fifty‐two arts start‐ups in Ontario, Canada, provides context from a wider group of such organizations with respect to their early stage funding.     

Essential competencies for program evaluators in a diverse cultural context

Essential evaluator competencies as identified by Stevahn, King, Ghere, and Minnema (2005) were studied in regard to how well they generalize to an Asian (Taiwan) context. A fuzzy Delphi survey with two iterations was used to collect data from 12 experts. While most competencies fit Taiwan, there were a number of unique ones. A complete set of results is provided along with the implications of the findings and what they might mean for evaluation in Taiwan particularly in relationship to the professionalization of evaluation.     

Essential competencies for program evaluators in a diverse cultural context

Essential evaluator competencies as identified by Stevahn, King, Ghere, and Minnema (2005) were studied in regard to how well they generalize to an Asian (Taiwan) context. A fuzzy Delphi survey with two iterations was used to collect data from 12 experts. While most competencies fit Taiwan, there were a number of unique ones. A complete set of results is provided along with the implications of the findings and what they might mean for evaluation in Taiwan particularly in relationship to the professionalization of evaluation.     

Should evaluators judge whether services are appropriate?

If an Aid for Dependent Children (AFDC) mother receives a cash reward for helping police apprehend a criminal, she automatically loses part of her children's benefits. Is this appropriate? More central to this article, does an evaluator of the AFDC program have the right to judge whether this is appropriate? Although decision makers' abilities to rectify such situations are increasingly constrained, they clearly need to know about such seemingly inappropriate aspects of services. But should evaluators be a source for this information? Yes — even though there are undeniable risks involved. This article discusses these issues and offers 18 suggestions for gathering and presenting information on appropriateness.     

Mr. Franks refuses surgery Cognition and values in competency determination in complex cases

The evaluation of medical decision-making competency is challenging and subject to low reliability for many reasons including petentially different life perspectives and cognitive processes between evaluators and patients, but especially because of the difficulties in defining and determining “competency” in complex cases. In this article, the case of an elderly man who is refusing surgery is presented to illustrate such difficulties. In these situations, the evaluator can assess both specific decisional tasks and personal values relevant to health care. Instruments designed to assess such specific decision-making abilities and values have benefits and limitations.     

Identifying competencies for communication practice: A needs assessment for curriculum development and selection in New Zealand

A hierarchical needs assessment model used to identify the competencies required by communication management practitioners in the New Zealand workplace generated three key findings. First, there was evidence of a perspective of CM as a distinct discipline founded on an agreed body of knowledge, skills and values. Second, despite an identified lack of consensus about required CM competencies in the literature, we found high levels of agreement between practitioners and academics. Third, there was, however, a significant difference in views between these two groups about the importance of the competency managing client reputation. As this was rated as one of the two most important competencies by respondents, there are implications for changing the focus of tertiary curricula to ensure graduates are appropriately prepared for the reality of workplace practice.     

Where do we stand? Recent AEA member views on professionalization

The professionalization of evaluation continues to be debated at numerous conferences in the U.S. and abroad. At this time, AEA member views on the potential benefits and negative side effects of professionalization are essential as the discussion evolves. This study provides recent views on major topics in professionalization, including potential benefits, negative side effects, processes, competencies, and procedures. Results from in-depth interviews and an online survey demonstrate that AEA members view potential benefits of professionalization to be stakeholder trust, evaluator reputation and identity, while concerns about a potential negative side effect known as the “narrowing effect” (i.e., some evaluators will be alienated based on their background, competencies, etc.) were expressed by participants. These recent findings can inform the ongoing discussion of professionalization, and suggest new directions for future research on evaluation.