Using practitioners' theories to document program results

A form of self-assessment, termed practitioner-centered evaluation, is proposed for human services agencies. In contrast to assessment methods that may require an outside evaluator, this qualitative technique focuses on the informal theories of behavioral change that underlie a program-as-implemented. It takes into account interacting cause-and-effect relationships and relies on the ability of staff members to document success and failure in the context of their own theories. As a vehicle for identifying program elements that are effective in achieving behavioral change and other goals, practitioner-centered evaluation can be the basis of franker, more productive conversations between funders and grantees and between staff members and their governing boards.     

Assessing competency-based evaluation course impacts: A mixed methods case study

The contexts in which evaluators develop and apply their expertise are increasingly complex; evaluator education needs to provide robust opportunities to support and assess the progressive, lifelong development of relevant knowledge and skills. This mixed methods case study begins to address the dearth of empirical evidence assessing the impacts and learner experiences of competency-based approaches to evaluator education. A decade-in-the-making doctoral evaluation course based on the Canadian Evaluation Society’s Competencies for Canadian Evaluation Practice created an opportune study setting. We applied a systems perspective to Stufflebeam’s Context, Input, Process, and Product (CIPP) model to frame the case study analysis and presentation of the complex impacts generated by pre/post course competency self-assessments and reflections, integrated with an end-of-course focus group. Seven insights shed new light on the effective course design and implementation features for developing intended and unintended evaluator competencies. We conclude with a discussion of the theoretical, practical, and methodological implications for effective competency-based evaluator education.

• •Keywords: Case study; Competency-based approach; Evaluator education; Mixed methods research

     

You got a problem with that? Exploring evaluators' disagreements about ethics

A random sample of American Evaluation Association (AEA) members were surveyed for their reactions to three case scenarios--informed consent, impartial reporting, and stakeholder involvement--in which an evaluator acts in a way that could be deemed ethically problematic. Significant disagreement among respondents was found for each of the scenarios, in terms of respondents' views of whether the evaluator had behaved unethically. Respondents' explanations of their judgments support the notion that general guidelines for professional behavior (such as AEA's Guiding Principles for Evaluators) can encompass sharply conflicting interpretations of how evaluators should behave in specific situations. Respondents employed in private business/consulting were less likely than those in other settings to believe that the scenarios portrayed unethical behavior by the evaluator, a finding that underscores the importance of taking contextual variables into account when analyzing evaluators' ethical perceptions. The need for increased dialogue among evaluators who represent varied perspectives on ethical issues is addressed.     

Ethical dilemmas in evaluating programs with family court related clients

Two ethical dilemmas are discussed; breaching client confidentiality in order to intervene in the client's illegal behavior, and defining the role of evaluator as expert witness. The evaluation context concerns a combined program development and evaluation project with clients involved in family court proceedings for child abuse or neglect. Different perspectives, approaches, and experiences in resolving each dilemma are reviewed. The conclusion argues that approaches to ethical dilemmas in evaluation projects must go beyond techniques to disguise data, to include an assessment of the values and value commitments connected to the different roles an evaluator must perform.     

The Investigation Component in Forensic Mental Health Evaluations

Abstract

Part of the data gathering and analysis in any parenting time evaluation involves investigation into sources of information that provide a factual basis for measuring key factors. However, this investigative function of the child custody evaluator has not been systematically described. In this article, investigative data collection is discussed in terms of holding a discriminating and skeptical eye in reviewing sources of information. The need to conduct forensic “detective work” and factfinding is discussed. The importance of the investigative mindset for the evaluator is to increase the completeness and accuracy of measurement of key variables, which increases the validity and accuracy of predictions on child developmental outcomes for the court.     

What do we know about evaluator education? A review of peer-reviewed publications (1978–2018)

This article reviews peer-reviewed writings on evaluator education from 1978 to 2018. Despite the topic’s presumed importance, scholars to date have not extensively addressed it in peer-reviewed publications. The article first describes the methods used to select articles and the conceptual framework for their analysis. It then presents the content of 64 articles selected for review, divided into two major categories: research studies, and reflective case narratives. We further divide research studies into program directories and empirical studies; the case narratives provide information on programs, curriculum, and instruction and minimally on students/faculty. The article concludes with thoughts on next steps for research on evaluator education.     

“Paradigms lost”: Some thoughts on choosing methods in evaluation research

Patton agrees with us concerning the need for a catholic approach to method choices in evaluation. However, Patton criticizes us for setting up a straw man when we argued that quantitative and qualitative methods are not inevitably linked to different “paradigms” and that allegiance to a particular paradigm does not force the evaluator into a choice between mutually exclusive qualitative and quantitative methods. The first part of this article shows how some of the past literature creates the impression both of an inevitable method-paradigm linkage and of a forcedchoice between qualitative and quantitative paradigms. It is argued that such impressions could have pernicious consequences for evaluation practice by artificially restricting method choices. In the final section, we consider some of the difficulties of making method choices and suggest that the purposefully developed strengths of different methods should not be completely ignored.     

Competency-based program evaluation: A contingency approach

What types and levels of professional competency should an evaluator have? Should requisite competencies differ depending on the nature of an evaluation? Can a practical competency assessment approach be developed for use in educating highly qualified evaluation professionals? This article addresses these concerns by presenting a contingency model of evaluation competencies. Four broad categories of competency — policy analysis, administrative, interpersonal /personal, and technical — are identified along with three different levels or criteria of acceptable performance. Then several features of the evaluation context, including the evaluation type, the phase and the evaluator role, are individually analyzed to determine the required competencies implied by each. Based on this, a contingency model is developed that postulates requisite competency types and levels depending on a combination of contextual features. The article concludes that the model can usefully serve as a heuristic or organizing mechanism for assessing evaluation contexts and orienting the education of professional evaluators.     

Relational responsibilities in responsive evaluation

This article explores how we can enhance our understanding of the moral responsibilities in daily, plural practices of responsive evaluation. It introduces an interpretive framework for understanding the moral aspects of evaluation practice. The framework supports responsive evaluators to better understand and handle their moral responsibilities. A case is introduced to illustrate our argument.Responsive evaluation contributes to the design and implementation of policy by working with stakeholders and coordinating the evaluation process as a relationally responsible practice. Responsive evaluation entails a democratic process in which the evaluator fosters and enters a partnership with stakeholders. The responsibilities of an evaluator generally involve issues such as ‘confidentiality’, ‘accountability’ and ‘privacy’. The responsive evaluator has specific responsibilities, for example to include stakeholders and vulnerable groups and to foster an ongoing dialogue. In addition, responsive evaluation involves a relational responsibility, which becomes present in daily situations in which stakeholders express expectations and voice demands. In our everyday work as evaluators, it is difficult to respond to all these demands at the same time. In addition, this article demonstrates that novice evaluators experience challenges concerning over- and underidenfitication with stakeholders. Guidelines and quality criteria on how to act are helpful, but need interpretation and application to the unique situation at hand.     

An evaluator of the future

For the evaluator who is part of the management of a human service organization, future expectations will be more demanding than at present. Operating staff, managers, and funders will already be familiar with the methods and the usefulness of capable program evaluation. Improved information tools will help the evaluator meet this challenge, by keeping the organization visible and manageable. A more fundamental change will be an improved “language of accountability”. Program standards, accreditation, and review now focus on structure and process characteristics. What is emerging is a growing consensus on definitions of client problems and program objectives in relation to these problems. This consensus spurs psychometric research on measures of problems and of the attainment of program objectives. Dependable, standard measures and meaningful comparative data will make the evaluation of program effectiveness possible.     

Expanding evaluator competency research: Exploring competencies for program evaluation using the context of non-formal education

The overlap of competencies between general program evaluation and specific contexts or content will always be reality because evaluators may need unique competencies to answer evaluation questions for particular contexts or content areas. Limited research exists that explores the essential competencies required by professionals who use evaluation as one part of their job portfolio, which leaves unanswered questions regarding the applicability of current evaluator competency models in such settings. We used a modified three-round Delphi technique to identify evaluator competencies for non-formal educators in Cooperative Extension (CE). Our panelists identified 36 competencies in the non-formal educational programming context for CE educators that they considered important to be included in evaluation capacity building efforts. We categorized our 36 identified competencies from the Delphi study into the five competency domains proposed by the American Evaluation Association. Our findings provide information to help guide professional development among non-formal educators related to program evaluation.     

Lessons learned using a values-engaged approach to attend to culture,diversity, and equity in a STEM program evaluation

Evaluation must attend meaningfully and respectfully to issues of culture, race, diversity, power, and equity. This attention is especially critical within the evaluation of science, technology, engineering, and mathematics (STEM) educational programming, which has an explicit agenda of broadening participation. The purpose of this article is to report lessons learned from the implementation of a values-engaged, educative (Greene et al., 2006) evaluation within a multi-year STEM education program setting. This meta-evaluation employed a case study design using data from evaluator weekly systematic reflections, review of evaluation and program artifacts, stakeholder interviews, and peer review and assessment. The main findings from this study are (a) explicit attention to culture, diversity, and equity was initially challenged by organizational culture and under-developed evaluator–stakeholder professional relationship and (b) evidence of successful engagement of culture, diversity, and equity emerged in formal evaluation criteria and documents, and informal dialogue and discussion with stakeholders. The paper concludes with lessons learned and implications for practice.     

Exploring Three Functions in Child Custody Evaluation for the Relocation Case: Prediction,Investigation, and Making Recommendations for a Long-Distance Parenting Plan

SUMMARY

The psycho-legal dilemmas posed by child custody relocation cases are discussed in terms of the four decisional alternatives facing the court and evaluator. Different legal contexts for relocation are reviewed in terms of their implications for the custody evaluation. Complexities involved in the evaluator's function of making predictions for the court are presented. The need to conduct careful investigation on both risk and pragmatic factors is highlighted by case illustrations. The obstacles of crafting of long distance parenting plans that will be in the best interests of the child are presented as governed by the goal of harm mitigation.     

Assessing Family Problems: An Evaluation of Key Elements of the Children’s Review Schedule

The importance of providing consistent methods for the assessment of need for practice and research in child and family social work has been accepted for some considerable time. One feature identified by key stakeholders, such as Directors of Children’s and Social Services, is the need for rigour in the development of such instruments. This paper reports on an evaluation of the Children’s Review Schedule, an instrument developed in collaboration with one Children’s Department. It focuses on a key aspect of the assessment of need: the measurement of the nature, range and severity of family problems.     

Changing stakeholder needs and changing evaluator roles: the Central Valley Partnership of the James Irvine foundation

This case describes the changing roles of an evaluator as the program he was evaluating changed and developed, and as the needs of his client and primary intended users changed over time. The focus in this case is on the relationship between the external evaluator and the staff of a philanthropic foundation as a major initiative unfolded. The focus of the evaluation was learning and improvement, and as the evaluator developed recommendations for organizational capacity-building, he was commissioned by the foundation to implement his own recommendations, a form of developmental evaluation. The case explores the dynamics and challenges of such an arrangement and relationship. Those involved have agreed to have their real names used because this is a highly visible initiative and attempting to disguise the foundation and key players might undermine, in this instance, the authenticity of the case. The initiative that is the focus of the case is the Central Valley Partnership of The James Irvine Foundation headquartered in San Francisco.     

Analysis approaches to community evaluation

Analysis approaches to the evaluation of community interventions must be sensitive to a wide variety of analytic contaminants that may bias the statistical assessment of changes in outcome measures. These contaminants include model misspecifications related to failures to control for community-specific time trends, temporal autocorrelated errors in equations, spatial autocorrelated errors among geographic units, and other failures of unit independence otherwise indexed by estimated intraclass correlations. Although an enormous amount of progress has been made toward the solution of many of these analytic problems over the past years, the contemporary evaluator of community interventions is left with a number of unenviable design and analysis choices; choices that inevitably force an assessment of the relative threats of different sources of error to the internal and external validity of the evaluation. This article describes the choices made for the evaluation of the Community Trial Project outcome data.     

Cognitive complexity of evaluator competencies

A critical aspect of evaluator education and professional learning is to educate evaluators who know the major evaluation models and learn how to manage relationships and solve complex problems when conducting, critiquing, developing and interpreting evaluations. The American and Australian Evaluation Associations have specified desired evaluator competencies, although developing a core curriculum for evaluation still seems elusive. It is suggested that these various competencies can be considered in terms of their levels of cognitive complexity.A model of cognitive complexity is utilised to explore the tasks and thinking of evaluators, leading to an important distinction between ‘knowing that’ and ‘knowing how’ in relation to evaluation tasks. As an illustration of this posited relationship, the Australian ‘Evaluators Professional Learning Competencies’ were coded according to their cognitive complexity. Two-thirds of these competencies were classed as ‘knowing that’ or surface thinking, and one third were classified as ‘knowing how’ or deeper thinking. A taxonomy is offered as a method to understand models of learning necessary for evaluator education and training, as well as for further development of professional evaluator competencies.     

Should evaluators judge whether services are appropriate?

If an Aid for Dependent Children (AFDC) mother receives a cash reward for helping police apprehend a criminal, she automatically loses part of her children's benefits. Is this appropriate? More central to this article, does an evaluator of the AFDC program have the right to judge whether this is appropriate? Although decision makers' abilities to rectify such situations are increasingly constrained, they clearly need to know about such seemingly inappropriate aspects of services. But should evaluators be a source for this information? Yes — even though there are undeniable risks involved. This article discusses these issues and offers 18 suggestions for gathering and presenting information on appropriateness.