Lessons learned using a values-engaged approach to attend to culture,diversity, and equity in a STEM program evaluation

Evaluation must attend meaningfully and respectfully to issues of culture, race, diversity, power, and equity. This attention is especially critical within the evaluation of science, technology, engineering, and mathematics (STEM) educational programming, which has an explicit agenda of broadening participation. The purpose of this article is to report lessons learned from the implementation of a values-engaged, educative (Greene et al., 2006) evaluation within a multi-year STEM education program setting. This meta-evaluation employed a case study design using data from evaluator weekly systematic reflections, review of evaluation and program artifacts, stakeholder interviews, and peer review and assessment. The main findings from this study are (a) explicit attention to culture, diversity, and equity was initially challenged by organizational culture and under-developed evaluator–stakeholder professional relationship and (b) evidence of successful engagement of culture, diversity, and equity emerged in formal evaluation criteria and documents, and informal dialogue and discussion with stakeholders. The paper concludes with lessons learned and implications for practice.     

Analyzing mental health evaluation: moral and ethical dimensions

Now that evaluators have been sensitized to the importance of moral and ethical issues in their work, it is time to move beyond generalities and examine the moral and ethical implications of specific evaluation models in specific settings. This paper proposes a framework that can be used to examine moral and ethical dimensions of evaluation and illustrates it by analyzing a selected model of mental health evaluation. Such a systematic identification of moral issues can improve evaluation practice both proactively, by shaping the training of evaluators, and retrospectively, by contributing to meta-evaluation.     

Zur Effektivität des Gordon-Eltern-Trainings (PET) — eine Meta-Evaluation

The article shows how meta-evaluation leads to evidence about the differential effectiveness of the parent effectiveness training (PET) by Thomas Gordon. It shows that the training has a high general effectiveness. There are large effects for the directly trained communication skills, medium effects for the change of parental attitudes toward child-rearing and the parent-child-communication and small effects in parental behavior and the children’s self-concept. The training also shows long-term effects. Moreover, PET seems to be better suited for gender homogenous parent groups and for parents of older children.     

Beyond the Core: The Hot Topic(al) Alternative to the Survey-Based Introduction to Sociology Course

In the following paper we argue that the conventional “Introduction to Sociology” survey course should be restructured because such courses try to survey an unsurveyable body of knowledge and they do not teach the application of sociological research. The conventional intro course should be replaced with an intro course that surveys the types of social dynamics that sociologists typically research and the methods they use to do so. We propose a semester-long intro course with four case study learning-units that are chosen for their coverage of the underlying sociological dynamics, methods, and core concepts. We contend that case study learning-units which concentrate on topical issues and core sociological concepts are better suited for an introduction course.     

A Case for the Case Study: How and Why They Matter

In this special issue we have asked the contributors to make a case for the case study. The guest editors, Jeffrey Longhofer, Jerry Floersch and Eric Hartmann, intergrate ideas from across the disciplines to explore the complexties of case study methods and theory. In education, Gary Thomas explores the importance of ethnographic case studies in understanding the relationships among schools, teachers, and students. Lance Dodes and Josh Dodes use the case study to articulate a psychoanalytic approach to addiction. In policy and generalist practice, Nancy Cartwright and Jeremy Hardie elaborate a model for a case-by-case approach to prediction and the swampy ground prediction serves up to practitioners. Christian Salas and Oliver Turnbull persuasively write about the role of the case study in neuro-psychoanalysis and illustrate it with a case vignette. In political science, Sanford Schram argues for a bottom up and ethnographic approach to studying policy implementation by describing a case of a home ownership program in Philadelphia. Eric Hartman queers the case study by articulating its role in deconstructing normative explanations of sexuality. In applied psychology, Daniel Fishman describes a comprehensive applied psychology perspective on the paradigmatic case study. Richard Miller and Miriam Jaffe offer us important ways of thinking about writing the case study and the use of multi-media. Each contributor brings a unique perspective to the use of the case study in their field, yet they share practical and philosophical assumptions.     

Case study as a research method to study life histories of elderly people: Some ideas and a case study of a case study

The article explores conditions under which life history interviewing, based on case study research, may become a valuable research instrument. Validity and reliability of case study as a form of qualitative research are discussed. It is shown that neither triangulation nor internal consistency methods can guarantee a single interpretation. As an example a case study of Zina, an immigrant from Russia, is used, and two explanations of her behavior are proposed. The choice between them is made through construction of a holistic image of Zina, which essentially means introduction of a new dimension—theory, worldview—as a criterion of validity and reliability of case studies. Relations of a case study to a theory are discussed, and the conclusion is made that a case study enables a researcher to utilize the most the diversity of life histories of elderly people.     

Evaluating social exclusion interventions in university-community partnerships

Most university-community partnerships (UCPs) involve elements of community-level social exclusion interventions. As such, they face substantial challenges in management and evaluation. This paper highlights the central challenges associated with evaluation of UCP and other social exclusion interventions at the community level, and suggests methods to overcome them. The main body of the paper presents a case study based on a four-year action research involving evaluation of a social exclusion intervention initiated and implemented by a UCP in Israel. The case study highlights the challenges faced by the evaluation team, the solutions provided, and the contribution of the evaluation to improvement and accountability.     

Assessing local capacity for health intervention

Because of their location within the practice realm, participatory, community-based public health coalitions offer many lessons about implementing and sustaining local interventions. This paper presents a case study of capacity assessment at the local level. Capacity evaluation methods are presented, with emphasis on the theoretical framework used to guide the evaluation. The capacity evaluation framework presented herein was theoretically based and designed to generate practical information to facilitate the adoption of a locally tailored youth obesity prevention program, VERB Summer Scorecard (VSS). Using multiple methods, four aspects of community capacity were assessed, including community, knowledge and skills, resources, and power. Within each category, factors that facilitated or impeded program implementation were distinguished. The evaluation protocol was designed to generate information increasing community capacity to sustain a community-based obesity prevention program. Capacity tables were used as a program-planning tool and as a system for sharing implementation and sustainability requirements with potential adopters. This case study also explores how to use capacity assessment results to empower coalitions to serve as catalysts for development of local programs in other communities.     

Assessing competency-based evaluation course impacts: A mixed methods case study

The contexts in which evaluators develop and apply their expertise are increasingly complex; evaluator education needs to provide robust opportunities to support and assess the progressive, lifelong development of relevant knowledge and skills. This mixed methods case study begins to address the dearth of empirical evidence assessing the impacts and learner experiences of competency-based approaches to evaluator education. A decade-in-the-making doctoral evaluation course based on the Canadian Evaluation Society’s Competencies for Canadian Evaluation Practice created an opportune study setting. We applied a systems perspective to Stufflebeam’s Context, Input, Process, and Product (CIPP) model to frame the case study analysis and presentation of the complex impacts generated by pre/post course competency self-assessments and reflections, integrated with an end-of-course focus group. Seven insights shed new light on the effective course design and implementation features for developing intended and unintended evaluator competencies. We conclude with a discussion of the theoretical, practical, and methodological implications for effective competency-based evaluator education.

• •Keywords: Case study; Competency-based approach; Evaluator education; Mixed methods research

     

History of Research on Process Relevant to Clinical Social Work

Case study research methods and holistic approaches have not appeared in the social work literature for at least two decades due to debate over what constitutes legitimate science. This debate has contributed to alienation of the work of the researcher and the day-to-day practice of the practitioner. Existing intellectual frameworks and measuring tools for understanding clients’ reactions and behaviors through case study research methods can help close the gap between researcher and practitioner. The purpose of this article is to provide a historical review highlighting certain tensions in the field of social work and to provide the foundation for a discussion of the contemporary state of process research—whereby new theory and methods from the natural sciences tender means for the field to pick up where previous process researchers have left off. The article concludes with recommendations for future work and anticipates novel and useful methods for process research.     

EMOTIONAL RETREAT AND SOCIAL EXCLUSION: TOWARDS BIOGRAPHICAL METHODS IN PROFESSIONAL TRAINING

This article argues the relevance of biographical case study methods as a tool for professional practice. The case of Laura, manager of a small hostel for homeless people, shows hidden reserves of emotional understanding which Laura finds too risky to bring into play. Analysis of the structure of the interview shows the enactment and lowering of Laura's defences in interaction with the interviewer. The article argues for support and supervision to allow professionals to recognise and use their capacities for emotional thinking, and that this kind of ‘experiential truth’ is key both for users and professionals in tackling social exclusion. Methodological parallels between biographical methods and effective training are also drawn through discussion of responses to a training video based on a fictionalised version of a critical incident from this interview.     

The voices people read: Orthography and the representation of non-standard speech

Combining features of matched guise tests with sociolinguistic interviewing and oral performance, this study investigates the social meanings carried by non-standard orthographies. Participant evaluations of the personas represented by non-standard orthographies showed that people connected orthography to social identities. Specifically, we found that people uncritically and spontaneously read non-standard orthographies as indices of low socioeconomic status. When we asked participants to read texts out loud, we found marked shifts in their reading performances of texts in standard versus non-standard orthographies. Through a case study analysis of participants' readings and attitudes, we identify two kinds of stances taken by participants towards stigmatized identities indexed by non-standard orthographies. Finally, we use these case studies to argue for the necessity of contextually rich qualitative research methods for the study of language attitudes.     

Employment services for persons with serious mental illness in northeastern Ontario: The case for partnerships

Objective: To better understand why employment success is low, a case study was conducted to examine the influence of place on access to employment for persons with serious mental illness (SMI) residing in two northeastern Ontario communities (Rebeiro, in progress). Methods: Community-based participatory research methods were used to engage persons who experience SMI, decision-makers and providers in the research. Forty-six interviews were conducted, complemented by primary and secondary quantitative data sources.Results: While most consumers consider employment to be a key element of their recovery, employment rates for persons with SMI remain limited in northeastern Ontario, Canada. The findings of this case study reveal the importance of collaborative partnerships to fostering better employment outcomes in northeastern Ontario. Conclusion: The challenges of collaboration due to rural and northern tensions, as well as various jurisdictional and funding tensions existing at the level of community support the case for partnerships in the provision of employment services in northern and rural places.     

Using simulation to develop testable functionalist explanations: a case study of church survival

Many arguments for and against functionalism have revolved around logical and methodological issues, discouraging attempts to develop testable functionalist theories. This paper takes a different approach, arguing that the problem with sociological functionalism is not its logic of explanation but the absence of appropriate techniques for formalizing functionalist theories. The solution presented --agent-based simulation --is illustrated by a case study: a functionalist account of church survival. The case study suggests that there is nothing wrong with the logic of functionalism properly formalized, illustrates how simulation can enable testing of functionalist explanations and suggests why existing sociological methods have not proved fruitful for this task. In the light of new techniques like simulation, the current neglect of functionalism may no longer be justified.     

Widening the Lens: The Efficacy of the Case Method in Helping Direct Practice MSW Students Understand and Apply Mezzo and Macro Dimensions of Practice

This study tested the effectiveness of the case method as a teaching and learning tool to help direct practice (micro) graduate students to more fully incorporate the mezzo and macro dimensions in clinical situations. Participants for this study included 115 MSW students enrolled in a second‐year clinical concentration who, as a requirement of the program, took an integrative seminar that was taught using the case method. Students completed a pre‐ and post‐test that measured the importance students placed on selected mezzo and macro dimensions of practice. The study also assessed whether the case method influenced students' perceived capacity for critical thinking, vertical integration of first‐ with second‐year content, and understanding the program's theoretical model. Results showed significant increases in several areas assessed. The paper begins by tracing the historical and contemporary writings in the social work literature on integrated learning, and discusses the impact of these trends on the learning process. Ideas on how to build integrative opportunities or methods of instruction into the social work curriculum are then explored. The case method is advocated as a viable tool in teaching with empirical evidence presented to support its efficacy.     

How to Construct a Mixed Methods Research Design

This article provides researchers with knowledge of how to design a high quality mixed methods research study. To design a mixed study, researchers must understand and carefully consider each of the dimensions of mixed methods design, and always keep an eye on the issue of validity. We explain the seven major design dimensions: purpose, theoretical drive, timing (simultaneity and dependency), point of integration, typological versus interactive design approaches, planned versus emergent design, and design complexity. There also are multiple secondary dimensions that need to be considered during the design process. We explain ten secondary dimensions of design to be considered for each research study. We also provide two case studies showing how the mixed designs were constructed.     

A research design for generalizing from multiple case studies

Many evaluations would be more useful to decision makers if their results were both explanatory and generalizable. In view of the limitations of traditional methods in meeting both these goals at the same time, the authors describe a research design explicitly intended for generalizing about the range of applicability of “explanatory patterns” (within a designated “target” population of cases). The paper presents an overview of the main features of a “multiple case study design,” shorthand for a multiple site, structured case study design. It then discusses the nature of explanatory patterns, how case study investigators pursue and recognize valid patterns, and how an analyst can apply the same logic to cross-site analysis in order to make valid inferences about the limiting conditions under which particular explanatory patterns apply to particular populations.     

Reasoning With Case Studies: Issues Of An Aging Workforce

Case studies are an ideal research approach to generate in-depth and contextualized data. Many specific data collection approaches can be incorporated to gather data through a case study, but the information so gathered is interpreted in terms of a single case. When a set of case studies is conducted on a particular research theme, the question arises as to what kinds of inferences can be made with a small number of cases. Problems of inference are increased when the cases have been selected opportunistically or haphazardly and when the universe of all potential cases cannot readily be defined. Charles Ragin has distinguished between the logic of multivariate analysis and the logic of comparative case analysis. Drawing on this distinction, this article will describe the ways in which inferences can be drawn, and the nature of such inferences, from a multiple case study approach. Reasoning from multiple case studies is an inductive process that uses comparisons to generate hypotheses and to provide limited tests through the principle of falsification. This approach was used in developing a synthesizing report based on seven large case studies from the project, “Issues of an Aging Workforce.” Five Canadian and two American case studies were conducted in corporate settings in different economic sectors. The case studies used qualitative methods (focus groups and key informant interviews) and survey data from employees and managers. Thus, the logic of multivariate analysis was used, along with inductive reasoning with qualitative data, within cases, and a logic of comparative case analysis when seeking to make and test generalizations based on the set of seven cases.