The development and field testing of a vocational education evaluation model

Reported is the development and field testing of an evaluation model used to evaluate the vocational education programs at the Metropolitan Community Colleges (MCC) of Kansas City, Missouri which can be implemented on vocational programs at other institutions. The model evaluated a vocational program on the following six criteria: (1) program's relationship to job market profile; (2) program's level of community support; (3) program's success in meeting vocational aspirations of clientele; (4) program's success in terms of student performance; (5) program 's cost-effectiveness; and (6) program 's success in reaching the handicapped and disadvantaged. The evaluation model provides a strategy and a set of procedures for assessing the quality of existing vocational education programs of study at any post-secondary institution. The model can be easily and inexpensively implemented, thus providing a data base which is useful for administrative planning and decision making.     

The problem of replicability in program evaluation. The component solution using the example of case management

Specificity and replicability of a program are crucial in order to rigorously evaluate the program's effectiveness. However, the traditional approaches to evaluation have ignored the problem that most social and mental health programs consist of vaguely specified interventions and cannot meet minimal standards of replicability. Consequently, the results of effectiveness evaluations of poorly defined programs have little general generalizability. Nevertheless, effectiveness information concerning su ch programs is much needed. One solution to this problem is to focus on program components, examining interventions at a level where operational definitions can be employed and replicability is possible. In this paper, this approach is expanded using the example of case management programs. A component-oriented approach ensures a systematic evaluation of program effectiveness for which the first question for evaluators is whether a program consists of interventions that reliably produce specific immediate outcomes. Then, and only then, is it possible to address questions concerning whether a program will reliably produce those outcomes that are the overall objectives of the program.     

Harnessing the reliability of outcome measures

A method is presented for engineering the necessary levels of measurement reliability for evaluating ongoing programs. Two studies of levels of client functioning at a community mental health center (CMHC), in which an outcome measure, the Global Assessment Scale (GAS), did not perform as expected, drew attention to the need for better control of outcome measure reliabilities. Drawing from generalizability theory, a study was conducted of three sources of GAS score variance — clients, raters, and training in the use of the scale. Several estimates of reliability (ERs) were developed, depending on the manner in which the GAS ratings were, or would be, obtained in the CMHC. The differences among these ERs clarified why the GAS had lower reliability when used in our setting. Finally, two hypothetical examples are described to illustrate the utility of applying generalizability theory to achieve higher reliabilities for outcome measures.     

Using client exit interviews to illuminate outcomes in program logic models: a case example

This paper describes how client exit interviews can assist human service administrators and workers to better understand the outcomes their programs are designed to accomplish. Specifically, the qualitative component of a demonstration family literacy program evaluation is used to illustrate how client input can be used to fine-tune the outcomes component of a program's logic model. An analysis of semi-structured exit interviews with 35 clients, who were randomly selected from all 89 served in the first year of the program, resulted in revision to the program's original logic model, creating explicit ‘testable’ pathways to achieving intended outcomes.     

An organization-environment framework for assessing program implementation

Evaluations of health programs are typically conducted from a rational bureaucratic framework in which client change and client outcome are seen as logical outcomes of program activity. This paper proposes a theoretical framework for examining organizational consequences of program implementation. Factors in the internal environment (e.g., the hospital) and the external environment (e.g., the community) which contribute to the program's success or failure, are considered. Observational and structured interview data are used to compare two state funded perinatal projects, one an organizational "success" and the other a costly failure. Attributes of external environments which may be important to successful implementation include demand for services, provider competition, access to information, social beliefs, and professional commitment. Important features of the internal environment include host organization commitment and conflict, and structural attributes: complexity, formalization, centralization, and coordination. The paper concludes with theoretically-based questions which serve as guidelines for process evaluation.     

Collaborative evaluation within a framework of stakeholder-oriented evaluation approaches

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation planning and implementation. Unlike "distanced" evaluation approaches, which reject stakeholder participation as evaluation team members, Collaborative Evaluation assumes that active, on-going engagement between evaluators and program staff, result in stronger evaluation designs, enhanced data collection and analysis, and results that stakeholder understand and use. Among similar "participant-oriented" evaluation approaches (Fitzpatrick, Sanders, & Worthen, 2011), Collaborative Evaluation distinguishes itself in that it uses a sliding scale for levels of collaboration. This means that different program evaluations will experience different levels of collaborative activity. The sliding scale is applied as the evaluator considers each program's evaluation needs, readiness, and resources. While Collaborative Evaluation is a term widely used in evaluation, its meaning varies considerably. Often used interchangeably with participatory and/or empowerment evaluation, the terms can be used to mean different things, which can be confusing. The articles use a comparative Collaborative Evaluation Framework to highlight how from a theoretical perspective, Collaborative Evaluation distinguishes itself from the other participatory evaluation approaches.     

Conducting evaluation in contested terrain: challenges, methodology and approach in an American context

Using case examples, the author describes circumstances under which violence and conflict, overt and covert, have impinged on his evaluation field work. When fear is pervasive, it constrains people's ability and willingness to participate in an evaluation and their candor if they do. It can also be dangerous. Safety and well-being of participants and researchers in his view take priority over the evaluation's goals. Evaluation involving people who are survivors of violence, torture and conflict requires special sensitivity and examples are offered of appropriate methodologies. Stakeholders in evaluations presumably have beneficent interest in a program's success. However in conflict situations ‘malevolent’ stakeholders, who standing to gain from a project's failure, will actively try to thwart it. Undermining the evaluation as well and impugning the evaluator's objectivity, skill, knowledge and credibility are among the tactics they employ. Evaluators are urged to stay attuned to field conditions, consider alternative methods and locations for collecting data and be self-conscious and deliberate about how their study and role are defined and understood by contending parties.     

Maltreatment prevention through early childhood intervention: A confirmatory evaluation of the Chicago Child-Parent Center preschool program

Increased recognition of the consequences associated with child maltreatment has led to greater emphasis on its prevention. Promising maltreatment prevention strategies have been identified, but research continues to suffer from methodological limitations and a narrow focus on select prevention models. This investigation uses data from the Chicago Longitudinal Study to examine mediating mechanisms that link the Chicago Child-Parent Center preschool program to a reduction in overall child maltreatment and, more specifically, child neglect. We use structural equation modeling to test child, family, and school measures hypothesized to mediate the effects of CPC participation on maltreatment and neglect. Results indicate that a substantial proportion of the program's impacts can be accounted for by family support processes, including increased parent involvement in school and maternal educational attainment as well as decreased family problems. The CPC program's association with reduced school mobility and increased attendance in higher-quality schools also significantly mediated its effects on maltreatment and neglect. Further, a decrease in troublemaking behavior contributed modestly to mediating the program's association with maltreatment but not neglect. We discuss the implications of these results for the field of maltreatment prevention.     

School-based prevention of bullying and relational aggression in adolescence: the fairplayer.manual

The fairplayer.manual is a school-based program to prevent bullying. The program consists of fifteen to seventeen consecutive ninety-minute lessons using cognitive-behavioral methods, methods targeting group norms and group dynamics, and discussions on moral dilemmas. Following a two-day training session, teachers, together with skilled fairplayer.teamers, implement fairplayer.manual in the classroom during regular school lessons. This chapter offers a summary of the program's conception and underlying prevention theory and summarizes the results from two evaluation studies. Standardized questionnaires showed a positive impact of the intervention program on several outcome variables.     

A theoretically derived model of premarital intervention: The building of a family system

Premarital intervention programs are on the rise in the United States despite the absence of persuasive empirical data that substantiate their effectiveness. In addition to this lack of strong empirical support, many premarital counseling programs fail to articulate any coherent theory of family process or family intervention which serve as conceptual guideposts for formulating program goals and devising program activities. In this article, the authors describe an experimental program that is conceptually grounded in theories of family process and family intervention. The theoretical rationale for each element included in the program is discussed and numerous examples of the program's content and exercises are offered. Finally, the authors discuss a number of issues central to the evaluation of premarital counseling programs.     

Mentoring At-Risk Latino Children and Their Parents: Impact on Social Skills and Problem Behaviors

The Family Mentoring Project, which provided approximately one year of mentoring for at-risk 10-year old Latino children and their parents, aimed to provide not only service but empirical evaluation of the program's impact. This University-community partnership offered individual mentoring, a group educational component for children and parents, and group social/recreational activities. A pre- and post-test analysis of 11 non-mentored and 20 mentored youth revealed positive gains on social skills for mentored children as reflected in self-ratings and mothers' ratings on the Social Skills Rating Scale (SSRS). Also based on the SSRS, mothers reported decreases in three problem behaviors for mentored children. In addition, by post-testing, the mentored children and their mothers compared very favorably with the SSRS standardized samples on both skills and problem behaviors. The findings suggest that bicultural competence may be fostered by programs that provide consistent and long-term mentoring, involve the children's families, include group educational components, and bring families and mentors together for social/recreational events.     

Volunteerism and social capital in policy implementation: evidence from the Long-Term Care Ombudsman Program

We assess the link between a program's volunteer support and state social capital in the case of the joint implementation of the federal Long-Term Care (LTC) Ombudsman Program by state and federal authorities. This program, which is designed to prevent elder abuse and ensure quality care in long-term facilities, is implemented at the state and local levels and relies heavily on volunteer staff. First, we find that volunteerism is vital to the efficacy of the program's monitoring and investigative functions. Second, we find that volunteerism in this program is tied to broader level conditions of a state's social capital. Last, we discuss the implications of our findings for volunteer-based programs devolved to the states.     

Implemented as intended? Recording family worker activities in a Families First program

In evaluating family preservation services, it is important not only to study the service outcomes and the family characteristics, but also what actually happens during the treatment. This requires a program model. This article describes how a program model works, prescribes how workers should carry it out, and describes how researchers should measure the program's characteristics. The authors use data from Families First in The Netherlands to test the model. Results show that the method of the program meets the specified characteristics. The results are important for treatment, policy, education, and evaluation research.     

The diversion of delinquent youth: family-frcused intervention

This paper describes the practices of the Adolescent Diversion Project and presents some research evaluating the program's effectiveness. The youth were referred to the project as an alternative to further court processing. These youth were randomly assigned to the project or to a control group which received treatment as usual by the court. Project youth were assigned to work with college undergraduates for 18 weeks. The two basic intervention techniques employed were behavioral contracting and advocacy. Project youth were further randomly assigned to one of two intervention conditions within the project. The Multi-Focus group applied the above strategies to a variety of social domains of the youth, while the Family Condition focused entirely on the family. Research results focused on standard outcome criteria (police, court, and school data) as well as on the degree to which the experimental conditions differed in practice.     

A model-guided process evaluation: Office-based prescribing and pharmacy dispensing of methadone

This article presents an exemplar of a model-guided process evaluation that specifies the treatment model, assesses its implementation, monitors the fidelity of the model throughout the project, assesses model exposure and absorption, and helps understand the program's intermediate effects (proximal outcomes) as well as final effects (distal outcomes). The New Mexico study on office-based prescribing and community pharmacy dispensing of methadone is a research demonstration project that phases a small group of female methadone maintenance patients out of methadone clinics and into a program where they will obtain their scheduled doses of methadone at pharmacies that work in collaboration with physicians and a social worker.The patient's methadone treatment will in this way become part of their overall health care. Early detection of problems of implementation (e.g., the omission of program content or the delivery of inaccurate information) enables the researcher to make adjustments before the problems become unmanageable and the integrity of the original research design is compromised. A model-guided process evaluation can critically inform health services research demonstrations designed for enabling continuous, ongoing feedback and improvement of client-related services.     

A cost-benefit analysis of a smoking cessation program

Smoking cessation programs are usually evaluated in terms of the percentage of participants who have stopped smoking at least 6 months after the program's completion. This paper shows how a relatively low rate of behavior change may actually result in a high ratio of benefits to costs. Cost-benefit analysis is done for a specific program but the procedures and cost estimates are generalizable to other smoking cessation programs. Benefits to the firm included reduced costs of insurance and the savings due to employee absenteeism and disability. Costs included the program costs as well as the opportunity costs of the participants. Data from the literature and from the specific program are combined to form conclusions about the effectiveness of the program.     

REPRODUCING CRIMINAL TYPES

Based upon observation in a cognitive treatment program for incarcerated violent offenders, this article uses a social constructionist approach to examine the program's rhetorical construction of inmates as particular "types'of beings. The program's practices contain ideological representations of criminals that, in turn, sustain and reproduce the program's logic. Specifically, the typification of inmates'"erroneous thoughts'and "anger'is examined as an example of how inmates are "talked into being'criminal types (Heritage 1984, p. 290). Inmates are asked to decontextualize and rearticulate their criminal pasts according to a criminological language of "cognitive distortions'deemed characteristic of criminal "personalities.'In this reductionist process, inmates'"techniques of neutralization'(Sykes and Matza 1957) are stripped away. Facilitators interpret inmates'social and situational counterrhetorics as indications of their criminal minds, thereby sustaining the individualistic typification in criminal classification and correctional philosophy. Thus, the micropolitics of the Cognitive Self-Change program reveals the dynamic, discursive construction of criminal selves according to institutional conditions and demands. In conclusion, it is argued that institutions engaged in managing human conduct require particular constructions of pathological individuals in order to do their work.     

Complexity of process and role in the evaluation of an interdisciplinary mental health education program: The deacon's masterpiece

Research and literature on program evaluation must attend to problems of application, which can have major effect on the outcome. Increasing numbers of innovative, multifaceted programs operating in complex settings are presenting for evaluation. Evaluation process must become flexible, creative, and multifaceted in order to produce valid and comprehensive results, and to meet the needs of this field. Evaluators and evaluation, the evaluated program, and the environment become a mutually interacting system in the process of evaluation. Evaluation goals and roles must be conceptualized, agreed upon, sanctioned, and appropriately implemented or conflict and ineffectiveness will result. Three models of evaluation emerge: objective and independent, objective but serving one interest, and pseudo-scientific public relations. Evaluation of an interdisciplinary mental health education program illustrates these issues. Evaluation technology, process, and roles must all be attended to or the weak link undermines the rest.     

Generalizing from social media data: a formal theory approach

ABSTRACT

Researchers increasingly draw on social media data to answer big questions about social patterns and dynamics. However, as with any data source, social media data present both opportunities and significant challenges. One major critique of social media data is that the data are not generalizable outside of the platforms from which the data originate. Problems of generalizability stem from non-universal participation rates on various platforms, demographically biased samples, as well as limited access to data based on infrastructural constraints and/or user privacy practices. We suggest that instead of empirical generalizability, social media data are theoretically generalizable in the formal theory tradition. Through a case example in which we use YouTube comments to test and extend a key tenet of identity theory, we show how social media data can instantiate theoretical variables and thus generalize to theoretical propositions. Mediated through formal theory, social media data maintain the capacity to address broad social questions while upholding methodological integrity.