The development and field testing of a vocational education evaluation model

Reported is the development and field testing of an evaluation model used to evaluate the vocational education programs at the Metropolitan Community Colleges (MCC) of Kansas City, Missouri which can be implemented on vocational programs at other institutions. The model evaluated a vocational program on the following six criteria: (1) program's relationship to job market profile; (2) program's level of community support; (3) program's success in meeting vocational aspirations of clientele; (4) program's success in terms of student performance; (5) program 's cost-effectiveness; and (6) program 's success in reaching the handicapped and disadvantaged. The evaluation model provides a strategy and a set of procedures for assessing the quality of existing vocational education programs of study at any post-secondary institution. The model can be easily and inexpensively implemented, thus providing a data base which is useful for administrative planning and decision making.     

A novel approach for evaluating programs designed to serve military veterans: Using an adapted common components analysis

Evaluators are challenged to keep pace with the vast array of Veteran support programs operating in the United States, resulting in a situation in which many programs lack any evidence of impact. Due to this lack of evidence, there is no efficient way to suggest which programs are most effective in helping Veterans in need of support. One potential solution to this dilemma is to reconceptualize program evaluation, by moving away from evaluating programs individually to evaluating what is common across programs. The Common Components Analysis (CCA) is one such technique that aggregates findings from programs that have undergone rigorous evaluation at the level of program components (e.g., content, process, barrier reduction). Given that many Veteran programs lack outcome evidence from rigorous studies, an adaptation to CCA is needed. This report examines cross-sectional data from a pilot study using an adapted CCA across five domains of well-being (i.e., employment, education, legal/financial/housing, mental/physical health, and social/personal relationships). The purpose of this preliminary study is to determine the feasibility of eliciting program nominations and program components from Veterans via an online survey. When coupled with a longitudinal research design, this adaptation to CCA will allow for stronger causal claims about the expected impact of different program components within and across a variety of domains.     

Distance Education Alumni: How Far Have They Gone?

SUMMARY

This evaluation compared 34 distance education (DE) and 38 on-campus alumni regarding employment-related outcomes; professional activities; satisfaction with MSW program components; and development of knowledge, skills, and values. Findings showed equivalent outcomes in most respects. DE alumni were generally positive about the extent to which the program enabled them to develop professionally.     

Applying national survey results for strategic planning and program improvement: The National Diabetes Education Program

Since the 1970s, the federal government has spearheaded major national education programs to reduce the burden of chronic diseases in the United States. These prevention and disease management programs communicate critical information to the public, those affected by the disease, and health care providers. The National Diabetes Education Program (NDEP), the leading federal program on diabetes sponsored by the National Institutes of Health (NIH) and the Centers for Disease Control and Prevention (CDC), uses primary and secondary quantitative data and qualitative audience research to guide program planning and evaluation. Since 2006, the NDEP has filled the gaps in existing quantitative data sources by conducting its own population-based survey, the NDEP National Diabetes Survey (NNDS). The NNDS is conducted every 2–3 years and tracks changes in knowledge, attitudes and practice indicators in key target audiences. This article describes how the NDEP has used the NNDS as a key component of its evaluation framework and how it applies the survey results for strategic planning and program improvement. The NDEP's use of the NNDS illustrates how a program evaluation framework that includes periodic population-based surveys can serve as an evaluation model for similar national health education programs.     

Algorithms of Oppression: How Search Engines Reinforce Racism

SUMMARY

This paper presents a literature review of social work programs' distance education evaluation studies with an emphasis on the strengths distance education programs offer to students relative to standard programs. The authors discuss quality issues in relation to teaching and learning in distance education programs with a panel of distance learners who will discuss their experiences.     

Evolutionary Evaluation: Implications for evaluators,researchers, practitioners,funders and the evidence-based program mandate

Evolutionary theory, developmental systems theory, and evolutionary epistemology provide deep theoretical foundations for understanding programs, their development over time, and the role of evaluation. This paper relates core concepts from these powerful bodies of theory to program evaluation. Evolutionary Evaluation is operationalized in terms of program and evaluation evolutionary phases, which are in turn aligned with multiple types of validity. The model of Evolutionary Evaluation incorporates Chen's conceptualization of bottom-up versus top-down program development. The resulting framework has important implications for many program management and evaluation issues. The paper illustrates how an Evolutionary Evaluation perspective can illuminate important controversies in evaluation using the example of the appropriate role of randomized controlled trials that encourages a rethinking of “evidence-based programs”. From an Evolutionary Evaluation perspective, prevailing interpretations of rigor and mandates for evidence-based programs pose significant challenges to program evolution. This perspective also illuminates the consequences of misalignment between program and evaluation phases; the importance of supporting both researcher-derived and practitioner-derived programs; and the need for variation and evolutionary phase diversity within portfolios of programs.     

An Empirically Based Framework for Content Selection and Management in Divorce Education Programs

Contemporary research has shown that divorce education programs are becoming mandated or recommended in numerous states throughout the nation. Cooccurring with the popularity of these programs is the variation in content of divorce education programs. The focus of this project was to systematically explore the research literature on factors related to divorce adjustment and identify findings to be used in content selection and development for divorce education programs. The divorce-stress-adjustment perspective was utilized to frame the divorce process affecting both children and adults. A systematic literature review resulted in identification of 103 published studies that fit the project criteria. Selected research findings identified in the literature review process were organized according to a conceptual framework designed to categorize content in a tiered approach useful for selecting and managing content in divorce education programs. Implications of the project findings for content identification in divorce education, program development, and program evaluation are explored.     

Evaluation of community-based AIDS education and risk reduction projects in ethnic and racial minority communities: A survey of projects funded by the U.S. public health service

The purpose of this study was to describe proposed program evaluation methods used to determine effectiveness of selected federally funded AIDS education and risk reduction interventions focused on ethnic and racial minority populations. Funding for AIDS prevention programs has grown significantly since 1985, when the first allocation of $11 million was earmarked for general AIDS prevention and education. In 1987, funds specifically targeting minority populations were set aside from AIDS prevention and education programs. Nine agencies/divisions within the Department of Health and Human Services (DHHS) were identified as having funded AIDS prevention and education projects targeted toward ethnic and racial minority populations. The grants and contracts reviewed for this study were limited to those programs receiving Federal funding through DHHS. Few programs provided information within the broader context of sex education and drug education. The modification of needle-sharing behavior among IV drug users and unprotected sex with an IV drug user were the two most frequent risk factors targeted. All 63 programs used at least one formative evaluation method. The most frequently used formative techniques were monitoring achievement of project objectives and counting the number of people participating in program activities. Forty programs counted the amount of AIDS literature distributed. Thirty-three programs conducted a baseline assessment of AIDS knowledge, and 33 programs conducted a baseline assessment of risk behaviors for HIV infection. Results from this study support the National Academy of Sciences recommendation that while summative evaluation will ultimately be valuable, it would be premature to begin developing outcome evaluation strategies at the present time. It is incumbent upon program planners and evaluators to ensure that premature program evaluations are not made in an effort to eliminate direct federal funding of minority community-based strategies to stop the spread of HIV infection.     

Randomize groups, not individuals: a strategy for improving early childhood programs

Until the past few years, our nation's approach to designing federal programs for preschool-age children lacked coherence and paid little attention to what had worked (and not worked) in the past. In this article, the authors propose that credible information useful for designing effective programs will require the ongoing, systematic development and evaluation of alternative approaches for the improvement of large-scale early childhood programs. The research should place greater reliance on experiments in which existing groups of individuals, such as intact classes or preschool agencies, are randomly assigned to implement competing early education programs or program components. Randomizing groups, rather than individual children, changes the research question from "What works?" to "What works better?" yielding more useful information than is currently available about which preschool approaches ought to be strongly embedded in our nation's social policy.     

Evaluating development programs: building joint activity

There has been an increasing plea for a participatory approach to development programs. It is argued that for programs to be sustainable, they need to be conceptualized, negotiated and run by all stakeholders jointly. Concurrent with this trend, there has been an increasing demand for program evaluation strategies, which encompass an evaluation of the nature and level of participation of the different stakeholders in the program. This paper proposes a theoretical framework for such evaluation and subsequently provides a practical illustration. Engeström's notions of ‘activity system’ and ‘learning by expanding’, are proposed as a framework for the evaluation of participation and an action evaluation methodology is suggested as the practical tool within such framework. The paper presents a case study of a conservation education program in Africa, which operationalizes the proposed theoretical framework.     

A comparison of linear and systems thinking approaches for program evaluation illustrated using the Indiana Interdisciplinary GK-12

Logic models are based on linear relationships between program resources, activities, and outcomes, and have been used widely to support both program development and evaluation. While useful in describing some programs, the linear nature of the logic model makes it difficult to capture the complex relationships within larger, multifaceted programs. Causal loop diagrams based on a systems thinking approach can better capture a multidimensional, layered program model while providing a more complete understanding of the relationship between program elements, which enables evaluators to examine influences and dependencies between and within program components. Few studies describe how to conceptualize and apply systems models for educational program evaluation. The goal of this paper is to use our NSF-funded, Interdisciplinary GK-12 project: Bringing Authentic Problem Solving in STEM to Rural Middle Schools to illustrate a systems thinking approach to model a complex educational program to aid in evaluation. GK-12 pairs eight teachers with eight STEM doctoral fellows per program year to implement curricula in middle schools. We demonstrate how systems thinking provides added value by modeling the participant groups, instruments, outcomes, and other factors in ways that enhance the interpretation of quantitative and qualitative data. Limitations of the model include added complexity. Implications include better understanding of interactions and outcomes and analyses reflecting interacting or conflicting variables.     

Puberty Education in a Global Context: Knowledge Gaps,Opportunities, and Implications for Policy

Youth programs and policies provide opportunities for institutions and societies to support healthy adolescent development. Puberty education programs are universally important, as they provide crucial knowledge and skills to help youth and their caregivers navigate the physical, emotional, and interpersonal changes of puberty with positive outcomes. However, few puberty programs have been rigorously evaluated, resulting in a lack of evidence‐based knowledge and practice in this area. This review examines the status of research on puberty education and related programs and draws on the broader intervention literature and recent research findings on adolescence to identify program features that might improve program effectiveness. Implications for policy are also discussed. The need for rigorous program evaluation is emphasized throughout.     

Research Note—Program Evaluation: Enhancing Academic Quality and Program Integrity During Rapid Off-Campus Growth

By creating a program evaluation model that assesses multiple components from several perspectives, a social work division is able to determine the quality, effectiveness, and integrity of its offerings and support its mission to develop social work students for practice. This task is critical in supporting an off-campus program’s ability to navigate change and uncertainty in higher education. This research note discusses how a small, bachelor of social work program with a dual-delivery system (traditional and community-site) used the Council on Social Work Education reaccreditation process to adapt its program evaluation to ensure academic quality and program integrity during a period of rapid growth.     

Delivering a Social Work MSW Program through Distance Education: An Innovative Collaboration between Two Universities,USA

The Joint Master of Social Work Program of Cleveland State University and The University of Akron was the first MSW program taught entirely via distance education that was fully accredited by the Council of Social Work Education. Over the past nine years, the Joint MSW Program has grown from one cohort of 24 full‐time students to a program with an enrollment of 186 full‐time, part‐time and Advanced Standing students. This innovative Joint MSW Program has required an ongoing process of collaboration, refinement, and flexibility. The purpose of this article is to provide a narrative of the experiences of delivering an entire MSW program via distance education, including obstacles that have been overcome as well as recommendations for other programs considering distance education.     

Lessons learned about collaborative evaluation using the Capacity for Applying Project Evaluation (CAPE) framework with school and district leaders

Schools, districts, and state-level educational organizations are experiencing a great shift in the way they do the business of education. This shift focuses on accountability, specifically through the expectation of the effective utilization of evaluative-focused efforts to guide and support decisions about educational program implementation. In as much, education leaders need specific guidance and training on how to plan, implement, and use evaluation to critically examine district and school-level initiatives. One specific effort intended to address this need is through the Capacity for Applying Project Evaluation (CAPE) framework. The CAPE framework is composed of three crucial components: a collection of evaluation resources; a professional development model; and a conceptual framework that guides the work to support evaluation planning and implementation in schools and districts. School and district teams serve as active participants in the professional development and ultimately as formative evaluators of their own school or district-level programs by working collaboratively with evaluation experts.The CAPE framework involves the school and district staff in planning and implementing their evaluation. They are the ones deciding what evaluation questions to ask, which instruments to use, what data to collect, and how and to whom results should be reported. Initially this work is done through careful scaffolding by evaluation experts, where supports are slowly pulled away as the educators gain experience and confidence in their knowledge and skills as evaluators. Since CAPE engages all stakeholders in all stages of the evaluation, the philosophical intentions of these efforts to build capacity for formative evaluation strictly aligns with the collaborative evaluation approach.     

Lessons learned about collaborative evaluation using the Capacity for Applying Project Evaluation (CAPE) framework with school and district leaders

Schools, districts, and state-level educational organizations are experiencing a great shift in the way they do the business of education. This shift focuses on accountability, specifically through the expectation of the effective utilization of evaluative-focused efforts to guide and support decisions about educational program implementation. In as much, education leaders need specific guidance and training on how to plan, implement, and use evaluation to critically examine district and school-level initiatives. One specific effort intended to address this need is through the Capacity for Applying Project Evaluation (CAPE) framework. The CAPE framework is composed of three crucial components: a collection of evaluation resources; a professional development model; and a conceptual framework that guides the work to support evaluation planning and implementation in schools and districts. School and district teams serve as active participants in the professional development and ultimately as formative evaluators of their own school or district-level programs by working collaboratively with evaluation experts. The CAPE framework involves the school and district staff in planning and implementing their evaluation. They are the ones deciding what evaluation questions to ask, which instruments to use, what data to collect, and how and to whom results should be reported. Initially this work is done through careful scaffolding by evaluation experts, where supports are slowly pulled away as the educators gain experience and confidence in their knowledge and skills as evaluators. Since CAPE engages all stakeholders in all stages of the evaluation, the philosophical intentions of these efforts to build capacity for formative evaluation strictly aligns with the collaborative evaluation approach.     

The use of metaevaluation to plan evaluations of conservation programs

The U.S. Department of Energy (DOE) is sponsoring and will be required to evaluate a wide range of conservation programs. The development of evaluation plans for such programs can often be improved by reviewing evaluations that have been done on similar programs. The Residential Conservation Service (RCS), sponsored by DOE, is discussed as an example of how conducting metaevaluations can improve evaluation planning. Many utilities have sponsored programs that are similar to the RCS. A review of the utility program evaluations suggested key methodological issues and data problems that should be considered in the development of an RCS evaluation plan. The implications of these metaevaluation findings for an RCS evaluation plan are discussed.     

Youth, drugs and resilience education

Billions of dollars are spent annually on school-based drug education programs, with youthful drug use remaining near peak levels since the 1980s. Institutional, policy, and program evidence presented here suggests that although primarily delivered in schools, the educational community rarely participates in the development of drug education; and despite the finding that "no-use" programs such as Drug Abuse Resistance Education (D.A.R.E.) or Life-Skills Training (LST) are likely ineffective, they continue to thrive in schools. This may be explained by insufficient educational participation and scientific discourse considering these issues; and the role of interest group politics, such as the symbiotic relationship between government, researchers, and the tobacco industry, in drug education. In addition to this evidence, resilience based program alternatives, evaluation, and implications are discussed.