Lessons learned about collaborative evaluation using the Capacity for Applying Project Evaluation (CAPE) framework with school and district leaders

Schools, districts, and state-level educational organizations are experiencing a great shift in the way they do the business of education. This shift focuses on accountability, specifically through the expectation of the effective utilization of evaluative-focused efforts to guide and support decisions about educational program implementation. In as much, education leaders need specific guidance and training on how to plan, implement, and use evaluation to critically examine district and school-level initiatives. One specific effort intended to address this need is through the Capacity for Applying Project Evaluation (CAPE) framework. The CAPE framework is composed of three crucial components: a collection of evaluation resources; a professional development model; and a conceptual framework that guides the work to support evaluation planning and implementation in schools and districts. School and district teams serve as active participants in the professional development and ultimately as formative evaluators of their own school or district-level programs by working collaboratively with evaluation experts. The CAPE framework involves the school and district staff in planning and implementing their evaluation. They are the ones deciding what evaluation questions to ask, which instruments to use, what data to collect, and how and to whom results should be reported. Initially this work is done through careful scaffolding by evaluation experts, where supports are slowly pulled away as the educators gain experience and confidence in their knowledge and skills as evaluators. Since CAPE engages all stakeholders in all stages of the evaluation, the philosophical intentions of these efforts to build capacity for formative evaluation strictly aligns with the collaborative evaluation approach.     

A principles-focused evaluation of community mental health short-term case management

Principles-focused evaluations allow evaluators to appraise each principle that guides an organization or program. This study completed a principles-focused evaluation of a new community mental health intervention called Short Term Case Management (STCM) in Toronto, Canada. STCM is a time limited intervention for clients to address unmet needs and personalized goals over 3 months. Findings showcase that a principles-focused evaluation, assessing whether program principles are guiding, useful, inspiring, developmental and/or evaluable (GUIDE), is a practical formative evaluation approach. Specifically, offering an understanding of a novel intervention, including its key components of assessment and planning, support plan implementation and evaluation and care transitions. Findings also highlight that STCM may work best for those clients ready to participate in achieving their own goals. Future research should explore how best to apply the GUIDE framework to complex interventions, including multiple principles, to increase evaluation feasibility and focus.     

The results framework--an innovative tool for program planning and evaluation.

This article constitutes a case study of the development and implementation of the "results framework," an innovative planning and evaluation tool that is rapidly becoming a standard requirement for United States Agency for International Development (USAID) projects. The framework is used in a USAID-funded regional initiative for HIV/AIDS prevention in Central America. This new program evaluation and monitoring tool provides many advantages over traditional evaluation approaches that use outside consultants to provide midterm and end-of-project evaluations. The results-framework process, which spans the life of the project, provides an opportunity for program staff, donors, partners, and evaluators to work as a team to collect and use rich, longitudinal data for project planning, implementation, and evaluation purposes.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

Planning for the delivery of special services in public schools : A multidimensional needs assessment framework

Public Law 94–142 requires that public school districts provide special service programs to handicapped and potentially handicapped children with diverse psychological, educational, and physical needs. In order for such programs to be developed and implemented in an appropriate and timely manner, procedures for assessment of pupil needs as well as assessment of school district resources that might address those needs, have been considered as important service delivery prerequisites. Toward that end, a Multidimensional Needs Assessment Framework, which has been developed by the authors and used in numerous public schools, is described in this paper. The Framework, which allows needs assessment to be seen as an integral part of the program planning process, requires that needs assessment and program planning activities occur in relation to three distinct, yet interrelated dimensions: Special Service Recipients, Special Service Resources, and Organizational Unit. Considerations necessary for design and implementation of needs assessment using the Framework are discussed, and suggestions for further development of the approach are noted.     

Updating the FORECAST formative evaluation approach and some implications for ameliorating theory failure,implementation failure,and evaluation failure

Historically, there has been considerable variability in how formative evaluation has been conceptualized and practiced. FORmative Evaluation Consultation And Systems Technique (FORECAST) is a formative evaluation approach that develops a set of models and processes that can be used across settings and times, while allowing for local adaptations and innovations. FORECAST integrates specific models and tools to improve limitations in program theory, implementation, and evaluation. In the period since its initial use in a federally funded community prevention project in the early 1990s, evaluators have incorporated important formative evaluation innovations into FORECAST, including the integration of feedback loops and proximal outcome evaluation. In addition, FORECAST has been applied in a randomized community research trial. In this article, we describe updates to FORECAST and the implications of FORECAST for ameliorating failures in program theory, implementation, and evaluation.     

The promotion of participatory methods for group events in Zambia: a process evaluation report for unicef 1994–1997

Beginning in early 1994, UNICEF Zambia made a commitment to emphasize the use of participatory methods in conducting its business. Specifically, this has meant adopting and experimenting with a participatory methodology named VIPP (Visualization in Participatory Programmes) which had been used in a wide variety of group events, ranging from planning and review meetings to training and team building. Three years later, a process evaluation was commissioned to examine the question: How has this initiative been accepted and what effect is it having on UNICEF-supported programmes in Zambia?Employing a widely-utilized framework for evaluating training effectiveness, the evaluation team traced the history of VIPP implementation in Zambia through document reviews and interviews. In addition, the evaluators collected data from three different programme sites in Zambia where VIPP had been used. Several participatory evaluation techniques were used to conduct this evaluation, the first exercise of its kind for assessing VIPP application in Africa.Through this evaluation exercise, the evaluation team has gained knowledge about the strategic benefits and application of VIPP that will help with future planning for VIPP in Zambia and that will also inform other UNICEF offices and organizations about the experience of implementing a participatory methodology for programme and organisational purposes. © 1998 Elsevier Science Ltd. All rights reserved.     

Zimbabwe's AIDS action programme for schools

AIDS education started very early in Zimbabwe and is now compulsory in all schools. This evaluation documents the development of the primary and secondary school AIDS Action Programme from 1991 to 1998. Six programme aspects were evaluated: programme start-up, planning and management, development of syllabi and materials, teacher training, research, monitoring and evaluation, and co-ordination. The Programme drew on resources from within the existing educational system. Due to broad-based consultation and participation, it was supported by Government and partners. Flexible management ensured implementation of mid-course corrections. It was concluded that school AIDS programmes should stress participatory teaching and learning methods and life-skills training. Curriculum writers and teachers needed training and supervision in participatory techniques. Cluster workshops between district and school levels were needed to strengthen the cascade model of teacher training that had been adopted. Systematic research, monitoring and evaluation were essential, including follow-ups to baseline surveys. The Zimbabwean programme was well-designed, relatively inexpensive and replicable.     

Demystifying reflective practice: Using the DATA model to enhance evaluators’ professional activities

Reflective practice (RP), one of six essential competency domains in evaluation identified by Stevahn, King, Ghere, and Minnema (2005), refers to thinking critically about one's evaluation practice, alone or with other people, and using critical insights to improve one's practice. Currently, evaluators have minimal guidance in navigating this essential professional competency, professed to be a necessary part of their practice. This article focuses on how RP can serve as a tool for evaluators through the use of the “DATA” integrated RP framework, developed by Peters, 1991, Peters, 2009. DATA is an acronym with each letter standing for a different step in the process of reflective practice. The “D” step of the acronym focuses on (D)escribing what is or has been happening in practice. The “A” step refers to (A)nalyzing the current state of practice—why is this happening the way it is? The “T” concentrates on a practice-oriented form of (T)heorizing, which comes from analysis and serves as a basis for the resulting (A)ct. The last “A” focuses on the specifics of an action plan to change one's evaluation practice in light of the practical theory developed through theorizing. This paper describes the DATA model and introduces the application of the framework in a practice context.     

SPECIAL EDUCATION OR RACIAL SEGREGATION: Understanding Variation in the Representation of Black Students in Educable Mentally Handicapped Programs

The disproportionate representation of black students in special education programs has been well documented, yet explanations for the overrepresentation are rare. Using a unique sample of U.S. public school districts (N = 981), this article examines the effects of local racial and political-economic structures, school district characteristics, and school desegregation politics on the representation of black students in educable mentally handicapped (EMH) programs. The findings demonstrate that a minority presence in the school district and economic resources and black political resources in the community as well as various dimensions of school desegregation politics are associated with the representation of black students in EMH programs. This research provides further evidence that racial differences in educational placement are related to structures of opportunity in school districts and the communities they serve and that the placement process should be the focus of continued study.     

Expanding evaluator competency research: Exploring competencies for program evaluation using the context of non-formal education

The overlap of competencies between general program evaluation and specific contexts or content will always be reality because evaluators may need unique competencies to answer evaluation questions for particular contexts or content areas. Limited research exists that explores the essential competencies required by professionals who use evaluation as one part of their job portfolio, which leaves unanswered questions regarding the applicability of current evaluator competency models in such settings. We used a modified three-round Delphi technique to identify evaluator competencies for non-formal educators in Cooperative Extension (CE). Our panelists identified 36 competencies in the non-formal educational programming context for CE educators that they considered important to be included in evaluation capacity building efforts. We categorized our 36 identified competencies from the Delphi study into the five competency domains proposed by the American Evaluation Association. Our findings provide information to help guide professional development among non-formal educators related to program evaluation.     

Stakeholder perceptions of contextual factors related to PBIS implementation in high need schools

Positive behavior intervention and supports (PBIS) is a thoroughly researched three-tiered framework for preventing and treating challenging behavior in schools. While more than 20 years of research contribute to the literature base, less is known about PBIS implementation in high-needs school settings, particularly those that serve students predominantly from low-income families and majority minority student populations across urban and rural districts. This study investigated the critical contextual factors of PBIS implementation from the stakeholder perspective. Specifically, we surveyed 161 participants including teachers, school staff members, and administrators who were PBIS team members and had received technical assistance in PBIS implementation. The completed surveys were analyzed with participants and their schools coded for district type, Title I funding, and position in order to determine how the perceptions of issues such as buy-in, administrator support, PBIS in secondary school settings, and poverty affect PBIS implementation and team needs. The factor analysis and resulting structure along with statistically significant results for each factor are presented along with implications.     

AN EMPIRICAL TEST OF THE MEDIAN VOTER MODEL

This paper uses data at the school district level for 257 Michigan school districts to estimate the Bowen equilibrium level of expenditures in each district. The estimation uses a theoretical model of millage referenda that is developed in the paper. Then, the estimated Bowen equilibrium level of expenditures in each district is compared with the actual level. In the average district, the actual level of expenditures is less than 3 percent away from the estimated Bowen equilibrium. This evidence strongly suggests that, for educational expenditures at least, the theoretical median voter model provides a good explanation of empirical reality.     

Assessing Competency in A Specialized Program

This paper discusses the results of fusing different social work education trends in planning a relevant curriculum. These trends include a focus on practice-specific competencies, specialization of graduate programs, and the emerging role of students as consumers and evaluators of their educational experiences. Recommendations for designing processes of evaluation are offered.     

上海市宝山区“十二五”经济社会发展思路的若干思考

“十一五”以来,上海市宝山区自觉实践科学发展观,围绕“经济实力更强、城乡环境更美、人民生活更好、社会更加和谐”的目标任务,调结构、稳增长、保民生、促和谐,协调推进经济社会发展,“十一五”规划确定的各项目标任务执行情况总体良好,“一带、两轴、三分区”的现代化滨江新城区框架已初步形成。在综合分析“十二五”期间所面临的各方面因素的基础上,提出上海宝山区“十二五”发展的基本思路和主要任务,加快“国际大都市现代城区”发展,并全面阐述相应的对策措施。     

Competing Cultural Narratives: Gender and the Representation of US K–12 Teachers in Contemporary School Reform Policies

Cultural narratives about the proper scope and focus of teaching are embedded in contemporary school reform policies. This review examines literature related to two competing cultural narratives about US primary and secondary teachers: that “good teachers” are autonomous saviors, defined by their abilities to act independently and against great odds to improve academic outcomes for low‐income and minority students, and that “good teachers” are disempowered technicians who follow the guidance of externally‐recognized experts in their efforts to reduce educational inequalities. A review of literature critiquing these narratives finds that scholars have often analyzed these narratives using theoretical frameworks associated with race, class, and/or neoliberalism. This review examines what historians of education and feminist scholars can contribute to a critical analysis of the representation of US teachers in political speech and popular culture. It demonstrates that gender, as part of an intersectional approach, is important to understanding how White middle‐class women teachers can be positioned simultaneously as “autonomous saviors” and as “disempowered technicians” and how these narratives influence the professional status and autonomy associated with the work of teaching.     

Evaluating multicultural curricula in the community college: A sociocultural framework

This paper addresses the need for models for assessing multicultural programs in the community college, the most culturally and socioeconomically diverse educational institutions in the country. A three-dimensional framework presents faculty, student, and curriculum variables critical to the implementation and outcomes of multicultural programs. The framework emerged from the formative evaluation of a new interdisciplinary social science curriculum and guided the design of the national field test of that curriculum in 30 community college classrooms. Three kinds of results are reported: implementation patterns; appropriateness to faculty members' teaching goals; and impact on reading behavior, interest, overall learning, and political efficacy of students with diverse ages, ethnic and sociocultural backgrounds, and political positions. Political efficacy gains of older students and students with lower socioeconomic backgrounds are discussed. The importance of such a framework in documenting the interaction between a curriculum and its sociocultural context is stressed.     

A contingent approach to strategy and tactics in formative and summative evaluations

A contingent approach to strategic and tactical choices in formative and summative evaluations is presented. Choices regarding evaluation phases and tactics were found to be dependent on evaluation goals (e.g., whether the evaluation has a formative or summative purpose) and contextual variables. A simulation was designed to pool the judgments of evaluators and users of evaluations to see how they felt evaluation strategy and tactics should change as the situation changes so as to increase the likelihood of evaluation success. Eight panels were constructed to examine eight different situations. The next step in this research is to determine whether following the pooled judgments does lead to greater evaluation success.     

Theory-based evaluation of a comprehensive Latino education initiative: an interactive evaluation approach

Latino student access to higher education has received significant national attention in recent years. This article describes a theory-based evaluation approach used with ENLACE of Hillsborough, a 5-year project funded by the W.K. Kellogg Foundation for the purpose of increasing Latino student graduation from high school and college. Theory-based evaluation guided planning, implementation as well as evaluation through the process of developing consensus on the Latino population of focus, adoption of culturally appropriate principles and values to guide the project, and identification of strategies to reach, engage, and impact outcomes for Latino students and their families. The approach included interactive development of logic models that focused the scope of interventions and guided evaluation designs for addressing three stages of the initiative. Challenges and opportunities created by the approach are discussed, as well as ways in which the initiative impacted Latino students and collaborating educational institutions.     

Sociology of School Boards: A Social Capital Perspective

School boards have received little attention in sociology of education. Drawing on organizational sociology, a model of school board social capital is proposed that treats brokerage (external ties) and closure (internal ties) as the key dimensions of board functioning. The model brings together insights from limited research on school boards concerning members' internal and external ties and recasts them as the building blocks of a unifying theoretical framework. It also proposes specific performance outcomes at the district level. The model is tested using the data from a representative sample of Pennsylvania districts. The analysis not only explores brokerage and closure patterns among school boards, but also examines their implications for academic outcomes. Findings suggest that eighth grade reading and mathematics performance are highest in districts, where boards exhibit high levels of both brokerage and closure, and lowest in districts, where brokerage and closure are both low.