Self-evaluation of assessment programs: a cross-case analysis

The goal of this article is to contribute to the validation of a self-evaluation method, which can be used by schools to evaluate the quality of their Competence Assessment Program (CAP). The outcomes of the self-evaluations of two schools are systematically compared: a novice school with little experience in competence-based education and assessment, and an innovative school with extensive experience. The self-evaluation was based on 12 quality criteria for CAPs, including both validity and reliability, and criteria stressing the importance of the formative function of assessment, such as meaningfulness and educational consequences. In each school, teachers, management and examination board participated. Results show that the two schools use different approaches to assure assessment quality. The innovative school seems to be more aware of its own strengths and weaknesses, to have a more positive attitude towards teachers, students, and educational innovations, and to explicitly involve stakeholders (i.e., teachers, students, and the work field) in their assessments. This school also had a more explicit vision of the goal of competence-based education and could design its assessments in accordance with these goals.     

Model CBAs and Community Benefits Ordinances as Tools for Negotiating Equitable Development: Three Critical Cases

Community benefits agreements (CBAs) allow coalitions of labor and grassroots organizations to negotiate for concessions in the development process. One goal of a CBA is to more broadly distribute the benefits of new development to distressed communities and historically disenfranchised groups. Since their emergence in the 1990s, CBAs have not been well institutionalized. To date, there have been roughly 30 CBAs adopted across the United States. A relatively new development in the CBA movement has been the push to adopt model CBAs and community benefits ordinances, that outline all of the linkages that local government expects a developer to include in a project and require developers to negotiate linked development agreements as part of the project approval process. Although the emergence of model CBAs and CBA ordinances is promising for local groups interested in promoting equitable development, little research has been done on them. This article examines three critical cases in the United States (Portland, OR; Cleveland, OH; Detroit, MI) and evaluates the degree to which they serve as tools to institutionalize equitable development in the urban planning process. This analysis is based on content analysis and semi-structured interviews with key stakeholders.     

Evaluation of competence-based teaching in higher education: From theory to practice

Competence-based teaching in higher education institutions and its evaluation have become a prevalent topic especially in the European Union. However, evaluation instruments are often limited, for example to single student competencies or specific elements of the teaching process. The present paper provides a more comprehensive evaluation concept that contributes to sustainable improvement of competence-based teaching in higher education institutions. The evaluation concept considers competence research developments as well as the participatory evaluation approach. The evaluation concept consists of three stages. The first stage evaluates whether the competencies students are supposed to acquire within the curriculum (ideal situation) are well defined. The second stage evaluates the teaching process and the competencies students have actually acquired (real situation). The third stage evaluates concrete aspects of the teaching process. Additionally, an implementation strategy is introduced to support the transfer from the theoretical evaluation concept to practice. The evaluation concept and its implementation strategy are designed for internal evaluations in higher education and primarily address higher education institutions that have already developed and conducted a competence-based curriculum.     

The Interaction Between Role Structures and Self-Conceptions in Fostering Different Types of Self-Esteem in Innovative and Traditional Elementary Schools*

A two-dimensional model of inner and outer self-esteem is discussed wherein inner self-esteem is hypothesized to be an outcome of self-evaluations of performance in prominent role identities. Outer self-esteem is hypothesized to be related to self-evaluations of both prominent and nonprominent role identities. The research context is a comparison of the different processes facilitating self-esteem in innovative schools (emphasis on student-initiated activities and control) and traditional (emphasis on pre-determined curriculum and teacher-administrator control). Data across school type demonstrate a modest correlation between positive self-evaluations of performance in prominent role identities and both types of self-esteem. The data suggest that high evaluation of performance in nonprominent role identities is related to outer self-esteem but not to inner self-esteem. Further empirical clarification of the concepts was established by viewing the relationships within school type. In the innovative schools both types of self-esteem were predominantly related to positive evaluations of prominent role identities, whereas in the traditional schools both types of self-esteem were related to both performance in prominent and nonprominent role identities. Thus the theoretical distinction gained partial support while the process of gaining self-esteem was shown to be essentially different between alternate school structures. Controls for social desirability, race, sex, and father's occupation did not change these findings.     

Attraktivität, Reziprozität und Lehrveranstaltungsevaluation

In this article we investigate the relationship between academic teachers’ attractiveness and their students’ evaluations of teaching. Furthermore it is examined, whether teachers’ grading influences students’ evaluations of teaching. Using data from an experimental design, hypotheses about attractiveness effects, frog-pond-effects, moderating gender effects and influences of grading are empirically tested. Controlling for confounding factors we find a very weak attractiveness effect, which can partly, but not only be ascribed to beauty’s productivity enhancing influence. Thereby, independently from the raters’ sex, female teachers profit from their looks. In contrast to that attractive male teachers are even sanctioned more strongly by their students for difficult tests than unattractive teachers. The frog-pond-hypothesis cannot empirically be supported: the attractiveness of teachers’ colleagues has no influence on the attractiveness effect. However, reciprocal evaluation behavior can consistently be found in the theoretical expected manner. It is substantially more relevant for students’ evaluations of teaching than attractiveness effects are. Alternative explanations for this grading effect can be excluded.     

Mein(schöner)Prof.de

Evaluations of academic teaching have become an instrument for the quality management in universities of constantly growing importance. This process is unproblematic as long as the quality of teaching is rated validly by the evaluations of students. The validity of the rating, however, becomes questionable as an influence of the teaching staffs’ physical attractiveness on the evaluation of teaching has been proved. This effect would only be compatible with an assumed valid rating of the teaching performance if attractive teachers would in fact show a better performance in academic teaching and are not just better rated for their attractiveness. We analyzed data from the German online platform MeinProf.de for the evaluation of academic teachers. The results show that attractive teachers very probably do not perform better. The attractiveness of female teachers does not have the expected influence on the evaluation of their teaching.     

Negotiating With the Growth Machine: Community Benefits Agreements and Value‐Conscious Growth

Scholars have extended, challenged, and molded growth machine theory to examine growth—in terms of population, basic industry, labor force participation, commerce, financial activity, and land development—in a variety of contexts. The theory's core, however, has remained the same: cities are conceptualized as growth machines, which consist of unified and powerful growth coalitions. These coalitions pursue a pro‐growth agenda, seeking to enhance the exchange value of local land and property. They often face opposition from local residents, who are more oriented toward use values of land. Resident opposition, however, tends to be unsuccessful in the face of large‐scale commercial development. Aware of this, communities across the country are pursuing new strategies to address development projects in their backyards. In particular, some have formed coalitions to negotiate for benefits from developers through legally binding community benefits agreements (CBAs). Drawing from a case study of Pittsburgh's first CBA, this article analyzes the implications of CBAs for pro‐growth agendas. Pittsburgh's CBA surrounded the construction of a professional sports facility, a development project that presents an ideal example of growth processes in today's cities. Ultimately, CBAs can achieve “value‐conscious” growth, but they do not fundamentally alter dominant standards of growth or growth machine processes.     

Learning about fishery management: Evaluation of a contextualized responsive evaluation approach

This article discusses the extent to which a responsive evaluation (RE) approach contributed to learning by stakeholders in a case of high complexity. Fishery management in Grand-Popo, Benin is characterized by ambiguity, that is contrasting views among fishery stakeholders about what should be done, why, how, where, and when to resolve fishery problems like the depletion of fish-stock and absence of income alternatives. It was also characterized by great gaps (mismatches) between interventionists’ plans and actions, despite generations of interventions and evaluations of their effectiveness. The RE approach aimed at facilitating interactions between interventionists and fishing people to stimulate learning and hence reduce the ambiguity and mismatches. In this article, we take distance and evaluate the results of this action research approach. We found that in the interaction some learning indeed occurred. The fishing people learned among others that intervention resources are limited and that they should organize themselves to lobby for and monitor interventions to solve their problems. Interventionists learned that they could share knowledge about their roles and limited resources with fishing people so that the latter could lobby for more resources. Fishing people however, did not learn to adopt more sustainable fishing practices. Also, interventionists did not learn to influence politicians and financial partners themselves for sufficient resources. Both categories of stakeholders developed ideas for how to collaborate to improve fishery management. We conclude that although some single-loop, double-loop and social learning occurred, the learning was limited and reflect on the related challenges for RE in natural resource management.     

Planning an integrated agriculture and health program and designing its evaluation: Experience from Western Kenya

Multi-sectoral programs that involve stakeholders in agriculture, nutrition and health care are essential for responding to nutrition problems such as vitamin A deficiency among pregnant and lactating women and their infants in many poor areas of lower income countries. Yet planning such multi-sectoral programs and designing appropriate evaluations, to respond to different disciplinary cultures of evidence, remain a challenge. We describe the context, program development process, and evaluation design of the Mama SASHA project (Sweetpotato Action for Security and Health in Africa) which promoted production and consumption of a bio-fortified, orange-fleshed sweetpotato (OFSP). In planning the program we drew upon information from needs assessments, stakeholder consultations, and a first round of the implementation evaluation of a pilot project. The multi-disciplinary team worked with partner organizations to develop a program theory of change and an impact pathway which identified aspects of the program that would be monitored and established evaluation methods. Responding to the growing demand for greater rigour in impact evaluations, we carried out quasi-experimental allocation by health facility catchment area, repeat village surveys for assessment of change in intervention and control areas, and longitudinal tracking of individual mother-child pairs. Mid-course corrections in program implementation were informed by program monitoring, regular feedback from implementers and partners’ meetings. To assess economic efficiency and provide evidence for scaling we collected data on resources used and project expenses. Managing the multi-sectoral program and the mixed methods evaluation involved bargaining and trade-offs that were deemed essential to respond to the array of stakeholders, program funders and disciplines involved.     

高职教育中能力本位精品课程建设探析——以无机化学课程建设为例

高职院校精品课程的建设,目的在于提高教学质量,使培养的人才更适合行业企业发展的需要。文章以乌海职业技术学院无机化学精品课程建设为例,从岗位群调研与分析、师资队伍建设、修订教学大纲,进行一体化教学,对课程进行精心设计、建立立体化考核机制等几方面探析了能力本位精品课程的建设。     

职业技能大赛对国贸专业教学改革的影响及作用研究

职业技能大赛对国贸专业人才培养模式的定位、课程体系的设置、师资队伍的建设、课程内容的选取、教学方法与手段的选择、考核体系的评价产生了深远的影响,同时职业技能大赛培养了学生的竞争意识、团队协作能力、创新意识等综合素质。     

Blazing an evaluation pathway: Lessons learned from applying utilization-focused evaluation to a conservation education program

In 2005, state fish and wildlife agency wanted to examine how one of its conservation education programs was providing science-based understanding and outdoor experiences by evaluating students’ knowledge, skills, attitudes and intended behavioral outcomes related to fish, fishing and aquatic habitats in Montana. A key factor in this study was the acceptance by program stakeholders to conduct the evaluation using a utilization-focused evaluation approach to promote usability and accuracy of evaluation results. Using a quasi-experimental non-equivalent group design, more than 2000 students in participating classrooms throughout Montana received a pre-survey, post-survey and an extended post-survey; 114 teachers participated in an Internet survey and 16 program instructors took part in a structured open-ended telephone interview. The participatory approach and mixed methods enhanced abilities to interpret results of student surveys in particular. The user-focused approach was discovered to be personal and situational, allowed the facilitation of the evaluation process with consideration for increased application of evaluation findings and implementation of recommendations from beginning to end. Further development of evaluation pathways is needed to more effectively evaluate outcomes and implement practical and transferable measures to determine if environmental education activities produce desired participant outcomes.     

Evaluations of Women and Men Among Black and White College Students1

Rainwater's assertion that blacks have low self-evaluations because they receive more negative evaluations from other blacks than whites receive from other whites is challenged here by reference to Heiss and Owens evidence that negative self evaluation among blacks is limited to work-related traits. Substantial support was found for our major hypotheses that, compared to whites, (1) blacks would report more negative evaluation of “most men” but not of “most women” (since the provider role is traditionally ascribed primarily to men), and (2) the more negative evaluation of “most men” by blacks would be limited to work-related traits.     

The unique character of involvement in multi-site evaluation settings

As the number of large federal programs increases, so, too, does the need for a more complete understanding of how to conduct evaluations of such complex programs. The research literature has documented the benefits of stakeholder participation in smaller-scale program evaluations. However, given the scope and diversity of projects in multi-site program evaluations, traditional notions of participatory evaluation do not apply. The purpose of this research is to determine the ways in which stakeholders are involved in large-scale, multi-site STEM evaluations. This article describes the findings from a survey of 313 program leaders and evaluators and from follow-up interviews with 12 of these individuals.Findings from this study indicate that attendance at meetings and conferences, planning discussions within the project related to use of the program evaluation, and participation in data collection should be added to the list of activities that foster feelings of evaluation involvement among stakeholders. In addition, perceptions of involvement may vary according to breadth or depth of evaluation activities, but not always both. Overall, this study suggests that despite the contextual challenges of large, multi-site evaluations, it is feasible to build feelings of involvement among stakeholders.     

Evaluation of a cross-sector community initiative partnership: Delivering a local sport program

Corporate community initiatives (CCI) are often established via cross-sector partnerships with nonprofit agencies to address critical social problems. While there is a growing body of literature exploring the effectiveness and social impact of these partnerships, there is a limited evaluative research on the implementation and execution processes of CCIs. In this paper, we examined the implementation and operational processes in the delivery of a professional sport organization's CCI initiative using program theory evaluation. The findings showed discrepancies between the associate organization and the implementers regarding understanding and fulfilling responsibilities with performing certain aspects (maintaining accurate records and program marketing) of the service delivery protocol. Despite program stakeholders being satisfied overall with the program delivery, contradictions between program stakeholders’ satisfaction in the quality of program delivery was found in critical components (marketing and communications) of the service delivery. We conclude that ongoing evaluations are necessary to pinpoint the catalyst of the discrepancies along with all partners valuing process evaluation in addition to outcome evaluation.     

Monitoring and evaluation of Guinea Worm Eradication

Dracunculiasis, or guinea worm disease is well-known for a number of reasons: its spectacular clinical manifestations (the emergence of a near-meter long worm from a painful cutaneous blister), the severe disease outbreaks which paralyze entire villages, and the striking traditional treatment which consists of laboriously rolling out inch after inch of the worm round a wooden stick. In the early 1980s, following the success of smallpox eradication, dracunculiasis was identified as the next candidate for eradication, since it could be tackled through simple preventive measures. Today, a successful global programme of eradication is underway in several countries. To appreciate the challenges of monitoring and evaluating the interventions, an understanding of the disease, the epidemiological situation and programme interventions is necessary; these aspects are therefore discussed. Then, based on the West and Central African experience, this article goes on to examine programme planning approaches and the monitoring and evaluation systems that were developed to track progress towards the global goal of eradication, notably community surveillance systems, formative evaluations, geo-reference database and mapping tools. Such monitoring and evaluation systems have several innovative elements unique to guinea worm eradication programmes and adapted to the requirements imposed by disease transmission patterns. There is evidence that the final efforts to eradicate guinea worm will be more costly, will take much longer than expected, and will require adequate and timely support in order to deal with the setbacks.     

Identifying and developing effective approaches to foster intercultural understanding in schools

This paper provides a systematic review of education literature focused on identifying school-based approaches for developing students’ intercultural understanding. Studies were assessed using selection criteria and then critically appraised for study quality. A key finding from the review is that developing students’ intercultural understanding beyond cultural awareness requires students and teachers to take a critical approach toward cultural diversity, as well as the opportunity for ongoing intercultural and intergroup contact. Studies reported that only building cultural awareness and knowledge is not enough to promote long-term changes in attitudes. There is a need for more rigorously evaluated longitudinal school-based interventions. Finally, studies consistently call for investment in teachers’ professional and personal intercultural capabilities. The paper concludes by calling for school-based interventions that are informed by best practice approaches at a whole school level in order to effectively develop students’ intercultural attitudes and skills.     

Integrity and Presidential Approval, 1980-2000

Do individuals' assessments of the president's integrity consistentlyaffect their evaluations of his job performance? Previous researchsuggests that they might, but extant studies typically do notdirectly examine the effects of these assessments. Those thatdo have examined only a few time points, leaving the questionof whether integrity assessments consistently affect approvalacross presidencies unresolved. Further, they do not examinethe effects of integrity on Bill Clinton's approval after theLewinsky scandal, a time when many argued that integrity assessmentswere irrelevant to evaluations of his job performance. Thisstudy examines the effects of integrity assessments on approvalof four presidents using 10 surveys from 1980 to 2000. Integrityassessments are found to influence approval throughout the period,although the magnitude of their effect varies somewhat. Further,integrity assessments affected approval for Clinton in aboutthe same way they shaped approval for previous presidents. Althoughmost studies of presidential approval focus on the economy andforeign affairs, these results suggest that evaluations of thepresident's job performance are in part evaluations of the presidenthimself. However, more politically substantive factors exertfar greater influence on approval. Therefore, approval is tiedto far more than just good character, which reflects favorablyon the quality of the public's evaluations of their presidents.     

Evaluating Area‐based Interventions: The Case of ‘Communities for Children’

Increasingly, governments in wealthy countries are designing early intervention initiatives around principles of ‘community regeneration’ or ‘place management’. Because these initiatives are multi‐site, aimed at long‐term systemic change, and implemented amidst a range of other initiatives, assessing their quality and outcomes demands departure from conventional programme evaluation approaches. This article analyses the challenges of evaluating area‐based interventions in the child welfare field, and shows how the National Evaluation of Australia's ‘Communities for Children’ initiative seeks to overcome these through its mixed method design and the longitudinal Stronger Families in Australia study.     

Analysing facilitating and hindering factors for implementing gender equality interventions in R&I: Structures and processes

This article analyses the facilitating and hindering factors that have affected the implementation of gender equality interventions in research and innovation in Europe. It applies the evaluation framework developed in the EFFORTI project that recognizes the complexity of evaluating gender equality interventions in R&I, the importance of factoring in context to any sound evaluation as well as the need to distinguish between the design and implementation of interventions in evaluations. It is based on the analysis of 19 empirical case studies carried out throughout Europe and focuses on those structural and procedural factors that have either facilitated or hindered the implementation process of these interventions. Findings include how the governance framework; top-management commitment; bottom-up participation; framing synergies with other initiatives, strategies for tackling resistance; resources; sustainability of actions; gender competence, experience and knowledge and transparency, targets, standards and monitoring; and accessible data and information all contributed to the successful implementation of the interventions.