The African Evaluation Guidelines: 2002. A checklist to assist in planning evaluations, negotiating clear contracts, reviewing progress and ensuring adequate completion of an evaluation

A review of the US ‘program evaluation standards’ (PES), undertaken in a series of workshops and meetings of networks of evaluators in Africa, resulted in modifications to those standards. The result was presented to a plenary session of the Inaugural Conference of the African Evaluation Association in September 1999, attended by over 300 evaluators from 35 countries. The AfrEA Conference decided that a systematic effort should be made to produce a list of African evaluation guidelines, similar to the PES, and that this checklist should be reviewed by national evaluation associations and networks in Africa and field tested in several countries. Ten national and regional networks and associations suggested modifications to the text and endorsed the final version of the guidelines.     

Dealing with context in logic model development: Reflections from a realist evaluation of a community health worker programme in Nigeria

Logic models (LMs) have been used in programme evaluation for over four decades. Current debate questions the ability of logic modelling techniques to incorporate contextual factors into logic models. We share experience of developing a logic model within an ongoing realist evaluation which assesses the extent to which, and under what circumstances a community health workers (CHW) programme promotes access to maternity services in Nigeria. The article contributes to logic modelling debate by: i) reflecting on how other scholars captured context during LM development in theory-driven evaluations; and ii) explaining how we explored context during logic model development for realist evaluation of the CHW programme in Nigeria. Data collection methods that informed our logic model development included documents review, email discussions and teleconferences with programme stakeholders and a technical workshop with researchers to clarify programme goals and untangle relationships among programme elements. One of the most important findings is that, rather than being an end in itself, logic model development is an essential step for identifying initial hypotheses for tentative relevant contexts, mechanisms and outcomes (CMOs) and CMO configurations of how programmes produce change. The logic model also informed development of a methodology handbook that is guiding verification and consolidation of underlying programme theories.     

政府间与非政府间国际组织在全球环境治理中的有效性评估

全球环境治理是当今国际社会的重要议题,其治理主体呈现多元化趋势,治理模式也由原有的国家中心治理模式向有限治理模式、网络治理模式发展,呈现多方向的新变化。除国家政府之外,就国际组织而言可分成政府间国际组织(IGO)与非政府间国际组织(INGO)两种。文章将探究在全球环境治理问题上是否政府间国际组织可以替代非政府组织的作用,并以绿色和平与联合国环境规划署两大组织为例,从国际社会对全球环境治理体制的功能需求方面进行案例对比分析与治理有效性的评估。     

A formative evaluation of the recovery public works programme in Blantyre City,Malawi

Public works programmes (PWPs) are popular social protection instruments in the context of chronic poverty but very little has been published in the way of implementation and outcomes of these programmes. This paper presents a formative process and outcome evaluation of the recovery PWP in Blantyre City, Malawi. The evaluation used longitudinal household survey data of PWP beneficiaries, programme records and interview responses from programme staff and community leaders. Largely, the process evaluation findings showed an agreement between actual and planned activities. The outcome evaluation found indications that the PWP community assets offered some potential benefits to the communities, and that PWP wages allowed the beneficiaries to purchase some food. This however, did not translate into more meals per day, nor did the earnings prevent the decline in household assets as expected. Given a plausible PWP theory and high implementation fidelity, the PWP wage rate or number of days was either just enough to smooth participant income, or insufficient altogether, to enable achievement of more distal outcomes.     

Evaluation of the impact of intervention programmes on education organisations: Application to a Quality Management System

This paper analyses the importance of evaluating the various components of the programmes or actions carried out by education organisations. It highlights the need to assess the impact of the intervention on the organisation and consider how changes are consolidated over time in interaction with the context. We propose an impact evaluation model and as an example have chosen the implementation of Quality Management Systems in schools. The paper analyses the results obtained in 40 schools in three regions (Spanish Autonomous Communities) with varying levels of implementation. The results show overall impact on these education centres as the teachers and management teams of the centres perceive it. This impact is more evident in some of the dimensions considered in the study than in others. The results also confirm the differences between regional contexts.     

Transforming a municipal school sports programme through a critical communicative methodology: The role of the of advisory committee

This research applies a communicative methodology (CM) to the transformation and improvement of the Municipal Comprehensive School Sports Programme in Segovia, Spain (MCSSP), using egalitarian dialogue, based on validity rather than power claims to achieve intersubjectivity and arrive at consensus between all of the Programme’s stakeholders through the intervention of an advisory committee (AC). The AC is a body comprising representatives of all stakeholder groups involved in the programme. During the 2013–2014 academic year the programme’s AC met four times, operating as a communicative focus group (CFG). The meetings focused on: (1) excluding dimensions (barriers preventing transformation) and transforming dimensions (ways of overcoming barriers), (2) the programme’s strengths, (3) the programme’s weaknesses and specific actions to remedy them, and (4) the resulting conclusions which were then incorporated into the subsequent programme contract signed between the University and the Segovia Local Authority for 2014–2018. The key conclusions were: (1) the recommendations of the AC widen the range of perspectives and help the research team to make key decisions and (2) the use of CM to fully evaluate the programme and to reach a consensus on how to improve it proved very valuable.     

Some practical issues in the evaluation of heterogeneous labour market programmes by matching methods

Summary.  Recently several studies have analysed active labour market policies by using a recently proposed matching estimator for multiple programmes. Since there is only very limited practical experience with this estimator, this paper checks its sensitivity with respect to issues that are of practical importance in this kind of evaluation study. The estimator turns out to be fairly robust to several features that concern its implementation. Furthermore, the paper demonstrates that the matching approach per se is no panacea for solving all the problems of evaluation studies, but that its success depends critically on the information that is available in the data. Finally, a comparison with a bootstrap distribution provides some justification for using a simplified approximation of the distribution of the estimator that ignores its sequential nature.     

Efficacy of programmes for the unemployed: discrete time modelling of duration data from a matched-comparison study

An evaluation is described of two UK Government programmes for the long-term unemployed in Great Britain, Employment Training and Employment Action, using discrete time hazard modelling of event histories. The study design employed a closely matched comparison group and carefully chosen control variables to minimize the effect of selection bias on conclusions. The effect of unobserved heterogeneity is investigated by using some standard random effect model formulations.     

Teachers' role in the primary prevention of child abuse dilemmas and problems

During the 1980s, many school-based primary prevention programmes were developed and adopted by schools. Much of the early enthusiasm which greeted these programmes dissipated in the light of often disappointing evaluations of the impact of the programmes on children. It is argued, however, that evaluating student outcomes should not occur before theextent of programme utilization is established. In this paper, an overview is given on the dilemmas and problems faced by teachers attempting to implement a primary prevention programme in South Australia. It is revealed that teachers selectively omit sections of the primary prevention programme as a way of resolving personal dilemmas presented by the programme. Some teachers are sensitive about teaching personal and controversial issues related to child abuse, while others deny the seriousness of child abuse in theirschool. The implications of the findings are that: (a) it cannot be assumed that primary prevention programmes are implemented by teachers in ways that are consistent with programme design; (b) teachers' personal beliefs, attitudes and feelings need to be addressed in any school-based primary prevention initiative; (c) training approaches and school support mechanisms need to focus on methods by which teachers can resolve the dilemmas raised by school-based primary prevention programmes in ways other than by radically dismembering the programme.     

Analysing partial ranks by using smoothed paired comparison methods: an investigation of value orientation in Europe

Summary. This paper introduces the paired comparison model as a suitable approach for the analysis of partially ranked data. For example, the Inglehart index, collected in international social surveys to examine shifts in post-materialistic values, generates such data on a set of attitude items. However, current analysis methods have failed to account for the complex shifts in individual item values, or to incorporate subject covariates. The paired comparison model is thus developed to allow for covariate subject effects at the individual level, and a reparameterization allows the inclusion of smooth non-linear effects of continuous covariates. The Inglehart index collected in the 1993 International Social Science Programme survey is analysed, and complex non-linear changes of item values with age, level of education and religion are identified. The model proposed provides a powerful tool for social scientists.