A meta-evaluation, or quality assessment, of the evaluations in this issue, based on the African Evaluation Guidelines: 2002

Fourteen evaluations of UNICEF and UNAIDS HIV/AIDS programmes underwent a quality control process called ‘meta-evaluation’, which is an evaluation of the quality of the evaluation. This process of evaluating an evaluation is considered professional good practice by African Evaluation Associations and Networks, as well as the American, Canadian, German, Korean, Swiss and other national professional associations of evaluators. The quality control criteria used were the African Evaluation Guidelines—a list of 30 quality-enhancing standards that can be used as a checklist by evaluators to ensure that evaluation studies are performed well. These quality control criteria can also be applied retrospectively as a quality control method, to assess completed evaluation studies. The results of this quality control review process, for the 14 papers in this issue of Evaluation and Program Planning, are presented and discussed. An indication of a possible area of weakness in HIV/AIDS programme evaluation emerges from this analysis.     

Evaluation in South Africa: An introduction

This article provides a brief background to the articles in this edition on evaluation in South Africa. The overall background is self-evident: South Africa is a country which has undergone a political revolution, with a government of national unity in place. The formulation and implementation of new policies, and the programs which form part of them, will be an important part of South African politics in the years to come.These articles also appear against the more immediate background of trying to place evaluation on a surer footing in this country. In this introduction the efforts of a group of South African evaluators to position themselves and their activities in such a way as to make evaluation an accepted part of the organizational landscape, and to play a useful role in a future South Africa, are described as well.     

Unsettling connections: transnational networks,development and African home associations

With the transnational turn in the social sciences attention has now turned to ‘global civil society’, ‘transnational civil society’, ‘transnational networks’ and, most recently, ‘migrant’ or ‘diasporic civil society’. Claims are being made about the developmental potential of these new configurations of civil society, and the global connections forged by migrant and diaspora associational life have been reified into things called ‘networks’ for the purpose of enrolling them into development policy. In this article, we challenge the network model through an analysis of transnational Cameroonian and Tanzanian home associations. The idea of a network suggests an overly robust and ordered set of linkages for what are in effect often loose and transient connections. African home associations draw attention to the historically‐embedded and mundane ways in which forms of associational life can be ‘transnational’ outside the formalized structures and Eurocentric development hierarchies created by international NGOs and other development institutions. Although they form largely invisible connections operating outside these hierarchies, African home associations unsettle assumptions about the geography of civil society and its relationship with development. Close attention to the histories and geographies of African home associations reveals that power and agency more often lie with migrants and elites within Africa than with the transnational diaspora.     

Evaluation of the Integrated Services Pilot Program from Western Australia

Independent evaluation of refugee-focused programs in developed nations is increasingly a mandatory requirement of funding bodies and government agencies. This paper presents an evaluation of the Integrated Services Centre (ISC) Pilot Project that was conducted in Australia in 2007 and early 2008. The purpose of the ISC program was to provide integrated support to humanitarian refugees in settlement, physical health, mental health and employment. The Pilot Project was based in two primary schools in Perth, Western Australia. The evaluation utilized a flexible qualitative ‘engaged’ methodology and included interviews, focus groups and telephone interviews with the key stakeholders, project staff and a small number of refugee families. The strength of the qualitative methodology (including data that is narrative rich) is that it highlights issues as perceived by each stakeholder and provides insights into the daily work by ISC staff that helped to uncover unintended outcomes. Despite the fact that the ISC evaluation was supposed to be a ‘before and after’ design, the researchers acknowledge a common weakness in many evaluations (including the ISC) that when baseline data is required, evaluators are recruited after the project has begun. This issue is discussed in the paper. It is critical that independent evaluators are able to begin collecting baseline data as soon as programs are launched, if not before.     

Practices and challenges in educational program evaluation in the Asia-Pacific region: results of a Delphi study

While educational program evaluation has become more important in recent years because of increasing governmental demands for accountability, little is known about the development of and issues in regard to this topic in the Asia-Pacific region. The findings from a Delphi study conducted in a number of relevant countries are provided in this article. Thirty panelists participated in three rounds of iterations to identify current status, challenges, and concerns in the near future. The results should be of value to educational evaluators, researchers, and decision makers.     

Access to higher education for disabled students in South Africa: a contradictory conjuncture of benevolence, rights and the social model of disability

The structures of support services for disabled students in the South African higher education system find themselves in a contradictory conjuncture of rights, benevolence and the social model of disability. To elucidate this argument, this paper (a) outlines the status of support provisions for disabled students in South Africa; (b) compares the state of these support provisions with those of the UK and the USA; (c) compares the different paths taken by South Africa and the developed countries in general towards disability rights. It concludes that South Africa seems to be moving along a contradictory path and that it should make a commitment to prioritize equal access to higher education for disabled students.     

Community-Based Program Research

ABSTRACT

At first glance, the worlds of program professionals and evaluators may seem quite separate. However, there are common issues whose resolution will enhance both program development and research on programs. Program professionals and evaluators have a great deal to learn from each other. As that learning occurs, both program delivery and evaluation research will benefit. Both are concerned about matters of validity, whether it pertains to the nature of the program intervention itself or to the nature of the assessment of the program intervention. To reinforce the development of program evaluation in partnership, this paper discusses key points about evaluation research relevant to both program professionals and researchers. These include the contextual influences on a program, the “readiness” of a program for evaluation, and whether the evaluation research “works” for the program. Underlying our discussion is a call for the development of partnerships around research on programs. In that light, this article elaborates the process steps that should be taken to build program evaluation partnerships, including a discussion of what evaluators and program professionals need to know about each other, and their respective values, interests, and professional perspectives.     

Towards migration research networking in Eastern-Southern African subregions

This article reports efforts made by a small group of Eastern-Southern African (ESA) subregion scholars to adopt a systematic approach to establishing a regional network Migration Network in Eastern and Southern Africa (MINESA). The approach involved: 1) holding a conference at which symptomatic types of internal and international migration would be discussed; 2) publication of the conference proceedings; and 3) establishment of MINESA as a network of policy-oriented research in the two subregions. The first stage has been accomplished, the second is nearly complete, and the third has yet to be undertaken. During the African Population Conference organized by the International Union for Scientific Study of Population in Dakar, Senegal, on 5-9 November 1988, a small group agreed on a timetable to establish MINESA. At the ESA conference, papers were presented on ESA issues; internal migration processes and mechanism; refugee movements and their implications for countries; the effects on the economies of Southern African states, of emigration to the Republic of South Africa (RSA). In a keynote address, Adepoju surveyed migration and development in Western-Central (Middle) Africa and Eastern-Southern Africa, which included colonial and post-colonial historical epochs, internal and international migration, and labor and refugee movements. A paper on Kenya by Oucho discussed the implications for rural-urban balance of internal migration based on 1969 and 1979 censuses. Rural-urban migration from the traditional economy to Nairobi and Mombasa in particular has created an unacceptable rural-urban imbalance, adversely affecting rural development. Eastern and Southern Africa has seen massive and wide spatial dispersal of refugees (victims of wars, drought, and famine). Two papers were presented on Tanzania and one on Uganda. The final set of papers addressed the effects of labor migration to the RSA on Swaziland and Lesotho.     

U.S. firms and black labor in South Africa: Creating a structure for change

Some have argued that U.S. firms should disinvest from South Africa as a means of putting pressure on the South African government to end apartheid. This argument, however, may ignore a dynamic of change in South Africa in which U.S. firms have played and are playing a large role: South African industrial relations have undergone significant evolution since 1979, the year in which major changes were introduced in that country’s labor legislation. Partly a result of these changes, black trade union membership has increased by 800 percent since 1979. By virtue of its unique exposure to varied constituencies, the multinational subsidiary in both home and host countries can be viewed as an agent of sociopolitical change in South Africa. The author wishes to thank Professors Herbert R. Northrup and Richard L. Rowan for helpful discussion.     

Mixed-methods library evaluation integrating the patron,library, and external perspectives: The case of Namibia regional libraries

The article presents a mixed-methods evaluation of regional libraries in Namibia, which incorporates three perspectives: the patron perspective (library users), the library perspective (library staff, management, and related officials), and the external perspective (including evaluators and monitoring data). Seven data collection methods were used: patron surveys, patron panel studies, focus group discussions, key informant and staff interviews, secondary data analysis, media analysis, and observations. The goal of the evaluation was to assess library performance for both formative and summative purposes by addressing evaluation questions on areas such as library services, use, and operations. Building upon the literature review of how mixed-methods approaches can contribute to library evaluation, the aim of this article is to show how a mixed-methods evaluation can be designed to examine multi-faceted library performance and to illustrate how the evaluation design allows information complementarity and can be utilized to present diverse viewpoints of the above three perspectives. The evaluation design, analysis process, and lessons learned from this study may be useful to evaluators engaged in evaluation of public services or programs (including public libraries) that examine multiple aspects of service performance and involve a variety of stakeholders.     

Evaluating development programs: building joint activity

There has been an increasing plea for a participatory approach to development programs. It is argued that for programs to be sustainable, they need to be conceptualized, negotiated and run by all stakeholders jointly. Concurrent with this trend, there has been an increasing demand for program evaluation strategies, which encompass an evaluation of the nature and level of participation of the different stakeholders in the program. This paper proposes a theoretical framework for such evaluation and subsequently provides a practical illustration. Engeström's notions of ‘activity system’ and ‘learning by expanding’, are proposed as a framework for the evaluation of participation and an action evaluation methodology is suggested as the practical tool within such framework. The paper presents a case study of a conservation education program in Africa, which operationalizes the proposed theoretical framework.     

Defining,illustrating and reflecting on logic analysis with an example from a professional development program

Program designers and evaluators should make a point of testing the validity of a program's intervention theory before investing either in implementation or in any type of evaluation. In this context, logic analysis can be a particularly useful option, since it can be used to test the plausibility of a program's intervention theory using scientific knowledge. Professional development in public health is one field among several that would truly benefit from logic analysis, as it appears to be generally lacking in theorization and evaluation. This article presents the application of this analysis method to an innovative public health professional development program, the Health Promotion Laboratory. More specifically, this paper aims to (1) define the logic analysis approach and differentiate it from similar evaluative methods; (2) illustrate the application of this method by a concrete example (logic analysis of a professional development program); and (3) reflect on the requirements of each phase of logic analysis, as well as on the advantages and disadvantages of such an evaluation method. Using logic analysis to evaluate the Health Promotion Laboratory showed that, generally speaking, the program's intervention theory appeared to have been well designed. By testing and critically discussing logic analysis, this article also contributes to further improving and clarifying the method.     

Reflections on the Green Growth Index for developing countries: A focus of selected African countries

This article presents a pilot and baseline African Green Growth Index (AGGI). The work is premised on the importance of Africa implementing green growth strategies. Baseline indicators allow countries to monitor progress towards green growth transition. The AGGI incorporates 48 indicators applied to 22 countries that had the requisite data. What emerges is that 18 out of the 22 African countries sampled scored 50 percentage points and above. The countries scoring below this threshold were: Egypt, Algeria, Nigeria and South Africa. The top five countries on the AGGI (as ranked from 1−5) were: Namibia, Zambia, Ghana, Tanzania and Togo. The authors recommend that for wider acceptance across the continent, the AGGI should work towards incorporating all 54 nations.     

Time matters: the potential and pitfalls of using mixed methods approaches in longitudinal program evaluation

While social programs are often assessed using short-term impact studies, longitudinal designs allow evaluators to capture change over time, identify longer-term outcomes, adapt instruments, and better understand participants in transition. A mixed methods design can be critical in understanding these dynamics; yet there is a lack of literature exploring the practical considerations of planning and conducting qualitative and quantitative data collection and analysis within longitudinal studies. This paper examines two different mixed methods frameworks used in a 5-year evaluation of three youth entrepreneurship programs in East Africa. We show how the evaluation team dealt with unique challenges across methods and over time, and how the design ultimately facilitated a richer understanding of program impacts and processes. Considerations for conducting this type of study are explored, related to the impact of longevity on analysis and research staff. Successfully using a longitudinal mixed methods approach requires researchers to be strategic and reflexive, and work in close collaboration.     

Relational responsibilities in responsive evaluation

This article explores how we can enhance our understanding of the moral responsibilities in daily, plural practices of responsive evaluation. It introduces an interpretive framework for understanding the moral aspects of evaluation practice. The framework supports responsive evaluators to better understand and handle their moral responsibilities. A case is introduced to illustrate our argument.Responsive evaluation contributes to the design and implementation of policy by working with stakeholders and coordinating the evaluation process as a relationally responsible practice. Responsive evaluation entails a democratic process in which the evaluator fosters and enters a partnership with stakeholders. The responsibilities of an evaluator generally involve issues such as ‘confidentiality’, ‘accountability’ and ‘privacy’. The responsive evaluator has specific responsibilities, for example to include stakeholders and vulnerable groups and to foster an ongoing dialogue. In addition, responsive evaluation involves a relational responsibility, which becomes present in daily situations in which stakeholders express expectations and voice demands. In our everyday work as evaluators, it is difficult to respond to all these demands at the same time. In addition, this article demonstrates that novice evaluators experience challenges concerning over- and underidenfitication with stakeholders. Guidelines and quality criteria on how to act are helpful, but need interpretation and application to the unique situation at hand.     

Revisiting a theory of negotiation: the utility of Markiewicz (2005) proposed six principles

People invited to participate in an evaluation process will inevitably come from a variety of personal backgrounds and hold different views based on their own lived experience. However, evaluators are in a privileged position because they have access to information from a wide range of sources and can play an important role in helping stakeholders to hear and appreciate one another's opinions and ideas. Indeed, in some cases a difference in perspective can be utilised by an evaluator to engage key stakeholders in fruitful discussion that can add value to the evaluation outcome. In other instances the evaluator finds that the task of facilitating positive interaction between multiple stakeholders is just ‘an uphill battle’ and so conflict, rather than consensus, occurs as the evaluation findings emerge and are debated.As noted by Owen [(2006) Program evaluation: Forms and approaches (3rd ed.). St. Leonards, NSW: Allen & Unwin] and other eminent evaluators before him [Fetterman, D. M. (1996). Empowerment evaluation: An introduction to theory and practice. In D. M. Fetterman, S. J. Kaftarian, & A. Wandersman (Eds.), Empowerment evaluation: Knowledge and tools for self-assessment and accountability (pp. 3–46). Thousand Oaks, CA: Sage Publications; Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage Publications; Stake, R. A. (1983). Stakeholder influence in the evaluation of cities-in-schools. New Directions for Program Evaluation, 17, 15–30], conflict in an evaluation process is not unexpected. The challenge is for evaluators to facilitate dialogue between people who hold strongly opposing views, with the aim of helping them to achieve a common understanding of the best way forward. However, this does not imply that consensus will be reached [Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage]. What is essential is that the evaluator assists the various stakeholders to recognise and accept their differences and be willing to move on.But the problem is that evaluators are not necessarily equipped with the technical or personal skills required for effective negotiation. In addition, the time and effort that are required to undertake this mediating role are often not sufficiently understood by those who commission a review. With such issues in mind Markiewicz, A. [(2005). A balancing act: Resolving multiple stakeholder interests in program evaluation. Evaluation Journal of Australasia, 4(1–2), 13–21] has proposed six principles upon which to build a case for negotiation to be integrated into the evaluation process. This paper critiques each of these principles in the context of an evaluation undertaken of a youth program. In doing so it challenges the view that stakeholder consensus is always possible if program improvement is to be achieved. This has led to some refinement and further extension of the proposed theory of negotiation that is seen to be instrumental to the role of an evaluator.