Government-NGO collaboration and sustainability of orphans and vulnerable children projects in southern Africa

Given current donor attention to orphans and children made vulnerable by HIV/AIDS, and the need for a new framework that recognizes the complementary roles of nations and non-governmental organizations (NGOs), this analysis reviews NGO-operated community-based orphans and vulnerable children (OVC) projects in Botswana, Lesotho, Namibia, South Africa, and Swaziland. There has been a lack of attention within the field of evaluation to inter-organizational relationships, specifically those with government agencies, as a factor in sustainability. We analyzed evaluations of nine OVC projects funded by the Bristol-Myers Squibb Foundation for the influence of government-NGO collaboration on project sustainability. For eight of the nine projects, evaluations provided evidence of the importance of the government partnership for sustainability. Government collaboration was important in projects designed to help families access government grants, initiate community-based solutions, and advocate for OVC rights through legislation. Government partnerships were also critical to the sustainability of two projects involved in placing children in foster care, but these showed signs of tension with government partners. In addition to the more common factors associated with sustainability, such as organizational characteristics, donors and NGOs should concentrate on developing strong partnerships with local and national government agencies for the sustainability of their projects.     

Improving and measuring transparency in NGOs: A disclosure index for activities and projects

Nongovernmental organizations (NGOs) are established not with the aim of making profits but rather to provide social values by implementing different projects and activities. Transmitting complete information about these projects to society is a key element of transparency, as they operate within an atmosphere of public trust. Although there is a large body of literature on transparency in NGOs from a global perspective, very little research has been conducted on transparency within the area of projects and activities. This study takes a deeper look at this line and contributes to the literature on transparency in NGOs by proposing an index to measure the information transparency of the projects implemented by these organizations. The index captures three dimensions of the information about the projects (technical, financial, and scope) and makes it possible to: analyze the level of transparency of the portfolio of projects, detect the specific aspects that could be improved in each organization, and carry out comparisons among organizations.     

Evaluation of community-based AIDS education and risk reduction projects in ethnic and racial minority communities: A survey of projects funded by the U.S. public health service

The purpose of this study was to describe proposed program evaluation methods used to determine effectiveness of selected federally funded AIDS education and risk reduction interventions focused on ethnic and racial minority populations. Funding for AIDS prevention programs has grown significantly since 1985, when the first allocation of $11 million was earmarked for general AIDS prevention and education. In 1987, funds specifically targeting minority populations were set aside from AIDS prevention and education programs. Nine agencies/divisions within the Department of Health and Human Services (DHHS) were identified as having funded AIDS prevention and education projects targeted toward ethnic and racial minority populations. The grants and contracts reviewed for this study were limited to those programs receiving Federal funding through DHHS. Few programs provided information within the broader context of sex education and drug education. The modification of needle-sharing behavior among IV drug users and unprotected sex with an IV drug user were the two most frequent risk factors targeted. All 63 programs used at least one formative evaluation method. The most frequently used formative techniques were monitoring achievement of project objectives and counting the number of people participating in program activities. Forty programs counted the amount of AIDS literature distributed. Thirty-three programs conducted a baseline assessment of AIDS knowledge, and 33 programs conducted a baseline assessment of risk behaviors for HIV infection. Results from this study support the National Academy of Sciences recommendation that while summative evaluation will ultimately be valuable, it would be premature to begin developing outcome evaluation strategies at the present time. It is incumbent upon program planners and evaluators to ensure that premature program evaluations are not made in an effort to eliminate direct federal funding of minority community-based strategies to stop the spread of HIV infection.     

Team effectiveness in Non-Governmental Organizations (NGOs) projects

The incorporation of team context into research and practice regarding team effectiveness in NGOs projects is a constant challenge. The research seeks to address the gap and identify the critical determinants of team effectiveness in projects undertaken by non-governmental organizations. Using a systematic process, the study involved both literature and focus group discussions to generate the required items. A total of 157 respondents (Team Members and Team Leaders) were part of the study that filled the questionnaires. Using exploratory factor analysis followed by confirmatory factor analysis, both convergent and discriminant validity was established. The present study found that team effectiveness in NGO social projects has a total of seven dimensions namely: Inter team coordination, community social linkage, team performance, knowledge, skills, and attitudes, leadership communication and engagement, decision making and information sharing, and team formation. There is a significant lack of research on team effectiveness in NGO projects. Where considerably large proportion of research on team effectiveness has focused on the corporate sector, the non-governmental teams have been neglected. This study clearly highlights the determinants that make up team effectiveness in NGOs. The determinants identified will help to specifically look at the effectiveness of teams in NGO projects. The study would help NGOs identify the dimensions in which they may be performing in a weaker manner and direct their energies in improving the factors.     

Foster youth evaluate the performance of group home services in California

In 2003 foster youth employed by a foster youth advocacy organization suggested that an evaluation of group home services to foster youth be conducted in Alameda County, California. This report presents the development and conduct of this evaluation study; how funding was obtained; and how foster youth were hired, trained, and employed to produce a timely and informative evaluation of the performance of 32 group homes where some of the foster youth formerly resided. The results of the study are described in another paper. This report contributes to evaluation practice in the newly emerging field of youth-led evaluations. The achievements of this project in utilizing group home clients to evaluate services with which they were familiar may stimulate other evaluators to develop similar projects, thereby enriching the development of our youth and promoting more informative evaluation findings.     

Evaluation capacity building in the nonformal education context: Challenges and strategies

Policymakers’ demand for increased accountability has compelled organizations to pay more attention to internal evaluation capacity building (ECB). The existing literature about ECB has focused on capacity building experiences and organizational research, with limited attention on challenges that internal evaluation specialists face in building organizational evaluative capacity. To address this knowledge gap, we conducted a Delphi study with evaluation specialists in the United States’ Cooperative Extension Service and developed a consensus on the most pervasive ECB challenges as well as the most useful strategies for overcoming ECB challenges. Challenges identified in this study include limited time and resources, limited understanding of the value of evaluation, evaluation considered as an afterthought, and limited support and buy-in from administrators. Alternatively, strategies found in the study include a shift in an organizational culture where evaluation is appreciated, buy-in and support from administration, clarifying the importance of quality than quantity of evaluations, and a strategic approach to ECB. The challenges identified in this study have persisted for decades, meaning administrators must understand the persistence of these issues and make an earnest investment (financial and human resource) to make noticeable progress. The Delphi approach can be used more often to prioritize ECB efforts.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

Evaluation of the orphans reunification project in Eritrea

As a result of the 30-years war of liberation with Ethiopia, Eritrea has been faced with a ruined infrastructure, devastated economy and damaged social fabric. A large number of children were orphaned (up to 90,000) due to the war. Since 1994 The Orphan Reunification Project has assisted some 13,768 orphans and their extended families (6947) with economic assets to shore up their vulnerability. This article reports on some key findings of a major evaluation of the project, conducted jointly by the Government of the State of Eritrea, UNICEF and two international NGOs. The evaluation focuses on: (a) the impact of the main strategic intervention alternative of direct economic support to target families; (b) the psychosocial status or outcome of the reintegrated orphans; (c) the outcome of the projects capacity building component; and (d) the cost-effectiveness of the project.Findings from the evaluation suggest that direct economic support, particularly provision of livestock to farm households, has led to a further increase in assets. The evidence clearly shows that the status of assisted orphans compares favorably with that of non-orphans, thus indicating that the psychosocial integration between the host families and the orphans has been successful. Although orphan boys were indicated to be slightly better off, the gender differences were not dramatic. Substantial evidence also suggests that the project has built capacities of the relevant government ministry on a number of key areas—research skills and appreciation of the importance of solid operational research before social interventions, participatory planning, monitoring and management tools, community mobilization and advocacy for child protection. Finally, the reunification project is found to be significantly more cost-effective, psychosocially acceptable and sustainable than the alternative strategy of keeping the children in orphanages.     

Experiences with systematic triangulation at the Global Environment Facility

Systematic triangulation may address common challenges in evaluation, such as the scarcity or unreliability of data, or the complexities of comparing and cross-checking evidence from diverse disciplines. Used to identify key evaluation findings, its application has proven to be effective in addressing the limitations encountered in country-level evaluation analysis conducted by the Independent Evaluation Office of the Global Environment Facility (GEF). These include the scarcity or unreliability of national statistics on environmental indicators and data series, especially in Least Developed Countries; challenges in evaluating the impacts of GEF projects; and inherent difficulties in defining the GEF portfolio of projects prior to the undertaking of the evaluation. In addition to responding to the need for further developing triangulation protocols, procedures and/or methodologies advocated by some authors, the approach offers a contribution to evaluation practice. This applies particularly to those evaluation units tasked with country-level evaluations in international organizations, facing similar constraints.     

Evaluation of a research mentorship program in community care

This article describes the results of a qualitative case study evaluating a research mentorship program in community care settings in Ontario, Canada. The purpose of the program was to build evaluation and research capacity among staff of community care agencies through a mentorship program. Data were collected through in-depth, semi-structured interviews, focus group interviews, and written evaluations. Three themes were identified: taking hold, fostering hold, and holding on. Mentees took hold of new evaluation and research skills. Factors fostering hold included positive mentoring relationships and participation in relevant research projects. Mentees struggled to hold on to, and apply their skills within a community care context of challenges such as constrained resources and a mandate to provide client-centered care. The lessons learned include having trainees participate in relevant, time-limited evaluation and research projects, and early implementation of mechanisms to sustain integration of trainees' knowledge and skills within community care agencies.     

NGO Strategies for Hungarian and Roma Minorities in Central Europe

NGOs sponsor a variety of innovative projects relating to the Hungarian and Roma minorities in Romania, Slovakia, and Ukraine, as well as for the Roma in Hungary. However, a study of 33 NGOs in this region reveals that the strategies behind these projects tend to vary according to the particular group being addressed. NGO projects emphasizing Hungarian minorities tend to utilize network strategies to increase contact between Hungarians and titulars (Romanians, Slovaks, and Ukrainians), while projects for Roma tend to avoid network strategies, focusing exclusively on status-raising strategies. This paper presents the promises and shortcomings of both approaches, and concludes with an analysis addressing why NGOs should be less hesitant to apply network strategies to Roma projects as well as to Hungarian projects.     

An evaluation paradox in social work? An empirical study of evaluation use in connection with temporary programmes in Swedish social work

This article contributes to the debate on evaluation use by analysing temporary national programmes in Swedish social work. Previous empirical research shows a gloomy picture of evaluation use, thus supporting an evaluation paradox: evaluations are not used for learning and development, but are continually prioritised. The aim of this article is to study, through interviews and document analysis, how evaluations of temporary programmes in social work are designed and used. The results support the paradox and we found that it has two dimensions. First, evaluations are prioritised, but they function as a ritual, characterised by routine and unclear intentions, with limited relevance for professional learning and improvement. The second dimension is about limited use by those commissioning the evaluations, but the evaluators are using data for new purposes and in new contexts. This is, at best, a form of conceptual use, hopefully contributing indirectly to social work practice. The article ends in a discussion on how evaluations could be designed for learning and improvement by focusing on three normative models: utilisation-focused evaluation, responsive evaluation and evidence-based practice approach (EBP).     

Alliances Across Ideologies: Networking with NGOs in a Tourism Dispute in Northern Bali

This paper explores how local actors form alliances and join networks with (inter-)national NGOs working on religious, political and environmental issues in a dispute over tourism development in northern Bali. All of these actors construct a particular concept of landscape which serves their respective goals to contest or defend the planned large-scale tourism projects. Three of these conceptual framings are identified as playing a major role in the conflict: (1) a spiritual Hindu-Balinese perspective mobilised by tourism protesters and NGOs; in alliance with (2) views on landscape based on sustainability and conservation powerfully represented by international NGOs; clashing with (3) a material and economic view of landscape by international investors, who consider land primarily as a commodity generating economic value. These three contrasting views and resulting networks provide an instrument of moral and political pressure for the contending parties in this conflict over natural resources and tourism development.     

The Influence of Maternal Socialization on Infants’ Social Evaluation in Two Cultures

Although a growing body of research has explored the early development of social evaluation, no research has directly compared social evaluations of infants between different cultures. In addition, there has been little understanding regarding socialization's effects on this ability. The goal of this study was to expand on earlier findings on social evaluation in infants by investigating a broader sample from two cultures, and to explore the influence of maternal socialization on infants’ social evaluation. Using the violation of expectations and the preferential reaching paradigm, four groups aged 6‐, 9‐, 12‐, and 15–18 months and their mothers from Japan and the United States (159 dyads) were compared in terms of spontaneous social evaluations. Japanese and European American infants showed similar performance in dishabituation to the inconsistent behavior and in their reaching preference for prosocial over antisocial agents, indicating that the emergence of spontaneous social evaluation is not culture‐specific. Furthermore, our study provides a novel finding regarding the relationship between mothers’ socially evaluative speech and infants’ preference for prosocial over antisocial agents. These results suggest that the development of sociomoral understanding results from complicated interactions among evolutionary, cognitive, and social factors.     

Institutional design and utilization of evaluation: a contribution to a theory of evaluation influence based on Swiss experience

Growing interest in the institutionalization of evaluation in the public administration raises the question as to which institutional arrangement offers optimal conditions for the utilization of evaluations. Institutional arrangement denotes the formal organization of processes and competencies, together with procedural rules, that are applicable independently of individual evaluation projects. It reflects the evaluation practice of an institution and defines the distance between evaluators and evaluees. This article outlines the results of a broad-based study of all 300 or so evaluations that the Swiss Federal Administration completed from 1999 to 2002. On this basis, it derives a theory of the influence of institutional factors on the utilization of evaluations.     

Evaluating homelessness – a comparative analysis of top 10 articles from the US and Europe

Homelessness as a political and social dominion has a number of features that make it particularly difficult to tackle. In addition to being a wicked problem, homelessness offers several analytical avenues. How do these characteristics affect the evaluation of programmes and projects aspiring to combat homelessness? Dissimilar pictures and solutions regarding homelessness in the US and in European countries will likely affect recommendations given as a result of evaluations performed. For this study, the empirical base is a sample of the most cited evaluations of homelessness programmes published in professional journals between 1996 and 2010. Most of these evaluations are from the US. These US evaluations more often use large-scale national programmes and quantitative methods, whereas European evaluations more often have smaller sample sizes and qualitative methods. In both regions, the evaluators seldom use theory from social science when analysing their findings. It is suggested that European evaluation research should be noticed to a larger extent since it is probably better suited for application in a European context.     

Explaining the Role of Culture and Traditions in Functioning of Civil Society Organizations in Kazakhstan

Recent studies find that Western-style professional nongovernmental organizations (NGOs) in Kazakhstan and other Central Asian countries are weak and unsustainable. Most of these NGOs developed strong dependency on foreign donors for funds, and did not develop local network of support. This study is conducted to understand the lack of effectiveness of NGOs in Kazakhstan and to test popular sentiments toward NGOs. The interview with local and foreign social science experts and public figures confirm that NGOs in Kazakhstan are weak and unsustainable. The explanations of institutional ineffectiveness lay in disconnect with local traditions, low visibility of NGOs, and unsupportive government. Survey of general population suggests that people in Kazakhstan know very little about NGOs and do not appreciate their utility. We explain the inability of civil society organizations to reach out to local people by cultural mismatch. By using the Hofstede national culture model (Culture's consequences: International differences in work-related values. Thousand Oaks: SAGE Publications, 1984), we argue that local culture is in striking dissonance with the culture of donor countries, which created the NGO agenda in Kazakhstan.     

NGO Hybridisation as an Outcome of HIV Services Delivery in Global Fund-Supported Programmes in Ukraine

Ukraine has one of the world’s fastest growing HIV rates and was one of the largest recipients of funding from the Global Fund to Fight AIDS, Tuberculosis and Malaria (GF). Doctoral research recently completed by the author investigates the conduct and practice of international and national nongovernmental organisations (NGOs) as Principal Recipients of GF grants in Ukraine from 2004 to 2012. The study aimed to understand how NGO-based services were implemented in the context of a state-owned health care system. An ethnographic enquiry including 50 participant interviews was conducted in three oblasts in Ukraine, and in its capital, Kyiv, between 2011 and 2013. The paper is based on a doctoral research and presents some of the findings that emerged from the analysis. The author argues that the accent on NGO-run services promoted by GF has rendered the original grass roots, community-based NGOs, to be undermined or replaced by ‘quasi’, hybrid NGOs created by health officials and AIDS centres head doctors. The outcome of such hybridization is a weakened civil society that is dependent on external funding and is unable to genuinely represent their communities.     

Civil Society Partnerships: Power Imbalance and Mutual Dependence in NGO Partnerships

This study examines the distinction between power imbalance and mutual dependence to better understand how NGOs manage resource dependencies in their relationships with civil society partners. The NGO leaders we interviewed emphasized mutual dependence in the relationships they developed with other NGOs regarding access to financial and information resources. In contrast, discourse about their relationships with IGOs focused on the acquisition of legitimacy and access, and was dominated by power imbalance. NGOs were largely accepting of both forms of dependence in pursuit of the community’s shared goals and for the greater good of constituents. Our finding that NGOs refrain from terminating suboptimal relationships also reflects the extent to which mutual dependence governs NGOs partnering strategies.     

A meta-evaluation, or quality assessment, of the evaluations in this issue, based on the African Evaluation Guidelines: 2002

Fourteen evaluations of UNICEF and UNAIDS HIV/AIDS programmes underwent a quality control process called ‘meta-evaluation’, which is an evaluation of the quality of the evaluation. This process of evaluating an evaluation is considered professional good practice by African Evaluation Associations and Networks, as well as the American, Canadian, German, Korean, Swiss and other national professional associations of evaluators. The quality control criteria used were the African Evaluation Guidelines—a list of 30 quality-enhancing standards that can be used as a checklist by evaluators to ensure that evaluation studies are performed well. These quality control criteria can also be applied retrospectively as a quality control method, to assess completed evaluation studies. The results of this quality control review process, for the 14 papers in this issue of Evaluation and Program Planning, are presented and discussed. An indication of a possible area of weakness in HIV/AIDS programme evaluation emerges from this analysis.