Evaluating development programs: building joint activity

There has been an increasing plea for a participatory approach to development programs. It is argued that for programs to be sustainable, they need to be conceptualized, negotiated and run by all stakeholders jointly. Concurrent with this trend, there has been an increasing demand for program evaluation strategies, which encompass an evaluation of the nature and level of participation of the different stakeholders in the program. This paper proposes a theoretical framework for such evaluation and subsequently provides a practical illustration. Engeström's notions of ‘activity system’ and ‘learning by expanding’, are proposed as a framework for the evaluation of participation and an action evaluation methodology is suggested as the practical tool within such framework. The paper presents a case study of a conservation education program in Africa, which operationalizes the proposed theoretical framework.     

Using client exit interviews to illuminate outcomes in program logic models: a case example

This paper describes how client exit interviews can assist human service administrators and workers to better understand the outcomes their programs are designed to accomplish. Specifically, the qualitative component of a demonstration family literacy program evaluation is used to illustrate how client input can be used to fine-tune the outcomes component of a program's logic model. An analysis of semi-structured exit interviews with 35 clients, who were randomly selected from all 89 served in the first year of the program, resulted in revision to the program's original logic model, creating explicit ‘testable’ pathways to achieving intended outcomes.     

Segregation and protectionism: Institutionalised views of Aboriginal rurality

Rurality is a complex and contested term, with multiple notions and gazes amid calls for theoretical pluralism. In Australia, the spatial categories of ‘remote’, ‘rural’, ‘regional’ and ‘urban’ are applied to places that vary in their distance from an economic and political core and have differing population densities. We argue that natural resources institutions in rural Australia demand an ‘authentic’ performance of Aboriginality that is framed within orthodox and stable constructions of an Indigeneity associated with the remote category. Dominant representations of remote Aboriginal people living on traditional homelands and engaged in ‘traditional’ environmental protection are assumed to hold for all places and transposed when natural resources institutions satisfy compulsory Indigenous engagement. Such institutional requirements for authenticity exclude alternative and multiple Indigenous voices in natural resources management. Rather, Aboriginal people seek engagement across a portfolio of natural resources activities typically found in rural areas (such as mining, grazing, forestry, water allocation planning, and natural resources service delivery and enterprise development), and not just isolated in natural and cultural heritage conservation. This broad participation would more completely match their expressed aspirations and the multiple lived realities of their fluid and networked rural worlds. Using the rural town of Eidsvold in Australia as a case study, we discuss the findings of participant observation and semi-structured interviews with Indigenous people at regional natural resources management meetings and at ‘home’ in Eidsvold. Rather than a generic institutional approach, a place-based approach to understanding the complex ruralities of Aboriginal people is needed.     

An Observational Study in Developing an Intergenerational Shared Site Program

Abstract

This article provides strategic insights into developing and evaluating an intergenerational shared site program at The Community Programs Center (CPC) in Port Jefferson, NY. Conducted under a grant from the Administration on Aging, attention is given to the meth odological challenges in conducting intergenerational evaluation research. The Project Evaluator found that the use of videotaping created a host of difficulties for both the staff and participants. Emphasis is given to chronicling the critical importance of developing a multi-pronged approach to gathering data, and the consequences of utilizing graduate students as evaluators. The evaluation component involved comparing two groups of child/elder participants within a day-care setting during an eight-month period of time. Each session was videotaped and later analyzed for changes in certain participant attributes. Based on the observations gathered, it was found that curriculum development, staff commitment, and programmatic flexibility were critical components in creating an intergenerational-shared site program. Also, the teaching orientation of the intergenerational facilitator was found to play a role in fostering communication among the participants. Important recommendations are provided for both intergenerational practitioners and research evaluators who want to replicate the CPC model. Although positive change was observed in select participants, future evaluation efforts must track change and activity outcomes over a greater period of time.     

A meta-evaluation, or quality assessment, of the evaluations in this issue, based on the African Evaluation Guidelines: 2002

Fourteen evaluations of UNICEF and UNAIDS HIV/AIDS programmes underwent a quality control process called ‘meta-evaluation’, which is an evaluation of the quality of the evaluation. This process of evaluating an evaluation is considered professional good practice by African Evaluation Associations and Networks, as well as the American, Canadian, German, Korean, Swiss and other national professional associations of evaluators. The quality control criteria used were the African Evaluation Guidelines—a list of 30 quality-enhancing standards that can be used as a checklist by evaluators to ensure that evaluation studies are performed well. These quality control criteria can also be applied retrospectively as a quality control method, to assess completed evaluation studies. The results of this quality control review process, for the 14 papers in this issue of Evaluation and Program Planning, are presented and discussed. An indication of a possible area of weakness in HIV/AIDS programme evaluation emerges from this analysis.     

The African Evaluation Guidelines: 2002. A checklist to assist in planning evaluations, negotiating clear contracts, reviewing progress and ensuring adequate completion of an evaluation

A review of the US ‘program evaluation standards’ (PES), undertaken in a series of workshops and meetings of networks of evaluators in Africa, resulted in modifications to those standards. The result was presented to a plenary session of the Inaugural Conference of the African Evaluation Association in September 1999, attended by over 300 evaluators from 35 countries. The AfrEA Conference decided that a systematic effort should be made to produce a list of African evaluation guidelines, similar to the PES, and that this checklist should be reviewed by national evaluation associations and networks in Africa and field tested in several countries. Ten national and regional networks and associations suggested modifications to the text and endorsed the final version of the guidelines.     

Malawi goes to scale with anti-AIDS clubs and popular media

This paper evaluates a project in Malawi that aimed to provide in-school and out-of-school youth with the necessary tools and skills to avoid high-risk sexual behavior, in order to reduce HIV transmission. Project components were school curriculum; extra-curricular activities; out-of-school youth clubs; enabling environmental support; youth reproductive health services; and research, monitoring and evaluation. Non-governmental organizations (NGOs) and Youth Technical Committees established 3200 anti-AIDS Clubs throughout Malawi. These Clubs positively influenced members' behavior and helped to ‘break the silence’. An enabling environment was created through two popular radio programs, advertisements and provision of sports equipment. The out-of-school youth clubs needed clearer criteria and objectives. Many objectives were achieved, but the aim of establishing health services for youth was over-ambitious, and not achieved. This program is exemplary for the national scale that it achieved, and for its approach of using diverse partners to implement locally adapted components.     

Mixed-methods library evaluation integrating the patron,library, and external perspectives: The case of Namibia regional libraries

The article presents a mixed-methods evaluation of regional libraries in Namibia, which incorporates three perspectives: the patron perspective (library users), the library perspective (library staff, management, and related officials), and the external perspective (including evaluators and monitoring data). Seven data collection methods were used: patron surveys, patron panel studies, focus group discussions, key informant and staff interviews, secondary data analysis, media analysis, and observations. The goal of the evaluation was to assess library performance for both formative and summative purposes by addressing evaluation questions on areas such as library services, use, and operations. Building upon the literature review of how mixed-methods approaches can contribute to library evaluation, the aim of this article is to show how a mixed-methods evaluation can be designed to examine multi-faceted library performance and to illustrate how the evaluation design allows information complementarity and can be utilized to present diverse viewpoints of the above three perspectives. The evaluation design, analysis process, and lessons learned from this study may be useful to evaluators engaged in evaluation of public services or programs (including public libraries) that examine multiple aspects of service performance and involve a variety of stakeholders.     

Evaluation of the Late Nite Basketball Project

The author's research team evaluated the Late Nite Basketball Project, an innovative recreational gang diversion program. Data were collected from in-depth interviews of over 100 gang members, family members, and staff and then analyzed using qualitative methodology. Reported benefits included: fun, jobs and scholarships, skill-building workshops, reduction of inter-gang violence, and formation of new relationships. Key recommendations included: institutionalization; expansion of services to younger children, girls, and non-athletes; linkage to suburban leagues; establishment of peer-counseling; increased parental involvement; improved selection, training, and ongoing support for staff; and improved integration with other programs.It's hard to shoot someone after you play ball with him (a gang member and participant in the Late Nite program)The project was funded by The Office of the Mayor, Salt Lake City and assisted by research assistants—Mark Beecher, Laurie Wilson, Vicki Windes, Susan Wrathall and Lee Wright, all of the Graduate School of Social Work, University of Utah.     

Understanding the use of rural space: the need for multi-methods

Although the late 1990s saw increasing use of qualitative data in rural studies and a turn towards issues such as identities and the construction of rurality, many rural researchers still rely on a range of different methods and use both qualitative and quantitative data. However, the challenge of combining quantitative and qualitative data and using different methods is a theme not often dealt with in rural studies, at least not explicitly. This paper (re-)turns the attention to implications of using various methods and combining different types of data for studying a subject matter called ‘the use of rural space’. It concerns both physical land use and the practice and values of individual actors influencing the land use. We emphasise interplay between methodology and philosophy throughout the research process and argue for using multi-methods without compromising the integrity of the different methods. The methodological approach is a combined study of practice and values of individual actors. Two examples—one concerning Senegalese pastoralists’ livelihoods and their use of mobility and one concerning landowners’ location of field afforestation in Denmark—illustrate how the approach facilitates quite different studies of both practice and values and how quantitative and qualitative data can be combined in a non-eclectic way.     

Community-Based Program Research

ABSTRACT

At first glance, the worlds of program professionals and evaluators may seem quite separate. However, there are common issues whose resolution will enhance both program development and research on programs. Program professionals and evaluators have a great deal to learn from each other. As that learning occurs, both program delivery and evaluation research will benefit. Both are concerned about matters of validity, whether it pertains to the nature of the program intervention itself or to the nature of the assessment of the program intervention. To reinforce the development of program evaluation in partnership, this paper discusses key points about evaluation research relevant to both program professionals and researchers. These include the contextual influences on a program, the “readiness” of a program for evaluation, and whether the evaluation research “works” for the program. Underlying our discussion is a call for the development of partnerships around research on programs. In that light, this article elaborates the process steps that should be taken to build program evaluation partnerships, including a discussion of what evaluators and program professionals need to know about each other, and their respective values, interests, and professional perspectives.     

Assumption-aware tools and agency; an interrogation of the primary artifacts of the program evaluation and design profession in working with complex evaluands and complex contexts

Like artisans in a professional guild, we evaluators create tools to suit our ever evolving practice. The tools we use as evaluators are the primary artifacts of our profession, reflect our practice and embody an amalgamation of paradigms and assumptions. With the increasing shifts in evaluation purposes from judging program worth to understanding how programs work, the evaluator’s role is changing to that of facilitating stakeholders in a learning process. This involves clarifying purposes and choices, as well as unearthing critical assumptions. In such a role, evaluators become major tool-users and begin to innovate with small refinements or produce completely new tools to fit a specific challenge or context.We interrogate the form and function of 12 tools used by evaluators when working with complex evaluands and complex contexts. The form is described in terms of traditional qualitative techniques and particular characteristics of the elements, use and presentation of each tool. Then the function of each tool is analyzed with respect to articulating assumptions and affecting the agency of evaluators and stakeholders in complex contexts.     

Tamworth, Australia's ‘country music capital’: place marketing, rurality, and resident reactions

Since the 1970s, Tamworth has become well known as Australia's ‘country music capital’. Its annual Country and Western Music Festival has become the leading event of its type in Australia, attracting over 60,000 visitors every year. The festival, and country music more generally, have become central to the town's identity and tourism marketing strategies. This article discusses the social constructions that have surrounded Tamworth's transition to ‘country music capital’—of the ‘rural’, and of ‘country’—within the context of debates about the politics of place marketing. Textual analysis of promotional material and built landscapes reveals representations of rurality (or ‘senses of the rural’). In their most commercial form, representations of rurality converge on a dominant notion of ‘country’, quite different from the ‘countryside’ and ‘rural idyll’ in England. This dominant, or normative ‘country’ forms the basis of imagery for the festival, the Town's marketing strategy, and associated advertising campaigns by major sponsors. It is predominantly masculine, white, working class and nationalist. But links between musical style and discourses of place are complex. Colonial British histories, Celtic musical traditions and North American popular culture all inform ‘country’ in Tamworth, dissipating nationalist interpretations. Normative constructions also contrast with other, heterogeneous ruralities in Australia, that include the lived experiences of rural Australians, and on stage—in country music—where multiple ‘ruralised’ identities are performed. Even those who stand to benefit from place promotion have been uncertain about country music and ‘the country’, because of associated discourses of Tamworth as ‘hick’ and ‘redneck’. In the final section of the paper, reactions of residents to constructions of Tamworth as country music capital are discussed, via the results of a simple resident survey. In contrast to previous studies of the disempowering politics of place marketing, Tamworth residents were on the whole supportive of the new associations and images for the town, despite ‘hick’ connotations, as it has become a centre for ‘country’, and for country music. Reasons for this are explored, and resistances discussed. The result is a complex and entangled politics of national identity, gender, race and class, where meanings for place are variously interpreted and negotiated.     

Lessons learned about collaborative evaluation using the Capacity for Applying Project Evaluation (CAPE) framework with school and district leaders

Schools, districts, and state-level educational organizations are experiencing a great shift in the way they do the business of education. This shift focuses on accountability, specifically through the expectation of the effective utilization of evaluative-focused efforts to guide and support decisions about educational program implementation. In as much, education leaders need specific guidance and training on how to plan, implement, and use evaluation to critically examine district and school-level initiatives. One specific effort intended to address this need is through the Capacity for Applying Project Evaluation (CAPE) framework. The CAPE framework is composed of three crucial components: a collection of evaluation resources; a professional development model; and a conceptual framework that guides the work to support evaluation planning and implementation in schools and districts. School and district teams serve as active participants in the professional development and ultimately as formative evaluators of their own school or district-level programs by working collaboratively with evaluation experts. The CAPE framework involves the school and district staff in planning and implementing their evaluation. They are the ones deciding what evaluation questions to ask, which instruments to use, what data to collect, and how and to whom results should be reported. Initially this work is done through careful scaffolding by evaluation experts, where supports are slowly pulled away as the educators gain experience and confidence in their knowledge and skills as evaluators. Since CAPE engages all stakeholders in all stages of the evaluation, the philosophical intentions of these efforts to build capacity for formative evaluation strictly aligns with the collaborative evaluation approach.     

Re-discovering the social: neo-liberalism and hybrid practices of governing in rural natural resource management

Since the 1980s, natural resource management (NRM) in rural Australia has been underpinned by rationalities and technologies of governing that constitute agricultural landscapes and resource managers in economically rational terms. While it is tempting to interpret these forms of regulation as part of a broad shift away from social forms of governing, this paper argues that ‘the social’ remains of crucial significance in understanding how both natural environments and the capacities of individuals to manage these environments are constructed. Drawing upon recent work in the Foucauldian-inspired literature on governmentality and, in particular, Stenson and Watt's (Urban Studies 36(1) (1999) 189–201) concept of hybrid governance, this paper examines how particular representations of ‘the social’ are assembled through strategies of NRM. Using the National Landcare Program (NLP) and Natural Heritage Trust (NHT) as examples, we consider how ‘social’ data is being incorporated into resource management strategies, and how this re-shapes both ‘the social’ and NRM as domains of governance. While the NLP and NHT incorporate concerns about social responsibility, they define these in terms of the capacity of individuals to respond to changing economic circumstances. This effectively defines land managers as socially and ecologically responsible only to the extent that they have the managerial capacities to pursue economically ‘rational’ practices. In concluding, we argue that hybrid practices of governing are indeed evident in NRM in Australia and that the concept of ‘hybrid governance’ requires further attention in understanding how rural spaces are made knowable and shaped as objects of knowledge.     

Program decisions : Evaluating a bird in the hand versus two in the bush

In new programs that have not reached full effectiveness, or in programs that are designed to produce future benefits (i.e., become “institutionalized”), evaluations that fail properly to assess future performance may bias decisions against the programs. For such programs, consideration of future benefits may significantly enhance the assessment of their cost-effectiveness. This paper describes an evaluation methodology based on decision theory that can take future benefits into account in evaluating a program. The methodology was applied to the Metropolitan Atlanta Crime Commission (MACC) Community Anti-Crime Project and indicated the potential for substantial future benefits subsequent to cessation of federal funding. The implications of use of this methodology and the MACC results for the overall Community Anti-Crime Program are also discussed.     

The complexity of roles in community action projects: The example of the evaluation of ‘alternatives’

Evaluation of community interventions is a special form of evaluation involving many roles that are key to accomplishing both the intervention and the accompanying research. The present paper describes the roles involved in the evaluation of a community intervention (‘Alternatives’) aimed at safer use of alcohol and depressant medications by seniors.The three core roles were the community developer who provided leadership in the implementation of the project, the researcher/evaluator, and the community committee that sanctioned and facilitated the project. Other roles on the project included: the project coordinator, the nurse-interviewers who collected the survey data for the evaluation, volunteers from seniors groups, local health service providers, the media, the funders, and others on the project team who provided specific technical expertise.The paper describes the competing interests among these roles as well as conflicts that arose and how these conflicts were handled. Although specific roles are described, the types of roles and the potential for competing interests and conflict are generally relevant to community level evaluations.     

State implementation of TANF: where do we stand in 2001?

To provide a context for understanding the evaluations discussed in this volume, this paper gives an overview of the federal TANF legislation and describes features of state TANF programs that are particularly relevant for appreciating the need for evaluation and for interpreting evaluations. These features include the states’ policies regarding work by applicants and recipients, policies to divert applicants from welfare, and rules designed to change individuals’ behavior regarding childbearing, childrearing, and marriage. Because the implementation of state policies depends on practices at the front lines of the welfare system, some features of the ‘culture’ of local welfare offices are also examined.