Evaluating development programs: building joint activity

There has been an increasing plea for a participatory approach to development programs. It is argued that for programs to be sustainable, they need to be conceptualized, negotiated and run by all stakeholders jointly. Concurrent with this trend, there has been an increasing demand for program evaluation strategies, which encompass an evaluation of the nature and level of participation of the different stakeholders in the program. This paper proposes a theoretical framework for such evaluation and subsequently provides a practical illustration. Engeström's notions of ‘activity system’ and ‘learning by expanding’, are proposed as a framework for the evaluation of participation and an action evaluation methodology is suggested as the practical tool within such framework. The paper presents a case study of a conservation education program in Africa, which operationalizes the proposed theoretical framework.     

Evaluating the engagement of universities in capacity building for sustainable development in local communities

Universities have the potential to play a leading role in enabling communities to develop more sustainable ways of living and working however, sustainable communities may only emerge with facilitation, community learning and continual efforts to build their capacities. Elements of programme planning and evaluation on the one hand, and capacity building on the other, are needed. The latter entails approaches and processes that may contribute to community empowerment; universities may either lead such approaches, or be key partners in an endeavour to empower communities to address the challenges posed by the need for sustainable development. Although capacity building and the promotion of sustainable development locally, are on the agenda for universities who take seriously regional engagement, very little is published that illustrates or describes the various forms of activities that take place. Further, there is a paucity of studies that have evaluated the work performed by universities in building capacity for sustainable development at the local level. This paper is an attempt to address this need, and entails an empirical study based on a sample of universities in the United Kingdom, Germany, Portugal and Brazil. The paper examines the extent to which capacity building for sustainable development is being undertaken, suggests the forms that this might take and evaluates some of the benefits for local communities. The paper concludes by reinforcing that universities have a critical role to play in community development; that role has to prioritise the sustainability agenda.     

Implications of nested designs in school-based mental health services research

Evaluating school-based mental health services for children and youth with emotional disturbance (ED) has been a challenge for researchers. One particular challenge is the study design of using the student as the statistical unit of analysis, which in certain cases may lead to a violation of the "independence of error" assumption. However, the alternative to this nested design, including fewer students and more schools, can be costly and administratively complex. This study examines data from two national studies including 314 students with ED and served in special education programs and their caregivers from 24 schools in the US to identify the extent to which nesting or design effects occur in this population. The results show that variables focusing on psychopathology are less affected by nesting but school-related variables such as academic functioning are more affected. Design effects varied by grade level, suggesting that grade should be considered when designing such evaluations.     

Organizational capacity building: addressing a research and practice gap

The purpose of this article is to address the gap between evaluation research, and the practice of capacity building with nonprofits. This study describes a 5-year capacity building initiative with grassroots organizations including a longitudinal evaluation of the implementation and outcomes achieved. Formative processes yielded many lessons that were used to improve the capacity building model of services. The results show that the majority of groups met a priori expectations for participation success. Organizational staff valued technology, consultants, and program funding the most. Increases were found in board membership and perceptions of visibility of the organization were enhanced. Executive directors reported greater awareness of needs and improved management knowledge. These small organizations fill many unmet needs and more capacity building evaluation studies are needed to understand the mechanisms that support their efforts and the impact on their sustainability.     

Toward building a typology for the evaluation of services in family support programs

This article briefly reviews the history, philosophy, practice principles, and foci of family support programs, examines the typologies currently in use to classify these programs, and discusses the difficulties these classifications pose for program evaluators. The authors introduce a new typology that deconstructs family support programs into their component services and discuss the potential of this typology for evaluation of family support services.     

Individualised Support and Funding: building blocks for capacity building and inclusion

The paradigm in disability supports is shifting away from institutional services and professional control towards self-determination and community involvement of people with disabilities. The assumption that the best way to provide disability supports is for government to give money to agencies or services, rather than directly to people with disabilities and their support networks, is being challenged. This article reports on findings and themes from a Canadian study that investigated individualised funding projects from different parts of the world. Ten of fifteen of the most 'promising initiatives' were selected for more detailed study and analysis. Projects analysed were from Canada, the US, and Australia. Themes emerging from the study included the fact that values and principles mattered, a policy framework provided coherence and equity, infrastructure supports for individuals were separate from service system, the facilitator-broker role differed from case management, allocation of individualised funds was designed to be equitable and account able to the funder and person, and a 'learn as you go' philosophy maximised positive outcomes. This research project demonstrates that individualised support and funding, when embedded in the new paradigm of disability and community, builds capacity of individuals, families, and communities.     

The politics of evaluation research: A case study of community development block grant funding for human services

This paper examines the ramifications of conducting evaluation research within a very politically sensitive area. The evaluation project discussed in this paper involved an assessment of six human service agencies within a city that were receiving Community Development Block Grant (CDBG) public service funds. The city's Office of Policy Planning and Management (OPPM) contracted for this research in order to evaluate the CDBG funded agencies and to provide technical assistance so they could make informed policy decisions concerning the allocation of the Seventh Year Community Development public service funds. The Department of Housing and Urban Development (HUD) had stipulated that there be a 20% limitation in CDBG public service funding. The city involved in this study had allocated over 25% of its CDBG funding to public services. Thus, the city was over the stipulated limitation and was therefore requested by HUD to either justify the expenditure or cut back its allocation of funds to the 20% level.As a result of this mandate by HUD, the city requested and received a technical assistance grant from HUD. This paper provides a specific example of how the conduct of evaluation can become entwined in the politics of human service agencies and their funding sources, Moreover, the implications of this experience for evaluation as a field of endeavor are explored.     

Development and validation of a measure of organizational capacity for implementing youth development programs

Accumulating evidence indicates that incorporating youth development (YD) principles, strategies, and supports into an organization promotes positive adult and youth outcomes. However, few validated measures assess this type of capacity. The YMCA commissioned a study to validate its Capacity Assessment for Youth Development Programming (Y-CAP), which examines the organizational infrastructure required to implement YD programs and processes in seven areas. Survey development was an iterative process informed by existing frameworks, instruments, and pilot testing of items. The Y-CAP was reviewed and revised three times prior to this study, with a final round of revisions made at the start of the validation phase as a result of thorough content, survey methodology, and psychometrics reviews. The revised Y-CAP was completed by 123 YMCA implementation teams. Rasch analyses were used to determine the extent to which validity evidence supports the use and interpretation of the Y-CAP scores. Convergent validity was assessed by comparing Y-CAP scales to the Algorhythm staff survey for youth-serving organizations, and focus groups informed the consequential validity of the Y-CAP. The results provide strong evidence for the reliability and validity of the Y-CAP, which can be used to guide continuous quality improvement initiatives that support capacity and functioning in youth-serving organizations and programs.     

The development of a new measure for work-family research: Community resource fit

We describe the development and validation of a quantitative measure of community resource fit; i.e., satisfaction with the extent to which community resources meet the needs of working families of school-aged children. The measure has good psychometric properties, and preliminary results suggest that the measure warrants further study. The measure is composed of six moderately intercorrelated subscales assessing resource fit in the areas of work, public transportation, school, school transportation, after-school programs and after-school transportation resources. We found interesting patterns of results linking community resource fit, especially in the areas of work and school resource fit, to a variety of quality-of-life and well-being outcomes among employed parents of school-aged children. These outcomes include work-to-family and family-to-work conflict and enhancement, psychological distress, job–role quality, likelihood of losing or leaving one's job and likelihood of leaving one's line of work.     

Evaluating public health perspectives of sexually transmitted diseases: The need for further research

While sexuality is the focus of extensive research, the concept has not been adequately defined. In this paper, sexuality is defined as the individual capacity to respond to physical experiences which are capable of producing body‐centered genital excitation, that only subsequently becomes associated with cognitive constructs (either anticipatory for new experiences or reflective of past experiences) independent of ongoing physical experiences. The discussion focuses on four components of this definition: that sexuality is an individual capacity, is experiential, is body‐oriented, and is directed toward genital excitation.     

Experiments in evaluation capacity building: Enhancing brain disorders research impact in Ontario

This paper is the introductory paper on a forum on evaluation capacity building for enhancing impacts of research on brain disorders. It describes challenges and opportunities of building evaluation capacity among community-based organizations in Ontario involved in enhancing brain health and supporting people living with a brain disorder. Using an example of a capacity building program called the “Evaluation Support Program”, which is run by the Ontario Brain Institute, this forum discusses multiple themes including evaluation capacity building, evaluation culture and evaluation methodologies appropriate for evaluating complex community interventions. The goal of the Evaluation Support Program is to help community-based organizations build the capacity to demonstrate the value that they offer in order to improve, sustain, and spread their programs and activities. One of the features of this forum is that perspectives on the Evaluation Support Program are provided by multiple stakeholders, including the community-based organizations, evaluation team members involved in capacity building, thought leaders in the fields of evaluation capacity building and evaluation culture, and the funders.     

Household technology: Implications for research and policy

If we are to have a fuller understanding of the social and economic context of the family, it is necessary to explore its technological environment. However, few scholars have examined the relationship between household technology and the functioning of the family. This article looks at which academic disciplines address household technology, what have been their findings, and why there is generally a paucity of research in this area. This article concludes with a discussion of the need for more research in household technology and the implications that this research may have for other family inquiry and for policy formation.Cathleen Zick, University of Utah, Richard Widdows, Purdue University, and Joan Ash, Central Washington University, provided helpful comments on earlier drafts of this article.His research interests include household technology, consumer policy, and consumer protection.     

Evaluating community capacity: Visual representation and interpretation

The purpose of this paper is to provide an interpretation ofprogramme experiences in the visual representation of strategiesused to build community capacity. The paper first discussesthe definition of community capacity in a programme contextand describes different approaches for the visual representationof the evaluation of community capacity. What is new about thispaper is that it provides a discussion about the interpretationof community capacity and in particular the use of the spiderweb configuration (so called because of the shape it resembles).This is illustrated by three examples taken from two differentcommunity development programmes, the first of which is in Fijiand the second in Kyrgyzstan. The paper concludes with a discussionabout the implications for the community development practiceof using visual representations of community capacity, includingthe difficulties, for example, of mapping changes in communitycapacities over time, and the benefits, for example, of promotinga free flow of information between stakeholders. The paper willassist practitioners to better plan for, evaluate and visuallyrepresent the findings of strategies used to build communitycapacity in community development programmes.     

Environmental assessment and evaluation research: examples from mental health and substance abuse programs.

Evaluation researchers have been broadening the traditional client input--"black box" treatment--client outcome paradigm that has guided many evaluations of mental health, substance abuse treatment, and other intervention programs. The points of expansion are in the areas of treatment implementation and treatment processes, as well as "extratreatment" influences on treatment selection, duration, and, especially, outcome. This review illustrates the application of environmental assessment procedures--particularly social climate measures--in four aspects of evaluation research suggested by the more comprehensive model: (a) evaluating treatment implementation; (b) exploring treatment processes; (c) identifying extratreatment influences on client posttreatment functioning; and (d) operationalizing outcome variables. Conceptual and methodological issues raised by these applications are discussed, and the benefits to be derived from an expanded model of evaluation research--especially the greater potential for program improvement--are considered.     

Rural accessibility problems and human resource development: Case study from Nigeria

This paper focuses on rural transport as a crucial factor in rural development. It examines the problems posed by poor transportation and the extent to which these constitute constraints to the mobilization of human resources for rural development. Using Kwara State. Nigeria, as a case study, the study shows the poor nature of rural transport network and its effect on the evacuation of agricultural products from the farm centres to markets. This accessibility problem was found to have negative effects on the farmers who are not in the least motivated to increase the acreage cultivated.