Learning about fishery management: Evaluation of a contextualized responsive evaluation approach

This article discusses the extent to which a responsive evaluation (RE) approach contributed to learning by stakeholders in a case of high complexity. Fishery management in Grand-Popo, Benin is characterized by ambiguity, that is contrasting views among fishery stakeholders about what should be done, why, how, where, and when to resolve fishery problems like the depletion of fish-stock and absence of income alternatives. It was also characterized by great gaps (mismatches) between interventionists’ plans and actions, despite generations of interventions and evaluations of their effectiveness. The RE approach aimed at facilitating interactions between interventionists and fishing people to stimulate learning and hence reduce the ambiguity and mismatches. In this article, we take distance and evaluate the results of this action research approach. We found that in the interaction some learning indeed occurred. The fishing people learned among others that intervention resources are limited and that they should organize themselves to lobby for and monitor interventions to solve their problems. Interventionists learned that they could share knowledge about their roles and limited resources with fishing people so that the latter could lobby for more resources. Fishing people however, did not learn to adopt more sustainable fishing practices. Also, interventionists did not learn to influence politicians and financial partners themselves for sufficient resources. Both categories of stakeholders developed ideas for how to collaborate to improve fishery management. We conclude that although some single-loop, double-loop and social learning occurred, the learning was limited and reflect on the related challenges for RE in natural resource management.     

Knowledge brokering in public health: A critical analysis of the results of a qualitative evaluation

Empirical data on the processes underlying knowledge brokering (KB) interventions, including their determining factors and effects, remain scarce. Furthermore, these interventions are rarely built on explicit theoretical foundations, making their critical analysis difficult, even a posteriori. For these reasons, it appeared relevant to revisit the results of a qualitative evaluation undertaken in the province of Quebec in parallel with a Canada-wide randomized controlled trial (RCT) evaluating various KB strategies in public health. This paper looks critically at the theoretical foundations of the KB interventions in light of two conceptual models: (1) the dissemination model underlying the KB interventions used in the Canadian trial and (2) a systemic KB model developed later. This critical analysis sheds light on the processes involved in KB interventions and the factors influencing their implementation and effects. The conclusions of the critical analysis are consistent with the systemic model, in which interpersonal contact is an essential condition for effective KB interventions. This analysis may advance knowledge in the field by enhancing our understanding of the role of knowledge brokers as essential mediators in KB processes and outcomes.     

Can willingness to pay capture the value of altruism? An exploration of Sen’s notion of commitment

Amartya Sen’s distinction between sympathy and commitment is used to examine the assertion that willingness to pay (WTP) can capture the value of altruism in health care. Willingness to pay values for a vaccination program stated to benefit only one’s self in the first instance and only other people in the second instance were elicited from a small convenience sample. Qualitative methods were then used to explore the meaning of the responses. Commitment was found to be a motivating factor and drove a wedge between willingness to pay and valuation of the program. The consequences of this for health economic research are discussed.     

Assessment of the monitoring and evaluation system of a population-based program: Theory-driven evaluation approach

The purpose of this study was to assess the performance of a monitoring and evaluation system (M&E) for a population-based program and to identify factors influencing its performance. The state health department provided a grant, guide, and training for health districts to implement the program and report quarterly and annually to the M&E. Some health districts did not meet this requirement. We also found that the M&E did not adequately capture the actual work of the districts in communities as related to the work plan. We identified several factors that negatively affected the performances of the M&E. For example, the state and health districts did not have a shared vision on how to develop and implement the program, and what to report to the M&E. We proposed strategies that could improve the timeliness and content of reports to the M&E, as well as its utilization in the future.     

Lapses,infidelities, and creative adaptations: Lessons from evaluation of a participatory market development approach in the Andes

Participatory approaches are frequently recommended for international development programs, but few have been evaluated. From 2007 to 2010 the Andean Change Alliance evaluated an agricultural research and development approach known as the “Participatory Market Chain Approach” (PMCA). Based on a study of four cases, this paper examines the fidelity of implementation, the factors that influenced implementation and results, and the PMCA change model. We identify three types of deviation from the intervention protocol (lapses, creative adaptations, and true infidelities) and five groups of variables that influenced PMCA implementation and results (attributes of the macro context, the market chain, the key actors, rules in use, and the capacity development strategy). There was insufficient information to test the validity of the PMCA change model, but results were greatest where the PMCA was implemented with highest fidelity. Our analysis suggests that the single most critical component of the PMCA is engagement of market agents – not just farmers – throughout the exercise. We present four lessons for planning and evaluating participatory approaches related to the use of action and change models, the importance of monitoring implementation fidelity, the limits of baseline survey data for outcome evaluation, and the importance of capacity development for implementers.     

Chapter 8: Responding to Deaths of Faculty,Staff and Students at UC,Berkeley

SUMMARY

This article describes an important initiative designed to address the deaths of faculty, staff, and students at the University of California at Berkeley. Work-life and EAP practitioners can play a significant role in reducing the distress and lost work time associated with workplace deaths by helping employers to plan a response to employee and client deaths, and by participating in implementing that response. A model framework is presented to demonstrate the vital need for coordination and integration among many of the employers' programs, services, and activities in order to address deaths.     

Recognizing,reporting, and responding to abuse,neglect, and self-neglect of vulnerable adults: an evaluation of the re:act adult protection worker basic curriculum

Abuse and neglect of vulnerable adults are significant problems, the identification and management of which is increasingly being made the legal responsibility of healthcare professionals. In response, some jurisdictions have developed training to assist professionals in these duties. This study evaluates the re:act Adult Protection Worker Basic Curriculum training from British Columbia, Canada. One hundred and fifty-seven participants, 109 training completers and 48 non-completers, answered a survey evaluating their confidence, knowledge, and applied knowledge in identifying, reporting, and investigating cases of suspected abuse, neglect, and self-neglect of vulnerable adults. Comparisons of training completers to non-completers showed significant gains in completers’ perceived confidence and knowledge, but no improvement in applied knowledge. The type of healthcare professional being trained and the length of training did not impact learner improvement. The results suggest ways in which both training and evaluation can be improved (e.g., supervised practice) to increase and detect gains in knowledge and practice.     

Expanding theory-based evaluation: Incorporating value creation in a theory of change

Evaluations of policies and programs often use a theory of change to articulate how the intervention is intended to function and the mechanisms by which it is supposed to generate outcomes. When an evaluation includes cost and efficiency considerations, economic and other concepts can be added to a theory of change to articulate a theory of value creation that articulates the mechanisms by which the intervention should use resources efficiently, effectively and create sufficient value to justify the resource use.This paper introduces some theories of value creation that are often implicit in program designs. Making these theories explicit can support clearer evaluative thinking about value for money - including specification of criteria and standards that are aligned with the theory, methods of inquiry that test the theory, and well-reasoned judgements that answer evaluative questions about value for money. Implications for evaluation practice will be discussed.     

Updating the FORECAST formative evaluation approach and some implications for ameliorating theory failure,implementation failure,and evaluation failure

Historically, there has been considerable variability in how formative evaluation has been conceptualized and practiced. FORmative Evaluation Consultation And Systems Technique (FORECAST) is a formative evaluation approach that develops a set of models and processes that can be used across settings and times, while allowing for local adaptations and innovations. FORECAST integrates specific models and tools to improve limitations in program theory, implementation, and evaluation. In the period since its initial use in a federally funded community prevention project in the early 1990s, evaluators have incorporated important formative evaluation innovations into FORECAST, including the integration of feedback loops and proximal outcome evaluation. In addition, FORECAST has been applied in a randomized community research trial. In this article, we describe updates to FORECAST and the implications of FORECAST for ameliorating failures in program theory, implementation, and evaluation.     

Process evaluation of a diversity training program: the value of a mixed method strategy

Patients’ health and health needs are influenced by categories of difference like sex, gender, ethnic origin and socioeconomic status (SES). To enhance awareness of this diversity among patients and to provide holistic care for them, health professionals should first be aware of the relation between dimensions of diversity and patients’ health and health demands. This paper presents a formative process evaluation of a diversity sensitivity training programme for healthcare professionals. The training was implemented in three healthcare settings (mental healthcare, nursing home and hospital care). Mixed methods were used to monitor the implementation of the training and its effects after three years. Findings demonstrate that the training stimulated participants’ awareness, knowledge and critical attitudes towards diversity. Their motivation and willingness to take action regarding diversity was also enhanced. Yet these developments were less apparent among nursing home participants who felt less satisfied and did not develop a critical perspective on this issue. Qualitative data were helpful to explain differences between the settings. By means of the combination of quantitative and qualitative data, we can conclude that individual learning was not enough to guarantee a sensitive approach to diversity at the organizational level.     

Effectively responding to the commercial sexual exploitation of children: a comprehensive approach to prevention, protection, and reintegration services.

The commercial sexual exploitation of children is one of the most destructive forms of abuse. Although some efforts have been made to stop the trafficking in children and youth, they are far from sufficient. A coordinated local and global response is needed if this "ultimate evil" is to be ended.     

Empowerment evaluation: An approach that has literally altered the landscape of evaluation

The quest for credible and actionable evidence to improve decision making, foster improvement, enhance self-determination, and promote social betterment is now a global phenomenon. Evaluation theorists and practitioners alike have responded to and overcome the challenges that limited the effectiveness and usefulness of traditional evaluation approaches primarily focused on seeking rigorous scientific knowledge about social programs and policies. No modern evaluation approach has received a more robust welcome from stakeholders across the globe than empowerment evaluation.Empowerment evaluation has been a leader in the development of stakeholder involvement approaches to evaluation, setting a high bar. In addition, empowerment evaluation’s respect for community knowledge and commitment to the people’s right to build their own evaluation capacity has influenced the evaluation mainstream, particularly concerning evaluation capacity building. Empowerment evaluation’s most significant contributions to the field have been to improving evaluation use and knowledge utilization.     

Value for money: A utilization-focused approach to extending the foundation and contribution of economic evaluation

Value for Money (VfM) is an evaluative question about the merit, worth, and significance of resource use in social programs. Although VfM is a critical component of evidence-based programming, it is often overlooked or avoided by evaluators and decision-makers. A framework for evaluating VfM across the dimensions of economy, effectiveness, efficiency, and equity has emerged in response to limitations of traditional economic evaluation. This framework for assessing VfM integrates methods for engaging stakeholders in evaluative thinking to increase acceptance and utilization of evaluations that address questions of resource use. In this review, we synthesize literature on the VfM framework and position it within a broader theory of Utilization-Focused Evaluation (UFE). We then examine mechanisms through which the VfM framework may contribute to increased evaluation use. Finally, we outline avenues for future research on VfM evaluation.     

Time matters: the potential and pitfalls of using mixed methods approaches in longitudinal program evaluation

While social programs are often assessed using short-term impact studies, longitudinal designs allow evaluators to capture change over time, identify longer-term outcomes, adapt instruments, and better understand participants in transition. A mixed methods design can be critical in understanding these dynamics; yet there is a lack of literature exploring the practical considerations of planning and conducting qualitative and quantitative data collection and analysis within longitudinal studies. This paper examines two different mixed methods frameworks used in a 5-year evaluation of three youth entrepreneurship programs in East Africa. We show how the evaluation team dealt with unique challenges across methods and over time, and how the design ultimately facilitated a richer understanding of program impacts and processes. Considerations for conducting this type of study are explored, related to the impact of longevity on analysis and research staff. Successfully using a longitudinal mixed methods approach requires researchers to be strategic and reflexive, and work in close collaboration.     

Responding to child deaths: the work of Australia's Victorian Child Death Review Committee

The Victorian Child Death Review Committee (VCDRC) in Australia is a multidisciplinary Ministerial Advisory Committee established to review the deaths of children either currently child protection clients or known to the statutory child protection system. The Committee provides advice about each child death as well as insights into what are the surrounding patterns and issues. Key to this role is examining the contribution of the service sectors to the protection of children and the routine practices that are in place to respond to children and vulnerability. This paper provides a snapshot of the cases referred to the VCDRC and the key messages for practice drawn from them. What emerges is that often the threshold for when statutory child protection services must be involved in child and family matters can be ambiguous and that agreement about intervention, the level and nature of need or risk, and when cumulative harm and neglect require statutory responses are not always shared between agencies. It is clear that the lack of common frameworks about what constitutes child protection intervention challenges services. It is recommended that there be and used agreed definitions and frameworks to ensure shared understandings and collaborative responses across the service and legal systems.     

An evaluation of undergraduate public administration programs at Turkish Universities: Reviewing the literature and proposing a model of cultural competence development

This study aims to identify the main predictors of cultural competence (CC). In the study, the cultural competence of the public administration students was analyzed. A total of 599 public administration students from twenty public and two foundation universities in eighteen different provinces of Turkey responded to a survey. The questionnaire survey was derived from a literature review; validated by exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). The study proposes a CC evaluation model with five principal attributes: political awareness, civic action, belief in collective action, social justice attitudes, and ethical leadership. Accordingly, the model included 29 items that address all aspects of cultural competence. The model can be used to evaluate the CC level of public administration students. Besides, this model can also be applied in public organizations to evaluate their responsiveness to the public. Because public institutions today are supposed to be responsive to the cultural needs of citizens.     

A qualitative study to examine how differences in motivation can inform the development of targeted physical activity interventions for African American women

Self-Determination Theory proposes that some people are motivated to initiate physical activity by externally-controlled reasons (e.g., physical appearance, approval from others), whereas others feel compelled by more autonomous sources of motivation (e.g., enjoyment, personal importance). This study used qualitative methods to examine whether individual differences in autonomous motivation offers a useful framework for developing targeted intervention programs for African American women. Focus groups (k = 6) were conducted to examine how different levels of autonomous motivation for physical activity related to differences in physical activity barriers and facilitators among African American women (N = 31). Sessions were audiotaped, transcribed, and coded by independent raters (r = 0.70). QSR NVivo 11 was used to analyze data, and themes were identified separately for women with low, medium, or high autonomous motivation. Those with high autonomous motivation focused on themes of wanting novelty, excitement, and competition, whereas those with low autonomous motivation focused on themes of wanting instrumental support and financial incentives for increasing engagement in physical activity. Implications for developing physical activity intervention programs for African American women that are targeted toward differences in autonomous motivation are discussed.     

From plan to action: Adapting evaluation to serve the developmental needs of a newly-funded multidisciplinary research center

The Southeastern Coastal Center for Agricultural Health and Safety (SCCAHS) is one of many newly-funded federal research centers, housing five multidisciplinary research projects and seven pilot projects, and serving a multi-state region. In the early stages of such a complex project, with multiple teams separated by geography and disciplines, the evaluation program has been integral in connecting internal and external stakeholders at the center and project levels. We used a developmental evaluation (DE) framework to respond to the complex political environment surrounding agricultural health and safety in the southeast; to engage external stakeholders in guiding the center’s research and outreach trajectories; to support center research teams in a co-creation process to develop logic models and tailored indicators; and to provide timely and feedback within the center to address communications gaps identified by the evaluation program. By using DE principles to shape monitoring and evaluation approaches, our evaluation program has adapted to the dynamic circumstances presented as our center’s progress has been translated from a plan in a grant proposal to implementation.