An evaluability assessment of a West Africa based Non-Governmental Organization's (NGO) progressive evaluation strategy

While program evaluations are increasingly valued by international organizations to inform practices and public policies, actual evaluation use (EU) in such contexts is inconsistent. Moreover, empirical literature on EU in the context of humanitarian Non-Governmental Organizations (NGOs) is very limited. The current article focuses on the evaluability assessment (EA) of a West-Africa based humanitarian NGO's progressive evaluation strategy. Since 2007, the NGO has established an evaluation strategy to inform its maternal and child health care user-fee exemption intervention. Using Wholey's (2004) framework, the current EA enabled us to clarify with the NGO's evaluation partners the intent of their evaluation strategy and to design its program logic model. The EA ascertained the plausibility of the evaluation strategy's objectives, the accessibility of relevant data, and the utility for intended users of evaluating both the evaluation strategy and the conditions that foster EU. Hence, key evaluability conditions for an EU study were assured. This article provides an example of EA procedures when such guidance is scant in the literature. It also offers an opportunity to analyze critically the use of EAs in the context of a humanitarian NGO's collaboration with evaluators and political actors.     

Evaluability assessment of a community development leisure program in Spain

This paper presents the results of an evaluability assessment (EA) conducted in Spain. EAs are assessments that minimize the uncertainty and risk inherent in a subsequent evaluation (e.g. an impact or results assessment) that may require excessive resources. We designed our own EA model by combining three theoretical approaches, with a holistic and pedagogic vision of the process and a flexible methodology to enable adaptation to different programs and contexts, thus fostering transferability. The EA was carried using a qualitative methodology: documentary analysis, seven semi-structured interviews with stakeholders, eight direct observation sessions and a check-list for politicians and experts. The outcome report helped promote organizational learning, provided the program with a logical and assessable theoretical model, detected its strengths and helped overcome the barriers to efficient assessment. Based on these achievements, an evaluation of the results was recommended, with repeated pretest-posttest measurements and an equivalent control group.     

Evaluability assessment of the adult education program (AEP): The results and their use

This paper describes the conduct and the results of an evaluability assessment (EA) of the Adult Education Program (AEP), PL 91-230, as amended (20 U.S.C. 1201–1211c). It presents the findings on the intended program, the actual program, the plausibility of the objectives, the measurement of the objectives, and the program evaluability and options for improvement. The paper concludes with a discussion of the management use of the results following the study and within the current context for the program.     

Evaluability assessment: Reflections on the process

As a formal process, evaluability assessment is 10 years old. In the Cooperative Extension Service (CES), the process is 5 years old. Since 1984, evaluability assessments have been implemented in five states with six different major programs in an effort to define the process in a practical methodological sense and to encourage its adoption in CES. Implementation proceeded iteratively and after each study, procedures were analyzed, clarified, and, wherever possible, simplified to make the process more usable and more “operator robust.” The purpose of this article is to reflect on the process as it evolved in applications in CES. Underlying philosophy and methodological considerations are discussed. To put these in perspective, background information is provided on the evaluability assessment process and on the studies conducted in CES.     

Paying for no reason? (Mis-)perceptions of product attributes in separate vs. joint product evaluation

Consumer decision-making involves the evaluation of options either in isolation or in relation to other alternatives present at the environment. According to Hsee’s evaluability hypothesis, it is easier to evaluate product attributes when they are juxtaposed (i.e., presented jointly) than when they are presented in isolation from each other. Recent research has provided some support to the evaluability hypothesis for the attribute of perceived product quantity. The present research tests the hypothesis in relation to the attribute of perceived fairness. In two experiments, we show that when participants evaluate products in isolation from each other, they err in their judgment of product quantity, and, consequently, they mis-attribute fairness to the seller. In a third experiment, we further show that the inclusion of constant yet unfair price information does not affect the fairness and price judgments. These findings provide evidence for the psychological plausibility of the evaluability hypothesis for the attributes of fairness and product quantity. Moreover, they suggest that isolated product evaluation may be systematically suboptimal for consumers, even when pricing information is included. Therefore, effective consumer decision-making will benefit by allowing the joint evaluation of alternatives.     

A model for evaluation of enterprise architecture quality

Today, most organizations use an enterprise architecture (EA) approach as a tool to increase the power of management on the organization's information technology. Enterprise architecture is a set of processes that helps an organization to translate its vision into an effective change in the organization's scope by providing a clear understanding of its current state. The purpose of this research is to identify EA quality attributes and its evaluation indicators in the organization. This study was conducted by using mixed method, including qualitative and quantitative parts. In the qualitative section, a variety of EA evaluation indicators were identified by a systematic literature review (SLR) approach, then in the quantitative section the survey data were collected by a questionnaire prepared based on the qualitative part and then exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) were performed. This research presents an EA evaluation model that has seven main quality attributes including alignment and integrity, quality of EA products and services, security, maintainability and portability, reliability, reusability and scalability, and 30 indicators that address all aspects of enterprise architecture. Through this model, organizations can evaluate the quality of implemented EA or AS-IS status of EA and take steps to improve it.     

Evaluation readiness: improved evaluation planning using a data inventory framework

Factors intrinsic to many programs, such as ambiguously stated objectives, inadequately defined performance measures, and incomplete or unreliable databases, often conspire to limit the evaluability of these programs. Current evaluation planning approaches are somewhat constrained in their ability to overcome these obstacles and to achieve full preparedness for evaluation. In this paper, the concept of evaluation readiness is introduced as a complement to other evaluation planning approaches, most notably that of evaluability assessment. The basic products of evaluation readiness--the formal program definition and the data inventory framework--are described, along with a guide for assuring more timely and appropriate evaluation response capability to support the decision making needs of program managers. The utility of evaluation readiness for program planning, as well as for effective management, is also discussed.     

An innovative medical and dental hygiene clinic for street youth: Results of a process evaluation

Canada has a noteworthy reputation for high quality health care. Nonetheless, street youth are one of our most vulnerable yet underserved populations. Consequently, a medical and dental clinic was created in downtown Ottawa, Ontario to respond to their needs. The purpose of this study is to describe a process evaluation of the clinic during its first year of operation with a focus on program fidelity, dose, reach, and satisfaction. A mixed methods approach was used involving interviews with providers, focus groups with street youth, analysis of Electronic Medical Record (EMR) data, and supplemental information such as document reviews. The evaluation identified areas that were working well along with challenges to program implementation. Areas of concerns and possible solutions were presented to the management team that then helped to plan and make improvements to the clinic. Our evaluation design and working relationship with clinic management promoted the integration of real-time evidence into program improvements.     

The sequence analysis of hospitals that established social work department under fuzzy environment

This paper reviews the existing literature on hospital social work and discusses intervention strategies for improving social work practice in hospital. The objective of this study was to improve the quality of medical care. But few studies have compared social work services between different hospitals. This study describes qualitative analysis under fuzzy environment, extracts the main influencing factors and establishes a comprehensive evaluation index system. It provides comprehensive evaluation for alternative hospitals by the fuzzy clustering method. This paper proposes a new mixed fuzzy clustering algorithm on the basis of analysing the axiomatic fuzzy set (AFS) and K-means algorithm, which is not affected by some complicated parameter issues and has higher statistical validity. Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) is applied for selecting the best option for each cluster and a comparative analysis is done. Results from a case study in Shanghai, China, confirm that the proposed approach is effective by using information entropy to test. By comparing AFS, K-means and C-means algorithms, the hybrid algorithm can find the two closest attributes of evaluation index of hospital social work, and the proposed approach can be easily help raise the level of hospital social work service.     

Process evaluation of an environmental walking and healthy eating pilot in small rural worksites

Small Steps are Easier Together (SS) was a pilot environmental intervention in small rural worksites in Upstate New York in collaboration with Extension educators. Worksite leaders teamed with co-workers to select and implement environmental changes to increase walking steps over individual baseline and to choose healthy eating options over 10 weeks. Participants were 226 primarily white, women employees in 5 sites. A mixed methods process evaluation, conducted to identify determinants of intervention effectiveness and to explain differences in outcomes across worksites, included surveys, self-reports of walking and eating, interviews, focus groups, and an intervention log. The evaluation assessed reach, characteristics of recruited participants, dose delivered, dose received, and context and compared sites on walking and eating outcomes. Emergent elements of participant-reported dose received included: active leadership, visible environmental changes, critical mass of participants, public display of accomplishments, accountability to co-workers, and group decision making. Participants at sites with high reach and dose were significantly more likely than sites with low reach and dose to achieve intervention goals. Although this small pilot needs replication, these findings describe how these evaluation methods can be applied and analyzed in an environmental intervention and provide information on trends in the data.     

Stressmanagement und Personalentwicklung

Stress management is a main topic of human resources development. This article discusses therefore characteristics of human resources development by means of stress management and starts by presenting theoretical foundations of stress management, intervention concepts, and evaluation results as well as contexts of intervention and target groups. Demands for interventions in the area of human resources development are then specified. These demands touch neglected target groups like low qualified workers and intervention contexts as well as interventions at the workplace. The further development needs of the intervention process are finally highlighted. The intervention process should systematically consider the motivation for participation and not only the motivation of the employees but also the motivation of the managerial decision-makers. Furthermore the relevance of the analysis and its instruments, the intervention methods, didactics and evaluation methods are pointed-out. A new developed intervention concept for stress and resource management for low qualified workers is presented, in which these demands are seized and used.     

Compassion Fatigue,Compassion Satisfaction,and Burnout

Summary

Over the past several decades the mental health field has become increasingly concerned about potential negative effects in professionals of providing crisis intervention and traumatic stress services to individuals and groups. The employee assistance (EA) field has not received adequate attention with regard to the study of these negative effects, such as compassion fatigue. This paper highlights results from a national research study of members of the Employee Assistance Professionals Association (EAPA) who were assessed for risk for compassion fatigue and burnout, as well as potential for compassion satisfaction. Additionally, coping methods for dealing with work-related stress resulting from the provision of workplace crisis intervention services were measured. Findings indicate that EA professionals who provide clinical services and/or crisis intervention services in the workplace are at low risk for burnout, moderate risk for compassion fatigue, and have high potential for compassion satisfaction. Implications for the EA field, in terms of training and practice, are discussed.     

Targeting the worst-off for free health care: a process evaluation in Burkina Faso

Effective mechanisms to exempt the indigent from user fees at health care facilities are rare in Africa. A State-led intervention (2004-2005) and two action research projects (2007-2010) were implemented in a health district in Burkina Faso to exempt the indigent from user fees. This article presents the results of the process evaluation of these three interventions. Individual and group interviews were organized with the key stakeholders (health staff, community members) to document the strengths and weaknesses of key components of the interventions (relevance and uptake of the intervention, worst-off selection and information, financial arrangements). Data was subjected to content analysis and thematic analysis. The results show that all three intervention processes can be improved. Community-based targeting was better accepted by the stakeholders than was the State-led intervention. The strengths of the community-based approach were in clearly defining the selection criteria, informing the waiver beneficiaries, using a participative process and using endogenous funding. A weakness was that using endogenous funding led to restrictive selection by the community. The community-based approach appears to be the most effective, but it needs to be improved and retested to generate more knowledge before scaling up.     

A principles-focused evaluation of community mental health short-term case management

Principles-focused evaluations allow evaluators to appraise each principle that guides an organization or program. This study completed a principles-focused evaluation of a new community mental health intervention called Short Term Case Management (STCM) in Toronto, Canada. STCM is a time limited intervention for clients to address unmet needs and personalized goals over 3 months. Findings showcase that a principles-focused evaluation, assessing whether program principles are guiding, useful, inspiring, developmental and/or evaluable (GUIDE), is a practical formative evaluation approach. Specifically, offering an understanding of a novel intervention, including its key components of assessment and planning, support plan implementation and evaluation and care transitions. Findings also highlight that STCM may work best for those clients ready to participate in achieving their own goals. Future research should explore how best to apply the GUIDE framework to complex interventions, including multiple principles, to increase evaluation feasibility and focus.     

Conceptual and methodological issues in evaluating emergent programs

This paper identifies and discusses conceptual and methodological issues related to five attributes of emergent programs that frequently apply across the spectrum of human services: undefined client populations, inadequate causal evidence relating inputs to outcomes, shifting objectives, identification of criteria for standardizing treatments, and temporal constraints on program development. A case study of the American Red Cross nutrition program is used to illuminate the evaluation problems associated with these issues to mitigate them. Strategies found to be useful in the evaluation of emergent programs include the examination of the relationship among observed client characteristics and needs, external reviews of objectives, identification of client preconditions that support or inhibit performance, examination of normal treatment variation, and a multi-stage process.     

Career Interventions for At-Risk Populations: Incorporating Social Cognitive Influences

This article discusses the career development and the career counseling needs of populations who have limited access to educational and occupational opportunities. The article reviews some of the limitations of current theories and interventions and offers recommendations for applying social cognitive career theory (SCCT; Lent, Brown, & Hackett, 1994) to populations that are at risk for experiencing employment and career barriers. To illustrate one application of SCCT, a career intervention program for adult female offenders is presented.     

A Brief Overview of Research Techniques Used to Evaluate Three Employee Assistance Programs Through the Family Service Association of Metropolitan Toronto Experience

This article describes three models of Employee Assistance Program (EAP) evaluations used by the Family Service Association of Metropolitan Toronto, a major provider of national EAPs in Canada. Each EAP and their subsequent evaluation is tailored to the specific needs of the different organizations. Example l focuses on a case by case review. Example 2 describes a method of determining how well the EAP reaches troubled employees and comments on the effectiveness of the intervention. Example 3 describes how a simple but effective user survey provides information for evaluation purposes. These three evaluation models are examples of how EAPs and their evaluations respond to the uniqueness of various organizations and their respective client groups.