An organization-environment framework for assessing program implementation

Evaluations of health programs are typically conducted from a rational bureaucratic framework in which client change and client outcome are seen as logical outcomes of program activity. This paper proposes a theoretical framework for examining organizational consequences of program implementation. Factors in the internal environment (e.g., the hospital) and the external environment (e.g., the community) which contribute to the program's success or failure, are considered. Observational and structured interview data are used to compare two state funded perinatal projects, one an organizational "success" and the other a costly failure. Attributes of external environments which may be important to successful implementation include demand for services, provider competition, access to information, social beliefs, and professional commitment. Important features of the internal environment include host organization commitment and conflict, and structural attributes: complexity, formalization, centralization, and coordination. The paper concludes with theoretically-based questions which serve as guidelines for process evaluation.     

Process evaluation of “Girls on the Run”: Exploring implementation in a physical activity-based positive youth development program

Many positive youth development programs rely on physical activity as a primary program component. Referred to as physical activity-based youth development programs, these program designs have great potential for promoting healthy youth development. This study examined how one such physical activity-based positive youth development program was implemented in order to identify design features critical to maximizing positive youth outcomes. This mixed method, multi-site process evaluation of Girls on the Run (GOTR) utilized focus groups, site visits, and self-report implementation checklists. Implementation scores were calculated to assess implementation fidelity across twenty-nine sites, and qualitative data were inductively analyzed to identify factors influential for implementation. Results reveal variability in how GOTR was implemented. Five themes emerged from the data that represented factors serving as facilitators or barriers to programmatic implementation. These included contextual/environmental factors (e.g., parental involvement, relationships with school personnel), organizational factors (e.g., implementation support and responsiveness of staff), program-specific factors (e.g., curriculum design), coach factors (e.g., existing relationships with participants, responsiveness to participant's needs), and youth factors (e.g., behavioral and discipline issues). Study findings have implications for improving the design of physical activity-based and other positive youth development programs, with relevance to evaluators, program planners, youth development leaders, and others working with children and youth.     

A novel approach for evaluating programs designed to serve military veterans: Using an adapted common components analysis

Evaluators are challenged to keep pace with the vast array of Veteran support programs operating in the United States, resulting in a situation in which many programs lack any evidence of impact. Due to this lack of evidence, there is no efficient way to suggest which programs are most effective in helping Veterans in need of support. One potential solution to this dilemma is to reconceptualize program evaluation, by moving away from evaluating programs individually to evaluating what is common across programs. The Common Components Analysis (CCA) is one such technique that aggregates findings from programs that have undergone rigorous evaluation at the level of program components (e.g., content, process, barrier reduction). Given that many Veteran programs lack outcome evidence from rigorous studies, an adaptation to CCA is needed. This report examines cross-sectional data from a pilot study using an adapted CCA across five domains of well-being (i.e., employment, education, legal/financial/housing, mental/physical health, and social/personal relationships). The purpose of this preliminary study is to determine the feasibility of eliciting program nominations and program components from Veterans via an online survey. When coupled with a longitudinal research design, this adaptation to CCA will allow for stronger causal claims about the expected impact of different program components within and across a variety of domains.     

Feasibility,Acceptability, and Initial Efficacy of an Online Sexual Health Promotion Program for LGBT Youth: The Queer Sex Ed Intervention

Lesbian, gay, bisexual, and transgender (LGBT) youth experience multiple sexual health inequities driven, in part, by deficits in parental and peer support, school-based sex education programs, and community services. Research suggests that the Internet may be an important resource in the development of sexual health among LGBT youth. We examined the feasibility of recruiting youth in same-sex relationships into an online sexual health intervention, evaluated intervention acceptability, and obtained initial estimates of intervention efficacy. LGBT youth (16 to 20 years old) completed Queer Sex Ed (QSE), an online, multimedia sexual health intervention consisting of five modules. The final sample (N = 202) completed the pretest, intervention, and posttest assessments. The primary study outcomes were sexual orientation identity and self-acceptance (e.g., coming-out self-efficacy), sexual health knowledge (e.g., sexual functioning), relationship variables (e.g., communication skills), and safer sex (e.g., sexual assertiveness). Analyses indicated that 15 of the 17 outcomes were found to be significant (p < .05). Effect sizes ranged from small for sexual orientation (e.g., internalized homophobia) and relationship variables (e.g., communication skills) to moderate for safer sex (e.g., contraceptive knowledge) outcomes. This study demonstrated the feasibility, acceptability, and initial efficacy of QSE, an innovative online comprehensive sexual health program for LGBT youth.     

An exploration of social support as a factor in the return-to-work process

Despite evidence that inter-personal relationships are important in human resource management, little is understood about the nature of workplace social support in a disability context, or what features of support are important to the success of return-to-work programs. The purpose of this qualitative study was to explore workplace disability support from worker and supervisory perspectives and to identify salient features for work re-entry. A total of 8 supervisors and 18 previously injured workers from a range of work units in a Canadian municipality were interviewed, and their views concerning supportive and unsupportive behaviours in work-re-entry situations were recorded and analyzed. A full range of social support dimensions were reported to be relevant, and were seen as arising from a variety of sources (e.g. supervisor, co-workers, disability manager, work unit, and outside of work). Respondents identified trust, communication and knowledge of disability as key precursors to a successful return-to-work process. Future research should explore the specific contributions of support to work rehabilitation outcomes as well as interventions to enhance available supports.     

Rural High School Youth's Access to and Use of School‐to‐Work Programs

Rural youth face numerous barriers that may limit career exposure. School‐to‐work (STW) programs are one tool used to increase exposure and facilitate connections between school and work. Using a nationally representative data set, the authors investigated whether rurality relates to the availability of STW programs and rural youth's program use. Results indicate that most schools provide school‐based services (e.g., career plans/career majors), but few schools offer work‐based services (e.g., internships). Rural schools were more likely to offer vocational–technical prep programs and job shadowing but were less likely to offer school‐based enterprise. After controlling for program availability, the authors found that rural students were less likely to take part in job shadowing and community service. Conclusions are offered for career development research, policy, and practice.     

Prevention validation and accounting platform: a framework for establishing accountability and performance measures of substance abuse prevention programs

The field of substance abuse prevention has neither an overarching conceptual framework nor a set of shared terminologies for establishing the accountability and performance outcome measures of substance abuse prevention services rendered. Hence, there is a wide gap between what we currently have as data on one hand and information that are required to meet the performance goals and accountability measures set by the Government Performance and Results Act of 1993 on the other. The task before us is: How can we establish the accountability and performance measures of substance abuse prevention programs and transform the field of prevention into prevention science? The intent of this volume is to serve that purpose and accelerate the processes of this transformation by identifying the requisite components of the transformation (i.e., theory, methodology, convention on terms, and data) and by introducing an open forum called, Prevention Validation and Accounting (PREVA) Platform. The entire PREVA Platform (for short, the Platform) is designed as an analytic framework, which is formulated by a collectivity of common concepts, terminologies, accounting units, protocols for counting the units, data elements, and operationalizations of various constructs, and other summary measures intended to bring about an efficient and effective measurement of process input, program capacity, process output, performance outcome, and societal impact of substance abuse prevention programs. The measurement units and summary data elements are designed to be measured across time and across jurisdictions, i.e., from local to regional to state to national levels. In the Platform, the process input is captured by two dimensions of time and capital. Time is conceptualized in terms of service delivery time and time spent for research and development. Capital is measured by the monies expended for the delivery of program activities during a fiscal or reporting period. Program capacity is captured by fourteen measurement units, tapping into the dimensions of staff resources and community assets. Staff resources are, in turn, operationalized in terms of staff size, staff certification status, staff turnover rate, and the accreditation status of a provider agency. Community assets are operationalized by the number of community centers accessible to the funded agency, number of formalized teams or antidrug coalitions active in the catchment area, and other social/human services providers with whom the prevention agency has formalized networks. The totality of process output from all sources of program activities is reduced to eighteen classes of measures. These are operationalized by thirty-three summary measures. Some of these include: total count of events facilitated; total number of clients served; average number of clients served per event; clients served by single and multiple program sessions; classification of target population in terms of the severity of risk as defined by the Institute of Medicine; age groups and race/ethnicity of clients served; number of program participants retained by recurring programs; number of clients who have completed the program; penetration rates to the target population; client attrition rates; average referral rates per provider per time interval; referral success rates; and so on. All process output measures specified in the Platform are derived from two broad classes of events classified as either products or services. The collectivity of these measures is expected to present a cost-effective, parsimonious, yet comprehensive picture of the entire spectrum of the process output, i.e., "what came out of the program as program activities". For the measurement of performance outcomes, two types of data are incorporated into the Platform: outcome data from individuals and the behavior (or performance) of social indicators from aggregated data bases. Individual data are used to evaluate the outcome of substance abuse programs     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

Treating Serious Emotional and Behavioural Problems Using Multisystemic Therapy

Multisystemic therapy (MST) is a family‐ and community‐based treatment of serious emotional and behavioural problems presented by adolescents. The development of this model began in the late 1970s, and today MST is a leading evidence‐based treatment of serious antisocial behaviour in youths, with programs transported to more than 30 states in the United States and 12 nations, including Australia and New Zealand. This article rerviews the theoretical rationale that underlies MST and presents the central clinical features of the intervention model. In addition, the training and quality assurance protocols used to promote MST program fidelity and treatment adherence are described. Finally, findings from approximately 15 published randomised and two quasi‐experimental clinical trials with youths presenting serious clinical problems (e.g., violence, substance abuse, serious emotional disturbance, sexual offending, and chronic illness) are summarised.     

Feedback from participants and leaders in an intervention program: the 1997-1998 opening doors process evaluation

Parents, students, and program leaders involved in an in-school drug prevention program called Opening Doors were assessed for their perceptions of the efficacy of the program. In general, the feedback indicated a very high program satisfaction level from respondents with parents indicating the highest level satisfaction (92%). Areas for improvement ranged from: parent attendance, invitation process, increased support from schools, and increasing awareness of scheduling and time involvement by leaders and school administration based on program leader feedback as well as invitation process and program duration from the parents (e.g., 37% thought the program was too short). Student satisfaction may be predicted from the perceived impact of the program on significant relationships (e.g., with peers and family) as well as affective reactions to the program (e.g., enjoyment of the program). Finally, a school-level analysis indicated that the application of the Opening Doors Program in different schools was provided uniformly.     

Posttreatment 12-Step Program Affiliation and Dropout: Theoretical Model and Qualitative Exploration

ABSTRACT

Treatment outcome research has repeatedly demonstrated that involvement in twelve-step (TS) programs (e.g., Alcoholics Anonymous or Narcotics Anonymous) following treatment is associated with improved substance use outcomes. Treatment providers widely encourage TS program attendance during treatment and for aftercare; yet, evidence suggests that minimal and sporadic attendance or complete dropout is the more common occurrence. Providers would benefit from knowledge that would contribute to a greater understanding of factors that influence posttreatment TS program affiliation or dropout to guide treatment strategies. This article reviews the literature on TS affiliation, advances theory and a model to explain affiliation, and reports findings from a qualitative research study focused on understanding what people in TS facilitative treatment report as good (the pros) and not so good (the cons) about TS program involvement. Results suggest that unresolved ambivalence as well as socio-cultural barriers to support contribute to poor TS program utilization.     

‘Tell Me and I Forget,Teach Me and I May Remember,Involve Me and I Learn. And That's What It's About.’ How a Co‐design Methodology is Used in the Delivery of Parents Building Solutions: A Qualitative Study

There are many parenting programs delivered in Australia and Parents Building Solutions (PBS) is one of these. Collaboratively designed by Parentzone staff of Anglicare Victoria, it has a twenty year history of building the evidence base. PBS stands apart from other programs because parents actively co‐design the sessions and content with skilled facilitators. Understanding the drivers, processes and practices that make the co‐design methodology work, is fundamental to the implementation science that underpins the program.This study aimed to examine the way co‐design was used in the delivery of PBS, specifically looking at program implementation and outcomes. Focus groups and interviews were conducted with Parentzone staff (team leaders and facilitators) and parents who had recently completed a program. Ethics approval was provided and analysis was conducted using NVivo software with co‐design as the unit of analysis. Three major themes about co‐design were present across team leaders, facilitators and parents: 1) responsiveness and flexibility are central to the functionality of co‐design within PBS; 2) facilitators implement the co‐design methodology using a suite of knowledge and skills; and 3) parents report tangible results from participating in a co‐designed parenting program. There was an overwhelming appreciation from both facilitators and parents about the co‐designed implementation style. Facilitators and team leaders highly valued the flexibility of the program which enabled their ability to respond to parent's needs. Furthermore, parents believed they achieved tangible outcomes derived from strategies and support delivered in the program. The co‐design methodology used in PBS was evident for all stakeholders and was a driver of program implementation and its outcomes. The methodology described in this has practice implications for family therapists and others who work directly parents and families.     

The Longevity of Community Aging Initiatives: A Framework for Describing NORC Programs’ Sustainability Goals and Strategies

Naturally Occurring Retirement Community Supportive Services programs (NORC programs) aim to bring together stakeholders within a local area with a large number of older adults (e.g., a neighborhood or apartment building) to facilitate activities and services that promote aging in place. Despite enthusiasm for these programs, there is ongoing concern regarding their long-term sustainability, especially as federal funding to support NORC programs nationally has dwindled in recent years. This exploratory study aimed to develop a framework to describe NORC programs’ sustainability goals and strategies. As part of a statewide study in New Jersey, this study drew on data from semi-structured in-depth interviews with staff of 10 lead organizations that represented 15 NORC programs. Using a grounded theory analytic approach, findings revealed three types of sustainability goals: continuing the program indefinitely within the lead agency (described by eight organizations); integrating the program functions across other organizations (described by one organization); and defining the program as a finite project (described by one organization). Lead organizations implemented and developed their programs in ways that reflected their goals. This framework can be used to guide efforts to maintain, expand, and evaluate NORC programs and related community aging initiatives.     

Unionism and changing employee views toward work

Using data from the 1977 Quality of Employment Survey and the 1984 Union Image Survey, this paper examines the changing importance of satisfaction with job elements that are not traditionally the focus of union activity (e.g., advancement, autonomy, interesting work) in the decision to unionize. The results provide modest evidence of an increase in the importance of satisfaction with these job elements as a determinant of union voting intent. Unions may achieve greater organizing success by giving more attention to such themes. We thank Charles McDonald of the AFL-CIO and James L. Medoff of Harvard University for making available the Union Image Survey data.     

Managing mental health organizations with 25 key performance indicators.

A set of 25 key performance indicators were developed to support managers and policy makers in assessing the performance of their program relative to others. The indicators cover four major areas: Revenue mix, client mix, staff mix, and service mix. Comparisons can be made among programs within or across a state or region, or among programs of similar circumstances (e.g., urban or rural).     

Measuring conflict management,emotional self-efficacy,and problem solving confidence in an evaluation of outdoor programs for inner-city youth in Baltimore,Maryland

Substantial evidence supports the value of outdoor education programs for promoting healthy adolescent development, yet measurement of program outcomes often lacks rigor. Accurately assessing the impacts of programs that seek to promote positive youth development is critical for determining whether youth are benefitting as intended, identifying best practices and areas for improvement, and informing decisions about which programs to invest in.We generated brief, customized instruments for measuring three outcomes among youth participants in Baltimore City Outward Bound programs: conflict management, emotional self-efficacy, and problem solving confidence. Measures were validated through exploratory and confirmatory factor analyses of pilot-testing data from two groups of program participants. We describe our process of identifying outcomes for measurement, developing and adapting measurement instruments, and validating these instruments.The finalized measures support evaluations of outdoor education programs serving urban adolescent youth. Such evaluations enhance accountability by determining if youth are benefiting from programs as intended, and strengthen the case for investment in programs with demonstrated success.     

Project CHOICE: Validity of Interest and Ability Measures for Student Choice of Vocational Program

This project sought to determine whether interest and ability measures can help high school students identify vocational education programs in which they may experience satisfaction and success. Interest measures (6) and ability measures (15) were administered to 2,101 seniors in 19 area vocational-technical schools. About 80% of the satisfied/successful seniors would have been referred, by a computer-based score interpretation, to the job cluster containing the vocational program they completed. Generally, score profiles were congruent with program content. Results indicate that computer-based interpretations can effectively and efficiently link interest and ability scores to vocational-technical program options.