Analyzing free-text survey responses: An accessible strategy for developing patient-centered programs and program evaluation

Despite widespread availability of yoga in the Veterans Health Administration (VA), it remains unclear how to best evaluate yoga programs. This is particularly problematic for programs aimed at veterans with mental health concerns, as evaluation typically focuses narrowly upon mental health symptom severity, even though program participants may have other health-related priorities. We analyzed responses to free-text questions on 237 surveys completed by veterans with mental health concerns enrolled in a yoga program at six VA clinics in Louisiana to characterize veteran participants’ experiences with yoga. Qualitative analysis resulted in 15 domains reflecting veterans’ individual health-related values and priorities. We use results to illustrate the potential for analysis of free-text responses to reveal valuable insights into patient experiences, demonstrating how these data can inform patient-centered program evaluation. The approach we present is more accessible to those responsible for decision-making about local programs than conventional methods of analyzing qualitive evaluation data.     

Evaluation approaches in mental health: Implications of the new federalism

Since the introduction of federal mental health legislation in 1963, there has been a changing emphasis on evaluation and accountability. With direct federal funding of community mental health services, accountability demands were met through expectations for local agency evaluation activities which were overseen by federal authorities. The advent of the New Federalism and the shift to block grant funding of mental health services to state mental health authorities have shifted responsibility for evaluation to the states and local programs. This paper reviews federal mental health statutes to trace the extent and locus of required evaluation activities and discusses two approaches to carrying out program evaluation: "top-down" where the evaluation topic, method, and data collection are mandated by an administering or funding body; and "bottom-up" approaches where the subject, method of study, and data to be collected are developed in response to a felt need at the local agency level. A case study of each approach as used at the state level in mental health is examined. Based on the literature and the case studies, conclusions are presented on the pro's and con's of each method in meeting accountability demands and the barriers which must be overcome for either method to be successful.     

Community mental health and mass media preventive education: the alternatives project.

An innovative mental health preventive education and public information program employing mass media was conducted in the Louisville, Kentucky, metropolitan area. Social work and mental health concepts of prevention were applied as a framework for developing project goals. Systematic evaluation, based on data from client-initiated telephone contacts and surveys of the community, demonstrated project success in increasing awareness and utilization of community mental health resources. The overall impact on assessment attitudes was positive; changes in resolution attitudes were not statistically significant. Implications for mass media utilization in mental health prevention are discussed.     

Building Community Capacity to Help Persons with Mental Illness

Abstract

In a time of crisis, persons with mental illness who encounter the police are often sent to jail or are involuntarily committed for psychiatric evaluation when mental health services are not readily available. To better serve these persons and the community, law enforcement and mental health professionals in one Iowa county joined together to ensure these individuals received the assessment and treatment needed. The results of a mixed methods program evaluation are detailed and suggest that the process to develop the program was as critical as the conduct of the program itself. Implications for program development are discussed.     

Mediators and moderators in the evaluation of programs for children. Current practice and agenda for improvement

The author examines the role of mediators and moderators in the evaluation of programs for children. The terms are defined and examples of each are presented. Using bibliometric analysis, the author examines how evaluators use mediators and moderators in treatment studies in education, juvenile justice, health care, child protection, and mental health. The use of mediators and moderators is sporadic and vague at best. An agenda for improvement is outlined that includes greater use of program theory and intensive case studies to find out why researchers in prevention and health promotion incorporate mediators and moderators more effectively in their evaluations.     

Complexity of process and role in the evaluation of an interdisciplinary mental health education program: The deacon's masterpiece

Research and literature on program evaluation must attend to problems of application, which can have major effect on the outcome. Increasing numbers of innovative, multifaceted programs operating in complex settings are presenting for evaluation. Evaluation process must become flexible, creative, and multifaceted in order to produce valid and comprehensive results, and to meet the needs of this field. Evaluators and evaluation, the evaluated program, and the environment become a mutually interacting system in the process of evaluation. Evaluation goals and roles must be conceptualized, agreed upon, sanctioned, and appropriately implemented or conflict and ineffectiveness will result. Three models of evaluation emerge: objective and independent, objective but serving one interest, and pseudo-scientific public relations. Evaluation of an interdisciplinary mental health education program illustrates these issues. Evaluation technology, process, and roles must all be attended to or the weak link undermines the rest.     

The problem of replicability in program evaluation. The component solution using the example of case management

Specificity and replicability of a program are crucial in order to rigorously evaluate the program's effectiveness. However, the traditional approaches to evaluation have ignored the problem that most social and mental health programs consist of vaguely specified interventions and cannot meet minimal standards of replicability. Consequently, the results of effectiveness evaluations of poorly defined programs have little general generalizability. Nevertheless, effectiveness information concerning su ch programs is much needed. One solution to this problem is to focus on program components, examining interventions at a level where operational definitions can be employed and replicability is possible. In this paper, this approach is expanded using the example of case management programs. A component-oriented approach ensures a systematic evaluation of program effectiveness for which the first question for evaluators is whether a program consists of interventions that reliably produce specific immediate outcomes. Then, and only then, is it possible to address questions concerning whether a program will reliably produce those outcomes that are the overall objectives of the program.     

Adoption Competent Clinical Practice: Defining Its Meaning and Development

The need for adoption competent mental health services has been well documented. However, the term “adoption-competent” has lacked a standardized, broadly accepted definition. This article reports findings from two related studies. The first examines how adoption competencies are demonstrated in practice by clinicians participating in an evidence-informed adoption competency training program. The second is an online survey designed to determine whether members of adoption kinship networks agree with a definition of an adoption competent mental health professional developed by experts. Both studies contribute to our understanding of what constitutes “adoption competent” clinical practice.     

Evaluation in a down-loaded mental health system.

Speculation that policy and funding shifts in the nation's mental health system would negatively impact program evaluation services in community mental health centers (CMHC) is substantiated. Performance of program evaluation activities was investigated in 71 CMHCs in 15 states over a two-year period. Twenty-five percent of the centers reported cutbacks, including staff attrition and diminished funding devoted to evaluation functions. The majority of centers reported no changes, but many directors representing these centers also indicated that they had never fully developed a capacity for performing evaluations and would cease to devote attention to these activities without available funding. In some centers evaluation functions are being transferred to clinical administrators; integrated into quality assurance activities; performed "as needed' by external consultants; or ignored altogether. Many centers are developing or enhancing a computerized information system to improve internal efficiency and to meet state accountability requirements. The professional issues raised to these trends as well as their implications for federal and state policy are discussed.     

Evaluation in the community mental health centers program: A bold new reproach?

The Federal Community Mental Health Centers Program (CMHC)-from 1963 to 1981-was heralded as a revolution in mental health care. Championed by many, and severely criticized by others, the actual impact of the program on the nation's mental health remains unclear. The authorization to evaluate the CMHC Program came originally from congressional legislation (PL 90-174), and later from the policies and regulations of NIMH under a series of Federal laws, notably PL 94-63. From 1976-1980, two dominant evaluation strategies were prevalent: funds expended by NIMH each year for studies of CMHC services or program-wide evaluations, and a much larger expenditure by CMHCs to conduct their own, independent evaluations following federal guidelines. As the Center's Program was turned over to the states in the form of block grants (PL 97-35), a group of professionals involved with setting and carrying out federal CMHC evaluation policy of both varieties met in public forum to debate the impact of these two evaluation approaches. While some participants cited gains in evaluation technology and impact upon local management of CMHCs, others found the lack of a coordinated and systematic approach to evaluating the CMHC Program to have been an opportunity missed. The impact of CMHC evaluation efforts are also discussed in terms of their major contribution to the field of evaluation research as a whole.     

Evaluating a community based mental health programme in West Bengal,India: Description of the methodology and lessons learned

Mental disorders impose an enormous burden on society. In developing countries like India, there is a lack of adequate number of trained mental health professionals to provide specialized care and 75–85 % of affected individuals do not have access to appropriate mental health services. The National Mental Health Programme (NMHP) is being implemented by the Government of India to support state governments in providing mental health services in the country. The Urban Mental Health Programme (UMHP) is a pilot initiative that has attempted the integration of mental health services in primary health care settings in two municipal wards in Kolkata, West Bengal, India. The overarching aim of this paper is to describe the methodology used for the evaluation of the community based mental health programme and to understand the processes of the programme in terms of barriers and facilitators. The current evaluation is based on a concurrent nested design, where qualitative and quantitative data are both collected at the same time but analysed separately and priority was given to qualitative data. This experience will contribute in helping other researchers to make some evaluations more effective, useful and manageable. Ethics approval was obtained from an institutional ethics committee of an organization (Ekjut) based in Ranchi, Jharkhand, India. The evaluation was undertaken by the George Institute for Global Health, New Delhi from February- June 2016.     

An examination of methods in the evaluation of volunteer programs.

Although many volunteer programs have been called "successful" and the use of paraprofessionals urged, few program evaluations with a benefit-cost analysis have been reported. Several major studies of volunteer utilization are reviewed, and particular needs with reference to program evaluation are discussed. Three alternative evaluation methodologies are proposed. The importance of utilizing a comprehensive evaluation incorporating all three methodologies is discussed. An ongoing volunteer program in a community mental health center is used as an illustration.     

Nursing in tele-mental health

This article discusses a cooperative project involving an academic health sciences center, a state university, a women's shelter, and a community mental health center in East Texas, a medically underserved, rural region of Texas. The U.S. Department of Commerce provided grant funding to develop a telehealth network to provide an evaluation, referral, and treatment program for victims of domestic violence. Nurses were involved in all phases of the project, from initial conception through development, implementation, and evaluation. The authors concluded that all of the women involved in the study had significant mental health issues and other health concerns that were not being addressed by the existing health care delivery system. Without the use of telehealth, these women would have had limited access to primary health care and virtually no access to mental health services. The East Texas Tele-Mental Health Network demonstrates the value of the clinical and organizational skills that nurses bring to innovative models of mental health service delivery.     

Evaluation of a complex, multisite, multilevel grants initiative

The Safe Schools/Healthy Students (SS/HS) national evaluation seeks to assess both the implementation process and the results of the SS/HS initiative, exploring factors that have contributed to or detracted from grantee success. Each site is required to forge partnerships with representatives from education, mental health, juvenile justice, and law enforcement, coordinating and integrating their efforts and working together to contribute to comparable outcomes (e.g., reduced violence and alcohol and drug use, improved mental health services). The evaluation uses multiple data collection techniques (archival data, surveys, site visits, interviews, and focus groups) from a variety of sources (project directors, community partners, schools, and students) over several years. Certain characteristics of the SS/HS initiative represent unique challenges for the evaluation, including the absence of common metrics for baseline, outcome data, and lack of comparison group. A unifying program theory was required to address these challenges and synthesize the large amounts of qualitative and quantitative information collected. This article stresses the role of program theory in guiding the evaluation.     

On considering the unintended impact of evaluation: reactive distortions in program goals and activities

The relationship between program planning and evaluation can be viewed as bidirectional; that is, evaluation methods, procedures, instruments, and criteria not only are determined by, but also influence, program goals and activities. Within the human services context, several factors or sources of reactivity between evaluation and program planning can be identified. These involve (a) quantification of goals and activities, (b) preferences by different audiences for various kinds of evaluation data, (c) values and evaluation criteria, and (d) evaluation requirements and resource availability. Effects of these reactive features are discussed and illustrated with examples drawn from mental health evaluation and accountability practices. It is argued that for evaluation to be a credible and useful practice, evaluators should plan their efforts and assess their own effectiveness within the larger context of human service systems.     

Continuity of Mental Health Care Across Military Transitions

Multiple deployments, a prominent characteristic of operations Enduring Freedom, Iraqi Freedom, and New Dawn, have been linked to increased risk for various mental health conditions. It is also well known that military transitions, such as a change of duty station or discharge, are an inevitable part of military life. While military transitions can be stressful for any service member or veteran, those with mental health conditions face additional challenges. Transition stress can exacerbate a mental health condition that in turn hinders ability to manage the transition, resulting in a cycle of stress. As a result, those receiving mental health care while transitioning often experience difficulty continuing treatment post-transition. This article describes a unique and innovative transition coaching program designed to address this emerging issue.     

Medicare Funded Mental Health Social Work: Better Access to What?

ABSTRACT

Better Access to Psychiatrists, Psychologists and General Practitioners through the Medicare Benefits Scheme initiative (Better Access), is an Australian Federal Government scheme aiming to improve access to mental health services. Accredited mental health social workers have been involved in the delivery of services under Better Access for more than a decade. In this time, there have been significant changes in the field of mental health services, with consequent increases in size and costs for Better Access. Better Access now represents public spending of more than one billion dollars per annum, yet there is no ongoing account of its impact. In this paper we consider the policy chronology and evaluation of the program, using current available data to question the impact of Better Access on both the service system and the nature of mental health social work practice. This case highlights the importance of a clear articulation of the scope of mental health social work, and ongoing monitoring of the impact of policy in a policy environment increasingly characterised by individualised funding and service delivery structured according to the principles of New Public Management.

IMPLICATIONS

• Social workers' increasing involvement with Better Access has reshaped the scope of mental health social work practice.

• The impact of Better Access on outcomes for service users and mental health social work is not currently a focus of research attention.

• Better Access is a case study illustrating the potential of the policy context to directly shape social work practice, therefore critical engagement with the impact of Better Access is an essential professional imperative.

     

Using an action research approach to involving service users in the assessment of professional competence

This paper reports on a piece of action research that has involved people who use mental health services in systematically providing feedback from a user perspective on participants’ assessed work completed for one module of a masters’ training programme in mental health. In an attempt to improve professional practice and include people who were accessing mental health services in so doing, it outlines how users were trained to provide feedback and the training methods employed. The findings summarise the kind of issues users raised in their feedback to participants about the evidence professionals provided to demonstrate their learning from the training programme. A focus group interview with professionals provides a contrasting insight into the participants’ experience of having their work commented upon from a user perspective. The paper draws on the experience of a five-year external evaluation of an interdisciplinary programme in community mental health at Birmingham University in the UK which has highlighted the involvement of people who use mental health services as a particular innovation in the design, delivery and evaluation of the curriculum.     

Case management: lessons from earlier efforts

Case management is a major, new initiative in the provision of community support services to implement public policy of maintaining mental patients in the community. A review of the case management function as used in the anti-poverty program in the 1960s, shows that a case manager works in a task environment consisting of a multitude of independent service agencies, works within the culture of a core service agency, and relates to chronic clients. The case manager's position is shaped by each of these relationships. Each provides dimensions for a formative evaluation of the program. Case management evaluation can identify issues, and document problems to contribute to the rational redesign of our service system.