Barriers and facilitators to evaluation of health policies and programs: Policymaker and researcher perspectives

Our research sought to identify the barriers and facilitators experienced by policymakers and evaluation researchers in the critical early stages of establishing an evaluation of a policy or program. We sought to determine the immediate barriers experienced at the point of initiating or commissioning evaluations and how these relate to broader system factors previously identified in the literature.We undertook 17 semi-structured interviews with a purposive sample of senior policymakers (n = 9) and senior evaluation researchers (n = 8) in Australia.Six themes were consistently raised by participants: political influence, funding, timeframes, a ‘culture of evaluation’, caution over anticipated results, and skills of policy agency staff. Participants also reflected on the dynamics of policy-researcher relationships including different motivations, physical and conceptual separation of the policy and researcher worlds, intellectual property concerns, and trust.We found that political and system factors act as macro level barriers to good evaluation practice that are manifested as time and funding constraints and contribute to organisational cultures that can come to fear evaluation. These factors then fed into meso and micro level factors. The dynamics of policy-researcher relationship provide a further challenge to evaluating government policies and programs.     

Process evaluation of a diversity training program: the value of a mixed method strategy

Patients’ health and health needs are influenced by categories of difference like sex, gender, ethnic origin and socioeconomic status (SES). To enhance awareness of this diversity among patients and to provide holistic care for them, health professionals should first be aware of the relation between dimensions of diversity and patients’ health and health demands. This paper presents a formative process evaluation of a diversity sensitivity training programme for healthcare professionals. The training was implemented in three healthcare settings (mental healthcare, nursing home and hospital care). Mixed methods were used to monitor the implementation of the training and its effects after three years. Findings demonstrate that the training stimulated participants’ awareness, knowledge and critical attitudes towards diversity. Their motivation and willingness to take action regarding diversity was also enhanced. Yet these developments were less apparent among nursing home participants who felt less satisfied and did not develop a critical perspective on this issue. Qualitative data were helpful to explain differences between the settings. By means of the combination of quantitative and qualitative data, we can conclude that individual learning was not enough to guarantee a sensitive approach to diversity at the organizational level.     

Implementing the ecological approach in tobacco control programs: results of a case study

The ‘new public health’ has stimulated calls to adopt an ecological approach for action aimed at improving population health. However, the implementation of ecological programs has often remained a challenge. The objective of this study was to identify factors facilitating the implementation of ecological tobacco control programming in two Public Health Departments in Québec (Canada) using Scheirer's model of program implementation as a conceptual framework and an exemplary case study design. Analysis revealed two distinct configurations of environmental, organizational and professional characteristics that were related to equally successful implementations of the framework. Key roles for factors such as provincial ministerial guidelines and financial resources, human resources, and relationships and alliances with other organizations, were highlighted. These results replicate findings from quantitative surveys and allow better understanding of the conditions required for planning ecological programming.     

Assessment of the monitoring and evaluation system of a population-based program: Theory-driven evaluation approach

The purpose of this study was to assess the performance of a monitoring and evaluation system (M&E) for a population-based program and to identify factors influencing its performance. The state health department provided a grant, guide, and training for health districts to implement the program and report quarterly and annually to the M&E. Some health districts did not meet this requirement. We also found that the M&E did not adequately capture the actual work of the districts in communities as related to the work plan. We identified several factors that negatively affected the performances of the M&E. For example, the state and health districts did not have a shared vision on how to develop and implement the program, and what to report to the M&E. We proposed strategies that could improve the timeliness and content of reports to the M&E, as well as its utilization in the future.     

Strategically creative: A case of the library planning process

Abstract

Best planning day ever, green, fun, play, flexible … are terms not usually associated with strategic planning. At the University of Technology, Sydney (UTS), plans to open a new library building in 2016 have ignited discussions about the library of the future, its meaning and role in a digital world, and the implications of further rapid changes. The Library's strategic planning process in 2010 was an opportunity to take the collective energy further by involving library staff in the strategic thinking and planning. The process aimed to provide conditions for open conversation through Gadamerian play, and creative modelling based on ideas of revealing and imagineering. The planning started with a playful engagement to generate ideas and promote divergent thinking, and moved through several stages of increasingly convergent thinking to arrive at strategic actions, which were created and supported by all participants. The formal goals of strategic planning were maintained throughout the process and resulted in a client-focused annual plan. This article considers issues of organizational creativity and strategic planning, and relates them to experiences with the strategic planning at the UTS Library. It argues for a systematic approach to fostering creativity and innovation in libraries.     

Governance and evaluation: The case of EU regional policy horizontal priorities

This paper argues that the framing of many programme evaluations tends to eschew any rigorous exploration of the governance aspects of programme design and implementation. This is illustrated by an examination of the method-based approach endorsed by the European Commission for the evaluation of its regional policy programmes. In line with previous work, this found that such evaluations, particularly those using management-based approaches, underplay the role of social and political agency in programme framing and implementation, and therefore in explaining outcomes. Acknowledging such factors is especially important in evaluating cross-cutting themes such as equal opportunities, environmental sustainability and social inclusion. Known as horizontal priorities in European Union policies, such issues at their heart reflect unease with the traditional functional organisation of the modern state, which while providing organisational efficiency gains for planning, is not able to cater for complex and multiple social realities in a postmodern world. Using evidence from South Yorkshire, United Kingdom, and in particular the Mid Term Evaluation of the 2000–2006 European Union Structural Funds Programme, the paper argues that a robust evaluation of such horizontal priorities has to incorporate the full range of governance factors. This is because their framing, implementation and possible achievements are contingent on, and shaped by, the mobilisation of social and political actors.     

Gender mainstreaming and evaluation in the EU: comparative perspectives from feminist institutionalism

This article presents a focused comparative analysis of the institutionalization of two governance practices in the European Commission that levy distinct challenges to the gender status quo: gender mainstreaming (which overtly challenges gender bias) and evaluation (which does not have explicit feminist aspirations). With reference to five dimensions, we identify evaluation as relatively strongly institutionalized, and gender mainstreaming as relatively weakly institutionalized. We draw on the explanatory power of feminist institutionalism to unpack these findings, arguing that a feminist institutional perspective can shed light on this variation, as it provides greater insight into the formal and informal institutions that constrain, enable and shape the implementation of evaluation and gender mainstreaming. We assert that the notion of path dependency, the logic of appropriateness, and the concept of layering serve as useful tools to understand the gendered nature of the European Commission. This research provides insights into the institutional factors that impact the implementation of gender equality strategies (such as gender mainstreaming). In turn, this contributes to the development of more effective strategies to promote institutional change toward greater equality.     

Breastfeeding interventions in Kansas: A qualitative process evaluation of program goals and objectives

Across the state of Kansas, eighteen public health departments received funding through the 2011 Breastfeeding Grant Initiative to start a breastfeeding intervention. The main objective of this study was to evaluate the progress toward program goals and objectives. This study was a process evaluation. Qualitative data were collected from recipient health departments at two time-points during the program year. Structured, open-ended questions were asked through telephone interviews. This study examined: (1) progress toward program goals and objectives, (2) problems encountered during implementation, and (3) evaluation measures employed to assess program impact. All health departments reported making significant progress toward program goals and objectives and reported successful collaboration with other healthcare providers. The use of breast pumps, educational classes, and professional training of staff were reported as providing the best outcome in the promotion of breastfeeding. The majority of respondents did not measure program impact. From a public health perspective, it is important that infants receive breast milk for the first six months of life. It appears that goals and objectives set a priori guided health departments with the administration of their breastfeeding program. Results may be used to enhance and sustain delivery of breastfeeding support programs in Kansas communities.     

A critical friend: Monitoring and evaluation systems,development cooperation and local government. The case of Tuscany

The role of monitoring and evaluation (M&E) systems in the field of development cooperation has globally increased in last decades. International and regional organizations, as well as states, local governments and NGOs have largely adopted the tools provided by M&E in order to enhance transparency, effectiveness and efficiency. The paper aims at verifying how and to what extent the implementation of M&E systems has affected the overall quality of international cooperation projects financed by a local government. After a literature review on M&E in development cooperation, the research analyzes the wide range of activities (evaluation ex-ante, mid-term, final, monitoring, consultancy) carried out by the Evaluation Team of the XY in the last eight years in behalf of an Italian local government: the Region of Tuscany. The paper reveals the strategic significance of adopting M&E systems in the medium-long term.     

Reviewing the ‘march to standards’ in public relations: a comparative analysis of four seminal measurement and evaluation initiatives

To many, development and adoption of professional standards for measurement and evaluation (M&E) is one of the most promising approaches for advancing public relations practice. In recent years, there has been a surge in efforts to develop standards for M&E in different parts of the world. Prominent examples of this include standard terminologies, metrics, principles for best practice in the field, and evaluation frameworks. Regardless of their alleged importance, however, the acceptance and application of such M&E standards in the practice varies significantly. To better understand the process by which standards in this field are developed and adopted, this article draws on recent concepts from organization studies (cf. Slager, Gond, & Moon, 2012) to analyze the trajectories of four seminal standards attempts: The Barcelona Principles and the AMEC Integrated Evaluation Framework on an international level, the DPRG/ICV Framework used in German-speaking countries, and the GCS Framework in the United Kingdom. The article reveals, by way of an interpretive qualitative approach, the various strategies undertaken to a) develop common sets of terms and rules, b) engage relevant actors in the design, promotion, and implementation of proposed standards, and c) to reinforcing standards symbolically.     

Cultural intermediaries and the circuit of culture: The Digital Ambassadors project in Johannesburg,South Africa

The question raised by Cheney and Christensen (2001) as to “what a non-Western, non-managerial and non-rationalist form of public relations will look like” (p.182), together with the call made by Gregory (2014) for public relations practitioners (PRPs) to be active social change agents, motivated this paper. The aim of this research is to follow a culture-centred approach and apply the circuit of culture (Curtin & Gaither, 2005) to investigate the meanings that young people (aged 18 to 34), as cultural intermediaries, ascribe to their participation in a digital empowerment project implemented by the City of Johannesburg, South Africa. This research forms part of a larger, ongoing multi-disciplinary research project.The research highlights the complexity of projects of this nature, the wide variety of stakeholders involved, the significance of structure, the importance of training of cultural intermediaries, and the benefits such intermediaries gain from their involvement. A key finding is that self-development and financial benefit contribute significantly to the success of cultural intermediaries. This paper contributes to the field of public relations by providing an analysis of cultural intermediaries’ contribution to the circuit of culture as influenced not only by production and consumption but also by identity and regulation as seen in the way they represented the meaning created during their experience.     

War outside, ceasefire inside: An analysis of the performance appraisal system of a public hospital in a zone of conflict

Our study examines the use of the performance appraisal system at Hebron Public Hospital (Palestine) during the second intifada, started in 2000. The aim of the article is to shed light on the reasons behind the use of performance appraisal systems in organizations operating in zones of conflicts, an area relatively neglected by HR scholars. To create the theoretical fundament we draw on mainstream literature on performance appraisal, contextualizing it to the Middle-Eastern context. From the literature analysis, we identify five guiding logics for the implementation and use of performance appraisal systems (Appendix A). We use a multi-method approach, qualitative and quantitative, to analyze the longitudinal performance evaluation data over the period 2000-2002 for about 250 individuals. These data are complemented with interviews and observations in the field.Our analysis shows that the trends evidenced in the quantitative analysis are similar to trends evident in Western contexts. However, these trends were not the consequence of the same five Western logics found in the literature. The qualitative study allows us to identify two additional logics for making sense of the performance appraisal system at Hebron Public Hospital: the need to find peace within the organization (organizational peacefulness logic); and the need to maintain order through the acceptance of the status quo (dominance logic). These results allow us to draw conclusions for theory and practice of HR management and to identify useful criteria for doing research in areas of conflicts.     

Public intellectuals,political satire and the birth of activist public relations: The case of Attic Comedy

The purpose of this article is to explore the origins of dissent public relations and to establish its emergence in the context of the ancient Greek comedy represented by Aristophanes. Indeed, Attic Comedy (also known as Old Comedy) was the first great example of today’s mass communication, in which political satire was used to dissent and protest against political and social circumstances in fifth-century BC Athens. This situation was conditioned by the Peloponnesian War and its political, economic and social consequences. From this perspective, this article also constitutes an investigation into the intellectual history of public relations, of which Aristophanes can be considered to be one of the first practitioners.     

Sociology, ethics, and the priority of the particular: learning from a case study of genetic deliberations

There are growing debates about the relationship between the two disciplines of sociology and ethics, particularly as they each become increasingly involved in research and policy formation on the life sciences, especially genetics. Much of this debate has been highly abstract, often stipulating the seemingly different character of the two disciplines and speculating on their theoretical potential--or lack thereof--for future collaborative work. This article uses an existing collaboration between a sociologist and an ethicist, on a study of participation in genetic databases, to explore some of the challenges, for both disciplines, of working together. Building upon this case study, we examine the suggestion that the Aristotelian concept of 'phronesis' provides the grounds for establishing one possible theoretical framework with which the disciplines can be bridged. Further exploration of this approach leads to suggestions for ways of thinking about the apparently fundamental divides between the disciplines and for ways of adding to notions of a 'public sociology'.     

The evaluation of the PACA regional public health plan: reconciling the managerial, scientific and democratic finalities

For more than a decade now, evaluation has developed considerably in France, thanks in particular to the Société Fran?aise de l'évaluation, whose charter sets out a number of principles designed to guide the work of evaluators. This article examines how the evaluation process surrounding a regional public health plan (referred to as PRSP)--itself being a new instrument for regional planning in France--accords with one of these principles, which specifies that evaluation must be framed according to "a three-fold logic involving public management, democracy and scientific debate." Our analysis shows that while this evaluation was driven primarily by managerial concerns (i.e., assessing the capacity of the plan to structure health policy in a region), it also provided an Opportunity for restoring dialogue with a range of actors by opening up a space of cooperation and consultation. In addition, in order to ensure the legitimacy of the evaluation's conclusions, the knowledge produced by the evaluators had to rest on an irreproachable methodology. This example illustrates how evaluation, in the French tradition, is a process that strives to reconcile the viewpoints and expectations of managers, scientists and the general public; it is also a process that brings out lines of tension and areas of complementariness between these three logics.     

Coorientation theory in international relations: The case of Slovenia and Croatia

In this paper, we question the possibility of applying the coorientational theory in researching the communication behavior between the ‘general’ publics of two nations, Slovenia and Croatia. We use these findings to offer practical advice on solving the potential communication problems.     

The civility of social capital: Public relations in the public sphere,civil society,and democracy

Scholars have analyzed public relations’ role in democracy via proxy concepts like the public sphere and civil society. However, some have critiqued the public sphere on grounds of equal access and portrayed civil society as a guise for first-world imperialism. These critiques have implications for the role of public relations in the public sphere and civil society. This article suggests the normative role of public relations in democracy is best perceived as creating the social capital that facilitates access to spheres of public discussion and in maintaining relationships among those organizations that check state power. To that end, the paper argues that social capital does much to advance public relations theory and prescribe the role of public relations in democracy. Several implications for public relations from a social capital perspective are offered, including the creation of generalized societal trust, the building of cross-cutting or “weak” ties, the engagement of media on behalf of subaltern counterpublics, and the (re)creation of community or a fully functioning society.     

Credibility of PR practitioners: The impact of professional journalism background on trustworthiness,expertness, and homophily evaluations

This study demonstrates that a professional journalism background contributes to the credibility evaluations of public relations practitioners in regards to expertness and the capability to meet journalistic values and expectations. However, having a professional journalism background does not necessarily render public relations practitioners any more trustworthy than not having such credentials.     

Explicating the public memory dialectic in public relations: The case of Donald Trump,The Oath Keepers,and January 6, 2021

The public relations practice participates in making, shaping, telling, and interpreting societal memory to influence issue positions and related actions. That claim implies several themes. One, societal memory is a useful concept for understanding the strategic processes of meaning and meaning making, narratives which constitute society. Two, the content of their prevailing memories shapes the choices individuals make separately and collectively. Public relations’ centrality to societal memory remains an underappreciated and underexplored research area. Scholars have explored the role public relations plays in societal memory by examining, for instance, the textuality of art as well as memorials and statues, their erection and erasure. To better understand this process, scholarship needs to examine strategic public relations efforts to “blur” societal memory as a means of creating alternative versions (revisionist history) of lived truth, values, norms, and policies. After examining memory, including blurring, we use the January 6, 2021 United States presidential election (Congressional Certification) Capitol “protest event” to demonstrate how blurring occurs and its potential adverse implications. We suggest and conceptualize in a model that societal memory occurs as a dialectic; some etched version of history (i.e., thesis) comes under question (i.e., antithesis), and through either a process of blurring, erasure, or retention, that etched history is then muddied, expunged, or lives on (i.e., synthesis). Meaning matters in the enactment and critical assessment of public relations’ role in societal agency.