An evaluation u-turn: From narrow organisational objectives to broad accountability

The development of evaluation scholarship progresses, but the slow and incremental steps taken are largely refinements on a direction of thinking that has become established and virtually taken for granted. Most frameworks and models which have gained recognition in the academy and practice follow programme logic methodology. They also point to the achievement of organisational objectives as being the principal, but not always sole, purpose of public relations programmes. The achievement of organisational objectives has therefore been the focus of evaluation.Parallel, to this, a second strand of thinking in the scholarship and practice has questioned the assumption that the meeting of organisational objectives is paramount and whether organisations alone should determine the measures of success. Other perspectives such as stakeholder judgements of success and the obligation of organisations to meet societal expectations have been discussed as legitimate ways in which to evaluate programmes and activities.This paper takes forward this second strand by proposing that the evaluation debate should be ‘flipped’. Instead of organisational objectives being a primary denominator for evaluation, the paper proposes that whether organisations discharge their accountabilities to stakeholders and society is of equal, if not more weight. An organisation will succeed only if it sustains its licence to operate. To achieve that it has to meet the reasonable expectations that licence-givers have and to obtain their active consent to support its actions. The key questions for public relations are therefore: who/what holds us to account? What are their expectations? How are their expectations fulfilled? Drawing on the management and public relations literature to answer these questions, the paper calls for a radical ‘turn’ from the prevailing thinking, which the authors suggest is more about public relations explaining and justifying itself, towards a model that aligns with contemporary management thinking and practice.The paper proposes a model and process that provides a way forwards based not only on organisational aspirations, but on a hierarchy of obligations: to society, stakeholders, the organisation, and to the professional function which runs campaigns. It offers a principles based evaluation approach. In doing so, it takes the evaluation debate in a new direction and thereby adds to theory-building. It also offers a new way forwards for the practice which has become increasingly mired in developing more comprehensive lists of indicators and metrics and expanding taxonomies of evaluation terms.     

Valuing internal communication; management and employee perspectives

Effective internal communication is a prerequisite for organisational success. Organisations need to evaluate and improve communication especially in increasingly difficult economic pressures. Assessment instruments enable organisations to monitor communication effectiveness. This review of academic and consultancy studies found over reliance on measuring satisfaction with the communication process. The analysis found management-centric rather than employee-centric approaches to assessment. This indicates a need to develop new approaches to assessing internal communication. A conceptual model is proposed here to encourage focus on employee communication needs in terms of content as well as channel. Future approaches to assessment should draw on a wider theoretical and conceptual framework. Assessment instruments need to reflect advances in practice. They need to assess the value of internal communication to employees as well as their organisations.     

Zu subtil?

Recent research on inter-ethnic relations in Germany discusses fundamental changes in patterns of attitudes and latent discrimination on the part of Germans towards ethnic minorities. It is argued that “modern”, rather subtle forms of attitudes and social distances emerged which cannot be measured with conventional indicators. Based on data from a Eurobarometer survey, Pettigrew and Meertens developed two scales — the Subtle Prejudice Scale and the Blatant Prejudice Scale — to close the gap. The aim of this article is to find out whether these scales are useful as supplement or substitution for commonly used measurement instruments in Germany. On the basis of data collected in 1999 the reliability and validity of the scales are tested. Moreover, the relations between the newly arranged indicators and the often used measurement instruments from the German General Social Survey (ALLBUS) are analysed for the first time. The empirical results show that especially the reliability and validity of the Subtle Prejudice Scale is questionable. It is concluded that the new scales in their current form provide no useful alternative to the commonly used measurement instruments.     

Institutionalising the evaluation function: A South African study of impartiality,use and cost

PurposeThis article explores the implications of outsourcing the evaluation function in South Africa, a context where there is a mismatch between evaluation supply and demand. It unpacks the tradeoffs between internal and external evaluation, and challenges some commonly held assumptions about both.ApproachBased on experiences as an internal evaluator, external evaluator, evaluation manager, and building evaluation capacity, the author explores how each role changes when evaluation is a scarce skill, and looks at implications outsourcing has for both the organization, and the evaluation.FindingsThe purpose of the evaluation must drive the decision to outsource. However, with changing models of collaboration, there may be hybrid options that allow organizations to build evaluation capacity.Practical implicationsOrganisations are faced with a trade-off between commissioning an evaluation, and building internal evaluation capacity. To better understand each approach, it is important to consider the purpose and context of the evaluation. This shifts some commonly held assumptions about internal and external evaluations. Re-examining these assumptions will help organizations make a more informed decision about an evaluation approach.Originality/valueThe field of evaluation is particularly concerned with evaluation use. Most of the literature on this has focused on the approach of individual evaluators, and insufficient attention has been paid to the institutional architecture of the evaluation. This article considers how some of the organisational structures around an evaluation contribute to evidence use, and the case study of South Africa also shifts the focus to the central but overlooked role of context in the debate.     

An Assessment of the Psychometric Properties of Italian Version of CPGI

The aim of this study was to adapt to the Italian context a very commonly used international instrument to detect problem gambling, the canadian problem gambling index (CPGI), and assess its psychometric properties. Cross-cultural adaptation of CPGI was performed in several steps and the questionnaire was administered as a survey among Italian general population (n = 5,292). Cronbach’s alpha reliability coefficient was 0.87 and can be considered to be highly reliable. Construct validity was assessed first by means of a principal component analysis and then by means of confirmatory factor analysis, showing that only one factor, problem gambling, was extracted from the CPGI questionnaire (an eigenvalues of 4,684 with percentage of variance 52 %). As far as convergent validity is concerned, CPGI was compared with Lie/Bet questionnaire, a two-item screening tool for detecting problem gamblers, and with both depression and stress scales. A short form DSM-IV CIDI questionnaire was used for depression and VRS scale, a rating scale, was used for rapid stress evaluation. A strong convergent validity with these instruments was found and these findings are consistent with past research on problem gambling, where another way to confirm the validity is to determine the extent to which it correlates with other qualities or measures known to be directly related to problem gambling. In sum, despite the lack of a direct comparison with a classic gold-standard such as DSM-IV, the Italian version of CPGI exhibits good psychometric properties and can be used among the Italian general population to identify at-risk problem gamblers.     

Further comment on the homosexual role: A reply to goode

The Herpes Attitudes Scale (HAS) and the Herpes Knowledge Scale (HKS) were developed as research instruments to investigate knowledge and attitudes among college students about herpes. The HAS is a 40‐item Likert‐type scale with an internal consistency reliability coefficient of .91, based on a sample of 148 students. The HKS is a 54‐item true‐false test with a reliability coefficient of .88, with the same sample. Content validity of both scales was evaluated by a panel of judges knowledgeable about herpes and/or instrument construction. Since the scales are designed to detect unfavorable attitudes and general misconceptions about herpes, they can function in research applications and as both needs assessment and evaluative tools in herpes education programs.     

Development and validation of a measure of organizational capacity for implementing youth development programs

Accumulating evidence indicates that incorporating youth development (YD) principles, strategies, and supports into an organization promotes positive adult and youth outcomes. However, few validated measures assess this type of capacity. The YMCA commissioned a study to validate its Capacity Assessment for Youth Development Programming (Y-CAP), which examines the organizational infrastructure required to implement YD programs and processes in seven areas. Survey development was an iterative process informed by existing frameworks, instruments, and pilot testing of items. The Y-CAP was reviewed and revised three times prior to this study, with a final round of revisions made at the start of the validation phase as a result of thorough content, survey methodology, and psychometrics reviews. The revised Y-CAP was completed by 123 YMCA implementation teams. Rasch analyses were used to determine the extent to which validity evidence supports the use and interpretation of the Y-CAP scores. Convergent validity was assessed by comparing Y-CAP scales to the Algorhythm staff survey for youth-serving organizations, and focus groups informed the consequential validity of the Y-CAP. The results provide strong evidence for the reliability and validity of the Y-CAP, which can be used to guide continuous quality improvement initiatives that support capacity and functioning in youth-serving organizations and programs.     

The Development and Validation of a Scale to Measure Attitudes Toward Women's Genitals

ABSTRACT

Sexual behaviors may be related to or influenced by a range of factors including individuals' attitudes toward women's genitals. To date, however, no reliable and valid measure exists with which researchers or clinicians can assess individuals' attitudes toward women's genitals. The purpose of this study was to develop a reliable and valid scale to measure attitudes toward women's genitals and to test its predictive capacity.

The process of developing the Attitudes Toward Women's Genitals Scale (ATWGS) was conducted in three phases: (1) elicitation, which involved a comprehensive literature review and surveys completed by 370 men and women; (2) development, review, and revision of an initial item pool; and (3) survey administration and psychometric analysis, which involved an initial survey administration to 604 men and women and a second administration, for the purposes of establishing temporal stability of the scale, to 16 individuals. The final ATWGS resulted in a 10-item scale that exhibited sufficient internal consistency (Cronbach's alpha = 0.848) and convergent validity, as it was correlated at 0.429 with the five-item Sexual Opinion Survey. The ATWGS also showed evidence of predictive capacity in that women's scale scores differed significantly based on their history of cunnilingus, gynecological care, vibrator use, and orgasm. For men, there were significant ATWGS score differences based on their experience with cunnilingus. Also, men scored significantly higher on the ATWGS than did women (p < .001), indicating more positive attitudes. The ATWGS was found to exhibit evidence of sufficient reliability, validity, and predictive capacity.     

Pathways to becoming an internal evaluator: Perspectives from the Australian non-government sector

There is a lack of clarity around intra-organisational evaluation roles and pathways into these roles in non-government organisations (NGOs). This article presents three auto-narratives from the authors who are working as internal evaluators in the NGO sector. We examine this phenomenon of role ambiguity by exploring our evaluation journeys and struggles to find identities in the formal evaluation community. Findings from the auto-narratives identify implications for the evaluation field regarding professionalisation. This article explores how aspects of professionalisation, such as clarification of roles and tasks of internal evaluators, could facilitate their recruitment, assess credibility and guide career trajectory. Elucidating internal evaluation career pathways contributes to the evaluation discipline by providing information relevant for evaluation capacity building, evaluator training, and the professionalisation movement.     

Multidimensional Resilience: Predictive and Construct Validity of the Individual,Family, and Community Resilience Profile

Few measures parallel the robust depth offered in the existing multidimensional and ecologically informed theories of resilience. This study sought to evaluate the test–retest reliability, construct, and predictive validity of the individual, family, and community resilience resource profile (IFCR-R). The IFCR-R measures a family’s access to resilience resources and protective factors across multiple ecological levels. Confirmatory factor analysis was used with a sample of n?=?810 low-income families. And 159 families completed multiple time point measures for test–retest reliability and predictive validity evaluation. Results of this study support the proposed multidimensional ecological structure of the IFCR-R and found that the IFCR-R offers an acceptable test–retest reliability and predictive validity for outcomes of mental and physical health.     

Development of the Mental Illness and Disorder Understanding Scale

Abstract: Although administrative bodies such as community health centers have implemented various awareness campaigns to promote community understanding of mental health and to reduce the stigma associated with mental illness and disorder, there have been few reports that have quantitatively analyzed the effects of these campaigns in Japan. Therefore, this study was conducted for the purpose of developing the evaluative “Mental Illness and Disorder Understanding Scale (MIDUS)” for awareness campaigns and assessing its reliability and validity. A survey was conducted on 1 004 community residents using a self‐administered questionnaire. An assessment of the reliability and validity of the scale was then conducted. The Cronbach a coefficient of the MIDUS was 0.78, thus indicating adequate reliability. Factor analysis of the MIDUS revealed three factors: “Treatability of illness”, “Efficacy of medication” and “Social recognition of illness”. In addition, a significant correlation was observed between the MIDUS and existing scales, and fixed criterion‐based validity was demonstrated. The MIDUS utilizes items worded in the form of affirmative expressions so as to avoid promoting new bias. It was confirmed to be both reliable and valid, thereby suggesting its usefulness as an evaluative scale of awareness campaigns.     

Exploring the Organisational Collage of Memetic Paradigms for (a) Change — A Research Note

Exploring organisations is the prerequisite for any intentional attempt to strategic change. Yet, what is it that we observe when we observe organisations? The argument chooses a narrative approach to exploring organisations. With Niklas Luhmann we look at the operations of organising which makes the organisation an organisation. The paper suggests the organisational collage (I.) of stories as a starting point of the exploration. The specific focus is on meaning-creation and sense-making as the genuine act of organisational self-observation. The disciplinary matrix (II.) reflects on how stories and narratives crystallise and rule the organisation in a paradigmatic way. Along Thomas Kuhn's understanding of paradigms (III.) management is referenced as an activity of a community of practice based on a disciplinary matrix of models, methods and instruments. Giorgio Agamben's conceptualisation of paradigms as reference giving examples allows the opening up of the implicit side of organisational culture. Memetics (V.) approaching reference giving examples as memes and culture as a meme-complex enable the observation of dynamics and cultural evolution over time. Concluding we come to understand the organisational implications (V.) of the conservative nature of organisational development and the systemic sensitivity that allows for management, learning and change. And as always, advances in research come at the price of new questions.     

Can a workbook work? Examining whether a practitioner evaluation toolkit can promote instrumental use

In large-scale, multi-site contexts, developing and disseminating practitioner-oriented evaluation toolkits are an increasingly common strategy for building evaluation capacity. Toolkits explain the evaluation process, present evaluation design choices, and offer step-by-step guidance to practitioners. To date, there has been limited research on whether such resources truly foster the successful design, implementation, and use of evaluation findings. In this paper, we describe a multi-site project in which we developed a practitioner evaluation toolkit and then studied the extent to which the toolkit and accompanying technical assistance was effective in promoting successful completion of local-level evaluations and fostering instrumental use of the findings (i.e., whether programs directly used their findings to improve practice, see Patton, 2008). Forensic nurse practitioners from six geographically dispersed service programs completed methodologically rigorous evaluations; furthermore, all six programs used the findings to create programmatic and community-level changes to improve local practice. Implications for evaluation capacity building are discussed.     

Evaluation of social network measurement instruments

This paper evaluates the reliability and validity of network measurement instruments for measuring social support. The authors present and discuss the results from eight experiments which were designed to analyze the quality of four measurement scales: (1) binary, (2) categorical, (3) categorical with labels, and (4) line production, as well as two measurement techniques for listing alters (free recall and recognition). Reliability and validity were estimated by the true score multitrait–multimethod (MTMM) approach. Meta-analysis of factors affecting the reliability and the validity of network measurement was done by multiple classification analysis (MCA). The results show that the binary scale and the first presentation of measurement instruments are the least reliable. Surprisingly, the two data collection techniques (free recall and recognition) yield equally reliable data.     

Sexual Self-Concept and Sexually Abused Males: Early Data on the Development of a Scale

ABSTRACT

This article describes the early development and psychometric piloting of the Sexual Self-Concept Scale for Sexually Abused Males, a one-dimensional measure assessing the gender-based self-perceptions of males with histories of childhood sexual abuse (CSA). A sample of 459 males with histories of CSA completed a demographic survey and responded to a pool of items using a Likert-type scale. Strong and promising evidence was rendered for face validity, content validity, convergent validity, criterion validity, discriminant validity, test-retest reliability, split-half reliability, internal consistency and item-total correlations. The scale may be used by helping professionals and clients to identify beliefs, emotions and behavior emanating from CSA with respect to gender-based sexual self-concept, design interventions to modify or ameliorate such effects, and to mark progress across phases of treatment.     

Application of an organizational evaluation capacity self-assessment instrument to different organizations: Similarities and lessons learned

Organizational evaluation capacity (EC) has received significant attention in the evaluation research literature in the past decade. Much of the focus has been on defining organizational evaluation capacity, which can be thought of as the competencies and structures required to conduct high-quality evaluation studies (capacity to do), as well as the organization's ability to integrate evaluation findings into its decision-making processes (capacity to use). This paper seeks to contribute to this growing body of knowledge through a multiple case study of EC across three different organizations (e.g., non-profit, provincial government and federal government, herein named sectors); the novelty of this particular study is that each case study is based on the use of a common measurement tool developed by Bourgeois, Toews, Whynot and Lamarche (2013). The cross-case analysis presented in the paper reveals that evaluation capacity tends to be higher, both in terms of capacity to do and capacity to use, in organizations that have developed systematic mechanisms to institute an evaluation culture within their walls. Interestingly, however, we also found that capacity to use does not first require capacity to do, as evidenced in the non-profit organization under study.     

Development and Validation of the Poverty Attributions Survey

This article describes the process of developing and testing the Poverty Attribution Survey (PAS), a measure of poverty attributions. The PAS is theory based and includes original items as well as items from previously tested poverty attribution instruments. The PAS was electronically administered to a sample of state-licensed professional social workers. The three scales of the PAS—individual, cultural, and structural—demonstrated strong internal consistency reliability. We conducted exploratory factor analyses and forced three-factor analyses and retained items with factor loadings at or above .50. The items on each scale loaded as predicted with few exceptions. The PAS is a new tool for standardizing poverty attribution research. We discuss implications for social work education, research, and practice.     

A measure of community members’ perceptions of the impacts of research partnerships in health and social services

Currently, there are no psychometrically sound outcome measures by which to assess the impacts of research partnerships. This article describes the development of a 33-item, survey questionnaire measuring community members’ perceptions of the impact of research partnerships addressing health or social issues. The Community Impacts of Research Oriented Partnerships (CIROP) was developed using information from the literatures on health promotion, community development, research utilization, and community-based participatory research, and from focus groups involving 29 key informants. Data from 174 community members were used to determine the factor structure, internal consistency, and test–retest reliability of the four CIROP scales, and to provide evidence of construct validity. The CIROP informs research partnerships about the extent of their impact in the areas of Personal Knowledge Development, Personal Research Skill Development, Organizational/Group Access To and Use of Information, and Community and Organizational Development, allowing them to demonstrate accountability to funding bodies. As well, the CIROP can be used as a research tool to assess the effectiveness of knowledge sharing approaches, determine the most influential activities of research partnerships, and determine structural characteristics of partnerships associated with various types of impact. The CIROP provides a better understanding of community members’ perspectives and expectations of research partnerships, with important implications for knowledge transfer and uptake.     

Tailored program evaluation: Past, present, future

This paper discusses measurement issues related to the evaluation of computer-tailored health behavior change programs. As the first generation of commercially available tailored products is utilized in health promotion programming, programmers and researchers are becoming aware of the unique challenges that the evaluation of these programs presents. A project is presented that used an online tailored health behavior assessment (HBA) in a worksite setting. Process and outcome evaluation methods are described and include the challenges faced, and strategies proposed and implemented, for meeting them. Implications for future research in tailored program development, implementation, and evaluation are also discussed.