Toward overcoming problems in evaluation research: A beginning perspective on power

An information feedback model and an ideology of evaluation is posed. The authors' experiences in actually doing evaluation research are then contrasted with the model and ideology. Suggestions are made for closing the gap between reality and the model, primarily by altering the structural constraints that present obstacles to achieving the model. The authors assume that impact on policy is the appropriate criterion for the success of an evlauation, and they conclude that political constraints are primary in preventing evaluation research from having this impact. It is proposed that evaluators need to develop more explicitly political ways of acquiring power of their own in order to maximize such impact. Some ways social researchers might conceptualize and construct an appropriate base of power are considered.     

Interpretive claims and methodological warrant in small-number qualitative,longitudinal research

This article discusses issues of methodological warrant in a qualitative, longitudinal study with only a small number of subjects. The 12 to 18 Project was designed to contribute to research on gender, class and schooling. The rationale for a design using only 26 students spread between four schools is explained. It is argued that even with small-number research, issues of selection and comparison are important--but emphasis on techniques of data-treatment and comparison is misplaced. The meaningfulness and contribution of studies of this type lie in multiple acts of design, comparison, reflexive interpretation and dialogue with the broader field, and the more that such studies emphasize technical analytic procedures, the more they undermine their warrant to be anything other than a report on a small sample. Illustrations of interpretations of the data and from the data are given.     

Toward ethical guidelines for network research in organizations

All sociological/anthropological research (network or otherwise) in organizations (particularly corporations) is complicated by the need to obtain consent not only from the potential respondent but from the organization itself. Whereas in ordinary research there are essentially two parties that must come to agreement – the researcher and the respondent – in the organizational research there are three. In addition, the fact of organizational hierarchy means that the employee's participation in the research entails considerably more risk than in other situations. At the same time, social network research involves special challenges due to the lack of anonymity at the questionnaire level and the sensitivity of some of the questions. This paper seeks to lay out some of the issues and to propose a set of standard guidelines for ethical research on networks in organizations. It is hoped that developing a set of standard guidelines and forms will help Institutional Review Boards (IRBs) to allow network research. Examples of proposed forms are included as appendices.     

Toward a more structured use of information technology in the research community

Information technology has great potential for supporting the activities of research networks. However, some fundamental problems must first be addressed to determine whether the technological support is necessary at all. Once that need has been determined, merely installing a set of isolated, generic information tools is not sufficient to address the full spectrum of network information needs. Therefore, a comprehensive and customized network information system is required. We argue that a specification method can help to structure the development of such an information system. Aldo de Moor finished his M.Sc. in information management in 1993, after which he worked at the University of Guelph, Canada, on the initiation of the Global Research Network on Sustainable Development. He is currently pursuing his Ph.D. at Tilburg University.     

Articulating influence: Toward a research agenda for interpreting the evaluation of soft power,public diplomacy and nation brands

While the terms soft power, public diplomacy (PD) and nation brands have cemented their place in academic discourse during the early 21st century, the evaluation of these activities has not been given anywhere near the same level of attention. When describing how campaigns are evaluated, scholars tend to make assumptions based on the goals or outputs of an initiative rather than on the basis of reliable, empirical data on its results. Strong positivist tendencies within current scholarship usually lead to evaluation being considered in terms of methodology and best practice, typically with the assumption that certain preferred outcomes will be demonstrable if an ideal model is followed. Most significantly, such approaches seem to underplay the interests and objectives that inform and constrain choices surrounding modes of communication and evaluation. I argue here that PD activities are rarely the product of rational choices about communication options, and nor is PD evaluation the result of applying the “best” methodology. Rather, questions of PD and evaluation practices are bound together in complex organizational and power structures that generate pragmatic responses both to the “problem of influence” and the reporting of results. Through use of the concept of articulation, this article outlines a framework for interpreting evaluation practices from a contextualized perspective, which grasps how and why soft power practices assume certain forms.     

Toward building a typology for the evaluation of services in family support programs

This article briefly reviews the history, philosophy, practice principles, and foci of family support programs, examines the typologies currently in use to classify these programs, and discusses the difficulties these classifications pose for program evaluators. The authors introduce a new typology that deconstructs family support programs into their component services and discuss the potential of this typology for evaluation of family support services.     

The problem of justification in social work

Abstract

In this paper we argue that the real test of professional social work practice is whether it can be plausibly, effectively and defensibly justified. Since the early 1950s social work in Australia has engaged in a strategy of professionalisation. This strategy and its implications were described by McDonald and Jones in 2000. This paper supports the concerns expressed. We argue that the way out of our profession's dilemma is by focusing on the problem of justification. The main questions addressed by McDonald and Jones appear to be: What is professional social work practice? Does the concept of professionalism serve us and our clients well now? What form should social work take in the future? Their answer is ‘that the “strategy of professionalisation”, as conventionally conceived by Australian social work, is no longer viable in the emerging milieu.’ We build on that answer by exploring the notion of justification in terms of the concepts Foundationalism, Coherentism and Reliabilism. We conclude by suggesting that the immediate task for social work in this new century is to solve the problem of justification.     

The next decade in evaluation research

The field of evaluation research has developed rapidly in the past few years. Its growth is marked by:(1) establishment of an'entrepreneurial industry of considerable proportions; (2) emergence of a new profession as evidenced by the founding of professional associations, publication of books and journals, and the development of specialized training programs; (3) recognition of the importance of evaluation in the federal bureaucracy by the establishment of key executive positions in the various offices of the Secretaries; (4) quality control efforts by a variety of stakeholders and watchdog groups; and (5) efforts to address methodological problems and to maximize the dissemination and utilization of studies. In the future, continued federal funding in health, education, and welfare will be dependent on demonstrated program impact, and “sunset legislation.” Moreover, there will be fewer important “policy” evaluations and more administrative and managerial efforts. Evaluation questions, increasingly, will be specified either by Congress or the administering agency, including the setting of design and methodological requirements for studies. Opportunities for investigator-inspired studies and basic methodological development will decrease.     

Toward a developmental family therapy: the clinical utility of research on adolescence

Tremendous advances have been made in our understanding of the intrapersonal, interpersonal, familial, and contextual characteristics and processes that contribute to adaptive as well as maladaptive developmental outcomes with high-risk and clinically referred adolescents. This empirical knowledge base offers clinically rich opportunities for systematic treatment development. An important step in this process is distinguishing which research findings in basic science areas such as developmental psychology and developmental psychopathology might have clinical relevance. Toward this goal, we review relevant but selective research in areas that are central to clinical work with adolescents (parent-adolescent relationship, biological aspects, and affect and cognition), and we offer examples of how basic research in these areas can inform treatment.     

The use of participant observation in evaluation research

In the last several years considerable interest has developed in the use of qualitative methods in evaluation research. Recent works indicate that the tradition of qualitative research has much that will be useful to evaluation researchers. This paper does not seek to cover the range of qualitative methods available to the evaluation researcher but rather is concerned with one methodology in that tradition, participant observation. The paper describes the methodology of participant observation for those who are interested in its characteristics and for those who would like to consider it for evaluation or other types of research. Examples are given of how it has been used and references to additional sources of information are included as well. The paper is organized into three sections: characteristics of participant observation, when it is appropriate to use it, and specific techniques of the method.     

Toward an integrated analysis of social movements and new religious movements research

Despite calls from researchers for intersectional studies between religion and social movements over the past few decades, scholars have not engaged in fruitful conversation about integrating the two disciplines. This article aims to facilitate such discussion by examining the topic of new religious movements (NRMs). I first review the existing literature on NRMs and discuss why NRM research has been neglected in social movement studies. Then, I explore a few research areas where both NRM studies and social movement research could intersect and benefit from a synthetic approach. Specifically, I suggest that social movement studies could advance through the examination of some relatively ignored subjects of research, such as persistent participation and disengagement, by drawing on empirical cases of NRMs. I also propose ways in which the application of social movement theories would enhance our understanding of different aspects of NRMs, such as their leadership and coalition practices. In making these arguments, I refer to one of the prominent, long-term NRMs, the Unification Church or Movement, to help illustrate my ideas.     

Using evaluation techniques and performance claims to demonstrate public relations impact: An Australian perspective

Public relations professionals use many methods to demonstrate their contribution to organizational goals, yet it is unclear how their attitudes towards evaluation and the reporting of success matches real outcomes. Ten years after the International Public Relations Association produced an evaluation gold paper, this study combines research on Australian practitioners’ evaluation practices and attitudes, and data from industry awards to identify how practitioners demonstrate their accountability. Data suggest that despite the attention paid to evaluation by the academy and industry, practitioners still focus on measuring outputs, not outcomes to demonstrate performance and continue to rely heavily on media-based evaluation methods.     

A methodological evaluation of public relations research

An examination of journal articles published from 1989 to 2007 reveals that although public relations scholars appear to have followed rigorous procedures, reporting on research protocols is inadequate in some quantitative and qualitative studies. Most areas of needed improvement concern sins of omission, not of commission.     

Evidence-based versus junk-based evaluation research: some lessons from 35 years of the evaluation review

Along with the late Howard Freeman, Richard Berk was a founding editor of Evaluation Review (then Evaluation Quarterly) in 1977. He resigned as editor of this journal at the end of 2010. In this article, he reflects on his experiences.