Clarifying the role of belief-motive explanations in multi-stakeholder realist evaluation

In realist evaluation, where researchers aim to make program theories explicit, they can encounter competing explanations as to how programs work. Managing explanatory tensions from different sources of evidence in multi-stakeholder projects can challenge external evaluators, especially when access to pertinent data, like client records, is mediated by program stakeholders. In this article, we consider two central questions: how can program stakeholder motives shape a realist evaluation project; and how might realist evaluators respond to stakeholders’ belief-motive explanations, including those about program effectiveness, based on factors such as supererogatory commitment or trying together in good faith? Drawing on our realist evaluation of a service reform initiative involving multiple agencies, we describe stakeholder motives at key phases, highlighting a need for tactics and skills that help to manage explanatory tensions. In conclusion, the relevance of stakeholders’ belief-motive explanations (‘we believe the program works’) in realist evaluation is clarified and discussed.     

CARRYOVER EFFECTS IN ATTITUDE SURVEYS

Answers to attitude questions in surveys can vary markedly dependingon the preceding items in the questionnaire. This study concernssuch context effects. More than 1,100 respondents were askedabout six target issues in a telephone survey. Before answeringthe target questions, most of the respondents had been askedabout one of two sets of related issues; the remainder receivedunrelated, "neutral" context items. For five of the six targetissues, the two groups receiving different sets of context itemsdiffered significantly in their answers to the target questions.However, only one of the substantive context groups differedsignificantly from the group that received the neutral contextitems. Additional results indicated that context effects werelarger when respondents reported that their beliefs about thetarget issue were both mixed and important. Coding of openendedresponses to follow-up questions suggested that the contextitems may have changed how respondents thought about the targetissues.     

A Longitudinal Study: Casino Gambling Attitudes, Motivations, and Gambling Patterns Among Urban Elders

Guided by self-determination theory, the main purpose of this study was to explore demographic characteristics, attitudes toward casinos, and self-reported intrinsic and extrinsic reasons for casino gambling by urban elders. The study hypothesized that individuals would more frequently report intrinsic motivations for casino gambling (e.g., entertainment, enjoyment) rather than extrinsic motivation (e.g., financial gain). This longitudinal sample included 247 urban elders older who were 60 years and older and who had participated in surveys in 2002 and 2004. The initial survey consisted of (a) demographic items, (b) five items to measure attitudes toward casino gambling, (c) questions inquiring about motivations for casino gambling, and (d) questions about gambling frequency. The follow-up survey was an expanded questionnaire which still included these items. The sample consisted of the 247 participants, over 200 of whom were African-Americans, 188 were female, and 98 of the participants had a post graduate education. About half were widowed, and the sample generally reported a low income. The results supported the theoretical perspective underlying the project. The hypothesis that more participants would endorse intrinsic motivations for casino gambling rather than extrinsic motivations was supported. The implications of these findings represent for social workers, gambling counselors and health care services providers an important step toward understanding the attitudes, behaviors, and motivational factors involved in casino gambling among older adults.     

INTERVIEWER GENDER AND GENDER ATTITUDES

This research addresses the effects of interviewer gender onresponses to a broad array of gender-related survey questions,using data from a probability sample of adults in the UnitedStates. We focus on whether gender-of-interviewer effects areevident, whether they vary by respondent gender, and whetherthey vary across several attitudinal domains relevant to genderinequality. While many items do not show statistically significantgender-of-interviewer effects, we document significant effectsacross a variety of items. When such effects are evident, theytend to involve both male and female respondents expressingmore egalitarian gender-related attitudes or greater criticismof existing gender inequalities to female interviewers. Malerespondents offer significantly different responses to maleand female interviewers on questions dealing with gender inequalityin employment. For female respondents, interviewer-gender effectsare evident for items addressing gender-related collective action,policy, and group interests. Using multivariate models thatallow us to represent both respondent-level and interviewer-levelvariables, we find that interviewer-gender effects are statisticallysignificant in most attitudinal domains but that the interactionbetween interviewer gender and respondent gender does not tendto be statistically significant. We consider the implicationsof these findings both for understanding the survey processand for understanding gender relations more generally.     

The Independence of Aptitudes,Interests, and Career Beliefs

The importance of client beliefs in career counseling depends on their ability to add unique information about the client over and above that contributed by aptitudes and interests. The Career Beliefs Inventory was administered to 200 Australian students in grade 10 together with measures of Holland's RIASEC interest themes and scholastic aptitudes. The correlations between scales from the three domains showed clearly that beliefs made a contribution distinct from that provided by aptitudes and interests. Even though the results may reflect possible sampling or method variance, career beliefs in this sample added unique information to that traditionally used in career counseling.     

The Devil is in the Detail: Clinicians,Academics and Researchers Resolving the Problems of Clinical Research

The difficulties associated with conducting valid family therapy research within a clinical practice discourage many potential researchers. This article will describe collaboration between a group of academics, researchers and clinicians who decided to explore the process and efficacy of systemic family therapy conducted within a working private practice. The specific questions we are addressing are, whether the requests clients bring to their first session of therapy can be reliably classified by practitioners, whether these requests change over time, and whether the nature of the request is associated with therapist and client ratings of therapeutic outcome. Additional questions about the form and nature of the therapeutic alliance as experienced by both client and practitioner are also being explored. This paper will map the passage of the work from inception to its current state where over 140 clients are active participants. In doing so attention will be paid to the obstacles encountered: practical, financial and ethical, and the solutions devised to address these.     

Community-Based Program Research

ABSTRACT

At first glance, the worlds of program professionals and evaluators may seem quite separate. However, there are common issues whose resolution will enhance both program development and research on programs. Program professionals and evaluators have a great deal to learn from each other. As that learning occurs, both program delivery and evaluation research will benefit. Both are concerned about matters of validity, whether it pertains to the nature of the program intervention itself or to the nature of the assessment of the program intervention. To reinforce the development of program evaluation in partnership, this paper discusses key points about evaluation research relevant to both program professionals and researchers. These include the contextual influences on a program, the “readiness” of a program for evaluation, and whether the evaluation research “works” for the program. Underlying our discussion is a call for the development of partnerships around research on programs. In that light, this article elaborates the process steps that should be taken to build program evaluation partnerships, including a discussion of what evaluators and program professionals need to know about each other, and their respective values, interests, and professional perspectives.     

BASIC FAMILY THERAPY SKILLS, IV: TRANSGENERATIONAL THEORIES OF FAMILY THERAPY

This pape focuses on the basic skills and compentencies of transgenerational approaches to family therapy, the fourth report of a program of research surveys by the Basic Family Therapy Skills Project. In the first survey, a panel of family therapy educators listed the most critical or basic skills or compentencies of beginning family therapist with a transgenerational orientation. In the third survey, self-selected respondents rated these items according to degree of importance for beginning family therapists. Rankings of the items according to mean scores of the Likert responses indicate that transgenerational family therapy skills are founded in theory and can be identified behaviorally. The most important skills are those that use the self of the therapist to understand and utilize transgenerational family therapy skills for both therapist and client. Also listed are skills that many raters deemed "generic" rather than specifically related to transgenerational family therapy. Other findings and their implications for the future training of family therapists are discusses. Although the paper is intended for family therapy educators interested in their colleagues' ideas about critical skills in trans-generational family therapy, theorists and clinicians may also find it useful.     

DO SURVEYS OVERSTATE PUBLIC CONCERNS?

Because the item, "How concerned are you about...?" asks respondentsto indicate their level of concern about an issue, some respondentsmay sense it assumes they are concerned or should be concernedabout the issue. Using a filter question to first determineif people are concerned about the issue before asking for theirdegree of concern may help solve this problem. To test thishypothesis, a split-ballot experiment was embedded in a nationalrandom digit dialing telephone survey on food-related issues.For the four items included in the experiment, the group receivingthe filter versions of the questions gave roughly double thepercentages of "not concerned" responses as the group receivingstandard items, and the filter group also gave fewer responsesat the upper end of the response scale.     

Issues in Psychotherapy with Clients Affiliated with the Opposing Side in a Violent Political Conflict

The therapeutic dyad of clinician and client affiliated with rival groups in a violent conflict shares many features that complicate psychotherapy with persons of different ethnic, racial, and cultural groups, including lack of knowledge, negative stereotyping, differences in fundamental values and world views, and power differentials. Although a great deal has been written about these matters, very little has been written about the therapeutic dynamic where therapist and client are affiliated with different sides of a violent political conflict. Based on the sparse literature on the subject and on the clinical experiences of the author and her colleagues, this paper discusses three features that characterize this dynamic, which do not appear in the same way in therapy involving dyads of different races, ethnic groups, or cultures. These are the presence of the “enemy” in the consulting room, the therapist’s feelings of mistrust towards the client as a representative of the opposing group, and the client’s feelings of guilt towards the therapist as a representative of the injured group. The impacts of each feature on the therapist, client, and the therapeutic encounter are explored and suggestions are made for dealing with them. The hope is that professionals’ awareness of these issues will help improve the quality of psychotherapy with persons on the rival side of a violent political conflict.     

The “Other” Effects of Psychotropic Medication: Social Workers’ Perspectives on the Psychosocial Effects of Medication Treatment on Adolescent Clients

This study explores social workers’ perspectives on the psychosocial effects of psychotropic drug treatment of their adolescent clients. Using a mail survey, we asked a national sample of experienced clinical social workers to answer questions about their perceptions of the effects of such treatment on the sense of self and social well-being of an adolescent client. The survey questions also explored associations between perceived psychosocial medication effects and characteristics of the client and the treatment. The findings suggested that social workers viewed medication treatment as having greater beneficial than harmful psychosocial effects on their adolescent clients, but that both effects existed simultaneously. The most important factors associated with the perceived effects of medication that emerged from social workers’ reports included the etiology of the disorder, the type of drug treatment and its effectiveness in addressing symptoms, the client’s “competence,” and the quality of the relationship between the client and the treating social worker. This study provides direction for future research on a neglected but important question. It would be helpful to broaden the scope of professional discourse on the advantages and disadvantages of psychopharmacologic treatment for youth beyond the effectiveness or safety of the treatment to include questions concerning clinicians’ perceptions of the effects of drug treatment on youths’ sense of self and social well-being.     

How Routine Patient‐Centered Monitoring Relates to Therapeutic Gains in Family Therapy: A Single‐Case Study

As routine outcome monitoring systems develop, questions emerge about how therapists incorporate feedback into their practice, and how this relates to therapeutic gains. A case of covert grief was monitored in each session with the Personal Questionnaire and the Helpful Aspects of Therapy instruments. At 4 months follow‐up, the Change Interview was administered. Individualized items facilitated access to the private views and needs of each member, which was useful for case formulation and ongoing personalization of the intervention. Qualitative feedback of treatment experiences helped therapists confirm the impacts of interventions, monitor therapeutic alliance, reformulate clinical hypothesis, and plan sessions. Therapists followed a critical triangulation process to decide the clinical meaning of feedback, according to his/her intervention model, expertise, and case‐specific context.     

Sex differences in how heterosexuals think about lesbians and gay men: Evidence from survey context effects

Two experiments were embedded in a 1997 telephone survey of U.S. households to assess possible differences in how heterosexuals think about lesbians versus gay men. In each experiment, one half of the sample first responded to one or more attitude items about lesbians, followed by comparable items about gay men. The other half received the gay male item(s) first. Results are reported separately for White (N = 976) and Black (N = 479) heterosexuals. For White and Black men alike, self‐reported attitudes toward lesbians tended to be more favorable when they were assessed without reference to gay men (i.e., lesbian items presented first). White men's reactions to gay men tended to be less negative when assessed after the questions about lesbians were presented, but Black men's responses did not consistently show this pattern. For some items, women gave more favorable ratings of lesbians and less favorable ratings of gay men when the lesbian items were presented first. The findings suggest possible gender differences in the cognitive organization of heterosexuals ‘ attitudes toward lesbians and gay men.     

PROBING "DONT KNOW" ANSWERS: EFFECTS ON SURVEY ESTIMATES AND VARIABLE RELATIONSHIPS

Don't Know responses to survey questions are ambiguous becausethe same words are used by respondents to mean different things—ignorance,indecision, or uncertainty about the meaning of the questionasked. In order to clarify the meaning of such answers, surveyinterviewers are frequently trained to probe Don't Know answersat least once before the answer is considered final. We arguethat unconditional probing of Don't Know answers may not bea desirable practice, particularly as regards knowledge items.Large unintended effects on responses to four knowledge itemsresulted when two groups of interviewers, who administered thesame survey questions, probed Don't Know responses at differentrates. Strong evidence is provided in the article that the probingof Don't Know answers encouraged guesswork on the part of uninformedrespondents, giving rise to significant distributional differencesand differences in means across half samples for the affectedvariables. However, relationships between variables appear tobe largely unaffected by probing effects. In order to formulatevalid probing guidelines for interviewer training, further researchis needed to establish whether the findings for knowledge questionsgeneralize to factual questions.     

Logic and evaluation theory

With this series of papers, evaluators are being called to substantiate the rationale and warrant for their own evaluative actions in ways parallel to how evaluators question the logic of program interventions, both as designed and as implemented. This endeavor is timely, appropriate, and important. In these comments, I raise modest questions about the logical constitution of an evaluation theory and about what is missing from a textual reading alone of such theory.     

The Effect of Question Framing and Response Options on the Relationship between Racial Attitudes and Beliefs about Genes as Causes of Behavior

Prior research suggests that the attribution of individual and group differences to genetic causes is correlated with prejudiced attitudes toward minority groups. Our study suggests that these findings may be due to the wording of the questions and to the choice of response options. Using a series of vignettes in an online survey, we find a relationship between racial attitudes and genetic attributions when respondents are asked to make causal attributions of differences between racial groups. However, when they are asked to make causal attributions for characteristics shown by individuals, no such relationship is found. The response scale used appears to make less, if any, difference in the results. These findings indicate that the way questions about genetic causation of behavior are framed makes a significant contribution to the answers obtained because it significantly changes the meaning of the questions. We argue that such framing needs to be carefully attended to, not only in posing research questions but also in discourse about genetics more generally.     

GENSI: A new graphical tool to collect ego-centered network data

This study (1) tested the effectiveness of a new survey tool to collect ego-centered network data and (2) assessed the impact of giving people feedback about their network on subsequent responses. The new tool, GENSI (Graphical Ego-centered Network Survey Interface), allows respondents to describe all network contacts at once via a graphical representation of their networks. In an online experiment, 434 American adults were randomly assigned to answer traditional network questions or GENSI and were randomly assigned to receive feedback about their network or not. The traditional questionnaire and GENSI took the same amount of time to complete, and measurements of racial composition of the network showed equivalent convergent validity in both survey tools. However, the new tool appears to solve what past researchers have considered to be a problem with online administration: exaggerated numbers of network connections. Moreover, respondents reported enjoying GENSI more than the traditional tool. Thus, using a graphical interface to collect ego-centered network data seems to be promising. However, telling respondents how their network compared to the average Americans reduced the convergent validity of measures administered after the feedback was provided, suggesting that such feedback should be avoided.     

ASKING SENSITIVE QUESTIONS: THE IMPACT OF DATA COLLECTION MODE, QUESTION FORMAT, AND QUESTION CONTEXT

This study compared three methods of collecting survey dataabout sexual behaviors and other sensitive topics: computer-assistedpersonal interviewing (CAPI), computer-assisted self-administeredinterviewing (CASI), and audio computer-assisted self-administeredinterviewing (ACASI). Interviews were conducted with an areaprobability sample of more than 300 adults in Cook County, Illinois.The experiment also compared open and closed questions aboutthe number of sex partners and varied the context in which thesex partner items were embedded. The three mode groups did notdiffer in response rates, but the mode of data collection didaffect the level of reporting of sensitive behaviors: both formsof self-administration tended to reduce the disparity betweenmen and women in the number of sex partners reported. Self-admimstration,especially via ACASI, also increased the proportion of respondentsadmitting that they had used illicit drugs. In addition, whenthe closed answer options emphasized the low end of the distribution,fewer sex partners were reported than when the options emphasizedthe high end of the distribution; responses to the open-endedversions of the sex partner items generally fell between responsesto the two closed versions.     

Spacing, Position, and Order: Interpretive Heuristics for Visual Features of Survey Questions

We present the results of six experiments that demonstrate theimpact of visual features of survey questions on the responsesthey elicit, the response process they initiate, or both. Allsix experiments were embedded in Web surveys. Experiments 1and 2 investigate the effects of the placement of nonsubstantiveresponse options (for example, "No opinion" and "Don’tknow" answer options) in relation to the substantive options.The results suggest that when these options are not differentiatedvisually (by a line or a space) from the substantive options,respondents may be misled about the midpoint of the scale; respondentsseemed to use the visual rather than the conceptual midpointof the scale as a reference point for responding. Experiment3, which varied the spacing of the substantive options, showeda similar result. Responses were pushed in the direction ofthe visual midpoint when it fell to one side of the conceptualmidpoint of the response scale. Experiment 4 examined the effectsof varying whether the response options, which were arrayedvertically, followed a logical progression from top to bottom.Respondents answered more quickly when the options followeda logical order. Experiment 5 examined the effects of the placementof an unfamiliar item among a series of similar items. For example,one set of items asked respondents to say whether several makesand models of cars were expensive or not. The answers for theunfamiliar items depended on the items that were nearby on thelist. Our last experiment varied whether a battery of relateditems was administered on a single screen, across two screens,or with each item on its own screen. The intercorrelations amongthe items were highest when they were all on the same screen.Respondents seem to apply interpretive heuristics in assigningmeaning to visual cues in questionnaires. They see the visualmidpoint of a scale as representing the typical or middle response;they expect options to be arrayed in a progression beginningwith the leftmost or topmost item; and they expect items thatare physically close to be related to each other conceptually.     

Mixed-methods library evaluation integrating the patron,library, and external perspectives: The case of Namibia regional libraries

The article presents a mixed-methods evaluation of regional libraries in Namibia, which incorporates three perspectives: the patron perspective (library users), the library perspective (library staff, management, and related officials), and the external perspective (including evaluators and monitoring data). Seven data collection methods were used: patron surveys, patron panel studies, focus group discussions, key informant and staff interviews, secondary data analysis, media analysis, and observations. The goal of the evaluation was to assess library performance for both formative and summative purposes by addressing evaluation questions on areas such as library services, use, and operations. Building upon the literature review of how mixed-methods approaches can contribute to library evaluation, the aim of this article is to show how a mixed-methods evaluation can be designed to examine multi-faceted library performance and to illustrate how the evaluation design allows information complementarity and can be utilized to present diverse viewpoints of the above three perspectives. The evaluation design, analysis process, and lessons learned from this study may be useful to evaluators engaged in evaluation of public services or programs (including public libraries) that examine multiple aspects of service performance and involve a variety of stakeholders.