The evaluation ranking scale: clarification of methodological and procedural issues

Compared with CSQ-8, a typical questionnaire approach to assessing global patient satisfaction, the Evaluation Ranking Scale (ERS) had equally good patient acceptability, yielded more normally distributed satisfaction scores, and results allowed comparative information about patients' evaluation of specific service dimensions. The study also addressed key questions that have emerged about the ERS procedure. Patients apparently do not distinguish conceptually between "importance" of dimensions and "satisfaction" with dimensions in the first phase (ranking) of the ERS. Results did confirm that the ERS sequence of ranking and then rating the dimensions is essential to achieving optimal utility of results. The ranking task seems to have an organizing effect on patients' approach to the rating task. This effect does not work to dictate results in the rating phase but rather seems to familiarize patients with the dimensions to be rated thereby yielding greater potential discriminative capacity for the ERS. Finally, results indicate that the ERS can be administered in a flexible fashion that yields additional information about the absolute importance of the six dimensions without loss of desired operating characteristics for the measure.     

Stigma mitigation and the importance of redundant treatments

Disgust can evoke strong behavioral responses. Sometimes these extreme visceral responses can lead to stigmatization—an overreaction to a risk. In fact, disgust may be so inhibiting that it leads people to refuse to consume completely safe items, such as treated drinking water, leading to important economic and policy implications. Using economic experiments, we provide a measure of the behavioral response to disgust. Our findings suggest that when monetary incentives are provided, the behavioral response may have been exaggerated by previous studies that have relied on survey methods. Furthermore, mitigation steps successfully reduce the stigma behavior. In fact, the results suggest that stigma is primarily reduced not by a specific mitigation step taken but by how many steps are taken consecutively. These results have important implications for policies addressing issues such as the global shortage of drinking water. Some efforts to resolve the shortage have involved recycled water that is completely safe to drink but is often rejected because of reactions of disgust.     

Comparison of indirect and direct approaches to measuring patient satisfaction

One indirect (PSQ) and two direct (CSQ-18B and ERS) measures of patient satisfaction were compared across a series of psychometric, acceptability, and concurrent validity criteria. None of the three measures was significantly related to health status or demographic characteristics. Each measure performed as expected from prior research on their psychometric properties. The scales contrasted, however, in regard to acceptability, with the PSQ being much less acceptable to patients than the other two. The PSQ also produced more missing data. The results on the PSQ reflected patient inconsistency on cross-check items and patients tended to be uncertain in their responses to many items. In multivariate analyses the three measures were used as predictors of patient-rated indices of global service satisfaction. The CSQ-18B and the ERS were significant predictors of patient-rated indices of global service satisfaction, whereas the PSQ was unrelated to these indices. It was concluded that the direct and indirect approaches measure different satisfaction domains. The PSQ likely assesses more generalized attitudes about health services while the CSQ-18B and the ERS efficiently reflect opinions about the specific setting in which they are administered.     

Measuring personal networks with daily contacts: a single-item survey question and the contact diary

This paper examines two extreme approaches that are alternatives to measure egocentric networks with network generators. The single-item approach to measure daily contacts differentiates the individuals effectively, corresponds closely with complex network measures, and reveals well how individuals vary in both expressive and instrumental returns, as supported by 14 large-scale probability surveys from three Chinese societies over a decade. This paper also draws upon three sets of sophisticated contact diaries, which yielded rich data about the circumstance of each contact, the alter's characteristics and the ego–alter relationship. Along with the diary approach, which offers sophisticated data about contacts, ties and networks, the single-item survey approach is another extreme yet straightforward measure of daily contacts.     

THE PERSISTENCE OF POLITICAL ATTITUDES AMONG 1960s CIVIL RIGHTS ACTIVISTS

A scattering of recent research has studied the current politicalbeliefs and attitudes of individuals identified as "1960s activists."In contrast to much of the treatment accorded such people inthe popular media, this research tends to find most of theseactivists currently liberal on a wide variety of political topics.However, in the absence of panel data, most of this researchhas had to assess any change in the activists' attitudes eitherby assuming the activists' past positions or by trusting totheir retrospective reports. In this paper we report on panel data from a large group ofwhite activists, mostly students, who spent the summer of 1965organizing voter registration drives in Southern black communities.In some specific areas on which the activists tended to holdrather extreme positions in 1965, they may have moderated by1984. However, their overall pattern of response on a wide varietyof issues is basically stable over this twenty-year period.     

RESPONSE-TIME MEASUREMENT IN SURVEY RESEARCH A METHOD FOR CATI AND A NEW LOOK AT NONATTITUDES

The paper introduces a methodology for measuring accuratelythe time it takes respondents to answer questions in computer-assistedtelephone surveys. The methodology, which is completely invisibleto respondents, comprises a computer "clock," capable of timingresponses with millisecond accuracy, and a "voice-key" thatconverts sounds emitted by respondents into signals capableof triggering the computer clock. Response times to questionson a range of attitude questions, including stable and unstableattitudes, were measured. The results revealed orderly normsin the latencies associated with various types of survey questions.The latencies associated with the expression of stable and unstableattitudes are discussed in the context of the notion of "nonattitudes"and shed new light on this controversial thesis. Overall, ourresults demonstrate that response latencies can be measuredprecisely and reliably in telephone surveys and that the datafrom such measurement open new windows on the cognitive dynamicsof survey responses.     

The evaluation ranking scale: a new methodology for assessing satisfaction

Most patient satisfaction scales produce high, undifferentiated levels of reported satisfaction that fail to detect program areas that consumers do not like. Methodological problems apparently contribute to these results. An alternative procedure, the Evaluation Ranking Scale (ERS), was formulated and tested. A 2 X 3 design was employed with 246 public health center patients randomly assigned to one of two measurement techniques (ERS vs. a global measure) and one of three informational sets. A secondary group of subjects was a convenience sample of 26 staff members, which allowed comparisons of staff evaluations of the health center to patients' evaluations of the health center. Compared to the global measure, the ERS provided more specific information about particular program components, was more discriminating, and resulted in mean satisfaction scores that were significantly lower. This new approach may be a more effective technique for assessing the psychosocial effectiveness of human service programs.     

Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey

Declining contact and cooperation rates in random digit dial(RDD) national telephone surveys raise serious concerns aboutthe validity of estimates drawn from such research. While researchin the 1990s indicated that nonresponse bias was relativelysmall, response rates have continued to fall since then. Thecurrent study replicates a 1997 methodological experiment thatcompared results from a "Standard" 5-day survey employing thePew Research Center’s usual methodology with results froma "Rigorous" survey conducted over a much longer field periodand achieving a significantly higher response rate. As withthe 1997 study, there is little to suggest that unit nonresponsewithin the range of response rates obtained seriously threatensthe quality of survey estimates. In 77 out of 84 comparableitems, the two surveys yielded results that were statisticallyindistinguishable. While the "Rigorous" study respondents tendedto be somewhat less politically engaged, they did not reportconsistently different behaviors or attitudes on other kindsof questions. With respect to sample composition, the Standardsurvey was closely aligned with estimates from the U.S. Censusand other large government surveys on most variables. We extendour analysis of nonresponse to include comparisons with thehardest-to-reach respondents and with respondents who terminatedthe interview prior to completion.     

Eliciting survey cooperation: incentives, self-interest, and norms of cooperation

Given the weaker ties to community as noted by scholars such as Robert Putnam, survey researchers should not be surprised by a decline in survey participation over the past 10 years. This research analyzes the use of incentives coupled with introductory themes emphasizing cooperation and helpfulness--cooperative norms in American society--to understand their effects on survey response. This article analyzes two separate experiments (one phone and one mail) that provide evidence that norms of cooperation matter in the decision to participate in a survey, suggesting that this is particularly true at the refusal conversion stage. These results indicate that survey researchers may use such themes to their advantage, especially when conducting a nonresponse follow-up in a mail survey.     

Understanding Symbolic Boundaries and Improving Quantitative Analysis of Social Exclusion by Improving the Operationalization of Boundary Work

In survey research, how can we measure actors' judgments concerning who is a member of a particular social group and who is not? How can we measure changes in this process over time? Within the sociology of culture, the operationalization of boundary work, or decisions of group membership, has been relatively overlooked. Similarly, measurement of changes in boundaries has been relatively overlooked as well. An important first step is to better match the operationalization of boundary work with theory. To do so, I argue that survey items concerning decisions of social exclusion measured in the Likert scale can be used. In addition to allowing scholars to compare those who exclude with those who include (often measured as Strongly Agree/Agree vs. Disagree/Strongly Disagree), the Likert scale offers a measurement of boundary strength: how strongly respondents are committed to their exclusionary judgments (Agree vs. Strongly Agree, Disagree vs. Strongly Disagree). A novel element of boundary change can then be measured as variation over time in commitment to exclusionary judgment in time series data. A boundary that has increased in strength would be one for which respondents increase in their emotional commitment to their exclusionary judgment over time.     

INTENSIFIERS IN BEHAVIORAL FREQUENCY QUESTIONS

Intensifiers, words such as "very" and "extremely", are usedto magnify the meaning of the phrases to which they are applied.In a series of studies we investigate how intensifiers in questionstems affect response patterns in social surveys. Our researchindicates that even apparently important differences in questionwording may in some situations have little or no impact; addingan intensifier to a root did not create a response shift forseveral intensifier/root combinations in large-scale surveys.Using both field and laboratory techniques we explore the situationsin which shifts do occur and go some lengths toward describingwhy this happens. Response shifts were observed for two situations.First, when "extreme" was applied to "physical pain" there wasa substantial and significant response shift. Second, when respondentswere asked a question without an intensifier and then had thequestion repeated immediately afterward with an intensifier,a response shift was produced. In addition to the practicalsignificance for survey methodologists, these results are importantto cognitive psychologists interested in word meaning.     

Violent behavior: a measure of emotional upset?

Over the past ten years, sociologists have broadened their view of what counts as an appropriate measure of mental health. This reflects a growing recognition that individuals express emotional upset in various ways. For example, some individuals are more likely to become depressed in response to stress while others are more likely to drink heavily. Contemporary studies often include measures of "internalizing" (i.e., more feeling-based measures) and "externalizing" (i.e., more behavioral measures) styles of psychopathology, especially when studying group differences in mental health. Alcohol abuse is the classic measure of externalized distress in sociological research. In this paper, we present a theoretical argument and supporting empirical evidence to argue that violent behavior should be included as a measure of externalized distress in response to stress. Our study suggests that violent behavior is a more likely response to stress among individuals with particular coping and appraisal tendencies. Specifically, violent behavior may be a more likely response to stress among individuals who tend to appraise situations as threatening while also repressing any emotional response to stress. We contend that, since some groups may be more likely than others to respond to stress with violence, it is particularly important to include measures of violent behavior when studying group difference in distress.     

AN ANALYSIS OF THE ACCURACY OF "TRIAL HEAT" POLLS DURING THE 1992 PRESIDENTIAL ELECTION

There is little systematic research on the multitude of factorsthat influence the accuracy of poll results. This article examinessix methodological factors directly under the survey researcher'scontrol and two exogenous factors concerning the nature of publicopinion as sources of survey error. Data for this study comefrom 56 "trial heat" polls conducted during the last month ofthe 1992 presidential election. The most important variablesinfluencing survey accuracy were the number of days a poll isin the field, which increased total accuracy one-half of a percentagepoint per day; conducting interviews only on weekdays (and thusonly during evening hours), which reduced overall accuracy ratesby more than 1 percentage point; and conducting a "tracking"poll, which increased accuracy by about 1.5 points. Sample sizewas not related to accuracy rates. Results also indicated thatsampling frames of "likely voters" (relative to "registeredvoters") tended to overestimate support for George Bush andunderestimate support for Ross Perot, that interviewing onlyon weekdays led to overestimates of support for Bush, and thatstrict methods of defining a respondent as "supporting" a candidatehurt the two newcomers, Perot and Bill Clinton, more than Bush.In light of these data it is recommended that the common practiceof reporting "margins of error" based solely on sample sizesbe abandoned as misleading and replaced by a more empiricallyjustifiable measure based more on response rates.     

SURVEY QUESTIONS ABOUT THE FREQUENCY OF VAGUELY DEFINED EVENTS: THE EFFECTS OF RESPONSE ALTERNATIVES

Experimental research has shown that the choice of responsealternatives can influence responses to questions about thefrequency of vaguely defined target events (e.g., feeling annoyed);the set of response alternatives is treated as information thatcontributes to the interpretation of the question. In a seriesof split ballot experiments we investigate whether such effectsoccur in a large-scale survey context for sets of response alternativesthat might be used interchangeably by survey researchers. Thepredicted response shifts were found in our field experiments.Those presented with response alternatives discriminating atlow frequencies reported fewer of the target episodes than thosepresented with higher-frequency response alternatives. However,the size of the observed shifts varied from zero to 13 percentand depended on a number of characteristics of the survey, forexample, the presence of "priming" questions and the orientationof the response scales. Response alternatives are not neutraland therefore must be carefully considered when constructinga survey question.     

The Feminist Consciousness

Feminism is more than a philosophy or ideology. It is a "vocabulary of motives" maintained by strong group support. Becoming a feminist leads to a transformation of consciousness and an alteration in the perception and interpretation of everyday life. I focused on "consciousness" as the organization of perceptions of women that included an awareness or self-consciousness of this organization.
Seven expected findings were developed comparing feminists and nonfeminists on three dimensions of consciousness: perceptions of women, autonomy and self-control, and overt feminist interpretation. A projective measure consisting of 14 pictures was administered to college women. Respondents wrote stories about the pictures in response to standard TAT questions. A feminist consciousness emerged as a recognizeable and distinct process. Feminists are more likely to use a feminist vocabulary of motives, introduce the general theme of sexism or specific feminist themes such as job discrimination. The stories of the two groups differed significantly in their degree of observable feminism. Women are portrayed by feminists as struggling for autonomy in life situations but are not perceived as having control over their lives in traditional and ambiguous settings. Feminists do not appear to be ideologically oriented; and while the feminist consciousness is distinctive, it is not monolithic or unidimensional. Feminists interject feminism into their interpretation of everyday life and perceive situations differently from nonfeminists leading to the conclusion that their subjective experience is different from that of nonfeminists.     

THE ANSWERING MACHINE POSES MANY QUESTIONS FOR TELEPHONE SURVEY RESEARCHERS

The proliferation of the telephone answering machine raisesa number of questions about the continued efficiency of thetelephone as a data-gathering mechanism. The most critical ofthese is what effect the use of these machines might have onestablishing contact with potential respondents. If individualsroutinely use these machines to screen calls, then their accessibilityto telephone survey researchers will be restricted. A secondquestion is the degree to which respondents who own answeringmachines and are reachable are likely to participate in a survey.A third question is the extent to which the incidence of theanswering machine as a response disposition might vary by thetime of calling and the respondent's place of residence. Thefindings of this study, based on a nationwide survey, pointto a significant proportion of answering machine owners beingreachable and willing to participate. Furthermore, the answeringmachine appears to be in use more on weekends than on weekdayevenings and in more urbanized areas than in areas with fewerinhabitants.     

Effects of anonymity and experimenter demand on client satisfaction with mental health services

Among the methodological issues associated with the measurement of client satisfaction are those concerning anonymity of response, experimenter demand effects, and reliability of instruments. In this study, 100 inpatients were randomly assigned in a 2 X 2 factorial design; subjects were asked or not asked to sign their names to the CSQ-8 questionnaire and were asked or not asked to "tell it like it is." The reported satisfaction of those given anonymity was significantly lower but there was no effect due to demand. Furthermore, subjects in the anonymity group had lower scores on all 8 items of the questionnaire. A factor analysis of the CSQ-8 revealed one major factor; the internal consistency of the scale was .93. The CSQ-8 was found to be a useful measure of overall client satisfaction but, because ceiling effects are a pervasive problem in client satisfaction research, anonymity should be assured whenever possible.     

When therapists do not want their clients to be homosexual: a response to Rosik's article

This commentary is a response to Rosik's "Motivational, Ethical, and Epistemological Foundations in the Treatment of Unwanted Homoerotic Attraction" (this issue). Such treatment raises complex questions that cannot be resolved by focusing on the therapist's conservative versus liberal values. Most such clients are deeply ambivalent about their homosexual attractions. The degree to which their homosexuality is "unwanted" is highly variable among them and sometimes within them over time. Clients who are exclusively homosexual are very unlikely to be able to change their sexual attractions, whereas some clients who are bisexual may be more able to "manage" their homoerotic attractions (acting only on their heterosexual feelings). Marriage and family therapists should be able to support a client along whatever sexual orientation path the client ultimately takes, and the client's sense of integrity and interpersonal relatedness are the most important goals of all.     

Measuring Religious Identities in Surveys

The purpose of this article is to acquaint the broader publicopinion research audience with what has been a salient issuewithin the community of scholars of religion. We address thequestion of how best to conceptualize and measure religiousidentities in research on contemporary American society. Weconsider the main approaches to the measurement of religiousidentification with regard to their backgrounds, their assumptionsabout the importance of understanding religious identities inhistorically relevant terms, and the practical considerationsof survey measurement. Using data from the General Social Survey,particularly recent innovative efforts to obtain informationon subjective association with particular religious traditionsand/or movements (e.g., Pentecostal, fundamentalist, evangelical,mainline, or liberal Protestant), we compare the two main approaches:the traditional "denominational" approach, where religious identitiesare assumed to be associated with religious denominations, andthe subjective approach, where religious identities are assumedto be captured by a set of "nondenominational" reference categorieslinked to particular historical religious traditions or socialmovements. We conclude that both approaches have substantialpredictive validity, and the most effective strategy for futureresearch may be one that uses a combination of approaches, ratherthan one that relies entirely on a single method of measurement.     

Responding to remediation and gatekeeping challenges in supervision

Thirty faculty in randomly selected Commission on Accreditation for Marriage and Family Therapy Education (COAMFTE) programs representing four geographical regions of the United States were asked to respond to a survey that included seven brief vignettes depicting gatekeeping and remediation challenges supervisors may face when working with therapists-in-training. Research participants were asked to select from among a range of 17 response options and were also asked to provide a rationale for the selection of their responses. The response rate for the mailed survey was 34%. Telephone interviews were conducted with three respondents who had provided especially detailed responses to the open-ended portions of the mailed survey and who also volunteered to be questioned regarding how supervision decisions are made. Results support the conclusion that COAMFTE faculty take their gatekeeping function seriously as they balance commitments to multiple stakeholders. When presented with hypothetical "bare-bones" vignettes, the supervisors in our study consistently recommended talking with the student in order to more fully understand the context of the student's performance problem before deciding how to proceed. Supervisors recommended a variety of remediation efforts, but reserved the most severe consequences, such as probation, dismissal, and filing an ethics complaint, for the vignette involving dishonesty and lack of personal integrity.