Using and interpreting analogue designs

Researchers in rehabilitation counseling and disability studies sometimes use analogue research, which involves materials that approximate or describe reality (e.g., written vignettes, videotaped exemplars) rather than investigating phenomena in real-world settings. Analogue research often utilizes experimental designs, and it thereby frequently possesses a high degree of internal validity. Analogue research allows investigators to exercise tight control over the implementation of the independent or treatment variable and over potentially confounding variables, which enables them to isolate the effects of those treatment variables on selected outcome measures. However, the simulated nature of analogue research presents an important threat to external validity. As such, the generalizability of analogue research to real-life settings and situations may be problematic. These and other issues germane to analogue research in vocational rehabilitation are discussed in this article, illustrated with examples from the contemporary literature.     

Multivariate methods in rehabilitation

The complexity of relationships between variables in a variety of rehabilitation settings can be appropriately addressed by the use of multivariate methods of statistical analysis. The purpose of this article is to describe (a) some basic concepts and principles of multivariate methods as well as (b) common misconceptions in the application of such methods. Specifically, the authors focus on the use of multivariate analysis of variance (MANOVA) in rehabilitation research and some pitfalls that rehabilitation researchers face when choosing between MANOVA for a set of outcome variables versus separate analyses of variance (ANOVAs) for each outcome variable. Examples of the use of MANOVA in rehabilitation research, as well as references to other multivariate methods, are also provided.     

The power of outcomes: FOTO Industrial Outcomes Tool -- Initial assessment

OBJECTIVE: To demonstrate how outcomes assessment can assist in describing clients receiving rehabilitation in occupational health rehabilitation clinics and to describe the preliminary assessment of internal consistency reliability and construct validity of the FOTO Industrial Outcomes Tool. METHODS: 266 adults referred for acute work rehabilitation (AWR), work conditioning/hardening (WC/WH) or a Functional Capacity Evaluation (FCE) comprised the data set. Clients were treated between July 1998 and January 1999 in 15 clinics from 6 states by 46 clinicians participating in the Focus on Therapeutic Outcomes (FOTO) national rehabilitation database beta test. For AWR and WC/WH, clients completed a health status questionnaire on intake and discharge, and health status was assessed prior to the FCE. Comprehensive demographic data were collected describing the clinics, clinicians, clients and work status collected 2 weeks following discharge. RESULTS: Internal consistency reliability coefficients for the health status scores ranged from 0.57 to 0.89. Construct validity was supported. CONCLUSION: Results demonstrate the power of collecting outcomes from a variety of constructs for clients receiving industrial rehabilitation services. Initial reliability and construct validity findings were adequate and support continuing data analyses.     

The validity of self-reported cost events by substance abusers. Limits, liabilities, and future directions

The following review considers data on the validity of self-reports in addict populations, and then it discusses (a) the types of cost-related questions and the assumptions underlying them that are useful to the evaluation of addictions treatment, (b) both internal and external sources of invalidity, (c) the limits on cost-related information that is gathered from administrative databases, (d) methods for assessing measure validity, and (e) the means for improving the validity of self-reports of cost events. With some important exceptions, addicts provide valid data about both medical and criminal cost events. Skilled socioeconomic researchers able to monetarize these events should be able to produce significant cost of illness, cost offset, cost-benefit, and cost-effectiveness research using self-report data.     

Partnering for workplace health and safety

Worker involvement in decision-making about the workplace can improve safety, health, productivity and the quality of organizational outputs. Australian work health and safety (WHS) legislation mandates worker involvement and gives elected Health and Safety Representatives (HSRs) specific powers, but there has been limited research about the impact of that regulatory framework on the nature, quality and outcomes of worker involvement. As part of a wider review of worker representation in WHS, we investigated the role and impact (positive and negative) of elected HSRs on WHS] in South Australia using a newly-constructed survey instrument. This paper reports on the development of the instrument and the initial findings of the research. The initial survey dimensions and items were developed from earlier research on consultation in South Australia and were refined and validated through this research. The survey has 9 constructs and 61 items and has both face validity and high internal consistency. This research is a step forward for researchers and policy makers seeking a means of determining the effectiveness of worker participation in WHS. It provides an instrument, pilot baseline data and a method that could be used internationally to enable this assessment.     

The theory-driven approach to validity

Current approaches for dealing with validity tend to focus on one type of validity and play down the importance of others. For example, internal validity is treated by Campbell and Stanley as “sine qua non,” whereas, Cronbach judges external validity to be prime importance in research. However, these single validity oriented approaches may achieve one type of validity at the expense of other types of validity.As an alternative, we propose a theory-driven approach to validity. Our central argument is that a model or theory should be formulated in a program evaluation and the modeling process should include the identification of potential threats to validity in research. In this sense, the purpose of applying a technique such as randomization is to strengthen rather than to replace the model or theory. That is, program evaluation should be model or theory-oriented instead of research design oriented. Randomization can be used in any research if it is applicable, but we need rely less on randomization as a safeguard to internal validity if the threats in a research can be adequately identified and controlled.Since the structural relationships among the relevant variables and intervening processes are carefully modeled, the theory-driven approach avoids the pitfalls of black-box evaluation and provides better understanding of the causal mechanisms underlying the relationships between treatment and effects. Unlike randomized experiments which aim only at internal validity, a theory-driven approach provides a general framework to deal with internal, external, construct and statistical conclusion validity.     

Crisis response strategy manipulation: A systematic review and a test of nuances

Although testing the effectiveness of crisis response strategies has been a prominent topic in crisis communication research, studies have rarely addressed whether the manipulation of these strategies accurately reflects their theoretical definitions in experiments. Through a systematic review, this study first identified nuances in the manipulation of apology, diminishing, and scapegoating strategies. The study then tested the effect of the variation of the same strategy on the public’s responses using three experiments. The findings revealed that an explicit apology was more effective than an implicit apology, while an apology with corrective action was even more effective than an apology alone. Internal scapegoating significantly harmed an organization compared to external scapegoating. Following the results of the study, researchers are encouraged to review their crisis response strategy manipulations to ensure the internal validity of experimental studies in crisis communication.     

Construction and validation of Rejection Sensitive Expectation,Perception, and Reaction Questionnaire—Partner (RSEPR-P)

The Rejection Sensitive Expectation, Perception, and Reaction Questionnaire—Partner (RSEPR-P) was developed to assess rejection expectation, perception of rejection, and reactions to perceived rejection among intimate partners. This article details the construction and validation procedures, including item pool generation, factor analysis, and the estimation of reliability and validity. The researchers examined the psychometric properties of RSEPR-P in a community sample of 151 participants. Confirmatory factor analysis (CFA) with the final 24 items suggested four-factor loading with each subscale loading on a separate factor. The subscales had good to excellent internal consistency. In support of the convergent validity, RSEPR-P exhibited robust correlations with rejection sensitivity-personal, mindfulness, marital adjustment, and self-esteem measures. A 6-month test–retest reliability was established. RSEPR-P is proposed to have potential therapeutic and research utility.     

Examining the Reliability and Validity of the Fisher Divorce Adjustment Scale: The Persian Version

This study was an attempt to determine the factor structure, reliability, validity, and adaptation of the Fisher Divorce Adjustment Scale (FDAS) for Iranian divorced individuals. The primary rationale for this study was the lack of such an instrument in an Iranian context to help researchers and therapists determine postdivorce adjustment and distinguish those in need of receiving psychological help. Participants of this study were 486 individuals (49.5% men, 50.5% women) who were divorced and were selected from the available population. Participants were asked to complete a sociodemographic questionnaire, the FDAS (Fisher, 1978), the Satisfaction With Life Scale (SWLS; Diener, Emmons, Larsen, & Griffin, 1985), and the General Health Questionnaire (GHQ; Ghanbarnejad & Turki, 2013). The results showed that the internal consistency of FDAS using Cronbach’s alpha was .93 and the split-half coefficient was .89, indicating fine consistency. Also, the internal consistency of FDAS subscales measuring self-worth, disentanglement from the ex-partner, anger, grief, social trust, and social self-worth was .81, .86, .89, .88, .86, and .69, respectively, using Cronbach’s alpha. Data analysis shows an adequate convergent validity with the SWLS and GHQ. It can be concluded that the FDAS has an acceptable factor structure, reliability, and validity, and can be used in Iran and other Persian-speaking countries.     

Gambling Screens: Does Shortening the Time Frame Affect their Psychometric Properties?

Gambling screening tools such as the South Oaks Gambling Screen (SOGS) and a DSM-IV Screen for Gambling Problems (NODS) developed by the National Opinion Research Council have psychometrically validated lifetime and past-year versions. As research questions often dictate shorter time intervals, researchers have adapted the time frames of these instruments to their specific purposes without examining whether changing the time frame affects the psychometric properties. In this study, 3-month versions of SOGS and NODS were administered to a sample of 80 pathological gamblers (59 men, 21 women, mean age 44) seeking treatment at a state-funded facility. The 3-months versions had good internal consistency, good convergent validity with each other, with gambling behaviors assessed via the timeline followback method, and with measures of impulsivity. The 3-month versions also showed good discriminant validity with demographic variables and a measure of verbal IQ. Together the data indicate that shortening the time frame to 3 months does not seem to have adverse effects on the psychometric properties of SOGS and NODS. Thus these adapted versions could profitably be used for shorter time intervals, including as pre/post-treatment and follow-up measures in treatment outcome studies.     

Measuring community connectedness among diverse sexual minority populations

Theory and research agree that connectedness to the lesbian, gay, bisexual, and transgender (LGBT) community is an important construct to account for in understanding issues related to health and well-being among gay and bisexual men. However, the measurement of this construct among lesbian and bisexual women or racial and ethnic minority individuals has not yet been adequately investigated. This study examined the reliability and validity of an existing measure of connectedness to the LGBT Community among a diverse group of sexual minority individuals in New York City, and whether differences in connectedness existed across gender and race or ethnicity. Scores on the measure demonstrated both internal consistency and construct stability across subgroups defined by gender and race or ethnicity. The subgroups did not differ in their mean levels of connectedness, and scores on the measure demonstrated factorial, convergent, and discriminant validity, both generally and within each of the subgroups. Inconsistencies were observed with regard to which scores on the measure demonstrated predictive validity in their associations with indicators of mental health and well-being. The scale is a useful tool for researchers and practitioners interested in understanding the role of community connectedness in the lives of diverse populations of sexual minority individuals.     

Women's Scores on the Sexual Inhibition/Sexual Excitation Scales (SIS/SES): Gender Similarities and Differences

Theory and research agree that connectedness to the lesbian, gay, bisexual, and transgender (LGBT) community is an important construct to account for in understanding issues related to health and well-being among gay and bisexual men. However, the measurement of this construct among lesbian and bisexual women or racial and ethnic minority individuals has not yet been adequately investigated. This study examined the reliability and validity of an existing measure of connectedness to the LGBT Community among a diverse group of sexual minority individuals in New York City, and whether differences in connectedness existed across gender and race or ethnicity. Scores on the measure demonstrated both internal consistency and construct stability across subgroups defined by gender and race or ethnicity. The subgroups did not differ in their mean levels of connectedness, and scores on the measure demonstrated factorial, convergent, and discriminant validity, both generally and within each of the subgroups. Inconsistencies were observed with regard to which scores on the measure demonstrated predictive validity in their associations with indicators of mental health and well-being. The scale is a useful tool for researchers and practitioners interested in understanding the role of community connectedness in the lives of diverse populations of sexual minority individuals.     

Exploring rigour in autoethnographic research

Abstract

The changing face of qualitative research has opened up discourses on the validity and rigour of research processes and products. A current debate is whether the criteria traditionally used to judge the rigour of quantitative research are appropriate to assess the academic integrity of qualitative research including autoethnography. The purpose of this article is to appraise views on the academic rigour, validity and scientific accountability of research in general and autoethnographic research in particular. The researcher reports on issues of rigour in conducting autoethnographic research from her own experience and that of published autoethnographic researchers. The issue of relevant appraisal criteria against which the research integrity and rigour of autoethnographic studies can be assessed is investigated. It is concluded that the academic rigour of autoethnographic studies should be established to enhance its credibility and value and that the criteria used need to be judiciously determined and applied.     

Sampling Richness and Qualitative Integrity: Challenges for Research With Families

Sampling is one of the most difficult and contentious aspects of qualitative research design. There are few guidelines for sampling decisions or for understanding saturation in qualitative family research. The authors frame the problematic of data quality in the selection of units of analysis and observation and consider how to enhance sample richness. They outline considerations for data quantity and sample size as well as case‐ and variable‐based approaches. With multiple examples from recent and classic studies to illustrate the consequences of sampling decisions, they explore links between saturation and validity. Finally, they encourage researchers to craft a coherent statement on qualitative integrity to demonstrate how their sampling decisions are rooted in epistemology, theory, and richness and quality of data.     

What Is Ecological Validity? A Dimensional Analysis

Ecological validity has typically been taken to refer to whether or not one can generalize from observed behavior in the laboratory to natural behavior in the world. Although common in current discussions of research, the idea of ecological validity has a long history in psychological thought. A brief historical examination of this idea reveals that concerns with ecological validity are evident in multiple dimensions of experimental work, including the nature of the experimental setting, the stimuli under investigation, and the observer's response employed as the measure. One problem with this multidimensionality, however, is that no explicit criteria have been offered for applying this concept to an evaluation of research. One consequence of this problem is that concerns with ecological validity can be raised in most experimental situations. This article includes a discussion of some demands of ecological validity and the nature of these different dimensions, as well as a critical evaluation of research on the development of mobility with respect to these constraints.     

“Hired hand research” revisited

This article reconsiders the arguments of Roth (1966) concerning “hired band research.” Problems associated with different types of employees, research tasks, and research organizations are distinguished. It is argued that researchers who hire assistants can minimize the hired hand mentality by: 1) hiring persons who are able to stay on the job long enough to develop commitment to the project; 2) involving both hired hand workers and researchers in the research process to the fullest extent possible; 3) designing a flexible research project; 4) justifying theoretically each variable, interview question, observation, etc.; 5) collecting qualitative as well as quantitative data; and 6) remaining highly involved in the research project. Her recent writings include “The Consequences of Professionalization and Formalization in the Pro-Choice Movement” (American Sociological Review 1988) and “Organizational and Environmental Influences on the Development of the Pro-Choice Movement” (Social Forces, forthcoming).     

Evaluating and rethinking the case study

This paper will first explore briefly how the case study fell out of favour as a legitimate research tool, and how case study researchers responded to the critique that case study research lacked internal and external validity. Some case study advocates attempted to meet this critique by using the case to compare different theoretical predictions, dividing the case into subcases, or treating the case as an experiment. A more effective response distinguishes between ‘extensive’ and ‘intensive’ research designs, and critiques the extensive model. The paper then provides a clarification of purpose and a revived emphasis on theory, history, and intervention to develop the case study method. This paper will begin to construct a comprehensive case study methodology independent of the old critiques, and will show the crucial place which case studies must occupy in our craft.     

Protecting Research Subjects and Unintended Consequences: The Effect of Guarantees of Confidentiality

There is significant concern among researchers and governmentadministrators that individuals who serve as research subjectsbe protected from risks associated with empirical inquiry. Surveyresearchers frequently administer guarantees of confidentialityand anonymity in an effort to minimize subjects1 psychologicalharm and to increase the validity ofdata. This study tests theeffect of guarantees of confidentiality and anonymity on (I)the evaluation apprehension of juvenile offenders who providedself-reports of past behavior soon after arrest, (2) their responserates to individual interview items, and (3) the quality ofthe subjects' responses. Findings are reported and implicationsfor survey researchers are discussed.