Internal validity in rehabilitation research

In experimental research, internal validity is the extent to which the researcher can conclude that the manipulation of the independent variable caused change in the dependent variable. Without causal inference, researchers and research consumers would not be able to meaningfully interpret the results of experiments. This article discusses (a) threats to internal validity in rehabilitation research an (b) experimental procedures that can aid rehabilitation researchers in maximizing the internal validity of their investigations.     

Using and interpreting analogue designs

Researchers in rehabilitation counseling and disability studies sometimes use analogue research, which involves materials that approximate or describe reality (e.g., written vignettes, videotaped exemplars) rather than investigating phenomena in real-world settings. Analogue research often utilizes experimental designs, and it thereby frequently possesses a high degree of internal validity. Analogue research allows investigators to exercise tight control over the implementation of the independent or treatment variable and over potentially confounding variables, which enables them to isolate the effects of those treatment variables on selected outcome measures. However, the simulated nature of analogue research presents an important threat to external validity. As such, the generalizability of analogue research to real-life settings and situations may be problematic. These and other issues germane to analogue research in vocational rehabilitation are discussed in this article, illustrated with examples from the contemporary literature.     

Multiple regression in rehabilitation research

This article examines applications of multiple regression in rehabilitation research. The authors address the logic and assumptions of correlational research, annotated with examples from the contemporary rehabilitation and allied health literature. As a means of testing relationships between a set of independent variables and one dependent variable, multiple regression is presented as a flexible and useful data analytic strategy.     

Counselor confounds in evaluations of vocational rehabilitation methods in substance dependency treatment

Evaluation research on vocational counseling in substance dependency treatment should distinguish between the effects of counselors and counseling methods on clients' employment outcomes. Three experimental designs permit investigation of possible confounds between these types of effects: (a) nested designs (each counselor delivers one counseling method so counselors are nested under methods), (b) crossed designs (each counselor delivers all counseling methods so counselors are crossed with methods), and (c) no-treatment control group designs (experimental group's counselors all deliver the same method). Each design is optimal for one stage of evaluation research. No-treatment control group designs are best for exploratory evaluations of new types of vocational counseling. Nested designs are best for outcome evaluations of different types of interventions. Crossed designs are best for (causal) process evaluations of counseling methods of demonstrated efficacy. Despite the importance of methodological issues and problems, vocational rehabilitation in substance dependency treatment has a greater need for stronger interventions than better evaluation designs.     

Single subject research methods in rehabilitation

The article describes single-subject research designs as an efficient and cost-effective way to assess the impact of targeted interventions on individual behavior. The authors overview the basic tenets of single-subject research, provide examples of several types of single-subject designs, and describe the utility of single-subject designs in rehabilitation research.     

Pretest-posttest designs and measurement of change

The article examines issues involved in comparing groups and measuring change with pretest and posttest data. Different pretest-posttest designs are presented in a manner that can help rehabilitation professionals to better understand and determine effects resulting from selected interventions. The reliability of gain scores in pretest-posttest measurement is also discussed in the context of rehabilitation research and practice.     

Qualitative research strategies in rehabilitation

This article presents an overview of the qualitative research methods that social scientists use to explore human phenomena. The authors describe the philosophical and historical foundations of qualitative research, coupled with illustrations of specific qualitative designs. Applications of qualitative methods in the contemporary rehabilitation literature are also presented.     

The Wider Benefits of Adult Learning: An Illustration of the Advantages of Multi‐method Research

This study presents a technique for enhancing causal validity in in‐depth interview research. The technique is based on the logic of control, as applied in various research contexts such as experiments and statistical analyses. According to this logic, to detect the real effect of an independent variable on a dependent variable, all other variables must be held constant. By using interview control questions (ICQs), this study applies this principle in in‐depth interviewing. An ICQ is a probe question that controls an independent variable in the participant’s thinking. Two modes are presented: simple control (neutralizing an independent variable to examine whether it is a necessary condition for an outcome) and parallel control (isolating an independent variable to examine whether it is a sufficient condition for an outcome). In both modes, the method allows for additional independent variables to surface. Applications in in‐depth interviews exploring attitudes toward victimless deviance are presented.     

Multivariate methods in rehabilitation

The complexity of relationships between variables in a variety of rehabilitation settings can be appropriately addressed by the use of multivariate methods of statistical analysis. The purpose of this article is to describe (a) some basic concepts and principles of multivariate methods as well as (b) common misconceptions in the application of such methods. Specifically, the authors focus on the use of multivariate analysis of variance (MANOVA) in rehabilitation research and some pitfalls that rehabilitation researchers face when choosing between MANOVA for a set of outcome variables versus separate analyses of variance (ANOVAs) for each outcome variable. Examples of the use of MANOVA in rehabilitation research, as well as references to other multivariate methods, are also provided.     

The empowerment of people with disabilities through qualitative research

The purpose of this article is to explore the use of qualitative research in vocational rehabilitation as a means of empowering people with disabilities to (a) take greater control of their lives and (b) directly inform public policy and vocational rehabilitation service delivery. After exploring various definitions of the construct empowerment, we provide a rationale for the use of qualitative research as an empowerment mechanism based on (a) philosophies espoused by leaders of the independent living movement, (b) disability legislation, and (c) a multi-constituency call for more consumer involvement in research. We then describe empirical properties of various qualitative methodologies that lend themselves to consumer empowerment. The article concludes with several examples of qualitative approaches to scientific inquiry from the rehabilitation literature that include many of the empowerment aspects described herein.     

倾向值匹配与因果推论:方法论述评

社会科学对于因果关系的关注使得倾向值匹配(propensity score matching)的方法受到越来越多的重视。本文从以下几个方面对这一方法进行述评。首先,本文系统梳理了倾向值匹配方法的历史、发展及其对于社会学调查研究的意义。其次,通过一个具体实例,本文展示了倾向值匹配的具体实施过程。第三,从密尔对因果关系的论述以及统计学的反事实框架(counterfactual framework)出发,本文阐释了倾向值匹配如何通过"控制"混淆变量从而满足非混淆假设(unconfoundedness)来帮助研究者提出因果性结论。第四,本文将倾向值匹配与回归中断设计和工具变量进行了比较。最后,在简要分析倾向值匹配的局限性之后,本文讨论了倾向值匹配的最新发展趋势。     

Child welfare and the challenge of causal inference

Causal inference refers to the assessment of cause and effect relationships in observational data—i.e., in situations where random assignment is impossible or impractical. Choices involving children in the child welfare system evoke core elements of causal inference—manipulation and the counterfactual. How would a child's circumstances differ if we changed her environment? This article begins with one mathematical approach to framing causal inference, the potential outcomes framework. This methodology is a cornerstone of newer approaches to causal inference. This framework makes clear the identification problem inherent in causal inference and highlights a key assumption often used to identify the model (ignorability or no unobserved confounding). The article then outlines the various approaches to causal inference and organizes them around whether they assume ignorability as well as other key features of each approach. The article then provides guidelines for producing good causal inference. These guidelines emerge from a review of methodological literature as broad ranging as epidemiology, statistics, economics, and policy analysis. These steps will be illustrated using an example from child welfare. The article concludes with suggestions for how the field could apply these newer methods.     

Multiple Segment Factorial Vignette Designs

The multiple segment factorial vignette design (MSFV) combines elements of experimental designs and probability sampling with the inductive, exploratory approach of qualitative research. MSFVs allow researchers to investigate topics that may be hard to study because of ethical or logistical concerns. Participants are presented with short stories about hypothetical characters. These stories comprise 2 – 5 segments in which randomly assigned dimensions are introduced sequentially. Researchers control the occurrence and timing of when dimensions are presented. Respondents are asked questions after each segment. MSFVs may include written, oral, and visual stimuli. These efficient designs elicit quantitative data about causal relations and qualitative data within complex, multidimensional contexts. We discuss how to develop MSFV and suggest ways to avoid problems.     

On the Reconceptualizing of Gender: Implications for Research Design

This article takes up the concept of gender as a specific case in point of spontaneous sociology. The contributory aim is to set forth a multidimensional framework for addressing gender in research. I summarize key developments in gender theory, tease out their implications for research design, and outline four interlocking strategies for how the conventional "gender=male/female" variable might be reconfigured. The first strategy involves shifting from a biological foundationalist paradigm of gender to a social constructionist one, and is the precondition upon which the other strategies are based. Second, research designs must address gender as multilayered and multivariable, not as a single, simple attribute. Third, more attention needs to be given to gender as active and as an outcome of social forces. Fourth is the suggestion to reconfigure the standard gender variable to one reflecting a continuum of positions. A possible terminological reconfiguration of the standard variable is suggested.     

Listening to Children in Care: A Review of Methodological and Theoretical Approaches to Understanding Looked after Children’s Perspectives

This article reviews 44 refereed journal articles published between 2003 and 2008. All of the articles attempt to directly uncover the experiences or perspectives of young people cared for by the state in foster, residential or kinship care homes. The review reveals that this field is developing a rich body of evidence derived from a broad range of methodological and theoretical frameworks. In this sample, research designs appeared to be influenced by placement settings, as well as theoretical orientations, but comparative and longitudinal designs were used in both quantitative and qualitative studies. Participative research designs were only present in qualitative designs within a broadly interpretive theoretical orientation. It is suggested in this article that researchers should critically reflect on the use of constructs that may negatively label young people in care. It is also suggested that there could be fuller reporting of ethical issues. Despite some identified gaps, such as the participation of young people in quantitative research designs and research with younger age groups, the article concludes that there appears to be an encouraging emergence of theoretical and methodological diversity in this research area.     

Using interviews to understand the assignment mechanism in a nonexperimental study: the case of eighth grade algebra

Many inquiries regarding the causal effects of policies or programs are based on research designs where the treatment assignment process is unknown, and thus valid inferences depend on tenuous assumptions about the assignment mechanism. This article draws attention to the importance of understanding the assignment mechanism in policy and program evaluation studies, and illustrates how information collected through interviews can develop a richer understanding of the assignment mechanism. Focusing on the issue of student assignment to algebra in 8th grade, I show how a preliminary data collection effort aimed at understanding the assignment mechanism is particularly beneficial in multisite observational studies in education. The findings, based on ten interviews and administrative data from a large school district, draw attention to the often ignored heterogeneity in the assignment mechanism across schools. These findings likely extend beyond the current research project in question to related educational policy issues such as ability grouping, tracking, differential course taking, and curricular intensity, as well as other social programs in which the assignment mechanism can differ across sites.     

The Current Efficiency of List-Assisted Telephone Sampling Designs

List-assisted random digit dialing (RDD) designs became popularin the late 1980s and early 1990s. Work done by the Bureau ofLabor Statistics and the University of Michigan in 1993 resultedin the development of the underlying theory for these designsas well as the evaluation of various alternative sampling plansto optimize the method. Recent research to reevaluate thesedesigns in light of the significant changes in the telephonesystem over the past decade is presented in this article. Thearticle provides background on the development of list-assisteddesigns, and recent changes in the U.S. telephone system arereviewed. Using 1999 data from Survey Sampling, Inc., an analysisof the current state of the telephone system is presented, anda reoptimization of the earlier designs is undertaken. Resultsfrom the earlier work are compared with findings from the 1999data.     

Why are mediation effects so small?

Identifying mediators in variable chains as part of a causal mediation analysis can shed light on issues of causation, assessment, and intervention. However, coefficients and effect sizes in a causal mediation analysis are nearly always small. This can lead those less familiar with the approach to reject the results of causal mediation analysis. The current paper highlights five factors that contribute to small path coefficients in mediation research: loss of information when measuring relationships across time, controlling for prior levels of a predicted variable, adding control variables to the analysis, ignoring measurement error in one’s variables, and using multiple mediators. It is argued that these issues are best handled by increasing the statistical power of the analysis, identifying the optimal temporal interval between variables, using bootstrapped confidence intervals to analyze the results, and finding alternate ways of assessing the meaningfulness of the indirect effect.