On the scientific study of small samples: Challenges confronting quantitative and qualitative methodologies

Often phenomena that are important to understand and predict are very rare. Rare events can prove difficult to analyze systematically because they do not generate many sampling observations. In this article I examine how small sample sizes can be studied scientifically. The article begins with an explanation of the distinction between research and science. I then bring to the fore the importance of counterfactual comparisons and outline the nature of the methodological problems posed by the study of small samples. These problems include challenges related to using a single case, small sample sizes, selecting on the dependent variable, regression toward the mean, explaining a variable with a constant, and using the same data to both generate and test hypotheses. I provide potential resolutions to these problems which are: (a) employing matched controls; (b) shifting or widen the category of inquiry; (c) selecting variables based on variance in the independent variable; (d) including counterfactuals; (e) ensuring that both independent and dependent variables demonstrate variation; and (f) testing potential hypotheses against data sets that are fully independent of those used to generate the hypotheses. I conclude with a discussion of future directions for undertaking a more scientific approach to using small samples.