Secondary observer effects: idiosyncratic errors in small-N secondary data analysis

In this paper, I extend the concept of observer effect into the realm of country-level secondary data analysis. When analyzing what appear to be the same secondary data using the same methods, macro-comparative researchers arrive at different results. I argue that this is a product of idiosyncratic variation directly or indirectly produced by the researchers. Even when this bias produces only small perturbations in results, the consequences may be very large. Using an influential study by Brooks and Manza I analyze this secondary observer effect (SOE). Two seemingly identical replications of their data by different researchers produced surprising variations. Reanalysis of these divergent values produces similar but not identical results. A rough calculation of the size of the SOEs suggests that they are about .32 standardized standard deviations across variable scores. Simulations of this size of error show that significant changes in findings occur as a result.