When social scientists use exemplary methods and report their findings accurately, we like to think that they have found out something about the social world. Furthermore, it then seems that if another social scientist conducted the same study again, with the same methods, their findings would be pretty much the same. In fact, “reproducibility” is one of the goals for social science research.
But in spite of widespread acceptance of the standard of reproducibility, few studies are carefully replicated. Perhaps that’s mostly because it’s more exciting to try to discover something new, rather than simply to confirm what someone else has already reported.
In a project that began in 2011, University of Virginia psychologist Brian Nosek and a large team decided to put reproducibility to a stringent test. They designed a research project to repeat 100 studies that had been published in leading psychology journals. Using the same methods, they were only able to reproduce the same results in 39% of the studies.
Very disappointing results that remind us both of the complexity of the social world we study and of the challenges of social research. The original studies seemed to have been carried out with rigorous methods.
What accounts for this low level of reproducibility? Do scientists skip over many of their findings and report only those that are “interesting” to journal editors? If so, could many reported findings just be due to chance? Or could social scientists shape their methods in subtle ways that make it more likely they can reach their favored conclusions?
What do you think explains the low rate of reproducibility of the psychology studies?
Do you think the rate of reproducibility is likely to be higher in sociological research that involves large representative samples from clearly defined populations?
Researching in a Social World 