Numerous functional magnetic resonance imaging (fMRI) studies have reported sex differences. To empirically evaluate for evidence of excessive significance bias in this literature, we searched for published fMRI studies of human brain to evaluate sex differences, regardless of the topic investigated, in Medline and Scopus over 10 years. We analyzed the prevalence of conclusions in favor of sex differences and the correlation between study sample sizes and number of significant foci identified. In the absence of bias, larger studies (better powered) should identify a larger number of significant foci. Across 179 papers, median sample size was n = 32 (interquartile range 23-47.5). A median of 5 foci related to sex differences were reported (interquartile range, 2-9.5). Few articles (n = 2) had titles focused on no differences or on similarities (n = 3) between sexes. Overall, 158 papers (88%) reached “positive” conclusions in their abstract and presented some foci related to sex differences. There was no statistically significant relationship between sample size and the number of foci (−0.048% increase for every 10 participants, p = 0.63). The extremely high prevalence of “positive” results and the lack of the expected relationship between sample size and the number of discovered foci reflect probable reporting bias and excess significance bias in this literature.
The nature of possible sex differences in behavior and brain structure and function has been a topic of debate in the scientific community for centuries1. Although the presence of Y sex chromosomes affects structural differentiation of some brain regions, such as the sexually dimorphic nucleus of the preoptic area, or “SDN”, in rodents2,3, neuroanatomical differences have not been consistently related to robust differences in human brain function4. In the field of human neuroimaging research, there are some who argue that sex differences in brain structure, chemistry and function are substantial and widespread5, while others claim that there is an overlapping continuum of brain structure and function rather than widespread stereotyped “gendered behavior”6. It is also speculated that there may be strong bias and major flaws, particularly in the corpus of neuroimaging literature7.
Recent systematic reviews and empirical evaluations of the human neuroimaging and animal studies literature suggest that publication and other reporting biases are prevalent and most studies are underpowered8, such that small sample sizes particularly for functional magnetic resonance imaging (fMRI) studies of the brain undermine the reliability and precision of results across the field9,10,11. Specifically, we previously reported evidence of too many statistically significant studies evaluating differences in morphometric measures of regions of interest studies for multiple neurological disease states12, and inflated numbers of statistically significant foci in small voxel-based morphometric studies (VBM)13 and fMRI studies of the brain14.
The goals of the present investigation are to (a) characterize the literature of fMRI studies of the brain that evaluated sex differences and (b) empirically evaluate for evidence of excessive significance bias, which may reflect selective reporting of “positive” (statistically significant) results in this complex and controversial field of neuroscience. The theoretical framework for the present investigation is based on the notion that studies with large samples have more power to detect abnormalities, therefore the number of reported foci should show a positive relationship with the sample size. Small studies should detect only a small proportion of the true signals, whilst larger studies should detect a larger proportion of the true signals. As shown in previous empirical evaluations of neuroimaging studies, a weak or null relationship could indicate potential reporting biases affecting the smaller studies more than the larger studies11,14. Moreover, we assessed whether there were any published studies in this field that concluded that there were no statistically significant sex-differences. Given that many studies in the field are very small, a substantial number of studies should find no sex-differences, even if genuine such differences exist. A very low proportion of such “negative” studies would also be cause for concern for similar selective reporting bias.
See full details