A “simple” method for spurious biostatistics: response to the manuscript submitted to The FASEB Journal by Hudes, et al .

The authors (hudes, et al.) of the manuscript have chosen to cluster together three distinct research groups of three senior scientists with different objectives and capabilities. The manner in which the published articles have been selected for analysis appears quite casual. Broad similarities such as “oxidative stress” or “rats” are hardly adequate for a meaningful comparison when there are so many distinguishing parameters in biomedical experiments. An unreasonable pooling of heterogeneous biochemical experimentations in terms of animal models (species, age, sex, body weight, number of animals in each group), pathologies, treatment groups, types of biochemical analyses, assay protocols, instrumentation, etc., have been attempted by the authors, claiming to have developed a “simple statistical methodology.” Besides, there might be substantial differences in the extent to which various biomedical studies take into account control measures in their experimental designs to avoid or reduce the nonspecific and confounding effects. For their analysis, Hudes et al. have pooled mean sd values of different analyses from the 100 articles they have cited. But the statistical methodology employed by them seems problematic because heterogenous data have been arbitrarily pooled, and a single set of 50% boundaries for the pooled coefficients of variation (CVs) has been constructed. They assume that these data derive from a single normally distributed population, so that convenient conclusions can be drawn about “unusual clustering” in some articles. It seems unacceptable to us to assume that the CVs of different biochemical analyses in a journal article would have a single distribution, as if they represented random samples of a single population. In other words, it seems wrong to define a single set of 50% boundaries for the pooled CVs of the heterogenous variables having different distributions, and to conclude that some of the journal articles have “unusual clustering.” At best, had the authors confined their analysis to a single analysis under similar experimental conditions, it is possible that they could have evaluated significant trends in biological variation. Even here, they would have to account for the contributions from analytical variations, intraand interindividual variations in healthy and diseased states. Examining the literatures on the biological variations and medical statistics, it is intuitively clear that the biomedical phenomena and parameters of the published articles are far too varied and complex to lend themselves to simplistic statistical premises. We worry about other assertions in the manuscript by Hudes et al. Their general assumptions and arguments are based on statistical methodology buttressed by only three references, with the remaining 116 references referring only to the journal articles analyzed. Moreover, the selection of articles that have been used as controls has no common ground. Some of the articles of the research groups from the medical biochemistry department have been used as controls and some from outside. Also, the number of articles used as controls is not equal to the test group. It is common knowledge that prior to publication, every manuscript is subjected to double-sieving by referees and editors for adjudicating the merit-worthiness. It is also common knowledge that validation of the data homogeneity and consistency through CV (holding 95% confidence interval) is part of assessing such merit-worthiness. Under these circumstances, analysis of the published articles followed by statements like “unusual clustering of CVs” and conclusions like “data . . . exhibit significantly less statistical variability than the minimum expected in a biological experiment” not only appears inappropriate but reflects on the judgment of the referees and/or editors. We were surprised that Hudes et al. have not found it necessary to communicate with the principal investigators while conducting a multiyear investigation of the published data. We are also troubled that biographical sketches of the principal investigators and qualitative remarks such as “small medical biochemistry department” were included in the study.