Interpreting Citations

I read with great interest the editorial by Dr. Siekevitz entitled “Citations and the Tenor of theTimes” (1). I am glad that he has raised a number of issues that are often ignored not only by practicing scientists but far worse by scientific administrators and policy makers who make pronouncements on matters of grave import using inadequate data. It is also vital to consider the issues of parochialism that Dr. Siekevitz raises for it certainly produces grave distortions at a global level. The real problem is that counting is relatively easy; any programmed computer can do it. To look beneath the numbers and reflect on the meaning is relatively hard. What one needs is an estimate of what Moravcsik and Murugesan (2) call the “quality of citations” as many of the subtleties of citations are related not only to the quality of the paper cited but the context in which that citation is made. This kind of analysis can only be undertaken by researchers who work in the field rather than by sociologists, librarians, and others who may not be able to understand the technical content of the papers cited. The authors (2) selected references in papers dealing with high-energy particle physics published in a single journal, Physical Review. One of the several questions they asked was whether the reference was organic (i.e., truly needed for the understanding of the referring paper) or perfunctory (i.e., an acknowledgement that some other work in the same general area has been performed). They concluded from their analysis that 41% of references were perfunctory. To quote the authors, “This raises serious doubts about the use of citations as a quality measure, since it is then quite possible for somebody or some group to chalk up high citation counts by simply writing barely publishable papers on fashionable subjects which will then be cited asperfunctory, ‘also ran’ references?’ One could argue that a single organic, relevant citation may be worth a dozen perfunctory ones. At the opposite extreme are those whose labors add substantially to the quality of a work reported but are rarely ever cited. This is particularly true for technical improvements. Again to take an example from physics, Goudsmit (3) notes “Scintillation counters and photomultipliers are generally used in experiments in nuclear and particle physics, but their inventors and dozens of researchers who have improved these essential tools to their present perfection are rarely cited?’ The relevance of citation analysis is far deeper than the “pats on the back” noted by Siekevitz (1). It has the potential for tracing the history or organic development of a discipline, and the citation index can be used to indicate “how each brick in the edifice of science is linked to all the others” (4). To do so objectively and fairly would raise thehistory of science to a genuine science where hypotheses can be posed and critically tested using hard data. Unfortunately, even a cursory use of the citation index for this purpose is bedeviled with errors. Several of these have been raised and discussed in MacRoberts and MacRoberts (5). These include among others parsimonious citations, miscitations, ceremonial citations, paying of intellectual debts, use ofsecondary sources, duplicate citing, etc. Whether the list of pitfalls noted by these authors precludes the use of citation analysis for comparative measurements can be debated (6, 7).Nevertheless, a proper use ofcitation analysis requires a good deal of thought and reflection and cannot be undertaken lightly. Garfield ended an early essayon the value of citation indexing with the words “Like most other scientific discoveries, this tool can be used wisely or abused. It is now up to the scientific community to prevent abuse of the SCI by devoting the necessary attention to its proper and judicious exploitation” (4). The editorial by Dr. Siekevitz (1) argues quite persuasively that the scientific and administrative communities have done no such thing. In fact they may have actively promoted its abuse. Whether given the temper of the times, scientists would make the time and effort to utilize such a powerful tool wisely and well is a moot point. I am pessimistic enough to believe that they will not. The citation index may end up having the same relevance for the quality of science that Nielsen ratings have for the quality of TV programming in North America. What a terrible pity!