Quantity, quality, and consistency as bibliometric indicators

Dear Sir, In his recent paper, Vinkler (2013, p. 1085) comes to the conclusion that “substantial theoretical work and several case studies are needed to arrive at a widely acceptable solution concerning the characterization of the eminence of publications of scientists and teams, both qualitatively and quantitatively, by a single indicator.” The quantity part is represented by the number of papers, P, in the publication set (for a team or individual scientist), and the quality part is represented by the impact, i = C/P, where C is the total number of citations received by the P papers. I argue that, in addition to quantity and quality, a third attribute, which I shall call “consistency,” ν, has to be introduced for a complete three-dimensional (3D) evaluation of the information production process. There is an interesting parallel with the “3Vs” metaphor of Laney (2011) on 3D data management. One can think of P as indicating volume, impact i as indicating velocity with which the ideas in P papers are communicated through citations, C, and consistency, ν, as indicating the variety (variation) in the quality of the individual papers in the portfolio. For a definition of these terms, let us start with Prathap (2011a, 2011b). Let ci, i = 1 to P, represent the citation sequence of all P papers of any portfolio (say, of an individual scientist or team). Then C = Σci, i = 1 to P is the total number of citations. The impact, i = C/P, becomes a proxy for quality (velocity). Prathap (2011a) showed that it is possible to define second-order energy terms such as E ci = ∑ 2 and X = iC. P itself serves as a measure of the quantity (volume) of effort and is a performance indicator of the zeroth-order. One can think of i and P as two orthogonal components of a 3D performance evaluation protocol. Then, C = Pi can be considered to be a performance indicator of the first order (Prathap, 2011a). If citation sequences are rearranged in monotonically decreasing order, very high skews are often seen because of a possible huge variation in the quality of each paper in the publication set. Thus, two different sets can have the same C, and one could have achieved this with far fewer papers, with a higher quality of overall performance, or with the same number of papers (i.e., same quality) but a higher degree of consistency or evenness. This suggests that C by itself may not be the final word on the measurement of performance. The product X = iC = iP becomes a higher order measure. It is a robust second-order performance indicator (Prathap, 2011a, 2011b). Apart from X, an additional indicator defined by E ci = ∑ 2 also appears as a second-order indicator. The coexistence of X and E allows us to introduce a third attribute that is neither quantity nor quality. In the context of 3D data management, the attribute variety is introduced as a third component (Laney, 2011). In a bibliometric context, the appellation “consistency” may be more meaningful. The simple ratio of X to E can be viewed as the third component of performance, namely, the consistency term ν = X/E. Perfect consistency (ν = 1, i.e., when X = E) is a case of absolutely uniform performance; that is, all papers in the set have the same number of citations, ci = c = i. The greater the skew, the larger is the concentration of the best work in a very few papers of extraordinary impact. The inverse of consistency thus becomes a measure of concentration. Thus, for a complete 3D evaluation of publication activity, we need P, i, and ν. These are the three components of a quantity–quality–consistency or volume–velocity–variety landscape.