An empirical evaluation of the G/Q/M method

Improvement of the software process is a major concern for many organizations. A critical part of such an endeavor is the definition of metrics. Despite the importance of metric definition, there have been no evaluations of existing methods for achieving this. It is generally taken for granted that a method with wide acceptance is suitable. A review of metric definition methods identities Basili's G/Q/M as one of the most widely used. This paper reports on an evaluation of the G/Q/M method. The evaluation is based on an actual application of the method in a process improvement effort. The resultant metrics (and instrument) are evaluated with respect to the following criteria: interpretability, validity, reliability, effectiveness, and transportability. The causes of problems found are identified. These causes are problems with the G/Q/M method itself. The evaluation indicates that the metrics resultant from the application of the G/Q/M method do not appropriately meet the above criteria, and research on evolving G/Q/M or developing alternative methods is emphasized.