Premium
How detrimental is coincidental correctness to coverage‐based fault detection and localization? An empirical study
Author(s) -
Abou Assi Rawad,
Masri Wes,
Trad Chadi
Publication year - 2021
Publication title -
software testing, verification and reliability
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.216
H-Index - 49
eISSN - 1099-1689
pISSN - 0960-0833
DOI - 10.1002/stvr.1762
Subject(s) - correctness , computer science , metric (unit) , fault (geology) , prioritization , algorithm , reliability engineering , biology , paleontology , operations management , management science , engineering , economics
According to the reachability–infection–propagation (RIP) model, three conditions must be satisfied for program failure to occur: (1) the defect's location must ber eached , (2) the program's state must becomei nfected and (3) the infection mustp ropagate to the output. Weak coincidental correctness (or weak CC ) occurs when the program produces the correct output, while condition (1) is satisfied but conditions (2) and (3) are not satisfied. Strong coincidental correctness (or strong CC ) occurs when the output is correct, while both conditions (1) and (2) are satisfied but not (3). The prevalence of CC was previously recognized. In addition, the potential for its negative effect on spectrum‐based fault localization (SBFL) was analytically demonstrated; however, this was not empirically validated. Using Defects4J , this paper empirically studies the impact of weak and strong CC on three well‐researched coverage‐based fault detection and localization techniques, namely, test suite reduction (TSR), test case prioritization (TCP) and SBFL. Our study, which involved 52 SBFL metrics, provides the following empirical evidence. (i) The negative impact of CC tests on TSR and TCP is very significant. In addition, cleansing the CC tests was observed to yield (a) a 100% TSR defect detection rate for all subject programs and (b) an improvement of TCP for over 92% of the subjects. (ii) The impact of CC tests on SBFL varies widely w.r.t. the metric used. The negative impact was strong for 11 metrics, mild for 37, non‐measurable for 1 and non‐existent for 3 metrics. Interestingly, the negative impact was mild for the 9 most popular and/or most effective SBFL metrics. In addition, cleansing the CC tests resulted in the deterioration of SBFL for a considerable number of subject programs. (iii) Increasing the proportion of CC tests has a limited impact on TSR, TCP and SBFL. Interestingly, for TSR and TCP and 11 SBFL metrics, small and large proportions of CC tests are strongly harmful. (iv) Lastly, weak and strong CC are equally detrimental in the context of TSR, TCP and SBFL.