Premium
Choosing the fitness function for the job: Automated generation of test suites that detect real faults
Author(s) -
Salahirad Alireza,
Almulla Hussein,
Gay Gregory
Publication year - 2019
Publication title -
software testing, verification and reliability
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.216
H-Index - 49
eISSN - 1099-1689
pISSN - 0960-0833
DOI - 10.1002/stvr.1701
Subject(s) - initialization , fitness function , computer science , unit testing , test (biology) , function (biology) , fitness approximation , machine learning , reliability engineering , fault (geology) , code (set theory) , artificial intelligence , genetic algorithm , engineering , software , paleontology , set (abstract data type) , evolutionary biology , seismology , biology , programming language , geology
Summary Search‐based unit test generation, if effective at fault detection, can lower the cost of testing. Such techniques rely on fitness functions to guide the search. Ultimately, such functions represent test goals that approximate—but do not ensure—fault detection. The need to rely on approximations leads to two questions— can fitness functions produce effective tests and, if so, which should be used to generate tests? To answer these questions, we have assessed the fault‐detection capabilities of unit test suites generated to satisfy eight white‐box fitness functions on 597 real faults from the Defects4J database. Our analysis has found that the strongest indicators of effectiveness are a high level of code coverage over the targeted class and high satisfaction of a criterion's obligations. Consequently, the branch coverage fitness function is the most effective. Our findings indicate that fitness functions that thoroughly explore system structure should be used as primary generation objectives—supported by secondary fitness functions that explore orthogonal, supporting scenarios. Our results also provide further evidence that future approaches to test generation should focus on attaining higher coverage of private code and better initialization and manipulation of class dependencies.