Abstraction in Assertion-Based Test Oracles

Yoonsik Cheon Department of Computer Science The University of Texas at El Paso El Paso, TX 79968-0518 ycheon@utep.edu Abstract Assertions can be used as test oracles. However, writ- ing effective assertions of right abstraction levels is difficult because on the one hand, detailed assertions are preferred for thorough testing (i.e., to detect as many errors as possi- ble), but on the other hand abstract assertions are preferred for readability, maintainability, and reusability. As asser- tions become a practical tool for testing and debugging pro- grams, this is an important and practical problem to solve for the effective use of assertions. We advocate the use of model variables--specification-only variables of which ab- stract values are given as mappings from concrete program states--to write abstract assertions for test oracles. We per- formed a mutation testing experiment to evaluate the effec- tiveness of the use of model variables in assertion-based test oracles. According to our experiment, assertions written in terms of model variables are as effective as assertions writ- ten without using model variables in detecting (injected) faults, and the execution time overhead of model variables are negligible. Our findings are applicable to other use of runtime checkable assertions.