Statistical Metamorphic Testing Testing Programs with Random Output by Means of Statistical Hypothesis Tests and Metamorphic Testing

Testing software with random output is a challenging task as the output corresponding to a given input dif- fers from execution to execution. Therefore, the usual ap- proaches to software testing are not applicable to random- ized software. Instead, statistical hypothesis tests have been proposed for testing those applications. To apply these sta- tistical hypothesis tests, either knowledge about the theo- retical values of statistical characteristics of the program output (e. g. the mean) or a reference implementation (e. g. a legacy system) are required to apply statistical hypothesis tests. But often, both are not available. In the present paper, it is discussed how a testing method called Metamorphic Testing can be used to construct sta- tistical hypothesis tests without knowing exact theoretical characteristics or having a reference implementation. For that purpose, two or more independent output sequences are generated by the implementation under test (IUT). Then, these sequences are compared according to the metamor- phic relation using statistical hypothesis tests.