Open AccessStatistical Methodology for Comparison of SAT Solvers
Author(s) -
Mladen Nikolić
Publication year - 2018
Publication title -
epic series in computing
Language(s) - English
Resource type - Conference proceedings
ISSN - 2398-7340
DOI - 10.29007/bhvj
Subject(s) - computer science , task (project management) , set (abstract data type) , point (geometry) , running time , theoretical computer science , algorithm , mathematics , programming language , geometry , management , economics
Evaluating improvements to modern SAT solvers and comparison of two arbitrary solvers is a challenging and important task. Relative performance of two solvers is usually assessed by running them on a set of SAT instances and comparing the number of solved instances and their running time in a straightforward manner. In this paper we point to shortcomings of this approach and advocate more reliable, statistically founded methodologies that could discriminate better between good and bad ideas. We present one such methodology and illustrate its application.