Gap between Prediction and Truth: A Case Study of False-Positives in Leakage Detection

Since leakage detection was introduced as a popular side-channel security assessment, it has been plagued by false-positives (a.k.a. type I errors). To fix this error, the previous solutions set detection thresholds based on an assumption-based prediction of false-positive rate (FPR). However, this study points out that such a prediction (of FPR) may be inaccurate. We notice that the prediction in EuroCrypt2016 is much smaller than (approximately 1 / 779 times) the true FPR. The gap between prediction and truth, called underpredicted false-positives (UFP), leads to severe false-positives in leakage detection. Then, we check the statistical distribution of test statistics to analyze the cause of UFP. Our analysis indicates that the overlap between cross-validation (CV) blocks gives rise to an assumption error in the distribution of the CV-based estimates of ρ -statistics, which is the root cause of UFP. Therefore, we tackle the UFP by eliminating the overlap between blocks. Specifically, we propose a profiling-shared validation (PSV) and utilize this validation to improve the detection of any-variate any-order leakages. Our experiments show that the PSV solves the UFP and saves more than 75% of the test time costs. In summary, this article reports a potential flaw in leakage detection and provides a complete analysis of the flaw for the first time.