Novel Evaluation Techniques for Outlier Detection Methods: A Case Study with RCOD

Outlier detection remains a key challenge in data analysis, with applications spanning cybersecurity, finance, medicine, and more. This paper introduces a comprehensive evaluation framework for comparing outlier detection methods, using the Random Clustering-based Outlier Detector (RCOD) as a case study. RCOD groups data points around randomly selected cluster centers and identifies outliers based on distance-based criteria and statistical thresholds. To enable more reliable assessment, two novel evaluation strategies are proposed: one based on deviations from the best-performing method per dataset, and another based on rank-based comparison. Experiments conducted on 30 benchmark datasets and 13 detection methods demonstrate RCOD’s superior performance and stability across accuracy, precision, and F1-score metrics. The proposed evaluation techniques provide a deeper insight into the effectiveness of outlier detectors than traditional performance metrics alone. Statistical validation confirms the significance of RCOD’s advantage, highlighting its robustness and applicability to diverse data environments.