Premium
An improved agglomerative hierarchical clustering anomaly detection method for scientific data
Author(s) -
Shi Peng,
Zhao Zhen,
Zhong Huaqiang,
Shen Hangyu,
Ding Lianhong
Publication year - 2020
Publication title -
concurrency and computation: practice and experience
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.309
H-Index - 67
eISSN - 1532-0634
pISSN - 1532-0626
DOI - 10.1002/cpe.6077
Subject(s) - cluster analysis , hierarchical clustering , data mining , computer science , generalization , anomaly detection , centroid , decision tree , pattern recognition (psychology) , measure (data warehouse) , anomaly (physics) , artificial intelligence , mathematics , mathematical analysis , physics , condensed matter physics
Summary Anomaly detection tries to find out the data that disobeys the rule of majority data or expected patterns. The traditional hierarchical clustering algorithms have been adopted to detect anomaly, but have the disadvantages of low effectiveness and unstability. So we propose an improved agglomerative hierarchical clustering method for anomaly detection. It dynamically adjusts the optimum clustering number according to the self‐defined criterion to save the trouble of manually picking clustering number, and determines the optimum clustering distance mode according to cophenetic correlation coefficient to reduce the procedure of manually testing the suitable distance mode in each iteration. The performances of proposed method are verified on tensile test, HTRU2 and credit card dataset. Compared with the traditional methods, our method possesses the most comprehensive performance (the highest F ‐measure with less iterations), which shows effectiveness of anomaly detection. And compared with the traditional methods (single, complete, average, and centroid mode), our method achieves the best performance on tensile test and HTRU2 dataset, showing stronger generalization. Compared with other methods (Decision + Gradient Boosted Tree, Decision Trees + Decision Stump, etc) on credit card dataset, our method obtains similar accuracy, and ranks in the top level in the aspect of sensitivity.