Information and Communications Security

Automated malware classification using deep learning techniques has been widely researched in recent years. However, existing studies addressing this problem are always based on the assumption of closed world, where all the categories are known and fixed. Thus, they lack robustness and do not have the ability to recognize novel malware instances. In this paper, we propose a prototype-based approach to perform robust malware traffic classification with novel class detection. We design a new objective function where a distance based cross entropy (DCE) loss term and a metric regularization (MR) term are included. The DCE term ensures the discrimination of different classes, and the MR term improves the within-class compactness and expands the betweenclass separateness in the deeply learned feature space, which enables the robustness of novel class detection. Extensive experiments have been conducted on datasets with real malware traffic. The experimental results demonstrate that our proposed approach outperforms the existing methods and achieves state-of-the-art results.