Unsupervised Clustering of Web Sessions to Detect Malicious and Non-malicious Website Users

Denial of Service (DoS) attacks are recognized as one of the most damaging attacks on the Internet security today. Recently, malicious web crawlers have been used to execute automated DoS attacks on web sites across the WWW. In this study, we examine the use of two unsupervised neural network (NN) learning algorithms for the purpose web-log analysis: the Self- Organizing Map (SOM) and Modified Adaptive Resonance Theory 2 (Modified ART2). In particular, through the use of SOM and Modified ART2, our work aims to obtain a better insight into the types and distribution of visitors to a public web-site based on their link-traversal behaviour, as well as to investigate the relative differences and/or similarities between malicious web crawlers and other non-malicious visitor groups. The results of our study show that, even though there is a pretty clear separation between malicious web-crawlers and other visitor groups, around 8% of malicious crawlers exhibit very ‘human-like’ browsing behaviour and as such pose a particular challenge for future web-site security systems