An Effective Fault-Tolerant Intrusion Detection System under Distributed Environment

Traditional intrusion detection system is limited to a single network or several hosts, which has been seriously unable to fulfill the growing information security problems. This paper uses the distributed technology to design and implement an intrusion detection system (IDS) based on the hybrid of Hadoop with some effective open-source technologies. On the one hand, it can efficiently realize the data acquisition and analysis under distributed environment. On the other hand, it can solve the problems of single-point fault-tolerant and the insufficient data processing capacity of the traditional intrusion detection system. In this IDS, RabbitMQ, Flume, and MongoDB are utilized to act as the middleware of this system to build the system environment which includes the collector, analyzer, and data storage. By detecting the CPU and memory usage of hosts, TCP connections, network bandwidth, web server operation logs, and the logs of user behavior, the proposed IDS especially focuses on monitoring the first four parts, which can better detect external distributed denial of service attacks and intrusions and send automatically alarm service information to the administrators.