Performance Evaluation of Reinforcement Learning-Based Intrusion Detection Systems

Reinforcement Learning (RL) offers an innovative approach to Intrusion Detection Systems (IDSs) by enabling agents to autonomously learn through dynamic interactions within network environments, without relying on pre-recorded datasets. Through RL, agents evaluate action values—rewards and penalties at each time step, aiming to converge toward optimal policies that enhance intrusion detection effectiveness. However, achieving reliable convergence in real-time environments remains uncertain, potentially impacting overall detection performance. This paper examines two prominent RL-based IDS architectures: a single-agent model and a game-theoretic multi-agent model, both employing the Deep Q-Network (DQN) algorithm, to highlight the challenges associated with convergence in real-time environments. Using OMNeT++ simulation, a semi-realistic network environment was implemented to test the performance of both architectures. Key performance metrics including accumulated Q-values, loss, epsilon decay, detection accuracy, and precision were analyzed to evaluate the two models. The findings reveal that both the single-agent and game-theoretic multi-agent architectures exhibit unstable convergence, leading to reduced detection accuracy and precision. The observed metrics highlight areas for improvement and underscore the challenges that must be addressed to achieve optimal real-time intrusion detection using reinforcement learning.