A Reinforcement Learning Based Hybrid Scheduling Mechanism for mmWave Networks

Millimeter-wave (mmWave) technology is expected to support next-generation wireless networks by expanding the available spectrum and supporting multi-gigabit services. While mmWave communications hold great promise, mmWave links are vulnerable against link blockages, which can severely impact their performance. This paper aims to develop resilient transmission mechanisms to suitably distribute traffic across multiple paths in mmWave networks. The main contributions include: (a) the development of proactive transmission mechanisms to build resilience against link blockages in advance, while achieving a high end-to-end packet rate; (b) the design of a heuristic path selection algorithm to efficiently select (in polynomial time in the network size) multiple proactively resilient paths that have high capacity; and (c) the development of a hybrid scheduling algorithm that combines the proposed path selection algorithm with a deep reinforcement learning (DRL) based online approach for decentralized adaptation to blockages. To achieve resilience against link blockages and to adapt the information flow through the network, a state-of-the-art Soft Actor-Critic DRL algorithm is investigated. The proposed scheduling algorithm robustly adapts to blockages and channel variations over different topologies, channels, and blockage realizations while outperforming alternative algorithms which include a conventional congestion control algorithm additive increase multiplicative decrease. Specifically, it achieves the desired packet rate in over 99% of the episodes in static networks with blockages (where up to 80% of the paths are blocked), and in 100% of the episodes in time-varying networks with blockages and link capacity variations – compared to 0.5% to 45% success rates achieved by the baseline methods under the same conditions.