Deep Neural Network-Based Detection of Modulated Jamming in Free-Space Optical Systems: Theory and Performance under Atmospheric Fading

Free-space optical (FSO) communication systems, though advantageous in terms of bandwidth and security, are highly susceptible to deliberate jamming attacks, particularly under intensity modulation and direct detection (IM/DD) constraints. This paper investigates the problem of detecting structured optical jamming introduced via a separate FSO link, where both legitimate and adversarial transmissions undergo independent Gamma-Gamma fading. We propose a deep neural network (DNN)-based binary classifier designed to discriminate between clean and jammed received frames. The DNN operates on a composite feature vector comprising raw signal samples, spectral content, energy statistics, and higher-order distributional descriptors, enabling robust detection under both modulated and persistent jamming scenarios. To benchmark the performance of the proposed architecture, we derive a closed-form upper bound on the Bayes classification error using the Bhattacharyya coefficient, expressed analytically in terms of Meijer-G functions. This bound reveals how key system parameters—including jammer power, noise variance, signal dimension, and turbulence severity–jointly influence detectability. Monte Carlo simulations are used to evaluate the DNN's performance under varying noise, fading, and jamming conditions. Results show that the proposed model approaches the theoretical limit at moderate dimensions, low noise, and high jammer power, while generalization performance is constrained at large dimensions by data sparsity and architectural capacity. The combination of theoretical bounds and feature-informed DNN design offers a principled framework for jamming detection in realistic FSO environments.