A Diffusion-based Expectation-Maximization Framework for Probabilistic Traffic Data Imputation

Traffic data imputation plays a crucial role in supporting the development of accurate forecasting models and the provision of reliable real-time information systems. Recent advances in generative modeling, particularly diffusion models, have introduced new solutions to probabilistic traffic data imputation, offering better understanding of imputation uncertainty compared with deterministic models and improved imputation accuracy over previous probabilistic models. Despite the success of these deep imputation models, they may suffer from performance degradation when confronted with complex missing patterns. Additionally, the artificial data corruption used in the prevalent self-supervised learning setting may not align with the missing patterns encountered in unseen test data. To address these challenges, this paper introduces a novel probabilistic approach, Diffusion-based EM framework for traffic data Imputation (DEMI), which integrates the diffusion model into the Expectation-Maximization (EM) algorithm. In contrast to existing methods, DEMI learns the underlying unconditional distribution of traffic data through the iterative EM framework, enabling both accurate imputation without requiring additional corruption of observations. It also allows conditional generation with arbitrary missing patterns, as well as unconditional generation as a data augmentation tool. The framework features a refined posterior sampling method to enhance the quality of conditional reverse sampling and incorporates a transformer-based denoising network to capture spatio-temporal dependencies. Experimental results on real-world traffic datasets demonstrate DEMI’s superior performance over state-of-the-art deterministic and probabilistic imputation models, particularly in scenarios with complicated missing patterns and pattern shifts.