Real-Time Probabilistic Data Fusion for Large-Scale IoT Applications

Internet of Things (IoT) data analytics is underpinning numerous applications, however, the task is still challenging predominantly due to heterogeneous IoT data streams, unreliable networks, and ever increasing size of the data. In this context, we propose a two-layer architecture for analyzing IoT data. The first layer provides a generic interface using a service oriented gateway to ingest data from multiple interfaces and IoT systems, store it in a scalable manner and analyze it in real-time to extract high-level events; whereas second layer is responsible for probabilistic fusion of these high-level events. In the second layer, we extend state-of-the-art event processing using Bayesian networks in order to take uncertainty into account while detecting complex events. We implement our proposed solution using open source components optimized for large-scale applications. We demonstrate our solution on real-world use-case in the domain of intelligent transportation system where we analyzed traffic, weather, and social media data streams from Madrid city in order to predict probability of congestion in real-time. The performance of the system is evaluated qualitatively using a web-interface where traffic administrators can provide the feedback about the quality of predictions and quantitatively using F-measure with an accuracy of over 80%.