Vanets Meet Autonomous Vehicles: Multimodal Surrounding Recognition Using Manifold Alignment

In the past two years, calls for developing synergistic links between the two worlds of vehicular ad-hoc networks (VANETs) and autonomous vehicles have significantly gone up to achieve further on-road safety and benefits for end-users. In this paper, we present our vision to create such a beneficial link by designing a multimodal scheme for object detection, recognition, and mapping based on the fusion of stereo camera frames, point cloud Velodyne LIDAR scans, and vehicle-to-vehicle (V2V) basic safety messages (BSMs) exchanges using VANET protocols. Exploiting the high similarities in the underlying manifold properties of the three data sets, and their high neighborhood correlation, the proposed scheme employs semi-supervised manifold alignment to merge the key features of rich texture descriptions of objects from 2-D images, depth and distance between objects provided by 3-D point cloud, and the awareness of self-declared vehicles from BSMs' 3-D information including the ones not seen by camera and LIDAR. The proposed scheme is applied to create joint pixel-to-point-cloud and pixel-to-V2V correspondences of objects in frames from the KITTI Vision Benchmark Suite, using a semi-supervised manifold alignment, to achieve camera-LIDAR and camera-V2V mapping of their recognized objects. We present the alignment accuracy results over two different driving sequences and show the additional acquired knowledge of objects from the various input modalities. We also study the effect of the number of neighbors employed in the alignment process on the alignment accuracy. With proper choice of parameters, the testing of our proposed scheme over two entire driving sequences exhibits 100% accuracy in the majority of cases, 74%-92% and 50%-72% average alignment accuracy for vehicles and pedestrians and up to 150% additional object recognition of the testing vehicle's surrounding.