Multimodal Fusion of Face and Gait for Person Identification in Automotive Applications

Smart and secure access to vehicles is a crucial aspect of the evolving automotive industry. This article focuses on the development of an end-to-end multimodal biometric recognition framework that identifies people walking toward a vehicle from an RGB video feed. The framework is based on a deep learning pipeline for person detection and tracking, face and gait feature extraction, and fusion of the two modalities at the score and feature level. Traditional face recognition (FR) systems can suffer from variations in lighting and occlusions. In order to deal with these issues, the proposed framework integrates face and gait features with the aim to enhance accuracy. The pipeline is modular, enabling seamless integration of new models for each step of person identification without the need for additional training. Baseline face and gait recognition (GR) models, as well as score- and feature-level fusion (FLF) techniques, are evaluated on subsets of the CASIA-A and CASIA-B datasets. Experimental results show that weighted mean score-level fusion (SLF) significantly improves both Rank-1 accuracy and verification accuracy (TAR@FAR $= \mathbf {10^{-5}}$ ) over unimodal baselines. Overall, the reported work provides insights into current limitations and suggests directions for future research about secure identity verification in vehicles.