DEEP eye contact detector: Robust eye contact bid detection using convolutional neural network

Eye contact (mutual gaze) is a foundation of human communication and social interactions; therefore, it is studied in many fields such as psychology, social science, and medicine. To support these application of eye-contact detection, much effort has been made for automated eye-contact detection by using image recognition techniques; however, they are difficult to use when facial-landmark tracking is not possible due to facial occlusions. To solve this issue, this paper proposes robust algorithms of detecting eye contacts leveraged using deep neural net to find the eye-contact skills needed for caregivers in dementia care. Since our algorithms do not depend on facial-landmark tracking and only use the images around the eyes, they are robust against facial occlusions and/or image noise. We prepared two eye contact facial image datasets and confirmed the performance of the proposed algorithms. The results show that our method are robust against facial occlusion in which a person is wearing a facemask or the person’s face is partially outside the camera viewing area. This study shows the potential to be able to obtain the eye-contact status only from the images around eyes.