Recognizing Grabbing Actions from Inertial and Video Sensor Data in a Warehouse Scenario

Modern industries are increasingly adapting to smart devices for aiding and improving their productivity and work flow. This includes logistics in warehouses where validation of correct items per order can be enhanced with mobile devices. Since handling incorrect orders is a big part of the costs of warehouse maintenance, reducing errors like missed or wrong items should be avoided. Thus, early identification of picking procedures and items picked is beneficial for reducing these errors. By using data glasses and a smartwatch we aim to reduce these errors while also enabling the picker to work hands-free. In this paper, we present an analysis of feature sets for classification of grabbing actions in the order picking process. For this purpose, we created a dataset containing inertial data and egocentric video from four participants performing picking tasks. As we previously worked with logistics companies, we modeled our test scenario close to real-world warehouse environments. Afterwards, we extracted features from the time and frequency domain for inertial data and color and descriptor features from the image data to learn grabbing actions. We were able to show that the combination of inertial and video data enables us to recognize grabbing actions in a picking scenario. We also show that the combination of different sensors improves the results, yielding an F-measure of 85.3% for recognizing grabbing actions