WritePad: Consecutive Number Writing on Your Hand With Smart Acoustic Sensing

Although the smartwatches have rapid development in recent years, inconvenient interaction mode limits the prevalence of smartwatches, which raises an urgent need for new interaction medium beyond the small built-in screen. For natural interaction with smart devices, full development of terminal sensing, and no dependence on hardware, we purpose to write numbers on the hand back to extend the input interface of the smartwatch. We have to address both the weakness of writing signal on the hand back and adaptability among various people. In this paper, we raise a passive acoustic sensing, where the smartwatches sample the ambient sound during writing. First, we employ the wavelet transformation to drop the interference of surrounding noise and devise the time-frequency features into images for machine learning-enabled processing. Then, we design a hybrid convolutional neural network model for number recognition, where three layers of convolution are followed by three layers of max pool. We choose the three most representative places to set up the experiment, namely, laboratory, dormitory, and canteen. The accuracy of number recognition could be over 95% when we adopt single data and be around 91% when 10 persons are incorporated. On promotion of writing speed under the limited area of hand back, we propose to write 2-3 consecutive numbers, where we could achieve the segmentation accuracy to be above 85%.