An Automated Academic Book Scanner with Deep Learning Powered Math Expression Detection and Recognition

Mathematical expressions are hard to read, which makes it hard to digitize printed STEM material. Conventional OCR methods do not work well on contents which comprise of text along with mathematical expressions. This paper proposes an automated book scanning system comprising of Raspberry Pi, page gripper and flipper set up and a smartphone camera for capturing the content. Once the content is captured, a combination of text recognition, mathematical expression detection and recognition modules work in synergy to provide the final digitized content. The mathematical expression detection module consists of a customized RoIHeads version of Faster R-CNN. The proposed model performed with 95.71% precision, 91.77% recall, 93.74% F1-score, and 87.42% mean IOU on the publicly available IBEM dataset. Once the expressions are detected, they are recognised using the Vision Encoder-Decoder Model architecture and the subsequent latex version of the mathematical expression is generated. This model is trained on the Im2Latex dataset and provided a BLEU score of 86.44 % on the test set. We also demonstrated the utility of the proposed method in academic STEM book scanning with accurate recovery of mathematical expressions. Our system gives a cost effective solution which makes it possible to extract text and mathematical content from printed books in a method that is scalable and accurate. This opens the door to large-scale digital preservation and machine-readable academic archiving.