Decibel Imaging for Generating Medical Report using Ultrasonic

Medical imaging is commonly used for diagnosis and care in clinical practice. Report-writing would be prone to mistakes for inexperienced physicians, and experienced physicians would be time consuming and boring. To handle these issues, we study the automated generation of medical imaging reports. This task presents several challenges. First, a complete report contains multiple heterogeneous types of information including findings and tags. Second, abnormal regions in medical images are difficult to spot. Third, usually, the reports are lengthy and contain multiple sentences. To deal with these challenges, we (1) build a multi-task learning framework which jointly performs the prediction of tags and therefore the generation of paragraphs, (2) propose a co-attention mechanism to localize regions containing abnormalities and generate narrations for them, (3) develop a hierarchical LSTM model to get long paragraphs. We show the efficacy of the proposed methods on two datasets which are publicly accessible.