Reinforcement Learning-Based Centerline Tracking with Cross-Sectional Similarity for Enhancing Mandibular Canal Segmentation

The mandibular canal (MC) is an anatomical structure surrounding inferior alveolar nerves (IAN) that deliver sensory information to the teeth, gums, and lips. The position of the MC is a critical factor for planning dental procedures and orthodontic surgery. Tracking the centerline of the MC can assist not only in surgical planning but also enhance segmentation performance. This study aims to accurately track the centerline of the MC from the mental foramen to the end of the MC in three-dimensional computed tomography images. To this end, we propose a deep reinforcement learning (DRL) framework utilizing cross-sections generated along the progression direction of the DRL agent. In our approach, we design a novel reward function with penalty factors for guiding the agent to smoothly track the curvature and centerline of the MC, which helps ensure both accuracy and robustness against the local minima. The proposed method is compared with other existing approaches for centerline extraction, using both tracking and segmentation metrics. Experimental results based on 480 public cone-beam computed tomography (CBCT) scans demonstrate that our method is highly robust and accurate relative to existing approaches across all evaluated metrics.