Object detection with different categories using YOLOv8 for optical remote sensing images

Universal models that facilitate the detection of objects of diverse categories in images obtained through optical remote sensing (ORS) hold the greatest practical value. The objective of this research was to develop models that enable accurate detection of objects belonging to five different categories, namely: aircraft, airport, helicopter, oil tank, and warship. These selected categories of objects are frequently analysed in both civil and military applications. This article focuses on developing solutions for object detection in ORS images using the YOLOv8 architecture. To avoid the problem of missing and false detections, algorithms such as Soft Non-Maximum Suppression (Soft-NMS) and the Confidence Propagation Cluster (CPC) were introduced into the YOLOv8 model. A data transfer technique was employed to enable the models to learn more quickly and generalise the acquired knowledge more effectively. Another approach involved freezing selected layers. This reduces overfitting when training with domain-specific datasets and decreases memory usage. The research also compares the results obtained using modern neural network architectures such as YOLOv5, RTMDet and DETR. Proposing new architectures and solutions alone is not enough to improve object detection on an imbalanced training set. Therefore, this research also focused on preparing an appropriate dataset to enable the effective training of applied models. In particular, the training dataset should include a wide variety of challenging scenes, such as densely packed objects and complex backgrounds, as well as objects of different categories and sizes within a single image. Consequently, a new dataset called DOTANA was created based on two existing, publicly available datasets: the Large-Scale Dataset for Detecting Objects in Aerial Images (DOTA) and the Remote Sensing Dataset for Geospatial Object Detection (RSD-GOD). The proposed solutions produce better results than previous studies for objects of the same categories from the RSD-GOD dataset. The best average precision metric (mAP) values obtained are mAP $_{50}$ =0.939 and mAP $_{50-95}$ =0.606 using the YOLOv8 model and the Soft-NMS algorithm.