Detecting the presence of meteors in images: new collection and results

In this paper, we present a new public and real dataset of labeled images of meteors and non-meteors that we recently used in a machine learning competition. We also present a comprehensive performance evaluation of several established machine learning methods and compare the results with a stacking approach – one of the winning solutions of the competition. We compared the performance obtained by the methods in the traditional repeated five-fold cross-validation with the ones obtained using the training and test partitions used in the competition. A careful analysis of the results indicates that, in general, the stacking based approach obtained the best performances compared to the baselines. Moreover, we found evidence that the validation strategy used by the platform that hosted the competition can lead to results that do not sustain in a cross-validation setup, which is recommendable in real-world scenarios.