Review and Performance Evaluation of Uncertainty Quantification in Data-Driven AI-Assisted Measurements

As Artificial Intelligence (AI) becomes more prevalent in measurement systems and synthetic instruments, quantifying the uncertainty of AI-assisted measurements becomes a crucial and necessary part of the measurement process. In this paper, we take a holistic approach towards both measurement science and AI’s formulation and implementation of uncertainty, and we review and categorize data-driven AI-assisted uncertainty quantification methods with a novel taxonomy. We also provide a one-stop shop identifying AI literature practices that are noncompliant with measurement standards, allowing readers to spot such noncompliances and understand their practical impact. Furthermore, specifically for classification-assisted measurements, we test the most common epistemic Type A uncertainty quantification methods with 12 diverse datasets, and we evaluate their indirect measurement accuracy, one of the most important metrics for engineering applications, as well as precision, recall, and F1 score, each in both macro and weighted modes. Finally, we study the multi-observation variation of misclassification probability and experimentally show that in some cases it can be an indication of uncertainty: an interesting fact considering misclassification probability itself is not uncertainty.