English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Within the last few years, there has been a move towards using statisticalmodels in conjunction with neural networks with the end goal of being able tobetter answer the question, "what do our models know?". From this trend,classical metrics such as Prediction Interval Coverage Probability (PICP) andnew metrics such as calibration error have entered the general repertoire ofmodel evaluation in order to gain better insight into how the uncertainty ofour model compares to reality. One important component of uncertainty modelingis model uncertainty (epistemic uncertainty), a measurement of what the modeldoes and does not know. However, current evaluation techniques tends toconflate model uncertainty with aleatoric uncertainty (irreducible error),leading to incorrect conclusions. In this paper, using posterior predictivechecks, we show how calibration error and its variants are almost alwaysincorrect to use given model uncertainty, and further show how this mistake canlead to trust in bad models and mistrust in good models. Though posteriorpredictive checks has often been used for in-sample evaluation of Bayesianmodels, we show it still has an important place in the modern deep learningworld.

Why Calibration Error is Wrong Given Model Uncertainty: Using Posterior Predictive Checks with Deep Learning