English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Leave-one-out cross-validation (LOO-CV) is a popular method for comparingBayesian models based on their estimated predictive performance on new, unseen,data. As leave-one-out cross-validation is based on finite observed data, thereis uncertainty about the expected predictive performance on new data. Bymodeling this uncertainty when comparing two models, we can compute theprobability that one model has a better predictive performance than the other.Modeling this uncertainty well is not trivial, and for example, it is knownthat the commonly used standard error estimate is often too small. We study theproperties of the Bayesian LOO-CV estimator and the related uncertaintyestimates when comparing two models. We provide new results of the propertiesboth theoretically in the linear regression case and empirically for multipledifferent models and discuss the challenges of modeling the uncertainty. Weshow that problematic cases include: comparing models with similar predictions,misspecified models, and small data. In these cases, there is a weak connectionin the skewness of the individual leave-one-out terms and the distribution ofthe error of the Bayesian LOO-CV estimator. We show that it is possible thatthe problematic skewness of the error distribution, which occurs when themodels make similar predictions, does not fade away when the data size grows toinfinity in certain situations. Based on the results, we also provide practicalrecommendations for the users of Bayesian LOO-CV for model comparison.

Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison