Online Laplace Model Selection Revisited

The Laplace approximation provides a closed-form model selection objectivefor neural networks (NN). Online variants, which optimise NN parameters jointlywith hyperparameters, like weight decay strength, have seen renewed interest inthe Bayesian deep learning community. However, these methods violate Laplace'smethod's critical assumption that the approximation is performed around a modeof the loss, calling into question their soundness. This work re-derives onlineLaplace methods, showing them to target a variational bound on a mode-correctedvariant of the Laplace evidence which does not make stationarity assumptions.Online Laplace and its mode-corrected counterpart share stationary points where1. the NN parameters are a maximum a posteriori, satisfying the Laplacemethod's assumption, and 2. the hyperparameters maximise the Laplace evidence,motivating online methods. We demonstrate that these optima are roughlyattained in practise by online algorithms using full-batch gradient descent onUCI regression datasets. The optimised hyperparameters prevent overfitting andoutperform validation-based early stopping.