English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Low-precision training has emerged as a promising low-cost technique toenhance the training efficiency of deep neural networks without sacrificingmuch accuracy. Its Bayesian counterpart can further provide uncertaintyquantification and improved generalization accuracy. This paper investigateslow-precision sampling via Stochastic Gradient Hamiltonian Monte Carlo (SGHMC)with low-precision and full-precision gradient accumulators for both stronglylog-concave and non-log-concave distributions. Theoretically, our results showthat, to achieve $\epsilon$-error in the 2-Wasserstein distance fornon-log-concave distributions, low-precision SGHMC achieves quadraticimprovement($\widetilde{\mathbf{O}}\left({\epsilon^{-2}{\mu^*}^{-2}\log^2\left({\epsilon^{-1}}\right)}\right)$)compared to the state-of-the-art low-precision sampler, Stochastic GradientLangevin Dynamics (SGLD)($\widetilde{\mathbf{O}}\left({{\epsilon}^{-4}{\lambda^{*}}^{-1}\log^5\left({\epsilon^{-1}}\right)}\right)$).Moreover, we prove that low-precision SGHMC is more robust to the quantizationerror compared to low-precision SGLD due to the robustness of themomentum-based update w.r.t. gradient noise. Empirically, we conductexperiments on synthetic data, and {MNIST, CIFAR-10 \& CIFAR-100} datasets,which validate our theoretical findings. Our study highlights the potential oflow-precision SGHMC as an efficient and accurate sampling method forlarge-scale and resource-limited machine learning.

Enhancing Low-Precision Sampling via Stochastic Gradient Hamiltonian Monte Carlo