English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This work examines the challenges of training neural networks using vectorquantization using straight-through estimation. We find that a primary cause oftraining instability is the discrepancy between the model embedding and thecode-vector distribution. We identify the factors that contribute to thisissue, including the codebook gradient sparsity and the asymmetric nature ofthe commitment loss, which leads to misaligned code-vector assignments. Wepropose to address this issue via affine re-parameterization of the codevectors. Additionally, we introduce an alternating optimization to reduce thegradient error introduced by the straight-through estimation. Moreover, wepropose an improvement to the commitment loss to ensure better alignmentbetween the codebook representation and the model embedding. These optimizationmethods improve the mathematical approximation of the straight-throughestimation and, ultimately, the model performance. We demonstrate theeffectiveness of our methods on several common model architectures, such asAlexNet, ResNet, and ViT, across various tasks, including image classificationand generative modeling.

Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks