English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Strictly enforcing orthonormality constraints on parameter matrices has beenshown advantageous in deep learning. This amounts to Riemannian optimization onthe Stiefel manifold, which, however, is computationally expensive. To addressthis challenge, we present two main contributions: (1) A new efficientretraction map based on an iterative Cayley transform for optimization updates,and (2) An implicit vector transport mechanism based on the combination of aprojection of the momentum and the Cayley transform on the Stiefel manifold. Wespecify two new optimization algorithms: Cayley SGD with momentum, and CayleyADAM on the Stiefel manifold. Convergence of Cayley SGD is theoreticallyanalyzed. Our experiments for CNN training demonstrate that both algorithms:(a) Use less running time per iteration relative to existing approaches thatenforce orthonormality of CNN parameters; and (b) Achieve faster convergencerates than the baseline SGD and ADAM algorithms without compromising theperformance of the CNN. Cayley SGD and Cayley ADAM are also shown to reduce thetraining time for optimizing the unitary transition matrices in RNNs.

Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform