English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

In recent years, implicit deep learning has emerged as a method to increasethe effective depth of deep neural networks. While their training ismemory-efficient, they are still significantly slower to train than theirexplicit counterparts. In Deep Equilibrium Models (DEQs), the training isperformed as a bi-level problem, and its computational complexity is partiallydriven by the iterative inversion of a huge Jacobian matrix. In this paper, wepropose a novel strategy to tackle this computational bottleneck from whichmany bi-level problems suffer. The main idea is to use the quasi-Newtonmatrices from the forward pass to efficiently approximate the inverse Jacobianmatrix in the direction needed for the gradient computation. We provide atheorem that motivates using our method with the original forward algorithms.In addition, by modifying these forward algorithms, we further providetheoretical guarantees that our method asymptotically estimates the trueimplicit gradient. We empirically study this approach and the recentJacobian-Free method in different settings, ranging from hyperparameteroptimization to large Multiscale DEQs (MDEQs) applied to CIFAR and ImageNet.Both methods reduce significantly the computational cost of the backward pass.While SHINE has a clear advantage on hyperparameter optimization problems, bothmethods attain similar computational performances for larger scale problemssuch as MDEQs at the cost of a limited performance drop compared to theoriginal models.

SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit models