English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Many tasks in deep learning involve optimizing over the \emph{inputs} to anetwork to minimize or maximize some objective; examples include optimizationover latent spaces in a generative model to match a target image, oradversarially perturbing an input to worsen classifier performance. Performingsuch optimization, however, is traditionally quite costly, as it involves acomplete forward and backward pass through the network for each gradient step.In a separate line of work, a recent thread of research has developed the deepequilibrium (DEQ) model, a class of models that foregoes traditional networkdepth and instead computes the output of a network by finding the fixed pointof a single nonlinear layer. In this paper, we show that there is a naturalsynergy between these two settings. Although, naively using DEQs for theseoptimization problems is expensive (owing to the time needed to compute a fixedpoint for each gradient step), we can leverage the fact that gradient-basedoptimization can \emph{itself} be cast as a fixed point iteration tosubstantially improve the overall speed. That is, we \emph{simultaneously} bothsolve for the DEQ fixed point \emph{and} optimize over network inputs, allwithin a single ``augmented'' DEQ model that jointly encodes both the originalnetwork and the optimization process. Indeed, the procedure is fast enough thatit allows us to efficiently \emph{train} DEQ models for tasks traditionallyrelying on an ``inner'' optimization loop. We demonstrate this strategy onvarious tasks such as training generative models while optimizing over latentcodes, training models for inverse problems like denoising and inpainting,adversarial training and gradient based meta-learning.

Joint inference and input optimization in equilibrium networks