English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

The adaptive momentum method (AdaMM), which uses past gradients to updatedescent directions and learning rates simultaneously, has become one of themost popular first-order optimization methods for solving machine learningproblems. However, AdaMM is not suited for solving black-box optimizationproblems, where explicit gradient forms are difficult or infeasible to obtain.In this paper, we propose a zeroth-order AdaMM (ZO-AdaMM) algorithm, thatgeneralizes AdaMM to the gradient-free regime. We show that the convergencerate of ZO-AdaMM for both convex and nonconvex optimization is roughly a factorof $O(\sqrt{d})$ worse than that of the first-order AdaMM algorithm, where $d$is problem size. In particular, we provide a deep understanding on whyMahalanobis distance matters in convergence of ZO-AdaMM and other AdaMM-typemethods. As a byproduct, our analysis makes the first step toward understandingadaptive learning rate methods for nonconvex constrained optimization.Furthermore, we demonstrate two applications, designing per-image and universaladversarial attacks from black-box neural networks, respectively. We performextensive experiments on ImageNet and empirically show that ZO-AdaMM convergesmuch faster to a solution of high accuracy compared with $6$ state-of-the-artZO optimization methods.

ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimization