English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Many powerful machine learning models are based on the composition ofmultiple processing layers, such as deep nets, which gives rise to nonconvexobjective functions. A general, recent approach to optimise such "nested"functions is the method of auxiliary coordinates (MAC). MAC introduces anauxiliary coordinate for each data point in order to decouple the nested modelinto independent submodels. This decomposes the optimisation into steps thatalternate between training single layers and updating the coordinates. It hasthe advantage that it reuses existing single-layer algorithms, introducesparallelism, and does not need to use chain-rule gradients, so it works withnondifferentiable layers. With large-scale problems, or when distributing thecomputation is necessary for faster training, the dataset may not fit in asingle machine. It is then essential to limit the amount of communicationbetween machines so it does not obliterate the benefit of parallelism. Wedescribe a general way to achieve this, ParMAC. ParMAC works on a cluster ofprocessing machines with a circular topology and alternates two steps untilconvergence: one step trains the submodels in parallel using stochasticupdates, and the other trains the coordinates in parallel. Only submodelparameters, no data or coordinates, are ever communicated between machines.ParMAC exhibits high parallelism, low communication overhead, and facilitatesdata shuffling, load balancing, fault tolerance and streaming data processing.We study the convergence of ParMAC and propose a theoretical model of itsruntime and parallel speedup. We develop ParMAC to learn binary autoencodersfor fast, approximate image retrieval. We implement it in MPI in a distributedsystem and demonstrate nearly perfect speedups in a 128-processor cluster witha training set of 100 million high-dimensional points.

ParMAC: distributed optimisation of nested functions, with application to learning binary autoencoders