English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Nonlinearity is crucial to the performance of a deep (neural) network (DN).To date there has been little progress understanding the menagerie of availablenonlinearities, but recently progress has been made on understanding the r\^oleplayed by piecewise affine and convex nonlinearities like the ReLU and absolutevalue activation functions and max-pooling. In particular, DN layersconstructed from these operations can be interpreted as {\em max-affine splineoperators} (MASOs) that have an elegant link to vector quantization (VQ) and$K$-means. While this is good theoretical progress, the entire MASO approach ispredicated on the requirement that the nonlinearities be piecewise affine andconvex, which precludes important activation functions like the sigmoid,hyperbolic tangent, and softmax. {\em This paper extends the MASO framework tothese and an infinitely large class of new nonlinearities by linkingdeterministic MASOs with probabilistic Gaussian Mixture Models (GMMs).} We showthat, under a GMM, piecewise affine, convex nonlinearities like ReLU, absolutevalue, and max-pooling can be interpreted as solutions to certain natural"hard" VQ inference problems, while sigmoid, hyperbolic tangent, and softmaxcan be interpreted as solutions to corresponding "soft" VQ inference problems.We further extend the framework by hybridizing the hard and soft VQoptimizations to create a $\beta$-VQ inference that interpolates between hard,soft, and linear VQ inference. A prime example of a $\beta$-VQ DN nonlinearityis the {\em swish} nonlinearity, which offers state-of-the-art performance in arange of computer vision tasks but was developed ad hoc by experimentation.Finally, we validate with experiments an important assertion of our theory,namely that DN performance can be significantly improved by enforcingorthogonality in its linear filters.

From Hard to Soft: Understanding Deep Network Nonlinearities via Vector Quantization and Statistical Inference