English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Despite their success deep neural networks (DNNs) are still largelyconsidered as black boxes. The main issue is that the linear and non-linearoperations are entangled in every layer, making it hard to interpret the hiddenlayer outputs. In this paper, we look at DNNs with rectified linear units(ReLUs), and focus on the gating property (`on/off' states) of the ReLUs. Weextend the recently developed dual view in which the computation is brokenpath-wise to show that learning in the gates is more crucial, and learning theweights given the gates is characterised analytically via the so called neuralpath kernel (NPK) which depends on inputs and gates. In this paper, we presentnovel results to show that convolution with global pooling and skip connectionprovide respectively rotational invariance and ensemble structure to the NPK.To address `black box'-ness, we propose a novel interpretable counterpart ofDNNs with ReLUs namely deep linearly gated networks (DLGN): the pre-activationsto the gates are generated by a deep linear network, and the gates are thenapplied as external masks to learn the weights in a different network. The DLGNis not an alternative architecture per se, but a disentanglement and aninterpretable re-arrangement of the computations in a DNN with ReLUs. The DLGNdisentangles the computations into two `mathematically' interpretablelinearities (i) the `primal' linearity between the input and thepre-activations in the gating network and (ii) the `dual' linearity in the pathspace in the weights network characterised by the NPK. We compare theperformance of DNN, DGN and DLGN on CIFAR-10 and CIFAR-100 to show that, theDLGN recovers more than $83.5\%$ of the performance of state-of-the-art DNNs.This brings us to an interesting question: `Is DLGN a universal spectralapproximator?'

Disentangling deep neural networks with rectified linear units using duality