English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Many state-of-the-art results obtained with deep networks are achieved withthe largest models that could be trained, and if more computation power wasavailable, we might be able to exploit much larger datasets in order to improvegeneralization ability. Whereas in learning algorithms such as decision treesthe ratio of capacity (e.g., the number of parameters) to computation is veryfavorable (up to exponentially more parameters than computation), the ratio isessentially 1 for deep neural networks. Conditional computation has beenproposed as a way to increase the capacity of a deep neural network withoutincreasing the amount of computation required, by activating some parametersand computation "on-demand", on a per-example basis. In this note, we propose anovel parametrization of weight matrices in neural networks which has thepotential to increase up to exponentially the ratio of the number of parametersto computation. The proposed approach is based on turning on some parameters(weight matrices) when specific bit patterns of hidden unit activations areobtained. In order to better control for the overfitting that might result, wepropose a parametrization that is tree-structured, where each node of the treecorresponds to a prefix of a sequence of sign bits, or gating units, associatedwith hidden units.

Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning