English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Deep neural networks are superior to shallow networks in learning complexrepresentations. As such, there is a fast-growing interest in utilizing them inlarge-scale settings. The training process of neural networks is already knownto be time-consuming, and having a deep architecture only aggravates the issue.This process consists mostly of matrix operations, among which matrixmultiplication is the bottleneck. Several sampling-based techniques have beenproposed for speeding up the training time of deep neural networks byapproximating the matrix products. These techniques fall under two categories:(i) sampling a subset of nodes in every hidden layer as active at everyiteration and (ii) sampling a subset of nodes from the previous layer toapproximate the current layer's activations using the edges from the samplednodes. In both cases, the matrix products are computed using only the selectedsamples. In this paper, we evaluate the scalability of these approaches on CPUmachines with limited computational resources. Making a connection between thetwo research directions as special cases of approximating matrixmultiplications in the context of neural networks, we provide a negativetheoretical analysis that shows feedforward approximation is an obstacleagainst scalability. We conduct comprehensive experimental evaluations thatdemonstrate the most pressing challenges and limitations associated with thestudied approaches. We observe that the hashing-based node selection method isnot scalable to a large number of layers, confirming our theoretical analysis.Finally, we identify directions for future research.

Sampling-Based Techniques for Training Deep Neural Networks with Limited Computational Resources: A Scalability Evaluation