English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

While post-training model compression can greatly reduce the inference costof a deep neural network, uncompressed training still consumes a huge amount ofhardware resources, run-time and energy. It is highly desirable to directlytrain a compact neural network from scratch with low memory and lowcomputational cost. Low-rank tensor decomposition is one of the most effectiveapproaches to reduce the memory and computing requirements of large-size neuralnetworks. However, directly training a low-rank tensorized neural network is avery challenging task because it is hard to determine a proper tensor rank {\ita priori}, which controls the model complexity and compression ratio in thetraining process. This paper presents a novel end-to-end framework for low-ranktensorized training of neural networks. We first develop a flexible Bayesianmodel that can handle various low-rank tensor formats (e.g., CP, Tucker, tensortrain and tensor-train matrix) that compress neural network parameters intraining. This model can automatically determine the tensor ranks inside anonlinear forward model, which is beyond the capability of existing Bayesiantensor methods. We further develop a scalable stochastic variational inferencesolver to estimate the posterior density of large-scale problems in training.Our work provides the first general-purpose rank-adaptive framework forend-to-end tensorized training. Our numerical results on various neural networkarchitectures show orders-of-magnitude parameter reduction and little accuracyloss (or even better accuracy) in the training process. Specifically, on a verylarge deep learning recommendation system with over $4.2\times 10^9$ modelparameters, our method can reduce the variables to only $1.6\times 10^5$automatically in the training process (i.e., by $2.6\times 10^4$ times) whileachieving almost the same accuracy.

Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination