English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Iterative differential approximation methods that rely upon backpropagationhave enabled the optimization of neural networks; however, at present, theyremain computationally expensive, especially when training models at scale. Inthis paper, we propose a computationally efficient alternative for optimizingneural networks that can both reduce the costs of scaling neural networks andprovide high-efficiency optimizations for low-resource applications. We derivean explicit solution to a simple feed-forward language model (LM) bymathematically analyzing its gradients. This solution generalizes fromsingle-layer LMs to the class of all single-layer feed-forwardsoftmax-activated neural models trained on positive-valued features, as isdemonstrated by our extension of this solution application to MNIST digitclassification. For both LM and digit classifiers, we find computationally thatexplicit solutions perform near-optimality in experiments showing that 1)iterative optimization only marginally improves the explicit solutionparameters and 2) randomly initialized parameters iteratively optimize towardsthe explicit solution. We also preliminarily apply the explicit solutionlocally by layer in multi-layer networks and discuss how the solution'scomputational savings increase with model complexity -- for both single- andmult-layer applications of the explicit solution, we emphasize that the optimaachieved cannot be reached by backpropagation alone, i.e., better optima appeardiscoverable only after explicit solutions are applied. Finally, we discuss thesolution's computational savings alongside its impact on model interpretabilityand suggest future directions for the derivation of explicit solutions tocomplex- and multi-layer architectures.

Reducing the Need for Backpropagation and Discovering Better Optima With Explicit Optimizations of Neural Networks