English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In modern deep learning, algorithmic choices (such as width, depth, andlearning rate) are known to modulate nuanced resource tradeoffs. This workinvestigates how these complexities necessarily arise for feature learning inthe presence of computational-statistical gaps. We begin by considering offlinesparse parity learning, a supervised classification problem which admits astatistical query lower bound for gradient-based training of a multilayerperceptron. This lower bound can be interpreted as a multi-resource tradeofffrontier: successful learning can only occur if one is sufficiently rich (largemodel), knowledgeable (large dataset), patient (many training iterations), orlucky (many random guesses). We show, theoretically and experimentally, thatsparse initialization and increasing network width yield significantimprovements in sample efficiency in this setting. Here, width plays the roleof parallel search: it amplifies the probability of finding "lottery ticket"neurons, which learn sparse features more sample-efficiently. Finally, we showthat the synthetic sparse parity task can be useful as a proxy for realproblems requiring axis-aligned feature learning. We demonstrate improvedsample efficiency on tabular classification benchmarks by using wide,sparsely-initialized MLP models; these networks sometimes outperform tunedrandom forests.

Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luck