English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This paper focuses on predicting the occurrence of grokking in neuralnetworks, a phenomenon in which perfect generalization emerges long after signsof overfitting or memorization are observed. It has been reported that grokkingcan only be observed with certain hyper-parameters. This makes it critical toidentify the parameters that lead to grokking. However, since grokking occursafter a large number of epochs, searching for the hyper-parameters that lead toit is time-consuming. In this paper, we propose a low-cost method to predictgrokking without training for a large number of epochs. In essence, by studyingthe learning curve of the first few epochs, we show that one can predictwhether grokking will occur later on. Specifically, if certain oscillationsoccur in the early epochs, one can expect grokking to occur if the model istrained for a much longer period of time. We propose using the spectralsignature of a learning curve derived by applying the Fourier transform toquantify the amplitude of low-frequency components to detect the presence ofsuch oscillations. We also present additional experiments aimed at explainingthe cause of these oscillations and characterizing the loss landscape.

Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok