English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The softmax function combined with a cross-entropy loss is a principledapproach to modeling probability distributions that has become ubiquitous indeep learning. The softmax function is defined by a lone hyperparameter, thetemperature, that is commonly set to one or regarded as a way to tune modelconfidence after training; however, less is known about how the temperatureimpacts training dynamics or generalization performance. In this work wedevelop a theory of early learning for models trained withsoftmax-cross-entropy loss and show that the learning dynamics depend cruciallyon the inverse-temperature $\beta$ as well as the magnitude of the logits atinitialization, $||\beta{\bf z}||_{2}$. We follow up these analytic resultswith a large-scale empirical study of a variety of model architectures trainedon CIFAR10, ImageNet, and IMDB sentiment analysis. We find that generalizationperformance depends strongly on the temperature, but only weakly on the initiallogit magnitude. We provide evidence that the dependence of generalization on$\beta$ is not due to changes in model confidence, but is a dynamicalphenomenon. It follows that the addition of $\beta$ as a tunable hyperparameteris key to maximizing model performance. Although we find the optimal $\beta$ tobe sensitive to the architecture, our results suggest that tuning $\beta$ overthe range $10^{-2}$ to $10^1$ improves performance over all architecturesstudied. We find that smaller $\beta$ may lead to better peak performance atthe cost of learning stability.

Temperature check: theory and practice for training models with softmax-cross-entropy losses