English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Vector embedding is a foundational building block of many deep learningmodels, especially in natural language processing. In this paper, we present atheoretical framework for understanding the effect of dimensionality on vectorembeddings. We observe that the distributional hypothesis, a governingprinciple of statistical semantics, requires a natural unitary-invariance forvector embeddings. Motivated by the unitary-invariance observation, we proposethe Pairwise Inner Product (PIP) loss, a unitary-invariant metric on thesimilarity between two embeddings. We demonstrate that the PIP loss capturesthe difference in functionality between embeddings, and that the PIP loss istightly connect with two basic properties of vector embeddings, namelysimilarity and compositionality. By formulating the embedding training processas matrix factorization with noise, we reveal a fundamental bias-variancetrade-off between the signal spectrum and noise power in the dimensionalityselection process. This bias-variance trade-off sheds light on many empiricalobservations which have not been thoroughly explained, for example theexistence of an optimal dimensionality. Moreover, we discover two new resultsabout vector embeddings, namely their robustness against over-parametrizationand their forward stability. The bias-variance trade-off of the PIP lossexplicitly answers the fundamental open problem of dimensionality selection forvector embeddings.

Understand Functionality and Dimensionality of Vector Embeddings: the  Distributional Hypothesis, the Pairwise Inner Product Loss and Its  Bias-Variance Trade-off