English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

A powerful concept behind much of the recent progress in machine learning isthe extraction of common features across data from heterogeneous sources ortasks. Intuitively, using all of one's data to learn a common representationfunction benefits both computational effort and statistical generalization byleaving a smaller number of parameters to fine-tune on a given task. Towardtheoretically grounding these merits, we propose a general setting ofrecovering linear operators $M$ from noisy vector measurements $y = Mx + w$,where the covariates $x$ may be both non-i.i.d. and non-isotropic. Wedemonstrate that existing isotropy-agnostic representation learning approachesincur biases on the representation update, which causes the scaling of thenoise terms to lose favorable dependence on the number of source tasks. This inturn can cause the sample complexity of representation learning to bebottlenecked by the single-task data size. We introduce an adaptation,$\texttt{De-bias & Feature-Whiten}$ ($\texttt{DFW}$), of the popularalternating minimization-descent scheme proposed independently in Collins etal., (2021) and Nayer and Vaswani (2022), and establish linear convergence tothe optimal representation with noise level scaling down with the$\textit{total}$ source data size. This leads to generalization bounds on thesame order as an oracle empirical risk minimizer. We verify the vitalimportance of $\texttt{DFW}$ on various numerical simulations. In particular,we show that vanilla alternating-minimization descent fails catastrophicallyeven for iid, but mildly non-isotropic data. Our analysis unifies andgeneralizes prior work, and provides a flexible framework for a wider range ofapplications, such as in controls and dynamical systems.

Sample-Efficient Linear Representation Learning from Non-IID Non-Isotropic Data