English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We propose expanding the shared Transformer module to produce and initializeTransformers of varying depths, enabling adaptation to diverse resourceconstraints. Drawing an analogy to genetic expansibility, we term such moduleas learngene. To identify the expansion mechanism, we delve into therelationship between the layer's position and its corresponding weight value,and find that linear function appropriately approximates this relationship.Building on this insight, we present Transformer as Linear Expansion oflearnGene (TLEG), a novel approach for flexibly producing and initializingTransformers of diverse depths. Specifically, to learn learngene, we firstlyconstruct an auxiliary Transformer linearly expanded from learngene, afterwhich we train it through employing soft distillation. Subsequently, we canproduce and initialize Transformers of varying depths via linearly expandingthe well-trained learngene, thereby supporting diverse downstream scenarios.Extensive experiments on ImageNet-1K demonstrate that TLEG achieves comparableor better performance in contrast to many individual models trained fromscratch, while reducing around 2x training cost. When transferring to severaldownstream classification datasets, TLEG surpasses existing initializationmethods by a large margin (e.g., +6.87% on iNat 2019 and +7.66% on CIFAR-100).Under the situation where we need to produce models of varying depths adaptingfor different resource constraints, TLEG achieves comparable results whilereducing around 19x parameters stored to initialize these models and around 5xpre-training costs, in contrast to the pre-training and fine-tuning approach.When transferring a fixed set of parameters to initialize different models,TLEG presents better flexibility and competitive performance while reducingaround 2.9x parameters stored to initialize, compared to the pre-trainingapproach.

Transformer as Linear Expansion of Learngene