English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent works attribute the capability of in-context learning (ICL) in largepre-trained language models to implicitly simulating and fine-tuning aninternal model (e.g., linear or 2-layer MLP) during inference. However, suchconstructions require large memory overhead, which makes simulation of moresophisticated internal models intractable. In this work, we propose anefficient construction, Transformer in Transformer (in short, TinT), thatallows a transformer to simulate and fine-tune complex models internally duringinference (e.g., pre-trained language models). In particular, we introduceinnovative approximation techniques that allow a TinT model with less than 2billion parameters to simulate and fine-tune a 125 million parametertransformer model within a single forward pass. TinT accommodates many commontransformer variants and its design ideas also improve the efficiency of pastinstantiations of simple models inside transformers. We conduct end-to-endexperiments to validate the internal fine-tuning procedure of TinT on variouslanguage modeling and downstream tasks. For example, even with a limitedone-step budget, we observe TinT for a OPT-125M model improves performance by4-16% absolute on average compared to OPT-125M. These findings suggest thatlarge pre-trained language models are capable of performing intricatesubroutines. To facilitate further work, a modular and extensible codebase forTinT is included.

Trainable Transformer in Transformer