English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Transformer model empowered architectures have become a pillar of cloudservices that keeps reshaping our society. However, the dynamic query loads andheterogeneous user requirements severely challenge current transformer servingsystems, which rely on pre-training multiple variants of a foundation model,i.e., with different sizes, to accommodate varying service demands.Unfortunately, such a mechanism is unsuitable for large transformer models dueto the additional training costs and excessive I/O delay. In this paper, weintroduce OTAS, the first elastic serving system specially tailored fortransformer models by exploring lightweight token management. We develop anovel idea called token adaptation that adds prompting tokens to improveaccuracy and removes redundant tokens to accelerate inference. To cope withfluctuating query loads and diverse user requests, we enhance OTAS withapplication-aware selective batching and online token adaptation. OTAS firstbatches incoming queries with similar service-level objectives to improve theingress throughput. Then, to strike a tradeoff between the overhead of tokenincrement and the potentials for accuracy improvement, OTAS adaptively adjuststhe token execution strategy by solving an optimization problem. We implementand evaluate a prototype of OTAS with multiple datasets, which show that OTASimproves the system utility by at least 18.2%.

OTAS: An Elastic Transformer Serving System via Token Adaptation