English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Transformer-based pre-trained language models (PLMs) mostly suffer fromexcessive overhead despite their advanced capacity. For resource-constraineddevices, there is an urgent need for a spatially and temporally efficient modelwhich retains the major capacity of PLMs. However, existing staticallycompressed models are unaware of the diverse complexities between inputinstances, potentially resulting in redundancy and inadequacy for simple andcomplex inputs. Also, miniature models with early exiting encounter challengesin the trade-off between making predictions and serving the deeper layers.Motivated by such considerations, we propose a collaborative optimization forPLMs that integrates static model compression and dynamic inferenceacceleration. Specifically, the PLM is slenderized in width while the depthremains intact, complementing layer-wise early exiting to speed up inferencedynamically. To address the trade-off of early exiting, we propose a jointtraining approach that calibrates slenderization and preserves contributivestructures to each exit instead of only the final layer. Experiments areconducted on GLUE benchmark and the results verify the Pareto optimality of ourapproach at high compression and acceleration rate with 1/8 parameters and 1/19FLOPs of BERT.

COST-EFF: Collaborative Optimization of Spatial and Temporal Efficiency with Slenderized Multi-exit Language Models