English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Scaling neural network models has delivered dramatic quality gains across MLproblems. However, this scaling has increased the reliance on efficientdistributed training techniques. Accordingly, as with other distributedcomputing scenarios, it is important to understand how will compute andcommunication scale relative to one another as models scale and hardwareevolves? A careful study which answers this question can better guide thedesign of future systems which can efficiently train future large models.  Accordingly, this work provides a comprehensive multi-axial (algorithmic,empirical, hardware evolution) analysis of compute vs. communication(Comp-vs.-Comm) scaling for future Transformer models on future hardware.First, our algorithmic analysis shows that compute generally enjoys an edgeover communication as models scale. However, since memory capacity scalesslower than compute, these trends are being stressed. Next, we quantify thisedge by empirically studying how Comp-vs.-Comm scales for future models onfuture hardware. To avoid profiling numerous Transformer models across manysetups, we extract execution regions and project costs using operator models.This allows a spectrum (hundreds) of future model/hardware scenarios to beaccurately studied ($<$15% error), and reduces profiling costs by 2100$\times$.Our experiments show that communication will be a significant portion (40-75%)of runtime as models and hardware evolve. Moreover, communication which ishidden by overlapped computation in today's models often cannot be hidden infuture, larger models. Overall, this work highlights the increasingly largerole communication will play as models scale and discusses techniques andupcoming technologies that can help address it.

Computation vs. Communication Scaling for Future Transformers on Future Hardware