English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This paper challenges the well-established paradigm for building any-to-anynetworks for training Large Language Models (LLMs). We show that LLMs exhibit aunique communication pattern where only small groups of GPUs requirehigh-bandwidth communication to achieve near-optimal training performance.Across these groups of GPUs, the communication is insignificant andhomogeneous. We propose a new network architecture that resembles thecommunication requirement of LLMs. Our architecture partitions the cluster intosets of GPUs interconnected with non-blocking any-to-any high-bandwidthinterconnects that we call HB domains. Across the HB domains, the network onlyconnects GPUs with non-zero communication demands. We develop an analyticalformulation of the training iteration time to evaluate our proposal. Ourformulation closely estimates the hardware floating-point utilization within0.15\% from the ground truth established in prior studies for larger models. Weshow that our proposed architecture reduces the network cost by 37% to 75%compared to the state-of-the-art any-to-any Clos networks without compromisingthe performance of LLM training.

How to Build Low-cost Networks for Large Language Models (without Sacrificing Performance)?