English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We present NetReduce, a novel RDMA-compatible in-network reductionarchitecture to accelerate distributed DNN training. Compared to existingdesigns, NetReduce maintains a reliable connection between end-hosts in theEthernet and does not terminate the connection in the network. The advantage ofdoing so is that we can fully reuse the designs of congestion control andreliability in RoCE. In the meanwhile, we do not need to implement a high-costnetwork protocol processing stack in the switch, as IB does. The prototypeimplemented by using FPGA is an out-of-box solution without modifying commoditydevices such as NICs or switches. For the coordination between the end-host andthe switch, NetReduce customizes the transport protocol only on the firstpacket in a data message to comply with RoCE v2. The special status monitoringmodule is designed to reuse the reliability mechanism of RoCE v2 for dealingwith packet loss. A message-level credit-based flow control algorithm is alsoproposed to fully utilize bandwidth and avoid buffer overflow. We study theeffects of intra bandwidth on the training performance in multi-machinesmulti-GPUs scenario and give sufficient conditions for hierarchical NetReduceto outperform other algorithms. We also extend the design from rack-levelaggregation to more general spine-leaf topology in the data center. NetReduceaccelerates the training up to 1.7x and 1.5x for CNN-based CV andtransformer-based NLP tasks, respectively. Simulations on large-scale systemsindicate the superior scalability of NetReduce to the state-of-the-art ringall-reduce.

NetReduce: RDMA-Compatible In-Network Reduction for Distributed DNN Training Acceleration