English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Current AI training infrastructure is dominated by single instructionmultiple data (SIMD) and systolic array architectures, such as GraphicsProcessing Units (GPUs) and Tensor Processing Units (TPUs), that excel ataccelerating parallel workloads and dense vector matrix multiplications.Potentially more efficient neural network models utilizing sparsity andrecurrence cannot leverage the full power of SIMD processor and are thus at asevere disadvantage compared to today's prominent parallel architectures likeTransformers and CNNs, thereby hindering the path towards more sustainable AI.To overcome this limitation, we explore sparse and recurrent model training ona massively parallel multiple instruction multiple data (MIMD) architecturewith distributed local memory. We implement a training routine based onbackpropagation through time (BPTT) for the brain-inspired class of SpikingNeural Networks (SNNs) that feature binary sparse activations. We observe amassive advantage in using sparse activation tensors with a MIMD processor, theIntelligence Processing Unit (IPU) compared to GPUs. On training workloads, ourresults demonstrate 5-10x throughput gains compared to A100 GPUs and up to 38xgains for higher levels of activation sparsity, without a significant slowdownin training convergence or reduction in final model performance. Furthermore,our results show highly promising trends for both single and multi IPUconfigurations as we scale up to larger model sizes. Our work paves the waytowards more efficient, non-standard models via AI training hardware beyondGPUs, and competitive large scale SNN models.

Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models