English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

DL inference queries play an important role in diverse internet services anda large fraction of datacenter cycles are spent on processing DL inferencequeries. Specifically, the matrix-matrix multiplication (GEMM) operations offully-connected MLP layers dominate many inference tasks. We find that the GEMMoperations for datacenter DL inference tasks are memory bandwidth bound,contrary to common assumptions: (1) strict query latency constraints forcesmall-batch operation, which limits reuse and increases bandwidth demands; and(2) large and colocated models require reading the large weight matrices frommain memory, again requiring high bandwidth without offering reuseopportunities. We demonstrate the large potential of accelerating thesesmall-batch GEMMs with processing in the main CPU memory. We develop a novelGEMM execution flow and corresponding memory-side address-generation logic thatexploits GEMM locality and enables long-running PIM kernels despite the complexaddress-mapping functions employed by the CPU that would otherwise destroylocality. Our evaluation of StepStone variants at the channel, device, andwithin-device PIM levels, along with optimizations that balance parallelismbenefits with data-distribution overheads demonstrate $12\times$ better minimumlatency than a CPU and $2.8\times$ greater throughput for strict query latencyconstraints. End-to-end performance analysis of recent recommendation andlanguage models shows that StepStone PIM outperforms a fast CPU (by up to$16\times$) and prior main-memory acceleration approaches (by up to $2.4\times$compared to the best prior approach).

Accelerating Bandwidth-Bound Deep Learning Inference with Main-Memory Accelerators