English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Model quantization represents both parameters (weights) and intermediatevalues (activations) in a more compact format, thereby directly reducing bothcomputational and memory cost in hardware. The quantization of recent largelanguage models (LLMs) faces challenges to achieve competitive memory densitycompared to other models such as convolutional neural networks, since values inLLMs require larger dynamic ranges.  Current hardware can expedite computation for LLMs using compact numericalformats such as low-bitwidth integers or floating-point numbers. Each hasadvantages: integer operations simplify circuit design, whereas floating-pointcalculations can enhance accuracy when a wider dynamic range is required. Inthis work, we seek an efficient data format that combines the best of bothworlds: Microscaling (MX) formats. MX formats are efficient data formats thatachieve both large dynamic ranges and high memory density.  In this paper, we propose a compiler named MASE for exploring mixed-precisionMX formats on dataflow hardware accelerators for LLM inference. Our maincontributions are twofold. First, we propose a novel orchestration abstractionto explore both software and hardware optimizations with new data formats.Second, MASE achieves LLM inference at an average precision of 4-bits, withminimal to no accuracy degradation. To our knowledge, MASE represents the firsteffort to harness fine-grain multi-precision MX formats in the design of LLMhardware accelerators. Over a range of LLMs and datasets, MASE achieves anaverage improvement of 24% in $\Delta$ accuracy with an overhead of only 3% inenergy efficiency compared to designs using 8-bit fixed-point numbers.

A Dataflow Compiler for Efficient LLM Inference using Custom Microscaling Formats