English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The past year has witnessed the increasing popularity of Large LanguageModels (LLMs). Their unprecedented scale and associated high hardware cost haveimpeded their broader adoption, calling for efficient hardware designs. Withthe large hardware needed to simply run LLM inference, evaluating differenthardware designs becomes a new bottleneck.  This work introduces LLMCompass, a hardware evaluation framework for LLMinference workloads. LLMCompass is fast, accurate, versatile, and able todescribe and evaluate different hardware designs. LLMCompass includes a mapperto automatically find performance-optimal mapping and scheduling. It alsoincorporates an area-based cost model to help architects reason about theirdesign choices. Compared to real-world hardware, LLMCompass' estimated latencyachieves an average 10.4% error rate across various operators with variousinput sizes and an average 4.1% error rate for LLM inference. With LLMCompass,simulating a 4-NVIDIA A100 GPU node running GPT-3 175B inference can be donewithin 16 minutes on commodity hardware, including 26,400 rounds of themapper's parameter search.  With the aid of LLMCompass, this work draws architectural implications andexplores new cost-effective hardware designs. By reducing the computecapability or replacing High Bandwidth Memory (HBM) with traditional DRAM,these new designs can achieve as much as 3.41x improvement in performance/costcompared to an NVIDIA A100, making them promising choices for democratizingLLMs.  LLMCompass is planned to be fully open-source.

A Hardware Evaluation Framework for Large Language Model Inference