English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large language models (LLMs) have exploded in popularity due to their newgenerative capabilities that go far beyond prior state-of-the-art. Thesetechnologies are increasingly being leveraged in various domains such as law,finance, and medicine. However, these models carry significant computationalchallenges, especially the compute and energy costs required for inference.Inference energy costs already receive less attention than the energy costs oftraining LLMs -- despite how often these large models are called on to conductinference in reality (e.g., ChatGPT). As these state-of-the-art LLMs seeincreasing usage and deployment in various domains, a better understanding oftheir resource utilization is crucial for cost-savings, scaling performance,efficient hardware usage, and optimal inference strategies.  In this paper, we describe experiments conducted to study the computationaland energy utilization of inference with LLMs. We benchmark and conduct apreliminary analysis of the inference performance and inference energy costs ofdifferent sizes of LLaMA -- a recent state-of-the-art LLM -- developed by MetaAI on two generations of popular GPUs (NVIDIA V100 \& A100) and two datasets(Alpaca and GSM8K) to reflect the diverse set of tasks/benchmarks for LLMs inresearch and practice. We present the results of multi-node, multi-GPUinference using model sharding across up to 32 GPUs. To our knowledge, our workis the one of the first to study LLM inference performance from the perspectiveof computational and energy resources at this scale.

From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference