English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

The high computational and memory requirements of generative large languagemodels (LLMs) make it challenging to serve them cheaply. This paper aims toreduce the monetary cost for serving LLMs by leveraging preemptible GPUinstances on modern clouds, which offer accesses to spare GPUs at a muchcheaper price than regular instances but may be preempted by the cloud at anytime. Serving LLMs on preemptible instances requires addressing challengesinduced by frequent instance preemptions and the necessity of migratinginstances to handle these preemptions.  This paper presents SpotServe, the first distributed LLM serving system onpreemptible instances. Several key techniques in SpotServe realize fast andreliable serving of generative LLMs on cheap preemptible instances. First,SpotServe dynamically adapts the LLM parallelization configuration for dynamicinstance availability and fluctuating workload, while balancing the trade-offamong the overall throughput, inference latency and monetary costs. Second, tominimize the cost of migrating instances for dynamic reparallelization, thetask of migrating instances is formulated as a bipartite graph matchingproblem, which uses the Kuhn-Munkres algorithm to identify an optimal migrationplan that minimizes communications. Finally, to take advantage of the graceperiod offered by modern clouds, we introduce stateful inference recovery, anew inference mechanism that commits inference progress at a much finergranularity and allows SpotServe to cheaply resume inference upon preemption.We evaluate on real spot instance preemption traces and various popular LLMsand show that SpotServe can reduce the P99 tail latency by 2.4 - 9.1x comparedwith the best existing LLM serving systems. We also show that SpotServe canleverage the price advantage of preemptive instances, saving 54% monetary costcompared with only using on-demand instances.

SpotServe: Serving Generative Large Language Models on Preemptible Instances