English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Existing LLM serving strategies can be categorized based on whether prefilland decode phases are disaggregated: non-disaggregated (NoDG) or fullydisaggregated (FuDG). However, the NoDG strategy leads to strong prefill-decodeinterference and the FuDG strategy highly relies on high-performanceinterconnects, making them less cost-effective.  We introduce EcoServe, a system that enables cost-effective LLM serving onclusters with commodity interconnects. EcoServe is built on the partiallydisaggregated (PaDG) strategy, applying temporal disaggregation and rollingactivation for proactive intra- and inter-instance scheduling. It firstdisaggregates the prefill and decode phases along the time dimension within asingle instance to mitigate inter-phase interference and enhance throughput.Next, it coordinates multiple instances and cyclically activates them to ensurethe continuous availability of prefill processing, thereby improving latency.Thus, EcoServe's basic serving unit is the macro instance, within whichmultiple instances collaborate. It further integrates an adaptive schedulingalgorithm to route requests in a macro instance and a mitosis scaling approachto enable fine-grained capacity scaling. Beyond delivering high goodput,EcoServe excels in load balancing, hardware cost, parallelism compatibility,and even engineering simplicity compared to existing solutions.  When serving 30B- and 70B-scale models on a production-level cluster with 32NVIDIA L20 GPUs using commodity Ethernet, EcoServe averagely improves goodputby 82.49%, 86.17%, 122.76%, and 126.96% over four representative NoDG and FuDGsystems.

EcoServe: Enabling Cost-effective LLM Serving with Proactive Intra- and Inter-Instance Orchestration