English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we investigate how to push the performance limits of servingDeep Neural Network (DNN) models on CPU-based servers. Specifically, we observethat while intra-operator parallelism across multiple threads is an effectiveway to reduce inference latency, it provides diminishing returns. Our primaryinsight is that instead of running a single instance of a model with allavailable threads on a server, running multiple instances each with smallerbatch sizes and fewer threads for intra-op parallelism can provide lowerinference latency. However, the right configuration is hard to determinemanually since it is workload- (DNN model and batch size used by the servingsystem) and deployment-dependent (number of CPU cores on server). We presentPackrat, a new serving system for online inference that given a model and batchsize ($B$) algorithmically picks the optimal number of instances ($i$), thenumber of threads each should be allocated ($t$), and the batch sizes eachshould operate on ($b$) that minimizes latency. Packrat is built as anextension to TorchServe and supports online reconfigurations to avoid servingdowntime. Averaged across a range of batch sizes, Packrat improves inferencelatency by 1.43$\times$ to 1.83$\times$ on a range of commonly used DNNs.

Packrat: Automatic Reconfiguration for Latency Minimization in CPU-based DNN Serving