English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Deep Neural Networks (DNNs) have become an essential component in manyapplication domains including web-based services. A variety of these servicesrequire high throughput and (close to) real-time features, for instance, torespond or react to users' requests or to process a stream of incoming data ontime. However, the trend in DNN design is toward larger models with many layersand parameters to achieve more accurate results. Although these models areoften pre-trained, the computational complexity in such large models can stillbe relatively significant, hindering low inference latency. Implementing acaching mechanism is a typical systems engineering solution for speeding up aservice response time. However, traditional caching is often not suitable forDNN-based services. In this paper, we propose an end-to-end automated solutionto improve the performance of DNN-based services in terms of theircomputational complexity and inference latency. Our caching method adopts theideas of self-distillation of DNN models and early exits. The proposed solutionis an automated online layer caching mechanism that allows early exiting of alarge model during inference time if the cache model in one of the early exitsis confident enough for final prediction. One of the main contributions of thispaper is that we have implemented the idea as an online caching, meaning thatthe cache models do not need access to training data and perform solely basedon the incoming data at run-time, making it suitable for applications usingpre-trained models. Our experiments results on two downstream tasks (face andobject classification) show that, on average, caching can reduce thecomputational complexity of those services up to 58\% (in terms of FLOPs count)and improve their inference latency up to 46\% with low to zero reduction inaccuracy.

Improving the Performance of DNN-based Software Services using Automated Layer Caching