English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Quantization is a popular technique used in Deep Neural Networks (DNN)inference to reduce the size of models and improve the overall numericalperformance by exploiting native hardware. This paper attempts to conduct anelaborate performance characterization of the benefits of using quantizationtechniques -- mainly FP16/INT8 variants with static and dynamic schemes --using the MLPerf Edge Inference benchmarking methodology. The study isconducted on Intel x86 processors and Raspberry Pi device with ARM processor.The paper uses a number of DNN inference frameworks, including OpenVINO (forIntel CPUs only), TensorFlow Lite (TFLite), ONNX, and PyTorch with MobileNetV2,VGG-19, and DenseNet-121. The single-stream, multi-stream, and offlinescenarios of the MLPerf Edge Inference benchmarks are used for measuringlatency and throughput in our experiments. Our evaluation reveals that OpenVINOand TFLite are the most optimized frameworks for Intel CPUs and Raspberry Pidevice, respectively. We observe no loss in accuracy except for the staticquantization techniques. We also observed the benefits of using quantizationfor these optimized frameworks. For example, INT8-based quantized modelsdeliver $3.3\times$ and $4\times$ better performance over FP32 using OpenVINOon Intel CPU and TFLite on Raspberry Pi device, respectively, for the MLPerfoffline scenario. To the best of our knowledge, this paper is the first onethat presents a unique characterization study characterizing the impact ofquantization for a range of DNN inference frameworks -- including OpenVINO,TFLite, PyTorch, and ONNX -- on Intel x86 processors and Raspberry Pi devicewith ARM processor using the MLPerf Edge Inference benchmark methodology.

Performance Characterization of using Quantization for DNN Inference on Edge Devices: Extended Version