Premium
Evaluating and analyzing the energy efficiency of CNN inference on high‐performance GPU
Author(s) -
Yao Chunrong,
Liu Wantao,
Tang Weiqing,
Guo Jinrong,
Hu Songlin,
Lu Yijun,
Jiang Wei
Publication year - 2020
Publication title -
concurrency and computation: practice and experience
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.309
H-Index - 67
eISSN - 1532-0634
pISSN - 1532-0626
DOI - 10.1002/cpe.6064
Subject(s) - computer science , efficient energy use , convolutional neural network , inference , energy consumption , parallel computing , cache , exploit , deep learning , convolution (computer science) , computer engineering , artificial neural network , artificial intelligence , ecology , computer security , electrical engineering , biology , engineering
Summary Convolutional neural network (CNN) inference usually runs on high‐performance graphic processing units (GPUs). Since GPU is a high power consumption unit, that makes the energy consumption increases sharply due to the deep learning tasks. The energy efficiency of CNN inference is not only related to the software and hardware configurations, but also closely related to the application requirements of inference tasks. However, it is not clear on GPUs at present. In this paper, we conduct a comprehensive study on the model‐level and layer‐level energy efficiency of popular CNN models. The results point out several opportunities for further optimization. We also analyze the parameter settings (i.e., batch size, dynamic voltage and frequency scaling) and propose a revenue model to allow an optimal trade‐off between energy efficiency and latency. Compared with the default settings, the optimal settings can improve revenue by up to 15.31 × . We obtain the following main findings: (i) GPUs do not exploit the parallelism from the model depth and small convolution kernels, resulting in low energy efficiency. (ii) Convolutional layers are the most energy‐consuming CNN layers. However, due to the cache, the power consumption of all layers is relatively balanced. (iii) The energy efficiency of TensorRT is 1.53 × than that of TensorFlow.