Open AccessOptimization Matters: Guidelines to Improve Representation Learning with Deep Networks
Author(s) -
Aline R. Becher,
Moacir Antonelli Ponti
Publication year - 2021
Language(s) - English
Resource type - Conference proceedings
DOI - 10.5753/eniac.2021.18287
Subject(s) - computer science , convergence (economics) , artificial intelligence , context (archaeology) , deep learning , software deployment , artificial neural network , machine learning , deep neural networks , representation (politics) , feature (linguistics) , feature learning , set (abstract data type) , quality (philosophy) , optimization problem , algorithm , software engineering , epistemology , politics , political science , law , economics , biology , programming language , economic growth , paleontology , linguistics , philosophy
Training deep neural networks is a relevant problem with open questions related to convergence and quality of learned representations. Gradient-based optimization methods are used in practice, but cases of failure and success are still to be investigated. In this context, we set out to better understand the convergence properties of different optimization strategies, under different parameter options. Our results show that (i) feature embeddings are impacted by different optimization settings, (ii) suboptimal results are achieved by the use of default parameters, (iii) significant improvement is obtained by making educated choices of parameters, (iv) learning rate decay should always be considered. Such findings offer guidelines for training and deployment of deep networks.