Open AccessA free-energy principle for representation learning
Author(s) -
Yansong Gao,
Pratik Chaudhari
Publication year - 2021
Publication title -
machine learning: science and technology
Language(s) - English
Resource type - Journals
ISSN - 2632-2153
DOI - 10.1088/2632-2153/abf984
Subject(s) - traverse , distortion (music) , computer science , task (project management) , representation (politics) , process (computing) , artificial intelligence , quality (philosophy) , regular polygon , surface (topology) , connection (principal bundle) , energy (signal processing) , rate–distortion theory , machine learning , pattern recognition (psychology) , mathematics , statistics , physics , geometry , law , computer network , amplifier , bandwidth (computing) , data compression , operating system , geodesy , quantum mechanics , political science , politics , geography , management , economics
This paper employs a formal connection of machine learning with thermodynamics to characterize the quality of learned representations for transfer learning. We discuss how information-theoretic functionals such as rate, distortion and classification loss of a model lie on a convex, so-called, equilibrium surface. We prescribe dynamical processes to traverse this surface under specific constraints; in particular we develop an iso-classification process that trades off rate and distortion to keep the classification loss unchanged. We demonstrate how this process can be used for transferring representations from a source task to a target task while keeping the classification loss constant. Experimental validation of the theoretical results is provided on image-classification datasets.