Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination

Over the past decade, deep learning has proven to be a highly effective toolfor learning meaningful features from raw data. However, it remains an openquestion how deep networks perform hierarchical feature learning across layers.In this work, we attempt to unveil this mystery by investigating the structuresof intermediate features. Motivated by our empirical findings that linearlayers mimic the roles of deep layers in nonlinear networks for featurelearning, we explore how deep linear networks transform input data into outputby investigating the output (i.e., features) of each layer after training inthe context of multi-class classification problems. Toward this goal, we firstdefine metrics to measure within-class compression and between-classdiscrimination of intermediate features, respectively. Through theoreticalanalysis of these two metrics, we show that the evolution of features follows asimple and quantitative pattern from shallow to deep layers when the input datais nearly orthogonal and the network weights are minimum-norm, balanced, andapproximate low-rank: Each layer of the linear network progressively compresseswithin-class features at a geometric rate and discriminates between-classfeatures at a linear rate with respect to the number of layers that data havepassed through. To the best of our knowledge, this is the first quantitativecharacterization of feature evolution in hierarchical representations of deeplinear networks. Empirically, our extensive experiments not only validate ourtheoretical results numerically but also reveal a similar pattern in deepnonlinear networks which aligns well with recent empirical studies. Moreover,we demonstrate the practical implications of our results in transfer learning.Our code is available at \url{https://github.com/Heimine/PNC_DLN}.