When the state of the art is ahead of the state of understanding: Unintuitive properties of deep neural networks

Deeplearning is an undeniably hot topic, not only within both academia andindustry, but also among society and the media. The reasons for theadvent of its popularity are manifold: unprecedented availability ofdata and computing power, some innovative methodologies, minor butsignificant technical tricks, etc. However, interestingly, the currentsuccess and practice of deep learning seems to be uncorrelated withits theoretical, more formal understanding. And with that, deeplearning’s state-of-the-art presents a number of unintuitiveproperties or situations. In this note, I highlight some of theseunintuitive properties, trying to show relevant recent work, andexpose the need to get insight into them, either by formal or moreempirical means.