Premium
Analyzing Machine‐Learned Representations: A Natural Language Case Study
Author(s) -
Dasgupta Ishita,
Guo Demi,
Gershman Samuel J.,
Goodman Noah D.
Publication year - 2020
Publication title -
cognitive science
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.498
H-Index - 114
eISSN - 1551-6709
pISSN - 0364-0213
DOI - 10.1111/cogs.12925
Subject(s) - parallels , generalization , heuristic , computer science , artificial intelligence , set (abstract data type) , natural language , natural (archaeology) , natural language processing , natural language understanding , cognitive science , psychology , epistemology , mechanical engineering , philosophy , archaeology , engineering , history , programming language
As modern deep networks become more complex, and get closer to human‐like capabilities in certain domains, the question arises as to how the representations and decision rules they learn compare to the ones in humans. In this work, we study representations of sentences in one such artificial system for natural language processing. We first present a diagnostic test dataset to examine the degree of abstract composable structure represented. Analyzing performance on these diagnostic tests indicates a lack of systematicity in representations and decision rules, and reveals a set of heuristic strategies. We then investigate the effect of training distribution on learning these heuristic strategies, and we study changes in these representations with various augmentations to the training set. Our results reveal parallels to the analogous representations in people. We find that these systems can learn abstract rules and generalize them to new contexts under certain circumstances—similar to human zero‐shot reasoning. However, we also note some shortcomings in this generalization behavior—similar to human judgment errors like belief bias. Studying these parallels suggests new ways to understand psychological phenomena in humans as well as informs best strategies for building artificial intelligence with human‐like language understanding.