Premium
Orthogonality is not a panacea: Backpropagation and “catastrophic interference”
Author(s) -
YAMAGUCHI MAKOTO
Publication year - 2006
Publication title -
scandinavian journal of psychology
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.743
H-Index - 72
eISSN - 1467-9450
pISSN - 0036-5564
DOI - 10.1111/j.1467-9450.2006.00528.x
Subject(s) - interference (communication) , orthogonality , backpropagation , coding (social sciences) , forgetting , computer science , connectionism , algorithm , artificial intelligence , artificial neural network , pattern recognition (psychology) , telecommunications , mathematics , statistics , psychology , cognitive psychology , channel (broadcasting) , geometry
Connectionist models with the backpropagation learning rule are said to exhibit catastrophic interference (or forgetting) with sequential training. Subsequent works showed that interference can be reduced by using orthogonal inputs. This study investigated, with a more rigorous assessment method, whether all orthogonal inputs lead to comparable extent of interference using three coding schemes. The results revealed large differences between the coding schemes. With larger networks, dense inputs led to severer interference compared with sparse inputs. With smaller networks, all the three schemes led to comparable extent of interference. Therefore, this study proved that not all the orthogonal inputs cause the same extent of interference, and that severity of interference depends on the interaction of the input coding scheme and the network size.