Delving Deeper Into Astromorphic Transformers

Preliminary attempts at incorporating the critical role of astrocytes - cellsthat constitute more than 50% of human brain cells - in brain-inspiredneuromorphic computing remain in infancy. This paper seeks to delve deeper intovarious key aspects of neuron-synapse-astrocyte interactions to mimicself-attention mechanisms in Transformers. The cross-layer perspective exploredin this work involves bio-plausible modeling of Hebbian and pre-synapticplasticities in neuron-astrocyte networks, incorporating effects ofnon-linearities and feedback along with algorithmic formulations to map theneuron-astrocyte computations to self-attention mechanism and evaluating theimpact of incorporating bio-realistic effects from the machine learningapplication side. Our analysis on sentiment and image classification tasks onthe IMDB and CIFAR10 datasets underscores the importance of constructingAstromorphic Transformers from both accuracy and learning speed improvementperspectives.