English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We investigate how neural networks can learn and process languages withhierarchical, compositional semantics. To this end, we define the artificialtask of processing nested arithmetic expressions, and study whether differenttypes of neural networks can learn to compute their meaning. We find thatrecursive neural networks can find a generalising solution to this problem, andwe visualise this solution by breaking it up in three steps: project, sum andsquash. As a next step, we investigate recurrent neural networks, and show thata gated recurrent unit, that processes its input incrementally, also performsvery well on this task. To develop an understanding of what the recurrentnetwork encodes, visualisation techniques alone do not suffice. Therefore, wedevelop an approach where we formulate and test multiple hypotheses on theinformation encoded and processed by the network. For each hypothesis, wederive predictions about features of the hidden state representations at eachtime step, and train 'diagnostic classifiers' to test those predictions. Ourresults indicate that the networks follow a strategy similar to ourhypothesised 'cumulative strategy', which explains the high accuracy of thenetwork on novel expressions, the generalisation to longer expressions thanseen in training, and the mild deterioration with increasing length. This isturn shows that diagnostic classifiers can be a useful technique for opening upthe black box of neural networks. We argue that diagnostic classification,unlike most visualisation techniques, does scale up from small networks in atoy domain, to larger and deeper recurrent networks dealing with real-lifedata, and may therefore contribute to a better understanding of the internaldynamics of current state-of-the-art models in natural language processing.

Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure