The relational processing limits of classic and contemporary neural network models of language processing

Guillermo Puebla Ramírez, Andrea E. Martin, Leonidas Doumas

Research output: Contribution to journalArticlepeer-review

Abstract

Whether neural networks can capture relational knowledge is a matter of long- standing controversy. Recently, some researchers have argued that (1) classic con- nectionist models can handle relational structure and (2) the success of deep learning approaches to natural language processing suggests that structured representations are unnecessary to model human language. We tested the Story Gestalt model, a classic connectionist model of text comprehension, and a Sequence-to-Sequence with Attention model, a modern deep learning architecture for natural language process- ing. Both models were trained to answer questions about stories based on abstract thematic roles. Two simulations varied the statistical structure of new stories while keeping their relational structure intact. The performance of each model fell be- low chance at least under one manipulation. We argue that both models fail our tests because they can’t perform dynamic binding. These results cast doubts on the suitability of traditional neural networks for explaining relational reasoning and language processing phenomena.
Original languageEnglish
JournalLanguage, Cognition and Neuroscience
Early online date21 Sep 2020
DOIs
Publication statusE-pub ahead of print - 21 Sep 2020

Keywords

  • relational reasoning
  • generalization
  • language processing
  • neural networks
  • deep lerning

Fingerprint

Dive into the research topics of 'The relational processing limits of classic and contemporary neural network models of language processing'. Together they form a unique fingerprint.

Cite this