Entity Decisions in Neural Language Modelling: Approaches and Problems

Jenny Kunz, Christian Hardmeier

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We explore different approaches to explicit entity modelling in language models (LM). We independently replicate two existing models in a controlled setup, introduce a simplified variant of one of the models and analyze their performance in direct comparison. Our results suggest that today's models are limited as several stochastic variables make learning difficult. We show that the most challenging point in the systems is the decision if the next token is an entity token. The low precision and recall for this variable will lead to severe cascading errors. Our own simplified approach dispenses with the need for latent variables and improves the performance in the entity yes/no decision. A standard well-tuned baseline RNN-LM with a larger number of hidden units outperforms all entity-enabled LMs in terms of perplexity.
Original languageEnglish
Title of host publicationProceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference
Subtitle of host publicationCRAC 2019
Place of PublicationMinneapolis, USA
PublisherAssociation for Computational Linguistics
Pages15-19
Number of pages5
ISBN (Print)978-1-948087-97-1
DOIs
Publication statusPublished - 7 Jun 2019
Event2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics - Minneapolis, United States
Duration: 2 Jun 20197 Jun 2019
https://naacl2019.org/

Conference

Conference2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics
Abbreviated titleNAACL-HLT 2019
CountryUnited States
CityMinneapolis
Period2/06/197/06/19
Internet address

Cite this