Sparsity Emerges Naturally in Neural Language Models

Naomi Saphra, Adam Lopez

Research output: Contribution to conferencePaperpeer-review

Abstract

Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks. If sparsity is important for NLP, might well-trained neural models naturally become roughly sparse? Using the Taxi-Euclidean norm to measure sparsity, we find that frequent input words are associated with concentrated or sparse activations, while frequent target words are associated with dispersed activations but concentrated gradients. We find that gradients associated with function words are more concentrated than the gradients of content words, even controlling for word frequency.
Original languageEnglish
Number of pages5
Publication statusPublished - 15 Jun 2019
EventICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena - Long Beach, United States
Duration: 15 Jun 201915 Jun 2019
http://deep-phenomena.org/

Workshop

WorkshopICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena
Abbreviated titleICML Deep Phenomena 2019
Country/TerritoryUnited States
CityLong Beach
Period15/06/1915/06/19
Internet address

Fingerprint

Dive into the research topics of 'Sparsity Emerges Naturally in Neural Language Models'. Together they form a unique fingerprint.

Cite this