Sparsity Emerges Naturally in Neural Language Models

Naomi Saphra; Adam Lopez

Sparsity Emerges Naturally in Neural Language Models

Research output: Contribution to conference › Paper › peer-review

Abstract

Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks. If sparsity is important for NLP, might well-trained neural models naturally become roughly sparse? Using the Taxi-Euclidean norm to measure sparsity, we find that frequent input words are associated with concentrated or sparse activations, while frequent target words are associated with dispersed activations but concentrated gradients. We find that gradients associated with function words are more concentrated than the gradients of content words, even controlling for word frequency.

Original language	English
Number of pages	5
Publication status	Published - 15 Jun 2019
Event	ICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena - Long Beach, United States Duration: 15 Jun 2019 → 15 Jun 2019 http://deep-phenomena.org/

Workshop

Workshop	ICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena
Abbreviated title	ICML Deep Phenomena 2019
Country/Territory	United States
City	Long Beach
Period	15/06/19 → 15/06/19
Internet address	http://deep-phenomena.org/

Access to Document

Sparsity Emerges Naturally _SAPHRA_DoA010619_AFVAccepted author manuscript, 421 KBLicence: All Rights Reserved

https://openreview.net/forum?id=H1ets1h56ELicence: All Rights Reserved

Cite this

@conference{3111573ce086461291254a9708227703,

title = "Sparsity Emerges Naturally in Neural Language Models",

abstract = "Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks. If sparsity is important for NLP, might well-trained neural models naturally become roughly sparse? Using the Taxi-Euclidean norm to measure sparsity, we find that frequent input words are associated with concentrated or sparse activations, while frequent target words are associated with dispersed activations but concentrated gradients. We find that gradients associated with function words are more concentrated than the gradients of content words, even controlling for word frequency.",

author = "Naomi Saphra and Adam Lopez",

year = "2019",

month = jun,

day = "15",

language = "English",

note = "ICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena, ICML Deep Phenomena 2019 ; Conference date: 15-06-2019 Through 15-06-2019",

url = "http://deep-phenomena.org/",

}

TY - CONF

T1 - Sparsity Emerges Naturally in Neural Language Models

AU - Saphra, Naomi

AU - Lopez, Adam

PY - 2019/6/15

Y1 - 2019/6/15

N2 - Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks. If sparsity is important for NLP, might well-trained neural models naturally become roughly sparse? Using the Taxi-Euclidean norm to measure sparsity, we find that frequent input words are associated with concentrated or sparse activations, while frequent target words are associated with dispersed activations but concentrated gradients. We find that gradients associated with function words are more concentrated than the gradients of content words, even controlling for word frequency.

AB - Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks. If sparsity is important for NLP, might well-trained neural models naturally become roughly sparse? Using the Taxi-Euclidean norm to measure sparsity, we find that frequent input words are associated with concentrated or sparse activations, while frequent target words are associated with dispersed activations but concentrated gradients. We find that gradients associated with function words are more concentrated than the gradients of content words, even controlling for word frequency.

M3 - Paper

T2 - ICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena

Y2 - 15 June 2019 through 15 June 2019

ER -

Sparsity Emerges Naturally in Neural Language Models

Abstract

Workshop

Access to Document

Fingerprint

Cite this