Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks. If sparsity is important for NLP, might well-trained neural models naturally become roughly sparse? Using the Taxi-Euclidean norm to measure sparsity, we find that frequent input words are associated with concentrated or sparse activations, while frequent target words are associated with dispersed activations but concentrated gradients. We find that gradients associated with function words are more concentrated than the gradients of content words, even controlling for word frequency.
|Number of pages||5|
|Publication status||Published - 15 Jun 2019|
|Event||ICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena - Long Beach, United States|
Duration: 15 Jun 2019 → 15 Jun 2019
|Workshop||ICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena|
|Abbreviated title||ICML Deep Phenomena 2019|
|Period||15/06/19 → 15/06/19|