Analogies Explained: Towards Understanding Word Embeddings

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Word embeddings generated by neural network methods such as word2vec (W2V) are well known to exhibit seemingly linear behaviour, e.g. the embeddings of analogy “woman is to queen as man is to king” approximately describe a parallelogram. This property is particularly intriguing since the embeddings are not trained to achieve it. Several explanations have been proposed, but each introduces assumptions that do not hold in practice. We derive a probabilistically grounded definition of paraphrasing that we re-interpret as word transformation, a mathematical description of “wx is to wy”. From these concepts we prove existence of linear relationships between W2V-type embeddings that underlie the analogical phenomenon, identifying explicit error terms.
Original languageEnglish
Title of host publicationProceedings of the 36th International Conference on Machine Learning (ICML)
EditorsKamalika Chaudhuri, Ruslan Salakhutdinov
Place of PublicationLong Beach, USA
PublisherPMLR
Pages223-231
Number of pages9
Volume97
Publication statusE-pub ahead of print - 3 Jul 2019
EventThirty-sixth International Conference on Machine Learning - Long Beach Convention Center, Long Beach, United States
Duration: 9 Jun 201915 Jun 2019
Conference number: 36
https://icml.cc/Conferences/2019

Publication series

NameProceedings of Machine Learning Research
PublisherPMLR
Volume97
ISSN (Electronic)2640-3498

Conference

ConferenceThirty-sixth International Conference on Machine Learning
Abbreviated titleICML 2019
CountryUnited States
CityLong Beach
Period9/06/1915/06/19
Internet address

Fingerprint

Dive into the research topics of 'Analogies Explained: Towards Understanding Word Embeddings'. Together they form a unique fingerprint.

Cite this