Covariance in Unsupervised Learning of Probabilistic Grammars

S. B. Cohen, Noah A. Smith

Research output: Contribution to journalArticlepeer-review

Abstract

Probabilistic grammars offer great flexibility in modeling discrete sequential data like natural language text. Their symbolic component is amenable to inspection by humans, while their probabilistic component helps resolve ambiguity. They also permit the use of well-understood, general-purpose learning algorithms. There has been an increased interest in using probabilistic grammars in the Bayesian setting. To date, most of the literature has focused on using a Dirichlet prior. The Dirichlet prior has several limitations, including that it cannot directly model covariance between the probabilistic grammar's parameters. Yet, various grammar parameters are expected to be correlated because the elements in language they represent share linguistic properties. In this paper, we suggest an alternative to the Dirichlet prior, a family of logistic normal distributions. We derive an inference algorithm for this family of distributions and experiment with the task of dependency grammar induction, demonstrating performance improvements with our priors on a set of six treebanks in different natural languages. Our covariance framework permits soft parameter tying within grammars and across grammars for text in different languages, and we show empirical gains in a novel learning setting using bilingual, non-parallel data.
Original languageEnglish
Pages (from-to)3017-3051
Number of pages35
JournalJournal of Machine Learning Research
Volume11
Issue numberNov
Publication statusPublished - 2010

Fingerprint

Dive into the research topics of 'Covariance in Unsupervised Learning of Probabilistic Grammars'. Together they form a unique fingerprint.

Cite this