Unsupervised Aggregation for Classification Problems with Large Numbers of Categories

Ivan Titov, Alexandre Klementiev, Kevin Small, Dan Roth

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Classification problems with a very large or unbounded set of output categories are common in many areas such as natural language and image processing. In order to improve accuracy on these tasks, it is natural for a decision-maker to combine predictions from various sources. However, supervised data needed to fit an aggregation model is often difficult to obtain, especially if needed for multiple domains. Therefore, we propose a generative model for unsupervised aggregation which exploits the agreement signal to estimate the expertise of individual judges. Due to the large output space size, this aggregation model cannot encode expertise of constituent judges with respect to every category for all problems. Consequently, we extend it by incorporating the notion of category types to account for variability of the judge expertise depending on the type. The viability of our approach is demonstrated both on synthetic experiments and on a practical task of syntactic parser aggregation.
Original languageEnglish
Title of host publicationProceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics
EditorsYee Whye Teh, Mike Titterington
Place of PublicationChia Laguna Resort, Sardinia, Italy
PublisherPMLR
Pages836-843
Number of pages8
Volume9
Publication statusPublished - 1 Jan 2010

Publication series

NameProceedings of Machine Learning Research
PublisherPMLR

Fingerprint Dive into the research topics of 'Unsupervised Aggregation for Classification Problems with Large Numbers of Categories'. Together they form a unique fingerprint.

Cite this