Classification problems with a very large or unbounded set of output categories are common in many areas such as natural language and image processing. In order to improve accuracy on these tasks, it is natural for a decision-maker to combine predictions from various sources. However, supervised data needed to fit an aggregation model is often difficult to obtain, especially if needed for multiple domains. Therefore, we propose a generative model for unsupervised aggregation which exploits the agreement signal to estimate the expertise of individual judges. Due to the large output space size, this aggregation model cannot encode expertise of constituent judges with respect to every category for all problems. Consequently, we extend it by incorporating the notion of category types to account for variability of the judge expertise depending on the type. The viability of our approach is demonstrated both on synthetic experiments and on a practical task of syntactic parser aggregation.
|Title of host publication||Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics|
|Editors||Yee Whye Teh, Mike Titterington|
|Place of Publication||Chia Laguna Resort, Sardinia, Italy|
|Number of pages||8|
|Publication status||Published - 1 Jan 2010|
|Name||Proceedings of Machine Learning Research|