Improved Estimation of Entropy for Evaluation of Word Sense Induction

Linlin Li, Ivan Titov, Caroline Sporleder

Research output: Contribution to journalArticlepeer-review


Information-theoretic measures are among the most standard techniques for evaluation of clustering methods including word sense induction (WSI) systems. Such measures rely on sample-based estimates of the entropy. However, the standard maximum likelihood estimates of the entropy are heavily biased with the bias dependent on, among other things, the number of clusters and the sample size. This makes the measures unreliable and unfair when the number of clusters produced by different systems vary and the sample size is not exceedingly large. This corresponds exactly to the setting of WSI evaluation where a ground-truth cluster sense number arguably does not exist and the standard evaluation scenarios use a small number of instances of each word to compute the score. We describe more accurate entropy estimators and analyze their performance both in simulations and on evaluation of WSI systems.
Original languageEnglish
Pages (from-to)671-685
Number of pages15
JournalComputational Linguistics
Issue number3
Publication statusPublished - Sep 2014


Dive into the research topics of 'Improved Estimation of Entropy for Evaluation of Word Sense Induction'. Together they form a unique fingerprint.

Cite this