Evaluation methods for topic models

Hanna M. Wallach; Iain Murray; Ruslan Salakhutdinov; David Mimno

doi:10.1145/1553374.1553515

Evaluation methods for topic models

Hanna M. Wallach, Iain Murray, Ruslan Salakhutdinov, David Mimno

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

A natural evaluation metric for statistical topic models is the probability of held-out documents given a trained model. While exact computation of this probability is intractable, several estimators for this probability have been used in the topic modeling literature, including the harmonic mean method and empirical likelihood method. In this paper, we demonstrate experimentally that commonly-used methods are unlikely to accurately estimate the probability of held-out documents, and propose two alternative methods that are both accurate and efficient.

Original language	English
Title of host publication	Proceedings of the 26th Annual International Conference on Machine Learning (ICML '09)
Place of Publication	New York, NY, USA
Publisher	ACM
Pages	1105-1112
Number of pages	8
ISBN (Print)	978-1-60558-516-1
DOIs	https://doi.org/10.1145/1553374.1553515
Publication status	Published - 2009

Access to Document

10.1145/1553374.1553515

p1105-wallachFinal published version, 689 KB

http://dl.acm.org/citation.cfm?doid=1553374.1553515

Cite this

@inproceedings{b6af6b5b61cb43148cea10eae25216e0,

title = "Evaluation methods for topic models",

abstract = "A natural evaluation metric for statistical topic models is the probability of held-out documents given a trained model. While exact computation of this probability is intractable, several estimators for this probability have been used in the topic modeling literature, including the harmonic mean method and empirical likelihood method. In this paper, we demonstrate experimentally that commonly-used methods are unlikely to accurately estimate the probability of held-out documents, and propose two alternative methods that are both accurate and efficient.",

author = "Wallach, \{Hanna M.\} and Iain Murray and Ruslan Salakhutdinov and David Mimno",

year = "2009",

doi = "10.1145/1553374.1553515",

language = "English",

isbn = "978-1-60558-516-1",

pages = "1105--1112",

booktitle = "Proceedings of the 26th Annual International Conference on Machine Learning (ICML '09)",

publisher = "ACM",

}

TY - GEN

T1 - Evaluation methods for topic models

AU - Wallach, Hanna M.

AU - Murray, Iain

AU - Salakhutdinov, Ruslan

AU - Mimno, David

PY - 2009

Y1 - 2009

N2 - A natural evaluation metric for statistical topic models is the probability of held-out documents given a trained model. While exact computation of this probability is intractable, several estimators for this probability have been used in the topic modeling literature, including the harmonic mean method and empirical likelihood method. In this paper, we demonstrate experimentally that commonly-used methods are unlikely to accurately estimate the probability of held-out documents, and propose two alternative methods that are both accurate and efficient.

AB - A natural evaluation metric for statistical topic models is the probability of held-out documents given a trained model. While exact computation of this probability is intractable, several estimators for this probability have been used in the topic modeling literature, including the harmonic mean method and empirical likelihood method. In this paper, we demonstrate experimentally that commonly-used methods are unlikely to accurately estimate the probability of held-out documents, and propose two alternative methods that are both accurate and efficient.

U2 - 10.1145/1553374.1553515

DO - 10.1145/1553374.1553515

M3 - Conference contribution

SN - 978-1-60558-516-1

SP - 1105

EP - 1112

BT - Proceedings of the 26th Annual International Conference on Machine Learning (ICML '09)

PB - ACM

CY - New York, NY, USA

ER -

Evaluation methods for topic models

Abstract

Access to Document

Fingerprint

Cite this