Contextual Dependencies in Unsupervised Word Segmentation

Sharon Goldwater; Thomas L. Griffiths; Mark Johnson

doi:10.3115/1220175.1220260

Contextual Dependencies in Unsupervised Word Segmentation

Sharon Goldwater, Thomas L. Griffiths, Mark Johnson

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

Developing better methods for segmenting continuous text into words is important for improving the processing of Asian languages, and may shed light on how humans learn to segment speech. We propose two new Bayesian word segmentation methods that assume unigram and bigram models of word dependencies respectively. The bigram model greatly out-performs the unigram model (and previous probabilistic models), demonstrating the importance of such dependencies for word segmentation. We also show that previous probabilistic models rely crucially on sub-optimal search procedures.

Original language	English
Title of host publication	Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics
Place of Publication	Sydney, Australia
Publisher	Association for Computational Linguistics
Pages	673-680
Number of pages	8
DOIs	https://doi.org/10.3115/1220175.1220260
Publication status	Published - 1 Jul 2006

Access to Document

10.3115/1220175.1220260

P06 1085Final published version, 127 KB

http://www.aclweb.org/anthology/P06-1085

Cite this

@inproceedings{bbfab021df7041ba8d0643273aeec9de,

title = "Contextual Dependencies in Unsupervised Word Segmentation",

abstract = "Developing better methods for segmenting continuous text into words is important for improving the processing of Asian languages, and may shed light on how humans learn to segment speech. We propose two new Bayesian word segmentation methods that assume unigram and bigram models of word dependencies respectively. The bigram model greatly out-performs the unigram model (and previous probabilistic models), demonstrating the importance of such dependencies for word segmentation. We also show that previous probabilistic models rely crucially on sub-optimal search procedures.",

author = "Sharon Goldwater and Griffiths, {Thomas L.} and Mark Johnson",

year = "2006",

month = jul,

day = "1",

doi = "10.3115/1220175.1220260",

language = "English",

pages = "673--680",

booktitle = "Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics",

}

Goldwater, S, Griffiths, TL & Johnson, M 2006, Contextual Dependencies in Unsupervised Word Segmentation. in Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Sydney, Australia, pp. 673-680. https://doi.org/10.3115/1220175.1220260

Contextual Dependencies in Unsupervised Word Segmentation. / Goldwater, Sharon; Griffiths, Thomas L.; Johnson, Mark.
Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics. Sydney, Australia: Association for Computational Linguistics, 2006. p. 673-680.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Contextual Dependencies in Unsupervised Word Segmentation

AU - Goldwater, Sharon

AU - Griffiths, Thomas L.

AU - Johnson, Mark

PY - 2006/7/1

Y1 - 2006/7/1

N2 - Developing better methods for segmenting continuous text into words is important for improving the processing of Asian languages, and may shed light on how humans learn to segment speech. We propose two new Bayesian word segmentation methods that assume unigram and bigram models of word dependencies respectively. The bigram model greatly out-performs the unigram model (and previous probabilistic models), demonstrating the importance of such dependencies for word segmentation. We also show that previous probabilistic models rely crucially on sub-optimal search procedures.

AB - Developing better methods for segmenting continuous text into words is important for improving the processing of Asian languages, and may shed light on how humans learn to segment speech. We propose two new Bayesian word segmentation methods that assume unigram and bigram models of word dependencies respectively. The bigram model greatly out-performs the unigram model (and previous probabilistic models), demonstrating the importance of such dependencies for word segmentation. We also show that previous probabilistic models rely crucially on sub-optimal search procedures.

U2 - 10.3115/1220175.1220260

DO - 10.3115/1220175.1220260

M3 - Conference contribution

SP - 673

EP - 680

BT - Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics

PB - Association for Computational Linguistics

CY - Sydney, Australia

ER -

Contextual Dependencies in Unsupervised Word Segmentation

Abstract / Description of output

Access to Document

Fingerprint

Cite this