A Bayesian framework for word segmentation: Exploring the effects of context

Sharon Goldwater, Thomas L. Griffiths, Mark Johnson

Research output: Contribution to journalArticlepeer-review

Abstract / Description of output

Since the experiments of Saffran et al. [Saffran, J., Aslin, R., & Newport, E. (1996). Statistical learning in 8-month-old infants. Science, 274, 1926-1928], there has been a great deal of interest in the question of how statistical regularities in the speech stream might be used by infants to begin to identify individual words. In this work, we use computational modeling to explore the effects of different assumptions the learner might make regarding the nature of words - in particular, how these assumptions affect the kinds of words that are segmented from a corpus of transcribed child-directed speech. We develop several models within a Bayesian ideal observer framework, and use them to examine the consequences of assuming either that words are independent units, or units that help to predict other units. We show through empirical and theoretical results that the assumption of independence causes the learner to undersegment the corpus, with many two- and three-word sequences (e.g. what's that, do you, in the house) misidentified as individual words. In contrast, when the learner assumes that words are predictive, the resulting segmentation is far more accurate. These results indicate that taking context into account is important for a statistical word segmentation strategy to be successful, and raise the possibility that even young infants may be able to exploit more subtle statistical patterns than have usually been considered.
Original languageEnglish
Pages (from-to)21-54
Number of pages34
Issue number1
Publication statusPublished - Jul 2009

Keywords / Materials (for Non-textual outputs)

  • Word segmentation
  • Language acquisition
  • Bayesian
  • Computational modeling


Dive into the research topics of 'A Bayesian framework for word segmentation: Exploring the effects of context'. Together they form a unique fingerprint.

Cite this