Spoken Dialog Challenge 2010: comparison of live and control test results

Alan W. Black, Susanne Burger, Alistair Conkie, Helen Hastie, Simon Keizer, Oliver Lemon, Nicolas Merigaud, Gabriel Parent, Gabriel Schubiner, Blaise Thomson, Jason D. Williams, Kai Yu, Steve Young, Maxine Eskenazi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

The Spoken Dialog Challenge 2010 was an exercise to investigate how different spoken dialog systems perform on the same task. The existing Let’s Go Pittsburgh Bus Information System was used as a task and four teams provided systems that were first tested in controlled conditions with speech researchers as users. The three most stable systems were then deployed to real callers. This paper presents the results of the live tests, and compares them with the control test results. Results show considerable variation both between systems and between the control and live tests. Interestingly, relatively high task completion for controlled tests did not always predict relatively high task completion for live tests. Moreover, even though the systems were quite different in their designs, we saw very similar correlations between word error rate and task completion for all the systems. The dialog data collected is available to the research community.
Original languageEnglish
Title of host publicationProceedings of the SIGDIAL 2011 Conference
PublisherAssociation for Computational Linguistics
Pages2-7
Number of pages6
ISBN (Print)9781937284107
Publication statusPublished - 2011
Event12th Annual Meeting of the Special Interest Group on Discourse and Dialogue - Portland, United States
Duration: 17 Jun 201118 Jun 2011
Conference number: 12
https://sigdial.org/files/workshops/sigdial2011/

Conference

Conference12th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Abbreviated titleSIGDIAL 2011
Country/TerritoryUnited States
CityPortland
Period17/06/1118/06/11
Internet address

Fingerprint

Dive into the research topics of 'Spoken Dialog Challenge 2010: comparison of live and control test results'. Together they form a unique fingerprint.

Cite this