Word, Syllable and Phoneme Based Metrics Do Not Correlate with Human Performance in ASR-Mediated Tasks

AnneH. Schneider; Johannes Hellrich; Saturnino Luz

doi:10.1007/978-3-319-10888-9_39

Word, Syllable and Phoneme Based Metrics Do Not Correlate with Human Performance in ASR-Mediated Tasks

AnneH. Schneider, Johannes Hellrich, Saturnino Luz

Research output: Chapter in Book/Report/Conference proceeding › Other chapter contribution

Abstract / Description of output

Automatic evaluation metrics should correlate with human judgement. We collected sixteen ASR mediated dialogues using a map task scenario. The material was assessed extrinsically (i.e. in context) through measures like time to task completion and intrinsically (i.e. out of context) using the word error rate and several variants thereof, which are based on smaller units. Extrinsic and intrinsic results did not correlate, neither for word error rate nor for metrics based on characters, syllables or phonemes.

Original language	Undefined/Unknown
Title of host publication	Advances in Natural Language Processing
Editors	Adam Przepiórkowski, Maciej Ogrodniczuk
Publisher	Springer
Pages	392-399
Number of pages	8
Volume	8686
ISBN (Print)	978-3-319-10887-2
DOIs	https://doi.org/10.1007/978-3-319-10888-9_39
Publication status	Published - 2014

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer

Access to Document

10.1007/978-3-319-10888-9_39

http://dx.doi.org/10.1007/978-3-319-10888-9_39

Cite this

@inbook{4ab3a1035db84cd0a9903c2c0546e919,

title = "Word, Syllable and Phoneme Based Metrics Do Not Correlate with Human Performance in ASR-Mediated Tasks",

abstract = "Automatic evaluation metrics should correlate with human judgement. We collected sixteen ASR mediated dialogues using a map task scenario. The material was assessed extrinsically (i.e. in context) through measures like time to task completion and intrinsically (i.e. out of context) using the word error rate and several variants thereof, which are based on smaller units. Extrinsic and intrinsic results did not correlate, neither for word error rate nor for metrics based on characters, syllables or phonemes.",

author = "AnneH. Schneider and Johannes Hellrich and Saturnino Luz",

year = "2014",

doi = "10.1007/978-3-319-10888-9_39",

language = "Undefined/Unknown",

isbn = "978-3-319-10887-2",

volume = "8686",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "392--399",

editor = "Adam Przepi{\'o}rkowski and Maciej Ogrodniczuk",

booktitle = "Advances in Natural Language Processing",

}

Word, Syllable and Phoneme Based Metrics Do Not Correlate with Human Performance in ASR-Mediated Tasks. / Schneider, AnneH.; Hellrich, Johannes; Luz, Saturnino.
Advances in Natural Language Processing. ed. / Adam Przepiórkowski; Maciej Ogrodniczuk. Vol. 8686 Springer, 2014. p. 392-399 (Lecture Notes in Computer Science).

Research output: Chapter in Book/Report/Conference proceeding › Other chapter contribution

TY - CHAP

T1 - Word, Syllable and Phoneme Based Metrics Do Not Correlate with Human Performance in ASR-Mediated Tasks

AU - Schneider, AnneH.

AU - Hellrich, Johannes

AU - Luz, Saturnino

PY - 2014

Y1 - 2014

N2 - Automatic evaluation metrics should correlate with human judgement. We collected sixteen ASR mediated dialogues using a map task scenario. The material was assessed extrinsically (i.e. in context) through measures like time to task completion and intrinsically (i.e. out of context) using the word error rate and several variants thereof, which are based on smaller units. Extrinsic and intrinsic results did not correlate, neither for word error rate nor for metrics based on characters, syllables or phonemes.

AB - Automatic evaluation metrics should correlate with human judgement. We collected sixteen ASR mediated dialogues using a map task scenario. The material was assessed extrinsically (i.e. in context) through measures like time to task completion and intrinsically (i.e. out of context) using the word error rate and several variants thereof, which are based on smaller units. Extrinsic and intrinsic results did not correlate, neither for word error rate nor for metrics based on characters, syllables or phonemes.

U2 - 10.1007/978-3-319-10888-9_39

DO - 10.1007/978-3-319-10888-9_39

M3 - Other chapter contribution

SN - 978-3-319-10887-2

VL - 8686

T3 - Lecture Notes in Computer Science

SP - 392

EP - 399

BT - Advances in Natural Language Processing

A2 - Przepiórkowski, Adam

A2 - Ogrodniczuk, Maciej

PB - Springer

ER -