Multi-reference WER for evaluating ASR for languages with no orthographic rule

Ahmed Ali; Walid Magdy; Peter Bell; Steve Renals

doi:10.1109/ASRU.2015.7404847

Multi-reference WER for evaluating ASR for languages with no orthographic rule

Ahmed Ali, Walid Magdy, Peter Bell, Steve Renals

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

Languages with no standard orthographic representation faces a challenge to evaluate the output from Automatic Speech Recognition (ASR). Since the reference transcription text can vary widely from one user to another. We propose an innovative approach for evaluating speech recognition using Multi-References. For each recognized speech segments, we ask five different users to transcribe the speech. We combine the alignment for the multiple references, and use the combined alignment to report a modified version of Word Error Rate (WER). This approach is in favor of accepting a recognized word if any of the references typed it in the same form. Results are reported using two Dialectal Arabic (DA) as a language with no standard orthographic; Egyptian, and North African speech. The average WER for the five references individually is 71.4%, and 80.1% respectively. When considering all references combined, the Multi-References MR-WER was found to be 39.7%, and 45.9% respectively.

Original language	English
Title of host publication	Proceedings of 2015 IEEE Automatic Speech Recognition and Understanding Workshop
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	576-580
Number of pages	5
ISBN (Print)	978-1-4799-7291-3
DOIs	https://doi.org/10.1109/ASRU.2015.7404847
Publication status	Published - 13 Dec 2015

Access to Document

10.1109/ASRU.2015.7404847

ahmed-asru2015Accepted author manuscript, 201 KBLicence: Other

http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7404847

Cite this

@inproceedings{4c65201222094eb8be5566b47525d5a7,

title = "Multi-reference WER for evaluating ASR for languages with no orthographic rule",

abstract = "Languages with no standard orthographic representation faces a challenge to evaluate the output from Automatic Speech Recognition (ASR). Since the reference transcription text can vary widely from one user to another. We propose an innovative approach for evaluating speech recognition using Multi-References. For each recognized speech segments, we ask five different users to transcribe the speech. We combine the alignment for the multiple references, and use the combined alignment to report a modified version of Word Error Rate (WER). This approach is in favor of accepting a recognized word if any of the references typed it in the same form. Results are reported using two Dialectal Arabic (DA) as a language with no standard orthographic; Egyptian, and North African speech. The average WER for the five references individually is 71.4%, and 80.1% respectively. When considering all references combined, the Multi-References MR-WER was found to be 39.7%, and 45.9% respectively.",

author = "Ahmed Ali and Walid Magdy and Peter Bell and Steve Renals",

year = "2015",

month = dec,

day = "13",

doi = "10.1109/ASRU.2015.7404847",

language = "English",

isbn = "978-1-4799-7291-3",

pages = "576--580",

booktitle = "Proceedings of 2015 IEEE Automatic Speech Recognition and Understanding Workshop",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

address = "United States",

}

Multi-reference WER for evaluating ASR for languages with no orthographic rule. / Ali, Ahmed; Magdy, Walid ; Bell, Peter et al.
Proceedings of 2015 IEEE Automatic Speech Recognition and Understanding Workshop. Institute of Electrical and Electronics Engineers (IEEE), 2015. p. 576-580.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Multi-reference WER for evaluating ASR for languages with no orthographic rule

AU - Ali, Ahmed

AU - Magdy, Walid

AU - Bell, Peter

AU - Renals, Steve

PY - 2015/12/13

Y1 - 2015/12/13

N2 - Languages with no standard orthographic representation faces a challenge to evaluate the output from Automatic Speech Recognition (ASR). Since the reference transcription text can vary widely from one user to another. We propose an innovative approach for evaluating speech recognition using Multi-References. For each recognized speech segments, we ask five different users to transcribe the speech. We combine the alignment for the multiple references, and use the combined alignment to report a modified version of Word Error Rate (WER). This approach is in favor of accepting a recognized word if any of the references typed it in the same form. Results are reported using two Dialectal Arabic (DA) as a language with no standard orthographic; Egyptian, and North African speech. The average WER for the five references individually is 71.4%, and 80.1% respectively. When considering all references combined, the Multi-References MR-WER was found to be 39.7%, and 45.9% respectively.

AB - Languages with no standard orthographic representation faces a challenge to evaluate the output from Automatic Speech Recognition (ASR). Since the reference transcription text can vary widely from one user to another. We propose an innovative approach for evaluating speech recognition using Multi-References. For each recognized speech segments, we ask five different users to transcribe the speech. We combine the alignment for the multiple references, and use the combined alignment to report a modified version of Word Error Rate (WER). This approach is in favor of accepting a recognized word if any of the references typed it in the same form. Results are reported using two Dialectal Arabic (DA) as a language with no standard orthographic; Egyptian, and North African speech. The average WER for the five references individually is 71.4%, and 80.1% respectively. When considering all references combined, the Multi-References MR-WER was found to be 39.7%, and 45.9% respectively.

U2 - 10.1109/ASRU.2015.7404847

DO - 10.1109/ASRU.2015.7404847

M3 - Conference contribution

SN - 978-1-4799-7291-3

SP - 576

EP - 580

BT - Proceedings of 2015 IEEE Automatic Speech Recognition and Understanding Workshop

PB - Institute of Electrical and Electronics Engineers (IEEE)

ER -

Multi-reference WER for evaluating ASR for languages with no orthographic rule

Abstract / Description of output

Access to Document

Fingerprint

Cite this