Are we using enough listeners? No! An empirically-supported critique of Interspeech 2014 TTS evaluations

Mirjam Wester; Cassia Valentini-Botinhao; Gustav Eje Henter

Are we using enough listeners? No! An empirically-supported critique of Interspeech 2014 TTS evaluations

Mirjam Wester, Cassia Valentini-Botinhao, Gustav Eje Henter

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Tallying the numbers of listeners that took part in subjective evaluations of synthetic speech at Interspeech 2014 showed that in more than 60% of papers conclusions are based on listening tests with less than 20 listeners. Our analysis of Blizzard 2013 data shows that for a MOS test measuring naturalness a stable level of significance is only reached when more than 30 listeners are used. In this paper, we set out a list of guidelines, i.e., a checklist for carrying out meaningful subjective evaluations. We further illustrate the importance of sentence coverage and number of listeners by presenting changes to rank order and number of significant pairs by re-analysing data from the Blizzard Challenge 2013.

Original language	English
Title of host publication	INTERSPEECH 2015
Place of Publication	Dresden
Publisher	International Speech Communication Association
Pages	3476-3480
Number of pages	5
ISBN (Print)	978-1-5108-1790-6
Publication status	Published - 10 Sep 2015
Event	Interspeech 2015 - Dresden, Germany Duration: 6 Sep 2015 → 9 Sep 2015

Publication series

Name	INTERSPEECH
Publisher	ISCA
ISSN (Electronic)	1990-9770

Conference

Conference	Interspeech 2015
Country/Territory	Germany
City	Dresden
Period	6/09/15 → 9/09/15

Access to Document

masterAccepted author manuscript, 721 KB
Are we using_WESTER_DOA01062015_VOR_All_RightsFinal published version, 666 KBLicence: All Rights Reserved

http://www.isca-speech.org/archive/interspeech_2015/i15_3476.htmlLicence: All Rights Reserved

Natural Speech Technology
Renals, S. & King, S.
EPSRC
1/05/11 → 31/07/16
Project: Research

Cassia Valentini Botinhao
- cvbotinhinf.ed.acuk
Person: Academic: Research Active

Cite this

@inproceedings{cf7b29ea3f77478596a32b5df768284c,

title = "Are we using enough listeners? No! An empirically-supported critique of Interspeech 2014 TTS evaluations",

abstract = "Tallying the numbers of listeners that took part in subjective evaluations of synthetic speech at Interspeech 2014 showed that in more than 60% of papers conclusions are based on listening tests with less than 20 listeners. Our analysis of Blizzard 2013 data shows that for a MOS test measuring naturalness a stable level of significance is only reached when more than 30 listeners are used. In this paper, we set out a list of guidelines, i.e., a checklist for carrying out meaningful subjective evaluations. We further illustrate the importance of sentence coverage and number of listeners by presenting changes to rank order and number of significant pairs by re-analysing data from the Blizzard Challenge 2013.",

author = "Mirjam Wester and Cassia Valentini-Botinhao and Henter, {Gustav Eje}",

year = "2015",

month = sep,

day = "10",

language = "English",

isbn = "978-1-5108-1790-6",

series = "INTERSPEECH",

publisher = "International Speech Communication Association",

pages = "3476--3480",

booktitle = "INTERSPEECH 2015",

note = "Interspeech 2015 ; Conference date: 06-09-2015 Through 09-09-2015",

}

Wester, M , Valentini-Botinhao, C & Henter, GE 2015, Are we using enough listeners? No! An empirically-supported critique of Interspeech 2014 TTS evaluations. in INTERSPEECH 2015. INTERSPEECH, International Speech Communication Association, Dresden, pp. 3476-3480, Interspeech 2015, Dresden, Germany, 6/09/15. <http://www.isca-speech.org/archive/interspeech_2015/i15_3476.html>

Are we using enough listeners? No! An empirically-supported critique of Interspeech 2014 TTS evaluations. / Wester, Mirjam ; Valentini-Botinhao, Cassia; Henter, Gustav Eje.

INTERSPEECH 2015. Dresden : International Speech Communication Association, 2015. p. 3476-3480 (INTERSPEECH).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Are we using enough listeners? No! An empirically-supported critique of Interspeech 2014 TTS evaluations

AU - Wester, Mirjam

AU - Valentini-Botinhao, Cassia

AU - Henter, Gustav Eje

PY - 2015/9/10

Y1 - 2015/9/10

N2 - Tallying the numbers of listeners that took part in subjective evaluations of synthetic speech at Interspeech 2014 showed that in more than 60% of papers conclusions are based on listening tests with less than 20 listeners. Our analysis of Blizzard 2013 data shows that for a MOS test measuring naturalness a stable level of significance is only reached when more than 30 listeners are used. In this paper, we set out a list of guidelines, i.e., a checklist for carrying out meaningful subjective evaluations. We further illustrate the importance of sentence coverage and number of listeners by presenting changes to rank order and number of significant pairs by re-analysing data from the Blizzard Challenge 2013.

AB - Tallying the numbers of listeners that took part in subjective evaluations of synthetic speech at Interspeech 2014 showed that in more than 60% of papers conclusions are based on listening tests with less than 20 listeners. Our analysis of Blizzard 2013 data shows that for a MOS test measuring naturalness a stable level of significance is only reached when more than 30 listeners are used. In this paper, we set out a list of guidelines, i.e., a checklist for carrying out meaningful subjective evaluations. We further illustrate the importance of sentence coverage and number of listeners by presenting changes to rank order and number of significant pairs by re-analysing data from the Blizzard Challenge 2013.

M3 - Conference contribution

SN - 978-1-5108-1790-6

T3 - INTERSPEECH

SP - 3476

EP - 3480

BT - INTERSPEECH 2015

PB - International Speech Communication Association

CY - Dresden

T2 - Interspeech 2015

Y2 - 6 September 2015 through 9 September 2015

ER -

Are we using enough listeners? No! An empirically-supported critique of Interspeech 2014 TTS evaluations

Abstract

Publication series

Conference

Access to Document

Fingerprint

Projects

Natural Speech Technology

Profiles

Cassia Valentini Botinhao

Cite this