A Spoofing Benchmark for the 2018 Voice Conversion Challenge: Leveraging from Spoofing Countermeasures for Speech Artifact Assessment

Tomi Kinnunen, Jaime Lorenzo-Trueba, Junichi Yamagishi, Tomoki Toda, Daisuke Saito, Fernando Villavicencio, Zhenhua Ling

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Voice conversion (VC) aims at conversion of speaker characteristic without altering content. Due to training data limitations and modeling imperfections, it is difficult to achieve believable speaker mimicry without introducing processing artifacts; performance assessment of VC, therefore, usually involves both speaker similarity and quality evaluation by a human panel. As a time-consuming, expensive, and non-reproducible process, it hinders rapid prototyping of new VC technology. We address artifact assessment using an alternative, objective approach leveraging from prior work on spoofing countermeasures (CMs) for automatic speaker verification. Therein, CMs are used for rejecting ‘fake’ inputs such as replayed, synthetic or converted speech but their potential for automatic speech artifact assessment remains unknown. This study serves to fill that gap. As a supplement to subjective results for the 2018 Voice Conversion Challenge (VCC’18) data, we configure a standard constant-Q cepstral coefficient CM to quantify the extent of processing artifacts. Equal error rate (EER) of the CM, a confusability index of VC samples with real human speech, serves as our artifact measure. Two clusters of VCC’18 entries are identified: low-quality ones with detectable artifacts (low EERs), and higher quality ones with less artifacts. None of the VCC’18 systems, however, is perfect: all EERs are< 30% (the ‘ideal’ value would be 50%). Our preliminary findings suggest potential of CMs outside of their original application, as a supplemental optimization and benchmarking tool to enhance VC technology.
Original languageEnglish
Title of host publicationSpeaker Odyssey 2018
Subtitle of host publicationThe Speaker and Language Recognition Workshop
Place of PublicationLes Sables d'Olonne, France
Number of pages8
Publication statusPublished - 29 Jun 2018
EventThe Speaker and Language Recognition Workshop - Les Sables d’Olonne, France
Duration: 26 Jun 201829 Jun 2018

Publication series

ISSN (Electronic)2312-2846


ConferenceThe Speaker and Language Recognition Workshop
Abbreviated titleSpeaker Odyssey 2018
CityLes Sables d’Olonne
Internet address


Dive into the research topics of 'A Spoofing Benchmark for the 2018 Voice Conversion Challenge: Leveraging from Spoofing Countermeasures for Speech Artifact Assessment'. Together they form a unique fingerprint.

Cite this