AVSE Challenge: Audio-Visual Speech Enhancement Challenge

Andrea Lorena Aldana Blanco; Cassia Valentini Botinhao; Ondrej Klejch; Mandar Gogate; Kia Dashtipour; Amir Hussain; Peter Bell

doi:10.1109/SLT54892.2023.10023284

AVSE Challenge: Audio-Visual Speech Enhancement Challenge

Andrea Lorena Aldana Blanco, Cassia Valentini Botinhao, Ondrej Klejch, Mandar Gogate, Kia Dashtipour, Amir Hussain, Peter Bell

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

Audio-visual speech enhancement is the task of improving the quality of a speech signal when video of the speaker is available. It opens-up the opportunity of improving speech intelligibility in adverse listening scenarios that are currently too challenging for audio-only speech enhancement models. The Audio-Visual Speech Enhancement (AVSE) challenge aims to set the first benchmark in this area. We provide participants with datasets and scripts to test their audio-visual speech enhancement models under a common framework for both training and evaluation. The data is derived from real-world videos, and comprises noisy mixes, in which audio from target speaker is mixed with either a competing speaker or a noise signal. The submitted systems are evaluated by conducting AV intelligibility tests involving human participants. We expect this challenge to be a platform for advancing the field of audio-visual speech-enhancement and to provide further insight about the scope and limitations of current AV speech enhancement approaches.

Original language	English
Title of host publication	Proceedings of the 2022 IEEE Spoken Language Technology Workshop
Publisher	Institute of Electrical and Electronics Engineers
Pages	465-471
Number of pages	7
ISBN (Electronic)	979-8-3503-9690-4, 979-8-3503-9689-8
ISBN (Print)	979-8-3503-9691-1
DOIs	https://doi.org/10.1109/SLT54892.2023.10023284
Publication status	Published - 27 Jan 2023
Event	The IEEE Spoken Language Technology Workshop, 2022 - Doha, Qatar Duration: 9 Jan 2023 → 12 Jan 2023 https://slt2022.org/

Workshop

Workshop	The IEEE Spoken Language Technology Workshop, 2022
Abbreviated title	SLT 2022
Country/Territory	Qatar
City	Doha
Period	9/01/23 → 12/01/23
Internet address	https://slt2022.org/

Keywords / Materials (for Non-textual outputs)

Audio-visual speech enhancement
subjective intelligibility
LRS3 dataset

Access to Document

10.1109/SLT54892.2023.10023284Licence: All Rights Reserved

AVSE Challenge_ALDANA_DOA30092022_AFVAccepted author manuscript, 1.46 MBLicence: Creative Commons: Attribution (CC-BY)

https://ieeexplore.ieee.org/document/10023284Licence: All Rights Reserved

Cite this

@inproceedings{9aa045cc739e445b8d69203e1b4f2eef,

title = "AVSE Challenge: Audio-Visual Speech Enhancement Challenge",

abstract = "Audio-visual speech enhancement is the task of improving the quality of a speech signal when video of the speaker is available. It opens-up the opportunity of improving speech intelligibility in adverse listening scenarios that are currently too challenging for audio-only speech enhancement models. The Audio-Visual Speech Enhancement (AVSE) challenge aims to set the first benchmark in this area. We provide participants with datasets and scripts to test their audio-visual speech enhancement models under a common framework for both training and evaluation. The data is derived from real-world videos, and comprises noisy mixes, in which audio from target speaker is mixed with either a competing speaker or a noise signal. The submitted systems are evaluated by conducting AV intelligibility tests involving human participants. We expect this challenge to be a platform for advancing the field of audio-visual speech-enhancement and to provide further insight about the scope and limitations of current AV speech enhancement approaches.",

keywords = "Audio-visual speech enhancement, subjective intelligibility, LRS3 dataset",

author = "{Aldana Blanco}, {Andrea Lorena} and {Valentini Botinhao}, Cassia and Ondrej Klejch and Mandar Gogate and Kia Dashtipour and Amir Hussain and Peter Bell",

note = "Funding Information: Acknowledgements This work is supported by the UK Engineering and Physical Sciences Research Council (EPSRC) programme grant: COG-MHEAR (EP/T021063/1). For the purpose of open access, the author has applied a Creative Commons Attribution (CC BY) licence to any Author Accepted Manuscript version arising from this submission. Publisher Copyright: {\textcopyright} 2023 IEEE.; The IEEE Spoken Language Technology Workshop, 2022, SLT 2022 ; Conference date: 09-01-2023 Through 12-01-2023",

year = "2023",

month = jan,

day = "27",

doi = "10.1109/SLT54892.2023.10023284",

language = "English",

isbn = "979-8-3503-9691-1",

pages = "465--471",

booktitle = "Proceedings of the 2022 IEEE Spoken Language Technology Workshop",

publisher = "Institute of Electrical and Electronics Engineers",

url = "https://slt2022.org/",

}

Aldana Blanco, AL , Valentini Botinhao, C , Klejch, O, Gogate, M, Dashtipour, K, Hussain, A & Bell, P 2023, AVSE Challenge: Audio-Visual Speech Enhancement Challenge. in Proceedings of the 2022 IEEE Spoken Language Technology Workshop. Institute of Electrical and Electronics Engineers, pp. 465-471, The IEEE Spoken Language Technology Workshop, 2022, Doha, Qatar, 9/01/23. https://doi.org/10.1109/SLT54892.2023.10023284

TY - GEN

T1 - AVSE Challenge: Audio-Visual Speech Enhancement Challenge

AU - Aldana Blanco, Andrea Lorena

AU - Valentini Botinhao, Cassia

AU - Klejch, Ondrej

AU - Gogate, Mandar

AU - Dashtipour, Kia

AU - Hussain, Amir

AU - Bell, Peter

N1 - Funding Information: Acknowledgements This work is supported by the UK Engineering and Physical Sciences Research Council (EPSRC) programme grant: COG-MHEAR (EP/T021063/1). For the purpose of open access, the author has applied a Creative Commons Attribution (CC BY) licence to any Author Accepted Manuscript version arising from this submission. Publisher Copyright: © 2023 IEEE.

PY - 2023/1/27

Y1 - 2023/1/27

N2 - Audio-visual speech enhancement is the task of improving the quality of a speech signal when video of the speaker is available. It opens-up the opportunity of improving speech intelligibility in adverse listening scenarios that are currently too challenging for audio-only speech enhancement models. The Audio-Visual Speech Enhancement (AVSE) challenge aims to set the first benchmark in this area. We provide participants with datasets and scripts to test their audio-visual speech enhancement models under a common framework for both training and evaluation. The data is derived from real-world videos, and comprises noisy mixes, in which audio from target speaker is mixed with either a competing speaker or a noise signal. The submitted systems are evaluated by conducting AV intelligibility tests involving human participants. We expect this challenge to be a platform for advancing the field of audio-visual speech-enhancement and to provide further insight about the scope and limitations of current AV speech enhancement approaches.

AB - Audio-visual speech enhancement is the task of improving the quality of a speech signal when video of the speaker is available. It opens-up the opportunity of improving speech intelligibility in adverse listening scenarios that are currently too challenging for audio-only speech enhancement models. The Audio-Visual Speech Enhancement (AVSE) challenge aims to set the first benchmark in this area. We provide participants with datasets and scripts to test their audio-visual speech enhancement models under a common framework for both training and evaluation. The data is derived from real-world videos, and comprises noisy mixes, in which audio from target speaker is mixed with either a competing speaker or a noise signal. The submitted systems are evaluated by conducting AV intelligibility tests involving human participants. We expect this challenge to be a platform for advancing the field of audio-visual speech-enhancement and to provide further insight about the scope and limitations of current AV speech enhancement approaches.

KW - Audio-visual speech enhancement

KW - subjective intelligibility

KW - LRS3 dataset

U2 - 10.1109/SLT54892.2023.10023284

DO - 10.1109/SLT54892.2023.10023284

M3 - Conference contribution

SN - 979-8-3503-9691-1

SP - 465

EP - 471

BT - Proceedings of the 2022 IEEE Spoken Language Technology Workshop

PB - Institute of Electrical and Electronics Engineers

T2 - The IEEE Spoken Language Technology Workshop, 2022

Y2 - 9 January 2023 through 12 January 2023

ER -

AVSE Challenge: Audio-Visual Speech Enhancement Challenge

Abstract / Description of output

Workshop

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Cite this