Silent speech recognition with articulator positions estimated from tongue ultrasound and lip video

Rachel Beeson; Korin Richmond

doi:10.21437/Interspeech.2023-1966

Silent speech recognition with articulator positions estimated from tongue ultrasound and lip video

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

We present a multi-speaker silent speech recognition system trained on articulator features derived from the Tongue and Lips corpus, a multi-speaker corpus of ultrasound tongue imaging and lip video data. We extracted articulator features using the pose estimation software DeepLabCut, then trained recognition models with these point-tracking features using Kaldi. We trained with voiced utterances, then tested performance on both voiced and silent utterances. Our multi-speaker SSR improved WER by 23.06% when compared to a previous similar multi-speaker SSR system which used image-based instead of point-tracking features. We also found great improvements (up to 15.45% decrease in WER) in recognition of silent speech using fMLLR adaptation compared to raw features. Finally, we investigated differences in articulator trajectories between voiced and silent speech and found that speakers tend to miss articulatory targets that are present in voiced speech when speaking silently.

Original language	English
Title of host publication	Proceedings of the Annual Conference of the International Speech Communication Association
Subtitle of host publication	Interspeech 2023
Editors	Naomi Harte, Julie Carson-Berndsen, Gareth Jones
Place of Publication	Dublin
Publisher	ISCA
Pages	1149-1153
DOIs	https://doi.org/10.21437/Interspeech.2023-1966
Publication status	Published - Sept 2023
Event	Interspeech 2023 - Dublin, Ireland Duration: 20 Aug 2023 → 24 Aug 2023 Conference number: 24 https://www.interspeech2023.org/

Publication series

Name	Interspeech - Annual Conference of the International Speech Communication Association
Publisher	ISCA
ISSN (Electronic)	2308-457X

Conference

Conference	Interspeech 2023
Country/Territory	Ireland
City	Dublin
Period	20/08/23 → 24/08/23
Internet address	https://www.interspeech2023.org/

Keywords / Materials (for Non-textual outputs)

silent speech interfaces
silent speech recognition
articulator pose estimation
ultrasound imaging
lip reading

Access to Document

10.21437/Interspeech.2023-1966

Beeson&Richmond2023SilentSpeechRecognitionAccepted author manuscript, 431 KBLicence: Creative Commons: Attribution (CC-BY)

UltraSuite Repository - sample data
Eshky, A. (Creator), Ribeiro, M. S. (Creator), Cleland, J. (Creator), Renals, S. (Creator), Richmond, K. (Creator), Roxburgh, Z. (Creator), Scobbie, J. (Creator) & Wrench, A. (Creator), Edinburgh DataShare, 11 Feb 2019
DOI: 10.7488/ds/2495, https://doi.org/10.21437/Interspeech.2018-1736
Dataset

Cite this

Beeson, R., & Richmond, K. (2023). Silent speech recognition with articulator positions estimated from tongue ultrasound and lip video. In N. Harte, J. Carson-Berndsen, & G. Jones (Eds.), Proceedings of the Annual Conference of the International Speech Communication Association: Interspeech 2023 (pp. 1149-1153). (Interspeech - Annual Conference of the International Speech Communication Association). ISCA. https://doi.org/10.21437/Interspeech.2023-1966

Beeson, Rachel ; Richmond, Korin. / Silent speech recognition with articulator positions estimated from tongue ultrasound and lip video. Proceedings of the Annual Conference of the International Speech Communication Association: Interspeech 2023. editor / Naomi Harte ; Julie Carson-Berndsen ; Gareth Jones. Dublin : ISCA, 2023. pp. 1149-1153 (Interspeech - Annual Conference of the International Speech Communication Association).

@inproceedings{f3122927cf214c8c9efd31c13b49b81e,

title = "Silent speech recognition with articulator positions estimated from tongue ultrasound and lip video",

abstract = "We present a multi-speaker silent speech recognition system trained on articulator features derived from the Tongue and Lips corpus, a multi-speaker corpus of ultrasound tongue imaging and lip video data. We extracted articulator features using the pose estimation software DeepLabCut, then trained recognition models with these point-tracking features using Kaldi. We trained with voiced utterances, then tested performance on both voiced and silent utterances. Our multi-speaker SSR improved WER by 23.06% when compared to a previous similar multi-speaker SSR system which used image-based instead of point-tracking features. We also found great improvements (up to 15.45% decrease in WER) in recognition of silent speech using fMLLR adaptation compared to raw features. Finally, we investigated differences in articulator trajectories between voiced and silent speech and found that speakers tend to miss articulatory targets that are present in voiced speech when speaking silently.",

keywords = "silent speech interfaces, silent speech recognition, articulator pose estimation, ultrasound imaging, lip reading",

author = "Rachel Beeson and Korin Richmond",

note = "/; Interspeech 2023 ; Conference date: 20-08-2023 Through 24-08-2023",

year = "2023",

month = sep,

doi = "10.21437/Interspeech.2023-1966",

language = "English",

series = "Interspeech - Annual Conference of the International Speech Communication Association",

publisher = "ISCA",

pages = "1149--1153",

editor = "Naomi Harte and Julie Carson-Berndsen and Gareth Jones",

booktitle = "Proceedings of the Annual Conference of the International Speech Communication Association",

url = "https://www.interspeech2023.org/",

}

Beeson, R & Richmond, K 2023, Silent speech recognition with articulator positions estimated from tongue ultrasound and lip video. in N Harte, J Carson-Berndsen & G Jones (eds), Proceedings of the Annual Conference of the International Speech Communication Association: Interspeech 2023. Interspeech - Annual Conference of the International Speech Communication Association, ISCA, Dublin, pp. 1149-1153, Interspeech 2023, Dublin, Ireland, 20/08/23. https://doi.org/10.21437/Interspeech.2023-1966

Silent speech recognition with articulator positions estimated from tongue ultrasound and lip video. / Beeson, Rachel; Richmond, Korin.
Proceedings of the Annual Conference of the International Speech Communication Association: Interspeech 2023. ed. / Naomi Harte; Julie Carson-Berndsen; Gareth Jones. Dublin: ISCA, 2023. p. 1149-1153 (Interspeech - Annual Conference of the International Speech Communication Association).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Silent speech recognition with articulator positions estimated from tongue ultrasound and lip video

AU - Beeson, Rachel

AU - Richmond, Korin

N1 - Conference code: 24

PY - 2023/9

Y1 - 2023/9

N2 - We present a multi-speaker silent speech recognition system trained on articulator features derived from the Tongue and Lips corpus, a multi-speaker corpus of ultrasound tongue imaging and lip video data. We extracted articulator features using the pose estimation software DeepLabCut, then trained recognition models with these point-tracking features using Kaldi. We trained with voiced utterances, then tested performance on both voiced and silent utterances. Our multi-speaker SSR improved WER by 23.06% when compared to a previous similar multi-speaker SSR system which used image-based instead of point-tracking features. We also found great improvements (up to 15.45% decrease in WER) in recognition of silent speech using fMLLR adaptation compared to raw features. Finally, we investigated differences in articulator trajectories between voiced and silent speech and found that speakers tend to miss articulatory targets that are present in voiced speech when speaking silently.

AB - We present a multi-speaker silent speech recognition system trained on articulator features derived from the Tongue and Lips corpus, a multi-speaker corpus of ultrasound tongue imaging and lip video data. We extracted articulator features using the pose estimation software DeepLabCut, then trained recognition models with these point-tracking features using Kaldi. We trained with voiced utterances, then tested performance on both voiced and silent utterances. Our multi-speaker SSR improved WER by 23.06% when compared to a previous similar multi-speaker SSR system which used image-based instead of point-tracking features. We also found great improvements (up to 15.45% decrease in WER) in recognition of silent speech using fMLLR adaptation compared to raw features. Finally, we investigated differences in articulator trajectories between voiced and silent speech and found that speakers tend to miss articulatory targets that are present in voiced speech when speaking silently.

KW - silent speech interfaces

KW - silent speech recognition

KW - articulator pose estimation

KW - ultrasound imaging

KW - lip reading

U2 - 10.21437/Interspeech.2023-1966

DO - 10.21437/Interspeech.2023-1966

M3 - Conference contribution

T3 - Interspeech - Annual Conference of the International Speech Communication Association

SP - 1149

EP - 1153

BT - Proceedings of the Annual Conference of the International Speech Communication Association

A2 - Harte, Naomi

A2 - Carson-Berndsen, Julie

A2 - Jones, Gareth

PB - ISCA

CY - Dublin

T2 - Interspeech 2023

Y2 - 20 August 2023 through 24 August 2023

ER -

Beeson R, Richmond K. Silent speech recognition with articulator positions estimated from tongue ultrasound and lip video. In Harte N, Carson-Berndsen J, Jones G, editors, Proceedings of the Annual Conference of the International Speech Communication Association: Interspeech 2023. Dublin: ISCA. 2023. p. 1149-1153. (Interspeech - Annual Conference of the International Speech Communication Association). Epub 2023 Aug 24. doi: 10.21437/Interspeech.2023-1966

Silent speech recognition with articulator positions estimated from tongue ultrasound and lip video

Abstract / Description of output

Publication series

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Datasets

UltraSuite Repository - sample data

Cite this