3D Lip Event Detection via Interframe Motion Divergence at Multiple Temporal Resolutions

Jie Zhang; Robert B Fisher

doi:10.1109/3DV53792.2021.00052

3D Lip Event Detection via Interframe Motion Divergence at Multiple Temporal Resolutions

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

The lip is a dominant dynamic facial unit when a person is speaking. Detecting lip events is beneficial to speech analysis and support for the hearing impaired. This paper proposes a 3D lip event detection pipeline that automatically determines the lip events from a 3D speaking lip sequence. We define a motion divergence measure using 3D lip landmarks to quantify the interframe dynamics of a 3D speaking lip. Then, we cast the interframe motion detection in a multi-temporal-resolution framework that allows the detection to be applicable to different speaking speeds. The experiments on the S3DFM Dataset investigate the overall 3D lip dynamics based on the proposed motion divergence. The proposed 3D pipeline is able to detect opening and closing lip events across 100 sequences, achieving a state-of-the-art performance.

Original language	English
Title of host publication	Proceedings of the 9th International Conference on 3D Vision (3DV 2021)
Pages	423-431
Number of pages	9
ISBN (Electronic)	978-1-6654-2688-6
DOIs	https://doi.org/10.1109/3DV53792.2021.00052
Publication status	Published - 6 Jan 2022
Event	9th International Conference on 3D Vision - Virtual, London, United Kingdom Duration: 1 Dec 2021 → 3 Dec 2021 https://3dv2021.surrey.ac.uk/

Publication series

Name	2021 International Conference on 3D Vision (3DV)
Publisher	IEEE
ISSN (Print)	2378-3826
ISSN (Electronic)	2475-7888

Conference

Conference	9th International Conference on 3D Vision
Abbreviated title	3DV 2021
Country/Territory	United Kingdom
City	London
Period	1/12/21 → 3/12/21
Internet address	https://3dv2021.surrey.ac.uk/

Access to Document

10.1109/3DV53792.2021.00052Licence: All Rights Reserved

3D Lip Event_ZHANG_DOA08102021_AFVAccepted author manuscript, 2.32 MBLicence: All Rights Reserved

https://ieeexplore.ieee.org/document/9665851Licence: All Rights Reserved

Cite this

@inproceedings{449b23cf4d944c859c024326fd4a4a7b,

title = "3D Lip Event Detection via Interframe Motion Divergence at Multiple Temporal Resolutions",

abstract = "The lip is a dominant dynamic facial unit when a person is speaking. Detecting lip events is beneficial to speech analysis and support for the hearing impaired. This paper proposes a 3D lip event detection pipeline that automatically determines the lip events from a 3D speaking lip sequence. We define a motion divergence measure using 3D lip landmarks to quantify the interframe dynamics of a 3D speaking lip. Then, we cast the interframe motion detection in a multi-temporal-resolution framework that allows the detection to be applicable to different speaking speeds. The experiments on the S3DFM Dataset investigate the overall 3D lip dynamics based on the proposed motion divergence. The proposed 3D pipeline is able to detect opening and closing lip events across 100 sequences, achieving a state-of-the-art performance.",

author = "Jie Zhang and Fisher, {Robert B}",

note = "Funding Information: The work is supported by Natural Science Foundation of China (NSFC) under Grant No.61906004. Publisher Copyright: {\textcopyright} 2021 IEEE.; 9th International Conference on 3D Vision, 3DV 2021 ; Conference date: 01-12-2021 Through 03-12-2021",

year = "2022",

month = jan,

day = "6",

doi = "10.1109/3DV53792.2021.00052",

language = "English",

isbn = "978-1-6654-2689-3",

series = "2021 International Conference on 3D Vision (3DV)",

publisher = "IEEE",

pages = "423--431",

booktitle = "Proceedings of the 9th International Conference on 3D Vision (3DV 2021)",

url = "https://3dv2021.surrey.ac.uk/",

}

Zhang, J & Fisher, RB 2022, 3D Lip Event Detection via Interframe Motion Divergence at Multiple Temporal Resolutions. in Proceedings of the 9th International Conference on 3D Vision (3DV 2021). 2021 International Conference on 3D Vision (3DV), pp. 423-431, 9th International Conference on 3D Vision, London, United Kingdom, 1/12/21. https://doi.org/10.1109/3DV53792.2021.00052

TY - GEN

T1 - 3D Lip Event Detection via Interframe Motion Divergence at Multiple Temporal Resolutions

AU - Zhang, Jie

AU - Fisher, Robert B

PY - 2022/1/6

Y1 - 2022/1/6

N2 - The lip is a dominant dynamic facial unit when a person is speaking. Detecting lip events is beneficial to speech analysis and support for the hearing impaired. This paper proposes a 3D lip event detection pipeline that automatically determines the lip events from a 3D speaking lip sequence. We define a motion divergence measure using 3D lip landmarks to quantify the interframe dynamics of a 3D speaking lip. Then, we cast the interframe motion detection in a multi-temporal-resolution framework that allows the detection to be applicable to different speaking speeds. The experiments on the S3DFM Dataset investigate the overall 3D lip dynamics based on the proposed motion divergence. The proposed 3D pipeline is able to detect opening and closing lip events across 100 sequences, achieving a state-of-the-art performance.

AB - The lip is a dominant dynamic facial unit when a person is speaking. Detecting lip events is beneficial to speech analysis and support for the hearing impaired. This paper proposes a 3D lip event detection pipeline that automatically determines the lip events from a 3D speaking lip sequence. We define a motion divergence measure using 3D lip landmarks to quantify the interframe dynamics of a 3D speaking lip. Then, we cast the interframe motion detection in a multi-temporal-resolution framework that allows the detection to be applicable to different speaking speeds. The experiments on the S3DFM Dataset investigate the overall 3D lip dynamics based on the proposed motion divergence. The proposed 3D pipeline is able to detect opening and closing lip events across 100 sequences, achieving a state-of-the-art performance.

U2 - 10.1109/3DV53792.2021.00052

DO - 10.1109/3DV53792.2021.00052

M3 - Conference contribution

SN - 978-1-6654-2689-3

T3 - 2021 International Conference on 3D Vision (3DV)

SP - 423

EP - 431

BT - Proceedings of the 9th International Conference on 3D Vision (3DV 2021)

T2 - 9th International Conference on 3D Vision

Y2 - 1 December 2021 through 3 December 2021

ER -

3D Lip Event Detection via Interframe Motion Divergence at Multiple Temporal Resolutions

Abstract

Publication series

Conference

Access to Document

Fingerprint

Cite this