Deep Multi-Task Learning to Recognise Subtle Facial Expressions of Mental States

Guosheng Hu, Li Liu, Yang Yuan, Zehao Yu, Yang Hua, Zhihong Zhang, Fumin Shen, Ling Shao, Timothy Hospedales, Neil Robertson, Yongxin Yang

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Facial expression recognition is a topical task. However, very little research investigates subtle expression recognition, which is important for mental activity analysis, deception detection, etc. We address subtle expression recognition through convolutional neural networks (CNNs) by developing multi-task learning (MTL) methods to effectively leverage a side task: facial landmark detection. Existing MTL methods follow a design pattern of shared bottom CNN layers and taskspecific top layers. However, the sharing architecture is usually heuristically chosen, as it is difficult to decide which layers should be shared. Our approach is composed of (1) a novel MTL framework that automatically learns which layers to share through optimisation under tensor trace norm regularisation and (2) an invariant representation learning approach that allows the CNN to leverage tasks defined on disjoint datasets without suffering from dataset distribution shift. To advance subtle expression recognition, we contribute a Large-scale Subtle Emotions and Mental States in the Wild database (LSEMSW). LSEMSW includes a variety of cognitive states as well as basic emotions. It contains 176K images, manually annotated with 13 emotions, and thus provides the first subtle expression dataset large enough for training deep CNNs. Evaluations on LSEMSW and 300-W (landmark) databases show the effectiveness of the proposed methods. In addition, we investigate transferring knowledge learned from LSEMSW database to traditional (nonsubtle) expression recognition. We achieve very competitive performance on Oulu-Casia NIR&Vis and CK+ databases via transfer learning.
Original languageEnglish
Title of host publicationEuropean Conference on Computer Vision (ECCV 2018)
Place of PublicationMunich, Germany
PublisherSpringer, Cham
Number of pages17
ISBN (Electronic)978-3-030-01258-8
ISBN (Print)978-3-030-01257-1
Publication statusPublished - 6 Oct 2018
EventEuropean Conference on Computer Vision 2018 - Munich, Germany
Duration: 8 Sep 201814 Sep 2018

Publication series

Name Lecture Notes in Computer Science
PublisherSpringer, Cham
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349
NameImage Processing, Computer Vision, Pattern Recognition, and Graphics


ConferenceEuropean Conference on Computer Vision 2018
Abbreviated titleECCV 2018
Internet address


Dive into the research topics of 'Deep Multi-Task Learning to Recognise Subtle Facial Expressions of Mental States'. Together they form a unique fingerprint.

Cite this