DNN Multimodal Fusion Techniques for Predicting Video Sentiment

Jennifer Williams; Ramona Comanescu; Oana Radu; Leimin Tian

DNN Multimodal Fusion Techniques for Predicting Video Sentiment

Jennifer Williams, Ramona Comanescu, Oana Radu, Leimin Tian

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

We present our work on sentiment prediction using the benchmark MOSI dataset from the CMU-MultimodalDataSDK. Previous work on multimodal sentiment analysis have been focused on input-level feature fusion or decision-level fusion for multimodal fusion. Here, we propose an intermediate-level feature fusion, which merges weights from each modality (audio, video, and text) during training with subsequent
additional training. Moreover, we tested principle component analysis (PCA) for feature selection. We found that applying PCA increases unimodal performance, and multimodal fusion outperforms unimodal models. Our experiments show that our proposed intermediate-level feature fusion outperforms other fusion techniques, and it achieves the best performance with an overall binary accuracy of 74.0% on video+text modalities. Our work also improves feature selection for unimodal sentiment analysis, while proposing a novel and effective multimodal fusion architecture for this task.

Original language	English
Title of host publication	Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML)
Publisher	Association for Computational Linguistics
Pages	64-72
Number of pages	9
Publication status	Published - Jul 2018
Event	Grand Challenge and Workshop on Human Multimodal Language - Melbourne, Australia Duration: 20 Jul 2018 → 20 Jul 2018 http://multicomp.cs.cmu.edu/acl2018multimodalchallenge/

Conference

Conference	Grand Challenge and Workshop on Human Multimodal Language
Abbreviated title	ACL 2018
Country/Territory	Australia
City	Melbourne
Period	20/07/18 → 20/07/18
Internet address	http://multicomp.cs.cmu.edu/acl2018multimodalchallenge/

Access to Document

Williams_et_al_2018a_sentimentAccepted author manuscript, 198 KB

Cite this

@inproceedings{a5bdb2e3d364425987b7d559d97dd61c,

title = "DNN Multimodal Fusion Techniques for Predicting Video Sentiment",

abstract = "We present our work on sentiment prediction using the benchmark MOSI dataset from the CMU-MultimodalDataSDK. Previous work on multimodal sentiment analysis have been focused on input-level feature fusion or decision-level fusion for multimodal fusion. Here, we propose an intermediate-level feature fusion, which merges weights from each modality (audio, video, and text) during training with subsequent additional training. Moreover, we tested principle component analysis (PCA) for feature selection. We found that applying PCA increases unimodal performance, and multimodal fusion outperforms unimodal models. Our experiments show that our proposed intermediate-level feature fusion outperforms other fusion techniques, and it achieves the best performance with an overall binary accuracy of 74.0% on video+text modalities. Our work also improves feature selection for unimodal sentiment analysis, while proposing a novel and effective multimodal fusion architecture for this task.",

author = "Jennifer Williams and Ramona Comanescu and Oana Radu and Leimin Tian",

year = "2018",

month = jul,

language = "English",

pages = "64--72",

booktitle = "Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML)",

publisher = "Association for Computational Linguistics",

note = "Grand Challenge and Workshop on Human Multimodal Language , ACL 2018 ; Conference date: 20-07-2018 Through 20-07-2018",

url = "http://multicomp.cs.cmu.edu/acl2018multimodalchallenge/",

}

Williams, J, Comanescu, R, Radu, O & Tian, L 2018, DNN Multimodal Fusion Techniques for Predicting Video Sentiment. in Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML). Association for Computational Linguistics, pp. 64-72, Grand Challenge and Workshop on Human Multimodal Language , Melbourne, Victoria, Australia, 20/07/18. <http://aclweb.org/anthology/W18-3309>

TY - GEN

T1 - DNN Multimodal Fusion Techniques for Predicting Video Sentiment

AU - Williams, Jennifer

AU - Comanescu, Ramona

AU - Radu, Oana

AU - Tian, Leimin

PY - 2018/7

Y1 - 2018/7

N2 - We present our work on sentiment prediction using the benchmark MOSI dataset from the CMU-MultimodalDataSDK. Previous work on multimodal sentiment analysis have been focused on input-level feature fusion or decision-level fusion for multimodal fusion. Here, we propose an intermediate-level feature fusion, which merges weights from each modality (audio, video, and text) during training with subsequent additional training. Moreover, we tested principle component analysis (PCA) for feature selection. We found that applying PCA increases unimodal performance, and multimodal fusion outperforms unimodal models. Our experiments show that our proposed intermediate-level feature fusion outperforms other fusion techniques, and it achieves the best performance with an overall binary accuracy of 74.0% on video+text modalities. Our work also improves feature selection for unimodal sentiment analysis, while proposing a novel and effective multimodal fusion architecture for this task.

AB - We present our work on sentiment prediction using the benchmark MOSI dataset from the CMU-MultimodalDataSDK. Previous work on multimodal sentiment analysis have been focused on input-level feature fusion or decision-level fusion for multimodal fusion. Here, we propose an intermediate-level feature fusion, which merges weights from each modality (audio, video, and text) during training with subsequent additional training. Moreover, we tested principle component analysis (PCA) for feature selection. We found that applying PCA increases unimodal performance, and multimodal fusion outperforms unimodal models. Our experiments show that our proposed intermediate-level feature fusion outperforms other fusion techniques, and it achieves the best performance with an overall binary accuracy of 74.0% on video+text modalities. Our work also improves feature selection for unimodal sentiment analysis, while proposing a novel and effective multimodal fusion architecture for this task.

M3 - Conference contribution

SP - 64

EP - 72

BT - Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML)

PB - Association for Computational Linguistics

T2 - Grand Challenge and Workshop on Human Multimodal Language

Y2 - 20 July 2018 through 20 July 2018

ER -

DNN Multimodal Fusion Techniques for Predicting Video Sentiment

Abstract / Description of output

Conference

Access to Document

Fingerprint

Cite this