Zero-Shot Visual Question Answering Using Knowledge Graph

Zhuo Chen, Jiaoyan Chen, Yuxia Geng, Jeff Z. Pan, Zonggang Yuan, Huajun Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Incorporating external knowledge to Visual Question Answering (VQA) has become a vital practical need. Existing methods mostly adopt pipeline approaches with different components for knowledge matching and extraction, feature learning, etc. However, such pipeline approaches suffer when some component does not perform well, which leads to error cascading and poor overall performance. Furthermore, the majority of existing approaches ignore the answer bias issue---many answers may have never appeared during training (i.e., unseen answers) in real-word application. To bridge these gaps, in this paper, we propose a Zero-shot VQA algorithm using knowledge graph and a mask-based learning mechanism for better incorporating external knowledge, and present new answer-based Zero-shot VQA splits for the F-VQA dataset. Experiments show that our method can achieve state-of-the-art performance in Zero-shot VQA with unseen answers, meanwhile dramatically augment existing end-to-end models on the normal F-VQA task.
Original languageEnglish
Title of host publicationThe Semantic Web -- ISWC 2021: 20th International Semantic Web Conference, ISWC 2021, Virtual Event, October 24–28, 2021, Proceedings
EditorsAndreas Hotho, Eva Blomqvist, Stefan Dietze, Achille Fokoue, Ying Ding, Payam Barnaghi, Armin Haller, Mauro Dragoni, Harith Alani
Place of PublicationCham
PublisherSpringer, Cham
Number of pages17
ISBN (Electronic)978-3-030-88361-4
ISBN (Print)978-3-030-88360-7
Publication statusPublished - 30 Sept 2021
EventThe 20th International Semantic Web Conference, 2021 - Online
Duration: 24 Oct 202128 Oct 2021
Conference number: 20

Publication series

NameLecture Notes in Computer Science
PublisherSpringer Cham
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


ConferenceThe 20th International Semantic Web Conference, 2021
Abbreviated titleISWC 2021
Internet address

Keywords / Materials (for Non-textual outputs)

  • Visual Question Answering
  • Zero-shot Learning
  • Knowledge Graph


Dive into the research topics of 'Zero-Shot Visual Question Answering Using Knowledge Graph'. Together they form a unique fingerprint.

Cite this