Detecting and grounding important characters in visual stories

Danyang Liu, Frank Keller

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Characters are essential to the plot of any story. Establishing the characters before writing a story can improve the clarity of the plot and the overall flow of the narrative. However, previous work on visual storytelling tends to focus on detecting objects in images and discovering relationships between them. In this approach, characters are not distinguished from other objects when they are fed into the generation pipeline. The result is a coherent sequence of events rather than a character-centric story. In order to address this limitation, we introduce the VIST-Character dataset, which provides rich character-centric annotations, including visual and textual co-reference chains and importance ratings for characters. Based on this dataset, we propose two new tasks: important character detection and character grounding in visual stories. For both tasks, we develop simple, unsupervised models based on distributional similarity and pre-trained vision-and-language models. Our new dataset, together with these models, can serve as the foundation for subsequent work on analysing and generating stories from a character-centric perspective.
Original languageEnglish
Title of host publicationProceedings of the 37th AAAI Conference on Artificial Intelligence
PublisherAAAI Press
Number of pages12
Publication statusAccepted/In press - 21 Nov 2022
Event37th AAAI Conference on Artificial Intelligence - Washington, DC, United States
Duration: 7 Feb 202314 Feb 2023

Publication series

NameProceedings of the AAAI Conference on Artificial Intelligence.
PublisherAAAI Press
ISSN (Electronic)2374-3468


Conference37th AAAI Conference on Artificial Intelligence
Abbreviated titleAAAI
Country/TerritoryUnited States
CityWashington, DC
Internet address


Dive into the research topics of 'Detecting and grounding important characters in visual stories'. Together they form a unique fingerprint.

Cite this