Who are you referring to?: Coreference resolution in image narrations

Arushi Goel; Basura Fernando; Frank Keller; Hakan Bilen

doi:10.1109/ICCV51070.2023.01399

Who are you referring to? Coreference resolution in image narrations

Arushi Goel, Basura Fernando, Frank Keller, Hakan Bilen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Coreference resolution aims to identify words and phrases which refer to the same entity in a text, a core task in natural language processing. In this paper, we extend this task to resolving coreferences in long-form narrations of visual scenes. First, we introduce a new dataset with annotated coreference chains and their bounding boxes, as most existing image-text datasets only contain short sentences without coreferring expressions or labeled chains. We propose a new technique that learns to identify coref-erence chains using weak supervision, only from image-text pairs and a regularization using prior linguistic knowledge. Our model yields large performance gains over several strong baselines in resolving coreferences. We also show that coreference resolution helps improve grounding narratives in images.

Original language	English
Title of host publication	2023 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher	Institute of Electrical and Electronics Engineers
Pages	15201-15212
Number of pages	12
ISBN (Electronic)	979-8-3503-0718-4
ISBN (Print)	979-8-3503-0719-1
DOIs	https://doi.org/10.1109/ICCV51070.2023.01399
Publication status	Published - 15 Jan 2024
Event	International Conference on Computer Vision 2023 - Paris, France Duration: 2 Oct 2023 → 6 Oct 2023 https://iccv2023.thecvf.com/

Publication series

Name	Proceedings of the IEEE International Conference on Computer Vision
Publisher	IEEE
ISSN (Print)	1550-5499
ISSN (Electronic)	2380-7504

Conference

Conference	International Conference on Computer Vision 2023
Abbreviated title	ICCV 2023
Country/Territory	France
City	Paris
Period	2/10/23 → 6/10/23
Internet address	https://iccv2023.thecvf.com/

Keywords / Materials (for Non-textual outputs)

cs.CV
cs.CL

Access to Document

10.1109/ICCV51070.2023.01399Licence: All Rights Reserved

2211.14563v2Accepted author manuscript, 3.44 MBLicence: Creative Commons: Attribution (CC-BY)

https://arxiv.org/abs/2211.14563Licence: Creative Commons: Attribution (CC-BY)
https://ieeexplore.ieee.org/document/10378027Licence: All Rights Reserved

Cite this

Goel, A., Fernando, B., Keller, F., & Bilen, H. (2024). Who are you referring to? Coreference resolution in image narrations. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 15201-15212). (Proceedings of the IEEE International Conference on Computer Vision). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICCV51070.2023.01399

@inproceedings{8858abcb84bb49ccbdf00d1dcb53d89e,

title = "Who are you referring to?: Coreference resolution in image narrations",

abstract = "Coreference resolution aims to identify words and phrases which refer to the same entity in a text, a core task in natural language processing. In this paper, we extend this task to resolving coreferences in long-form narrations of visual scenes. First, we introduce a new dataset with annotated coreference chains and their bounding boxes, as most existing image-text datasets only contain short sentences without coreferring expressions or labeled chains. We propose a new technique that learns to identify coref-erence chains using weak supervision, only from image-text pairs and a regularization using prior linguistic knowledge. Our model yields large performance gains over several strong baselines in resolving coreferences. We also show that coreference resolution helps improve grounding narratives in images.",

keywords = "cs.CV, cs.CL",

author = "Arushi Goel and Basura Fernando and Frank Keller and Hakan Bilen",

note = "15 pages; International Conference on Computer Vision 2023, ICCV 2023 ; Conference date: 02-10-2023 Through 06-10-2023",

year = "2024",

month = jan,

day = "15",

doi = "10.1109/ICCV51070.2023.01399",

language = "English",

isbn = "979-8-3503-0719-1",

series = "Proceedings of the IEEE International Conference on Computer Vision",

publisher = "Institute of Electrical and Electronics Engineers",

pages = "15201--15212",

booktitle = "2023 IEEE/CVF International Conference on Computer Vision (ICCV)",

address = "United States",

url = "https://iccv2023.thecvf.com/",

}

Goel, A, Fernando, B, Keller, F & Bilen, H 2024, Who are you referring to? Coreference resolution in image narrations. in 2023 IEEE/CVF International Conference on Computer Vision (ICCV). Proceedings of the IEEE International Conference on Computer Vision, Institute of Electrical and Electronics Engineers, pp. 15201-15212, International Conference on Computer Vision 2023, Paris, France, 2/10/23. https://doi.org/10.1109/ICCV51070.2023.01399

Who are you referring to? Coreference resolution in image narrations. / Goel, Arushi; Fernando, Basura; Keller, Frank et al.
2023 IEEE/CVF International Conference on Computer Vision (ICCV). Institute of Electrical and Electronics Engineers, 2024. p. 15201-15212 (Proceedings of the IEEE International Conference on Computer Vision).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Who are you referring to?

T2 - International Conference on Computer Vision 2023

AU - Goel, Arushi

AU - Fernando, Basura

AU - Keller, Frank

AU - Bilen, Hakan

N1 - 15 pages

PY - 2024/1/15

Y1 - 2024/1/15

N2 - Coreference resolution aims to identify words and phrases which refer to the same entity in a text, a core task in natural language processing. In this paper, we extend this task to resolving coreferences in long-form narrations of visual scenes. First, we introduce a new dataset with annotated coreference chains and their bounding boxes, as most existing image-text datasets only contain short sentences without coreferring expressions or labeled chains. We propose a new technique that learns to identify coref-erence chains using weak supervision, only from image-text pairs and a regularization using prior linguistic knowledge. Our model yields large performance gains over several strong baselines in resolving coreferences. We also show that coreference resolution helps improve grounding narratives in images.

AB - Coreference resolution aims to identify words and phrases which refer to the same entity in a text, a core task in natural language processing. In this paper, we extend this task to resolving coreferences in long-form narrations of visual scenes. First, we introduce a new dataset with annotated coreference chains and their bounding boxes, as most existing image-text datasets only contain short sentences without coreferring expressions or labeled chains. We propose a new technique that learns to identify coref-erence chains using weak supervision, only from image-text pairs and a regularization using prior linguistic knowledge. Our model yields large performance gains over several strong baselines in resolving coreferences. We also show that coreference resolution helps improve grounding narratives in images.

KW - cs.CV

KW - cs.CL

U2 - 10.1109/ICCV51070.2023.01399

DO - 10.1109/ICCV51070.2023.01399

M3 - Conference contribution

SN - 979-8-3503-0719-1

T3 - Proceedings of the IEEE International Conference on Computer Vision

SP - 15201

EP - 15212

BT - 2023 IEEE/CVF International Conference on Computer Vision (ICCV)

PB - Institute of Electrical and Electronics Engineers

Y2 - 2 October 2023 through 6 October 2023

ER -