Learning to Interpret and Describe Abstract Scenes

Luis Gilberto Mateos Ortiz; Clemens Wolff; Mirella Lapata

Learning to Interpret and Describe Abstract Scenes

Luis Gilberto Mateos Ortiz, Clemens Wolff, Mirella Lapata

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

Given a (static) scene, a human can effortlessly describe what is going on (who is doing what to whom, how, and why). The process requires knowledge about the world, how it is perceived, and described. In this paper we study the problem of interpreting and verbalizing visual information using abstract scenes created from collections of clip art images. We propose a model inspired by machine translation operating over a large parallel corpus of visual relations and linguistic descriptions. We demonstrate that this approach produces human-like scene descriptions which are both fluent and relevant, outperforming a number of competitive alternatives based on templates, sentence-based retrieval, and a multi-modal neural language model.

Original language	English
Title of host publication	Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Place of Publication	Denver, Colorado
Publisher	Association for Computational Linguistics
Pages	1505-1515
Number of pages	11
Publication status	Published - 1 May 2015

Access to Document

N15-1174Final published version, 2 MB

http://www.aclweb.org/anthology/N15-1174

Cite this

@inproceedings{2ee17249464f49aa80d8ad9b8231705a,

title = "Learning to Interpret and Describe Abstract Scenes",

abstract = "Given a (static) scene, a human can effortlessly describe what is going on (who is doing what to whom, how, and why). The process requires knowledge about the world, how it is perceived, and described. In this paper we study the problem of interpreting and verbalizing visual information using abstract scenes created from collections of clip art images. We propose a model inspired by machine translation operating over a large parallel corpus of visual relations and linguistic descriptions. We demonstrate that this approach produces human-like scene descriptions which are both fluent and relevant, outperforming a number of competitive alternatives based on templates, sentence-based retrieval, and a multi-modal neural language model.",

author = "{Gilberto Mateos Ortiz}, Luis and Clemens Wolff and Mirella Lapata",

year = "2015",

month = may,

day = "1",

language = "English",

pages = "1505--1515",

booktitle = "Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",

publisher = "Association for Computational Linguistics",

}

Gilberto Mateos Ortiz, L, Wolff, C & Lapata, M 2015, Learning to Interpret and Describe Abstract Scenes. in Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Denver, Colorado, pp. 1505-1515. <http://www.aclweb.org/anthology/N15-1174>

Learning to Interpret and Describe Abstract Scenes. / Gilberto Mateos Ortiz, Luis; Wolff, Clemens; Lapata, Mirella.
Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Denver, Colorado: Association for Computational Linguistics, 2015. p. 1505-1515.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Learning to Interpret and Describe Abstract Scenes

AU - Gilberto Mateos Ortiz, Luis

AU - Wolff, Clemens

AU - Lapata, Mirella

PY - 2015/5/1

Y1 - 2015/5/1

N2 - Given a (static) scene, a human can effortlessly describe what is going on (who is doing what to whom, how, and why). The process requires knowledge about the world, how it is perceived, and described. In this paper we study the problem of interpreting and verbalizing visual information using abstract scenes created from collections of clip art images. We propose a model inspired by machine translation operating over a large parallel corpus of visual relations and linguistic descriptions. We demonstrate that this approach produces human-like scene descriptions which are both fluent and relevant, outperforming a number of competitive alternatives based on templates, sentence-based retrieval, and a multi-modal neural language model.

AB - Given a (static) scene, a human can effortlessly describe what is going on (who is doing what to whom, how, and why). The process requires knowledge about the world, how it is perceived, and described. In this paper we study the problem of interpreting and verbalizing visual information using abstract scenes created from collections of clip art images. We propose a model inspired by machine translation operating over a large parallel corpus of visual relations and linguistic descriptions. We demonstrate that this approach produces human-like scene descriptions which are both fluent and relevant, outperforming a number of competitive alternatives based on templates, sentence-based retrieval, and a multi-modal neural language model.

M3 - Conference contribution

SP - 1505

EP - 1515

BT - Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

PB - Association for Computational Linguistics

CY - Denver, Colorado

ER -

Learning to Interpret and Describe Abstract Scenes

Abstract / Description of output

Access to Document

Fingerprint

Cite this