An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games

Alessandro Suglia; Yonatan Bisk; Ioannis Konstas; Antonio Vergari; Emanuele Bastianelli; Andrea Vanzo; Oliver Lemon

doi:10.18653/v1/2021.eacl-main.183

An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games

Alessandro Suglia, Yonatan Bisk, Ioannis Konstas, Antonio Vergari, Emanuele Bastianelli, Andrea Vanzo, Oliver Lemon

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

Guessing games are a prototypical instance of the ``learning by interacting'' paradigm. This work investigates how well an artificial agent can benefit from playing guessing games when later asked to perform on novel NLP downstream tasks such as Visual Question Answering (VQA). We propose two ways to exploit playing guessing games: 1) a supervised learning scenario in which the agent learns to mimic successful guessing games and 2) a novel way for an agent to play by itself, called Self-play via Iterated Experience Learning (SPIEL). We evaluate the ability of both procedures to generalise: an in-domain evaluation shows an increased accuracy (+7.79) compared with competitors on the evaluation suite CompGuessWhat?!; a transfer evaluation shows improved performance for VQA on the TDIUC dataset in terms of harmonic average accuracy (+5.31) thanks to more fine-grained object representations learned via SPIEL.

Original language	English
Title of host publication	Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Editors	Paola Merlo, Jorg Tiedemann, Reut Tsarfaty
Place of Publication	Stroudsburg, PA, USA
Publisher	Association for Computational Linguistics
Pages	2135-2144
Number of pages	10
ISBN (Electronic)	978-1-954085-02-2
DOIs	https://doi.org/10.18653/v1/2021.eacl-main.183
Publication status	Published - 19 Apr 2021
Event	16th Conference of the European Chapter of the Association for Computational Linguistics - Online Duration: 19 Apr 2021 → 23 Apr 2021 https://2021.eacl.org/

Conference

Conference	16th Conference of the European Chapter of the Association for Computational Linguistics
Abbreviated title	EACL 2021
Period	19/04/21 → 23/04/21
Internet address	https://2021.eacl.org/

Access to Document

10.18653/v1/2021.eacl-main.183Licence: Creative Commons: Attribution (CC-BY)

An Empirical Study_SUGLIA_DOA11012021_VOR_CC-BYFinal published version, 2.38 MBLicence: Creative Commons: Attribution (CC-BY)

https://aclanthology.org/2021.eacl-main.183Licence: Creative Commons: Attribution (CC-BY)

Cite this

Suglia, A., Bisk, Y., Konstas, I., Vergari, A., Bastianelli, E., Vanzo, A., & Lemon, O. (2021). An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games. In P. Merlo, J. Tiedemann, & R. Tsarfaty (Eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (pp. 2135-2144). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.eacl-main.183

Suglia, Alessandro ; Bisk, Yonatan ; Konstas, Ioannis et al. / An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. editor / Paola Merlo ; Jorg Tiedemann ; Reut Tsarfaty. Stroudsburg, PA, USA : Association for Computational Linguistics, 2021. pp. 2135-2144

@inproceedings{e951fcf85b2b463f91b372758293eb0c,

title = "An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games",

abstract = "Guessing games are a prototypical instance of the ``learning by interacting'' paradigm. This work investigates how well an artificial agent can benefit from playing guessing games when later asked to perform on novel NLP downstream tasks such as Visual Question Answering (VQA). We propose two ways to exploit playing guessing games: 1) a supervised learning scenario in which the agent learns to mimic successful guessing games and 2) a novel way for an agent to play by itself, called Self-play via Iterated Experience Learning (SPIEL). We evaluate the ability of both procedures to generalise: an in-domain evaluation shows an increased accuracy (+7.79) compared with competitors on the evaluation suite CompGuessWhat?!; a transfer evaluation shows improved performance for VQA on the TDIUC dataset in terms of harmonic average accuracy (+5.31) thanks to more fine-grained object representations learned via SPIEL.",

author = "Alessandro Suglia and Yonatan Bisk and Ioannis Konstas and Antonio Vergari and Emanuele Bastianelli and Andrea Vanzo and Oliver Lemon",

year = "2021",

month = apr,

day = "19",

doi = "10.18653/v1/2021.eacl-main.183",

language = "English",

pages = "2135--2144",

editor = "Paola Merlo and Jorg Tiedemann and Reut Tsarfaty",

booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume",

publisher = "Association for Computational Linguistics",

note = "16th Conference of the European Chapter of the Association for Computational Linguistics , EACL 2021 ; Conference date: 19-04-2021 Through 23-04-2021",

url = "https://2021.eacl.org/",

}

Suglia, A, Bisk, Y, Konstas, I, Vergari, A, Bastianelli, E, Vanzo, A & Lemon, O 2021, An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games. in P Merlo, J Tiedemann & R Tsarfaty (eds), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. Association for Computational Linguistics, Stroudsburg, PA, USA, pp. 2135-2144, 16th Conference of the European Chapter of the Association for Computational Linguistics , 19/04/21. https://doi.org/10.18653/v1/2021.eacl-main.183

An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games. / Suglia, Alessandro; Bisk, Yonatan; Konstas, Ioannis et al.
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. ed. / Paola Merlo; Jorg Tiedemann; Reut Tsarfaty. Stroudsburg, PA, USA: Association for Computational Linguistics, 2021. p. 2135-2144.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games

AU - Suglia, Alessandro

AU - Bisk, Yonatan

AU - Konstas, Ioannis

AU - Vergari, Antonio

AU - Bastianelli, Emanuele

AU - Vanzo, Andrea

AU - Lemon, Oliver

PY - 2021/4/19

Y1 - 2021/4/19

N2 - Guessing games are a prototypical instance of the ``learning by interacting'' paradigm. This work investigates how well an artificial agent can benefit from playing guessing games when later asked to perform on novel NLP downstream tasks such as Visual Question Answering (VQA). We propose two ways to exploit playing guessing games: 1) a supervised learning scenario in which the agent learns to mimic successful guessing games and 2) a novel way for an agent to play by itself, called Self-play via Iterated Experience Learning (SPIEL). We evaluate the ability of both procedures to generalise: an in-domain evaluation shows an increased accuracy (+7.79) compared with competitors on the evaluation suite CompGuessWhat?!; a transfer evaluation shows improved performance for VQA on the TDIUC dataset in terms of harmonic average accuracy (+5.31) thanks to more fine-grained object representations learned via SPIEL.

AB - Guessing games are a prototypical instance of the ``learning by interacting'' paradigm. This work investigates how well an artificial agent can benefit from playing guessing games when later asked to perform on novel NLP downstream tasks such as Visual Question Answering (VQA). We propose two ways to exploit playing guessing games: 1) a supervised learning scenario in which the agent learns to mimic successful guessing games and 2) a novel way for an agent to play by itself, called Self-play via Iterated Experience Learning (SPIEL). We evaluate the ability of both procedures to generalise: an in-domain evaluation shows an increased accuracy (+7.79) compared with competitors on the evaluation suite CompGuessWhat?!; a transfer evaluation shows improved performance for VQA on the TDIUC dataset in terms of harmonic average accuracy (+5.31) thanks to more fine-grained object representations learned via SPIEL.

U2 - 10.18653/v1/2021.eacl-main.183

DO - 10.18653/v1/2021.eacl-main.183

M3 - Conference contribution

SP - 2135

EP - 2144

BT - Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

A2 - Merlo, Paola

A2 - Tiedemann, Jorg

A2 - Tsarfaty, Reut

PB - Association for Computational Linguistics

CY - Stroudsburg, PA, USA

T2 - 16th Conference of the European Chapter of the Association for Computational Linguistics

Y2 - 19 April 2021 through 23 April 2021

ER -

Suglia A, Bisk Y, Konstas I, Vergari A, Bastianelli E, Vanzo A et al. An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games. In Merlo P, Tiedemann J, Tsarfaty R, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. Stroudsburg, PA, USA: Association for Computational Linguistics. 2021. p. 2135-2144 doi: 10.18653/v1/2021.eacl-main.183

An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games

Abstract / Description of output

Conference

Access to Document

Fingerprint

Cite this