StyleBabel: Artistic style tagging and captioning

Dan Ruta; Andrew Gilbert; Pranav Aggarwal; Naveen Marri; Ajinkya Kale; Jo Briggs; Chris Speed; Hailin Jin; Baldo Faieta; Alex Filipkowski; Zhe Lin; John Collomosse

doi:10.1007/978-3-031-20074-8_13

StyleBabel: Artistic style tagging and captioning

Dan Ruta, Andrew Gilbert, Pranav Aggarwal, Naveen Marri, Ajinkya Kale, Jo Briggs, Chris Speed, Hailin Jin, Baldo Faieta, Alex Filipkowski, Zhe Lin, John Collomosse

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

We present StyleBabel, a unique open access dataset of natural language captions and free-form tags describing the artistic style of over 135K digital artworks, collected via a novel participatory method from experts studying at specialist art and design schools. StyleBabel was collected via an iterative method, inspired by ‘Grounded Theory’: a qualitative approach that enables annotation while co-evolving a shared language for fine-grained artistic style attribute description. We demonstrate several downstream tasks for StyleBabel, adapting the recent ALADIN architecture for fine-grained style similarity, to train cross-modal embeddings for: 1) free-form tag generation; 2) natural language description of artistic style; 3) fine-grained text search of style. To do so, we extend ALADIN with recent advances in Visual Transformer (ViT) and cross-modal representation learning, achieving a state of the art accuracy in fine-grained style retrieval.

Original language	English
Title of host publication	Computer Vision – ECCV 2022
Editors	Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella
Publisher	Springer
Pages	219-236
Number of pages	18
ISBN (Electronic)	9783031200748
ISBN (Print)	9783031200731
DOIs	https://doi.org/10.1007/978-3-031-20074-8_13
Publication status	Published - 12 Nov 2022
Event	European Conference on Computer Vision 2022 - Israel, Tel Aviv, Israel Duration: 23 Oct 2022 → 27 Oct 2022 https://eccv2022.ecva.net/

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Computer Vision 2022
Abbreviated title	ECCV 2022
Country/Territory	Israel
City	Tel Aviv
Period	23/10/22 → 27/10/22
Internet address	https://eccv2022.ecva.net/

Keywords / Materials (for Non-textual outputs)

datasets and evaluation
image and video retrieval
vision and language
vision applications and systems

Access to Document

10.1007/978-3-031-20074-8_13

RutaEtal2022ECCVStyleBabelArtisticStyleTaggingAccepted author manuscript, 6.64 MBLicence: Creative Commons: Attribution (CC-BY)

Cite this

Ruta, D., Gilbert, A., Aggarwal, P., Marri, N., Kale, A., Briggs, J., Speed, C., Jin, H., Faieta, B., Filipkowski, A., Lin, Z., & Collomosse, J. (2022). StyleBabel: Artistic style tagging and captioning. In S. Avidan, G. Brostow, M. Cissé, & G. M. Farinella (Eds.), Computer Vision – ECCV 2022 (pp. 219-236). (Lecture Notes in Computer Science). Springer. https://doi.org/10.1007/978-3-031-20074-8_13

@inproceedings{56cf07555fbf45828efd655cd7464a1b,

title = "StyleBabel: Artistic style tagging and captioning",

abstract = "We present StyleBabel, a unique open access dataset of natural language captions and free-form tags describing the artistic style of over 135K digital artworks, collected via a novel participatory method from experts studying at specialist art and design schools. StyleBabel was collected via an iterative method, inspired by {\textquoteleft}Grounded Theory{\textquoteright}: a qualitative approach that enables annotation while co-evolving a shared language for fine-grained artistic style attribute description. We demonstrate several downstream tasks for StyleBabel, adapting the recent ALADIN architecture for fine-grained style similarity, to train cross-modal embeddings for: 1) free-form tag generation; 2) natural language description of artistic style; 3) fine-grained text search of style. To do so, we extend ALADIN with recent advances in Visual Transformer (ViT) and cross-modal representation learning, achieving a state of the art accuracy in fine-grained style retrieval.",

keywords = "datasets and evaluation, image and video retrieval, vision and language, vision applications and systems",

author = "Dan Ruta and Andrew Gilbert and Pranav Aggarwal and Naveen Marri and Ajinkya Kale and Jo Briggs and Chris Speed and Hailin Jin and Baldo Faieta and Alex Filipkowski and Zhe Lin and John Collomosse",

year = "2022",

month = nov,

day = "12",

doi = "10.1007/978-3-031-20074-8_13",

language = "English",

isbn = "9783031200731",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "219--236",

editor = "Shai Avidan and Gabriel Brostow and Moustapha Ciss{\'e} and Farinella, {Giovanni Maria}",

booktitle = "Computer Vision – ECCV 2022",

address = "United Kingdom",

note = "European Conference on Computer Vision 2022, ECCV 2022 ; Conference date: 23-10-2022 Through 27-10-2022",

url = "https://eccv2022.ecva.net/",

}

Ruta, D, Gilbert, A, Aggarwal, P, Marri, N, Kale, A, Briggs, J, Speed, C, Jin, H, Faieta, B, Filipkowski, A, Lin, Z & Collomosse, J 2022, StyleBabel: Artistic style tagging and captioning. in S Avidan, G Brostow, M Cissé & GM Farinella (eds), Computer Vision – ECCV 2022. Lecture Notes in Computer Science, Springer, pp. 219-236, European Conference on Computer Vision 2022, Tel Aviv, Israel, 23/10/22. https://doi.org/10.1007/978-3-031-20074-8_13

TY - GEN

T1 - StyleBabel

T2 - European Conference on Computer Vision 2022

AU - Ruta, Dan

AU - Gilbert, Andrew

AU - Aggarwal, Pranav

AU - Marri, Naveen

AU - Kale, Ajinkya

AU - Briggs, Jo

AU - Speed, Chris

AU - Jin, Hailin

AU - Faieta, Baldo

AU - Filipkowski, Alex

AU - Lin, Zhe

AU - Collomosse, John

PY - 2022/11/12

Y1 - 2022/11/12

N2 - We present StyleBabel, a unique open access dataset of natural language captions and free-form tags describing the artistic style of over 135K digital artworks, collected via a novel participatory method from experts studying at specialist art and design schools. StyleBabel was collected via an iterative method, inspired by ‘Grounded Theory’: a qualitative approach that enables annotation while co-evolving a shared language for fine-grained artistic style attribute description. We demonstrate several downstream tasks for StyleBabel, adapting the recent ALADIN architecture for fine-grained style similarity, to train cross-modal embeddings for: 1) free-form tag generation; 2) natural language description of artistic style; 3) fine-grained text search of style. To do so, we extend ALADIN with recent advances in Visual Transformer (ViT) and cross-modal representation learning, achieving a state of the art accuracy in fine-grained style retrieval.

AB - We present StyleBabel, a unique open access dataset of natural language captions and free-form tags describing the artistic style of over 135K digital artworks, collected via a novel participatory method from experts studying at specialist art and design schools. StyleBabel was collected via an iterative method, inspired by ‘Grounded Theory’: a qualitative approach that enables annotation while co-evolving a shared language for fine-grained artistic style attribute description. We demonstrate several downstream tasks for StyleBabel, adapting the recent ALADIN architecture for fine-grained style similarity, to train cross-modal embeddings for: 1) free-form tag generation; 2) natural language description of artistic style; 3) fine-grained text search of style. To do so, we extend ALADIN with recent advances in Visual Transformer (ViT) and cross-modal representation learning, achieving a state of the art accuracy in fine-grained style retrieval.

KW - datasets and evaluation

KW - image and video retrieval

KW - vision and language

KW - vision applications and systems

UR - https://eccv2022.ecva.net/

U2 - 10.1007/978-3-031-20074-8_13

DO - 10.1007/978-3-031-20074-8_13

M3 - Conference contribution

SN - 9783031200731

T3 - Lecture Notes in Computer Science

SP - 219

EP - 236

BT - Computer Vision – ECCV 2022

A2 - Avidan, Shai

A2 - Brostow, Gabriel

A2 - Cissé, Moustapha

A2 - Farinella, Giovanni Maria

PB - Springer

Y2 - 23 October 2022 through 27 October 2022

ER -

StyleBabel: Artistic style tagging and captioning

Abstract / Description of output

Publication series

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Other files and links

Fingerprint

Cite this