A hunt for the Snark: Annotator Diversity in Data Practices

Shivani Kapania; Alex S. Taylor; Ding Wang

doi:10.1145/3544548.3580645

A hunt for the Snark: Annotator Diversity in Data Practices

Shivani Kapania, Alex S. Taylor, Ding Wang

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

Diversity in datasets is a key component to building responsible AI/ML. Despite this recognition, we know little about the diversity among the annotators involved in data production. We investigated the approaches to annotator diversity through 16 semi-structured interviews and a survey with 44 AI/ML practitioners. While practitioners described nuanced understandings of annotator diversity, they rarely designed dataset production to account for diversity in the annotation process. The lack of action was explained through operational barriers: from the lack of visibility in the annotator hiring process, to the conceptual difficulty in incorporating worker diversity. We argue that such operational barriers and the widespread resistance to accommodating annotator diversity surface a prevailing logic in data practices - where neutrality, objectivity and 'representationalist thinking' dominate. By understanding this logic to be part of a regime of existence, we explore alternative ways of accounting for annotator subjectivity and diversity in data practices.

Original language	English
Title of host publication	CHI '23
Subtitle of host publication	Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Publisher	Association for Computing Machinery
Pages	1-15
Number of pages	15
ISBN (Electronic)	9781450394215
DOIs	https://doi.org/10.1145/3544548.3580645
Publication status	Published - 19 Apr 2023
Event	2023 CHI Conference on Human Factors in Computing Systems - Hamburg, Germany Duration: 23 Apr 2023 → 28 Apr 2023 https://chi2023.acm.org/

Publication series

Name	Conference on Human Factors in Computing Systems - Proceedings

Conference

Conference	2023 CHI Conference on Human Factors in Computing Systems
Abbreviated title	CHI 2023
Country/Territory	Germany
City	Hamburg
Period	23/04/23 → 28/04/23
Internet address	https://chi2023.acm.org/

Keywords / Materials (for Non-textual outputs)

annotator diversity
data annotation
data production
data work
diversity
machine learning
ML datasets

Access to Document

10.1145/3544548.3580645Licence: Creative Commons: Attribution (CC-BY)

A hunt for_KAPANIA_VOR_CC BYFinal published version, 870 KBLicence: Creative Commons: Attribution (CC-BY)

https://dl.acm.org/doi/10.1145/3544548.3580645Licence: Creative Commons: Attribution (CC-BY)

Cite this

@inproceedings{cb816be3412241629a29fa14f349607b,

title = "A hunt for the Snark: Annotator Diversity in Data Practices",

abstract = "Diversity in datasets is a key component to building responsible AI/ML. Despite this recognition, we know little about the diversity among the annotators involved in data production. We investigated the approaches to annotator diversity through 16 semi-structured interviews and a survey with 44 AI/ML practitioners. While practitioners described nuanced understandings of annotator diversity, they rarely designed dataset production to account for diversity in the annotation process. The lack of action was explained through operational barriers: from the lack of visibility in the annotator hiring process, to the conceptual difficulty in incorporating worker diversity. We argue that such operational barriers and the widespread resistance to accommodating annotator diversity surface a prevailing logic in data practices - where neutrality, objectivity and 'representationalist thinking' dominate. By understanding this logic to be part of a regime of existence, we explore alternative ways of accounting for annotator subjectivity and diversity in data practices.",

keywords = "annotator diversity, data annotation, data production, data work, diversity, machine learning, ML datasets",

author = "Shivani Kapania and Taylor, {Alex S.} and Ding Wang",

year = "2023",

month = apr,

day = "19",

doi = "10.1145/3544548.3580645",

language = "English",

series = "Conference on Human Factors in Computing Systems - Proceedings",

publisher = "Association for Computing Machinery",

pages = "1--15",

booktitle = "CHI '23",

note = "2023 CHI Conference on Human Factors in Computing Systems, CHI 2023 ; Conference date: 23-04-2023 Through 28-04-2023",

url = "https://chi2023.acm.org/",

}

Kapania, S, Taylor, AS & Wang, D 2023, A hunt for the Snark: Annotator Diversity in Data Practices. in CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems., 133, Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, pp. 1-15, 2023 CHI Conference on Human Factors in Computing Systems, Hamburg, Hamburg, Germany, 23/04/23. https://doi.org/10.1145/3544548.3580645

A hunt for the Snark: Annotator Diversity in Data Practices. / Kapania, Shivani; Taylor, Alex S.; Wang, Ding.
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2023. p. 1-15 133 (Conference on Human Factors in Computing Systems - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A hunt for the Snark

T2 - 2023 CHI Conference on Human Factors in Computing Systems

AU - Kapania, Shivani

AU - Taylor, Alex S.

AU - Wang, Ding

PY - 2023/4/19

Y1 - 2023/4/19

N2 - Diversity in datasets is a key component to building responsible AI/ML. Despite this recognition, we know little about the diversity among the annotators involved in data production. We investigated the approaches to annotator diversity through 16 semi-structured interviews and a survey with 44 AI/ML practitioners. While practitioners described nuanced understandings of annotator diversity, they rarely designed dataset production to account for diversity in the annotation process. The lack of action was explained through operational barriers: from the lack of visibility in the annotator hiring process, to the conceptual difficulty in incorporating worker diversity. We argue that such operational barriers and the widespread resistance to accommodating annotator diversity surface a prevailing logic in data practices - where neutrality, objectivity and 'representationalist thinking' dominate. By understanding this logic to be part of a regime of existence, we explore alternative ways of accounting for annotator subjectivity and diversity in data practices.

AB - Diversity in datasets is a key component to building responsible AI/ML. Despite this recognition, we know little about the diversity among the annotators involved in data production. We investigated the approaches to annotator diversity through 16 semi-structured interviews and a survey with 44 AI/ML practitioners. While practitioners described nuanced understandings of annotator diversity, they rarely designed dataset production to account for diversity in the annotation process. The lack of action was explained through operational barriers: from the lack of visibility in the annotator hiring process, to the conceptual difficulty in incorporating worker diversity. We argue that such operational barriers and the widespread resistance to accommodating annotator diversity surface a prevailing logic in data practices - where neutrality, objectivity and 'representationalist thinking' dominate. By understanding this logic to be part of a regime of existence, we explore alternative ways of accounting for annotator subjectivity and diversity in data practices.

KW - annotator diversity

KW - data annotation

KW - data production

KW - data work

KW - diversity

KW - machine learning

KW - ML datasets

U2 - 10.1145/3544548.3580645

DO - 10.1145/3544548.3580645

M3 - Conference contribution

AN - SCOPUS:85159220681

T3 - Conference on Human Factors in Computing Systems - Proceedings

SP - 1

EP - 15

BT - CHI '23

PB - Association for Computing Machinery

Y2 - 23 April 2023 through 28 April 2023

ER -

A hunt for the Snark: Annotator Diversity in Data Practices

Abstract / Description of output

Publication series

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Cite this