Mapping the multilingual margins: Intersectional biases of sentiment analysis systems in English, Spanish, and Arabic

António Câmara; Nina Taneja; Tamjeed Azad; Emily Allaway; Richard Zemel

doi:10.18653/v1/2022.ltedi-1.11

Mapping the multilingual margins: Intersectional biases of sentiment analysis systems in English, Spanish, and Arabic

António Câmara
, Nina Taneja
, Tamjeed Azad
, Emily Allaway
, Richard Zemel

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

As natural language processing systems become more widespread, it is necessary to address fairness issues in their implementation and deployment to ensure that their negative impacts on society are understood and minimized. However, there is limited work that studies fairness using a multilingual and intersectional framework or on downstream tasks. In this paper, we introduce four multilingual Equity Evaluation Corpora, supplementary test sets designed to measure social biases, and a novel statistical framework for studying unisectional and intersectional social biases in natural language processing. We use these tools to measure gender, racial, ethnic, and intersectional social biases across five models trained on emotion regression tasks in English, Spanish, and Arabic. We find that many systems demonstrate statistically significant unisectional and intersectional social biases. We make our code and datasets available for download.

Original language	English
Title of host publication	Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion
Editors	Bharathi Raja Chakravarthi, B Bharathi, John P McCrae, Manel Zarrouk, Kalika Bali, Paul Buitelaar
Place of Publication	Dublin, Ireland
Publisher	Association for Computational Linguistics
Pages	90–106
Number of pages	17
ISBN (Electronic)	9781955917438
DOIs	https://doi.org/10.18653/v1/2022.ltedi-1.11
Publication status	Published - 27 May 2022
Event	2nd Workshop on Language Technology for Equality, Diversity, Inclusion 2022 - Convention Centre Dublin, Dublin, Ireland Duration: 27 May 2022 → 27 May 2022 Conference number: 2 https://sites.google.com/view/lt-edi-2022/home

Workshop

Workshop	2nd Workshop on Language Technology for Equality, Diversity, Inclusion 2022
Abbreviated title	LT-EDI 2022
Country/Territory	Ireland
City	Dublin
Period	27/05/22 → 27/05/22
Internet address	https://sites.google.com/view/lt-edi-2022/home

Access to Document

10.18653/v1/2022.ltedi-1.11Licence: Creative Commons: Attribution (CC-BY)

CâmaraEtalLTEDI2022MappingtheMultilingualMarginsFinal published version, 236 KBLicence: Creative Commons: Attribution (CC-BY)

Cite this

Câmara, A., Taneja, N., Azad, T., Allaway, E., & Zemel, R. (2022). Mapping the multilingual margins: Intersectional biases of sentiment analysis systems in English, Spanish, and Arabic. In B. R. Chakravarthi, B. Bharathi, J. P. McCrae, M. Zarrouk, K. Bali, & P. Buitelaar (Eds.), Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion (pp. 90–106). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.ltedi-1.11

Câmara, António ; Taneja, Nina ; Azad, Tamjeed et al. / Mapping the multilingual margins : Intersectional biases of sentiment analysis systems in English, Spanish, and Arabic. Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion. editor / Bharathi Raja Chakravarthi ; B Bharathi ; John P McCrae ; Manel Zarrouk ; Kalika Bali ; Paul Buitelaar. Dublin, Ireland : Association for Computational Linguistics, 2022. pp. 90–106

@inproceedings{36e29f9890ff4ec3aa5f0957585348b1,

title = "Mapping the multilingual margins: Intersectional biases of sentiment analysis systems in English, Spanish, and Arabic",

abstract = "As natural language processing systems become more widespread, it is necessary to address fairness issues in their implementation and deployment to ensure that their negative impacts on society are understood and minimized. However, there is limited work that studies fairness using a multilingual and intersectional framework or on downstream tasks. In this paper, we introduce four multilingual Equity Evaluation Corpora, supplementary test sets designed to measure social biases, and a novel statistical framework for studying unisectional and intersectional social biases in natural language processing. We use these tools to measure gender, racial, ethnic, and intersectional social biases across five models trained on emotion regression tasks in English, Spanish, and Arabic. We find that many systems demonstrate statistically significant unisectional and intersectional social biases. We make our code and datasets available for download.",

author = "Ant{\'o}nio C{\^a}mara and Nina Taneja and Tamjeed Azad and Emily Allaway and Richard Zemel",

year = "2022",

month = may,

day = "27",

doi = "10.18653/v1/2022.ltedi-1.11",

language = "English",

pages = "90–106",

editor = "Chakravarthi, \{Bharathi Raja\} and B Bharathi and McCrae, \{John P\} and Manel Zarrouk and Kalika Bali and Paul Buitelaar",

booktitle = "Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion",

publisher = "Association for Computational Linguistics",

note = "2nd Workshop on Language Technology for Equality, Diversity, Inclusion 2022, LT-EDI 2022 ; Conference date: 27-05-2022 Through 27-05-2022",

url = "https://sites.google.com/view/lt-edi-2022/home",

}

Câmara, A, Taneja, N, Azad, T, Allaway, E & Zemel, R 2022, Mapping the multilingual margins: Intersectional biases of sentiment analysis systems in English, Spanish, and Arabic. in BR Chakravarthi, B Bharathi, JP McCrae, M Zarrouk, K Bali & P Buitelaar (eds), Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion. Association for Computational Linguistics, Dublin, Ireland, pp. 90–106, 2nd Workshop on Language Technology for Equality, Diversity, Inclusion 2022, Dublin, Ireland, 27/05/22. https://doi.org/10.18653/v1/2022.ltedi-1.11

Mapping the multilingual margins: Intersectional biases of sentiment analysis systems in English, Spanish, and Arabic. / Câmara, António; Taneja, Nina; Azad, Tamjeed et al.
Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion. ed. / Bharathi Raja Chakravarthi; B Bharathi; John P McCrae; Manel Zarrouk; Kalika Bali; Paul Buitelaar. Dublin, Ireland: Association for Computational Linguistics, 2022. p. 90–106.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Mapping the multilingual margins

T2 - 2nd Workshop on Language Technology for Equality, Diversity, Inclusion 2022

AU - Câmara, António

AU - Taneja, Nina

AU - Azad, Tamjeed

AU - Allaway, Emily

AU - Zemel, Richard

N1 - Conference code: 2

PY - 2022/5/27

Y1 - 2022/5/27

N2 - As natural language processing systems become more widespread, it is necessary to address fairness issues in their implementation and deployment to ensure that their negative impacts on society are understood and minimized. However, there is limited work that studies fairness using a multilingual and intersectional framework or on downstream tasks. In this paper, we introduce four multilingual Equity Evaluation Corpora, supplementary test sets designed to measure social biases, and a novel statistical framework for studying unisectional and intersectional social biases in natural language processing. We use these tools to measure gender, racial, ethnic, and intersectional social biases across five models trained on emotion regression tasks in English, Spanish, and Arabic. We find that many systems demonstrate statistically significant unisectional and intersectional social biases. We make our code and datasets available for download.

AB - As natural language processing systems become more widespread, it is necessary to address fairness issues in their implementation and deployment to ensure that their negative impacts on society are understood and minimized. However, there is limited work that studies fairness using a multilingual and intersectional framework or on downstream tasks. In this paper, we introduce four multilingual Equity Evaluation Corpora, supplementary test sets designed to measure social biases, and a novel statistical framework for studying unisectional and intersectional social biases in natural language processing. We use these tools to measure gender, racial, ethnic, and intersectional social biases across five models trained on emotion regression tasks in English, Spanish, and Arabic. We find that many systems demonstrate statistically significant unisectional and intersectional social biases. We make our code and datasets available for download.

U2 - 10.18653/v1/2022.ltedi-1.11

DO - 10.18653/v1/2022.ltedi-1.11

M3 - Conference contribution

SP - 90

EP - 106

BT - Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion

A2 - Chakravarthi, Bharathi Raja

A2 - Bharathi, B

A2 - McCrae, John P

A2 - Zarrouk, Manel

A2 - Bali, Kalika

A2 - Buitelaar, Paul

PB - Association for Computational Linguistics

CY - Dublin, Ireland

Y2 - 27 May 2022 through 27 May 2022

ER -

Câmara A, Taneja N, Azad T, Allaway E, Zemel R. Mapping the multilingual margins: Intersectional biases of sentiment analysis systems in English, Spanish, and Arabic. In Chakravarthi BR, Bharathi B, McCrae JP, Zarrouk M, Bali K, Buitelaar P, editors, Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion. Dublin, Ireland: Association for Computational Linguistics. 2022. p. 90–106 doi: 10.18653/v1/2022.ltedi-1.11

Mapping the multilingual margins: Intersectional biases of sentiment analysis systems in English, Spanish, and Arabic

Abstract

Workshop

Access to Document

Fingerprint

Cite this