Bias Beyond English: Counterfactual Tests for Bias in Sentiment Analysis in Four Languages

Seraphina Goldfarb-Tarrant, Adam Lopez, Roi Blanco, Diego Marcheggiani

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Sentiment analysis (SA) systems are used in many products and hundreds of languages. Gender and racial biases are well-studied in English SA systems, but understudied in other languages, with few resources for such studies. To remedy this, we build a counterfactual evaluation corpus for gender and racial/migrant bias in four languages. We demonstrate its usefulness by answering a simple but important question that an engineer might need to answer when deploying a system: What biases do systems import from pre-trained models when compared to a baseline with no pre-training? Our evaluation corpus, by virtue of being counterfactual, not only reveals which models have less bias, but also pinpoints changes in model bias behaviour, which enables more targeted mitigation strategies. We release our code and evaluation corpora to facilitate future research.
Original languageEnglish
Title of host publicationFindings of the Association for Computational Linguistics: ACL 2023
EditorsAnna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
Place of PublicationToronto, Canada
PublisherAssociation for Computational Linguistics
Pages4458-4468
Number of pages11
DOIs
Publication statusPublished - 1 Jul 2023
Event61st Annual Meeting of the Association for Computational Linguistics - Toronto, Canada
Duration: 9 Jul 202314 Jul 2023
Conference number: 61
https://2023.aclweb.org/

Conference

Conference61st Annual Meeting of the Association for Computational Linguistics
Abbreviated titleACL 2023
Country/TerritoryCanada
CityToronto
Period9/07/2314/07/23
Internet address

Fingerprint

Dive into the research topics of 'Bias Beyond English: Counterfactual Tests for Bias in Sentiment Analysis in Four Languages'. Together they form a unique fingerprint.

Cite this