Arabic dialect identification under scrutiny: Limitations of single-label classification

Amr Keleg; Walid Magdy

doi:10.18653/v1/2023.arabicnlp-1.31

Arabic dialect identification under scrutiny: Limitations of single-label classification

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Automatic Arabic Dialect Identification (ADI) of text has gained great popularity since it was introduced in the early 2010s. Multiple datasets were developed, and yearly shared tasks have been running since 2018. However, ADI systems are reported to fail in distinguishing between the micro-dialects of Arabic. We argue that the currently adopted framing of the ADI task as a single-label classification problem is one of the main reasons for that. We highlight the limitation of the incompleteness of the Dialect labels and demonstrate how it impacts the evaluation of ADI systems. A manual error analysis for the predictions of an ADI, performed by 7 native speakers of different Arabic dialects, revealed that ≈ 67% of the validated errors are not true errors. Consequently, we propose framing ADI as a multi-label classification task and give recommendations for designing new ADI datasets.

Original language	English
Title of host publication	Proceedings of ArabicNLP 2023
Editors	Hassan Sawaf, Samhaa El-Beltagy, Wajdi Zaghouani, Walid Magdy, Ahmed Abdelali, Nadi Tomeh, Ibrahim Abu Farza, Nizar Habash, Salam Khalifa, Amr Keleg, Hatem Haddad, Imed Zitouni, Khalil Mrini, Rawan Almatham
Publisher	Association for Computational Linguistics
Pages	385-398
Number of pages	14
Edition	1
ISBN (Electronic)	9781959429272
DOIs	https://doi.org/10.18653/v1/2023.arabicnlp-1.31
Publication status	Published - 7 Dec 2023
Event	The First Arabic Natural Language Processing Conference - Sentosa, Singapore Duration: 7 Dec 2023 → 7 Dec 2023 Conference number: 1 https://arabicnlp2023.sigarab.org/

Conference

Conference	The First Arabic Natural Language Processing Conference
Abbreviated title	ArabicNLP 2023
Country/Territory	Singapore
City	Sentosa
Period	7/12/23 → 7/12/23
Internet address	https://arabicnlp2023.sigarab.org/

Access to Document

10.18653/v1/2023.arabicnlp-1.31Licence: Creative Commons: Attribution (CC-BY)

KelegAndMagdyACL2023ArabicDialectIdentificationUnderScrutinyFinal published version, 838 KBLicence: Creative Commons: Attribution (CC-BY)

Cite this

Keleg, A., & Magdy, W. (2023). Arabic dialect identification under scrutiny: Limitations of single-label classification. In H. Sawaf, S. El-Beltagy, W. Zaghouani, W. Magdy, A. Abdelali, N. Tomeh, I. A. Farza, N. Habash, S. Khalifa, A. Keleg, H. Haddad, I. Zitouni, K. Mrini, & R. Almatham (Eds.), Proceedings of ArabicNLP 2023 (1 ed., pp. 385-398). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.arabicnlp-1.31

Keleg, Amr ; Magdy, Walid. / Arabic dialect identification under scrutiny : Limitations of single-label classification. Proceedings of ArabicNLP 2023. editor / Hassan Sawaf ; Samhaa El-Beltagy ; Wajdi Zaghouani ; Walid Magdy ; Ahmed Abdelali ; Nadi Tomeh ; Ibrahim Abu Farza ; Nizar Habash ; Salam Khalifa ; Amr Keleg ; Hatem Haddad ; Imed Zitouni ; Khalil Mrini ; Rawan Almatham. 1. ed. Association for Computational Linguistics, 2023. pp. 385-398

@inproceedings{f3efef903eb24288b58bd1046b4b122b,

title = "Arabic dialect identification under scrutiny: Limitations of single-label classification",

abstract = "Automatic Arabic Dialect Identification (ADI) of text has gained great popularity since it was introduced in the early 2010s. Multiple datasets were developed, and yearly shared tasks have been running since 2018. However, ADI systems are reported to fail in distinguishing between the micro-dialects of Arabic. We argue that the currently adopted framing of the ADI task as a single-label classification problem is one of the main reasons for that. We highlight the limitation of the incompleteness of the Dialect labels and demonstrate how it impacts the evaluation of ADI systems. A manual error analysis for the predictions of an ADI, performed by 7 native speakers of different Arabic dialects, revealed that ≈ 67\% of the validated errors are not true errors. Consequently, we propose framing ADI as a multi-label classification task and give recommendations for designing new ADI datasets.",

author = "Amr Keleg and Walid Magdy",

year = "2023",

month = dec,

day = "7",

doi = "10.18653/v1/2023.arabicnlp-1.31",

language = "English",

pages = "385--398",

editor = "Hassan Sawaf and Samhaa El-Beltagy and Wajdi Zaghouani and Walid Magdy and Ahmed Abdelali and Nadi Tomeh and Farza, \{Ibrahim Abu\} and Nizar Habash and Salam Khalifa and Amr Keleg and Hatem Haddad and Imed Zitouni and Khalil Mrini and Rawan Almatham",

booktitle = "Proceedings of ArabicNLP 2023",

publisher = "Association for Computational Linguistics",

edition = "1",

note = "The First Arabic Natural Language Processing Conference, ArabicNLP 2023 ; Conference date: 07-12-2023 Through 07-12-2023",

url = "https://arabicnlp2023.sigarab.org/",

}

Keleg, A & Magdy, W 2023, Arabic dialect identification under scrutiny: Limitations of single-label classification. in H Sawaf, S El-Beltagy, W Zaghouani, W Magdy, A Abdelali, N Tomeh, IA Farza, N Habash, S Khalifa, A Keleg, H Haddad, I Zitouni, K Mrini & R Almatham (eds), Proceedings of ArabicNLP 2023. 1 edn, Association for Computational Linguistics, pp. 385-398, The First Arabic Natural Language Processing Conference, Sentosa, Singapore, 7/12/23. https://doi.org/10.18653/v1/2023.arabicnlp-1.31

Arabic dialect identification under scrutiny: Limitations of single-label classification. / Keleg, Amr; Magdy, Walid.
Proceedings of ArabicNLP 2023. ed. / Hassan Sawaf; Samhaa El-Beltagy; Wajdi Zaghouani; Walid Magdy; Ahmed Abdelali; Nadi Tomeh; Ibrahim Abu Farza; Nizar Habash; Salam Khalifa; Amr Keleg; Hatem Haddad; Imed Zitouni; Khalil Mrini; Rawan Almatham. 1. ed. Association for Computational Linguistics, 2023. p. 385-398.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Arabic dialect identification under scrutiny

T2 - The First Arabic Natural Language Processing Conference

AU - Keleg, Amr

AU - Magdy, Walid

N1 - Conference code: 1

PY - 2023/12/7

Y1 - 2023/12/7

N2 - Automatic Arabic Dialect Identification (ADI) of text has gained great popularity since it was introduced in the early 2010s. Multiple datasets were developed, and yearly shared tasks have been running since 2018. However, ADI systems are reported to fail in distinguishing between the micro-dialects of Arabic. We argue that the currently adopted framing of the ADI task as a single-label classification problem is one of the main reasons for that. We highlight the limitation of the incompleteness of the Dialect labels and demonstrate how it impacts the evaluation of ADI systems. A manual error analysis for the predictions of an ADI, performed by 7 native speakers of different Arabic dialects, revealed that ≈ 67% of the validated errors are not true errors. Consequently, we propose framing ADI as a multi-label classification task and give recommendations for designing new ADI datasets.

AB - Automatic Arabic Dialect Identification (ADI) of text has gained great popularity since it was introduced in the early 2010s. Multiple datasets were developed, and yearly shared tasks have been running since 2018. However, ADI systems are reported to fail in distinguishing between the micro-dialects of Arabic. We argue that the currently adopted framing of the ADI task as a single-label classification problem is one of the main reasons for that. We highlight the limitation of the incompleteness of the Dialect labels and demonstrate how it impacts the evaluation of ADI systems. A manual error analysis for the predictions of an ADI, performed by 7 native speakers of different Arabic dialects, revealed that ≈ 67% of the validated errors are not true errors. Consequently, we propose framing ADI as a multi-label classification task and give recommendations for designing new ADI datasets.

U2 - 10.18653/v1/2023.arabicnlp-1.31

DO - 10.18653/v1/2023.arabicnlp-1.31

M3 - Conference contribution

SP - 385

EP - 398

BT - Proceedings of ArabicNLP 2023

A2 - Sawaf, Hassan

A2 - El-Beltagy, Samhaa

A2 - Zaghouani, Wajdi

A2 - Magdy, Walid

A2 - Abdelali, Ahmed

A2 - Tomeh, Nadi

A2 - Farza, Ibrahim Abu

A2 - Habash, Nizar

A2 - Khalifa, Salam

A2 - Keleg, Amr

A2 - Haddad, Hatem

A2 - Zitouni, Imed

A2 - Mrini, Khalil

A2 - Almatham, Rawan

PB - Association for Computational Linguistics

Y2 - 7 December 2023 through 7 December 2023

ER -

Keleg A, Magdy W. Arabic dialect identification under scrutiny: Limitations of single-label classification. In Sawaf H, El-Beltagy S, Zaghouani W, Magdy W, Abdelali A, Tomeh N, Farza IA, Habash N, Khalifa S, Keleg A, Haddad H, Zitouni I, Mrini K, Almatham R, editors, Proceedings of ArabicNLP 2023. 1 ed. Association for Computational Linguistics. 2023. p. 385-398 doi: 10.18653/v1/2023.arabicnlp-1.31

Arabic dialect identification under scrutiny: Limitations of single-label classification

Abstract

Conference

Access to Document

Fingerprint

Cite this