SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages

Tiago Pimentel; Maria Ryskina; Sabrina J. Mielke; Shijie Wu; Eleanor Chodroff; Brian Leonard; Garrett Nicolai; Yustinus Ghanggo Ate; Salam Khalifa; Nizar Habash; Charbel El-Khaissi; Omer Goldman; Michael Gasser; William Lane; Matt Coler; Arturo Oncevay; Jaime Rafael Montoya Samame; Gema Celeste Silva Villegas; Adam Ek; Jean-Philippe Bernardy; Andrey Shcherbakov; Aziyana Bayyr-ool; Karina Sheifer; Sofya Ganieva; Matvey Plugaryov; Elena Klyachko; Ali Salehi; Andrew Krizhanovsky; Natalia Krizhanovsky; Clara Vania; Sardana Ivanova; Aelita Salchak; Christopher Straughn; Zoey Liu; Jonathan North Washington; Duygu Ataman; Witold Kieraś; Marcin Woliński; Totok Suhardijanto; Niklas Stoehr; Zahroh Nuriah; Shyam Ratan; Francis M. Tyers; Edoardo M. Ponti; Grant Aiton; Richard J. Hatcher; Emily Prud'hommeaux; Ritesh Kumar; Mans Hulden; Botond Barta; Dorina Lakatos; Gábor Szolnok; Judit Ács; Mohit Raj; David Yarowsky; Ryan Cotterell; Ben Ambridge; Ekaterina Vylomova

doi:10.18653/v1/2021.sigmorphon-1.25

SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages

Tiago Pimentel, Maria Ryskina, Sabrina J. Mielke, Shijie Wu, Eleanor Chodroff, Brian Leonard, Garrett Nicolai, Yustinus Ghanggo Ate, Salam Khalifa, Nizar Habash, Charbel El-Khaissi, Omer Goldman, Michael Gasser, William Lane, Matt Coler, Arturo Oncevay, Jaime Rafael Montoya Samame, Gema Celeste Silva Villegas, Adam Ek, Jean-Philippe BernardyAndrey Shcherbakov, Aziyana Bayyr-ool, Karina Sheifer, Sofya Ganieva, Matvey Plugaryov, Elena Klyachko, Ali Salehi, Andrew Krizhanovsky, Natalia Krizhanovsky, Clara Vania, Sardana Ivanova, Aelita Salchak, Christopher Straughn, Zoey Liu, Jonathan North Washington, Duygu Ataman, Witold Kieraś, Marcin Woliński, Totok Suhardijanto, Niklas Stoehr, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Richard J. Hatcher, Emily Prud'hommeaux, Ritesh Kumar, Mans Hulden, Botond Barta, Dorina Lakatos, Gábor Szolnok, Judit Ács, Mohit Raj, David Yarowsky, Ryan Cotterell, Ben Ambridge, Ekaterina Vylomova

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

This year's iteration of the SIGMORPHON Shared Task on morphological reinflection focuses on typological diversity and cross-lingual variation of morphosyntactic features. In terms of the task, we enrich UniMorph with new data for 32 languages from 13 language families, with most of them being under-resourced: Kunwinjku, Classical Syriac, Arabic (Modern Standard, Egyptian, Gulf), Hebrew, Amharic, Aymara, Magahi, Braj, Kurdish (Central, Northern, Southern), Polish, Karelian, Livvi, Ludic, Veps, Võro, Evenki, Xibe, Tuvan, Sakha, Turkish, Indonesian, Kodi, Seneca, Asháninka, Yanesha, Chukchi, Itelmen, Eibela. We evaluate six systems on the new data and conduct an extensive error analysis of the systems' predictions. Transformer-based models generally demonstrate superior performance on the majority of languages, achieving textgreater90% accuracy on 65% of them. The languages on which systems yielded low accuracy are mainly under-resourced, with a limited amount of data. Most errors made by the systems are due to allomorphy, honorificity, and form variation. In addition, we observe that systems especially struggle to inflect multiword lemmas. The systems also produce misspelled forms or end up in repetitive loops (e.g., RNN-based models). Finally, we report a large drop in systems' performance on previously unseen lemmas.

Original language	English
Title of host publication	Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology
Editors	Garrett Nicolai, Kyle Gorman, Ryan Cotterell
Place of Publication	Stroudsburg, PA, USA
Publisher	Association for Computational Linguistics
Pages	229-259
Number of pages	31
ISBN (Electronic)	978-1-954085-62-6
DOIs	https://doi.org/10.18653/v1/2021.sigmorphon-1.25
Publication status	Published - 5 Aug 2021
Event	The 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology - Online Duration: 5 Aug 2021 → 5 Aug 2021 Conference number: 18 https://sigmorphon.github.io/workshops/2021/

Workshop

Workshop	The 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology
Abbreviated title	SIGMORPHON 2021
Period	5/08/21 → 5/08/21
Internet address	https://sigmorphon.github.io/workshops/2021/

Access to Document

10.18653/v1/2021.sigmorphon-1.25Licence: Creative Commons: Attribution (CC-BY)

SIGMORPHON_PIMENTEL_DOA28052021_VOR_CC-BYFinal published version, 358 KBLicence: Creative Commons: Attribution (CC-BY)

https://aclanthology.org/2021.sigmorphon-1.25Licence: Creative Commons: Attribution (CC-BY)

Cite this

Pimentel, T., Ryskina, M., Mielke, S. J., Wu, S., Chodroff, E., Leonard, B., Nicolai, G., Ghanggo Ate, Y., Khalifa, S., Habash, N., El-Khaissi, C., Goldman, O., Gasser, M., Lane, W., Coler, M., Oncevay, A., Montoya Samame, J. R., Silva Villegas, G. C., Ek, A., ... Vylomova, E. (2021). SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages. In G. Nicolai, K. Gorman, & R. Cotterell (Eds.), Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology (pp. 229-259). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.sigmorphon-1.25

Pimentel, Tiago ; Ryskina, Maria ; Mielke, Sabrina J. et al. / SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages. Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. editor / Garrett Nicolai ; Kyle Gorman ; Ryan Cotterell. Stroudsburg, PA, USA : Association for Computational Linguistics, 2021. pp. 229-259

@inproceedings{39239d4bf62f402f9b9d971dc25b6b68,

title = "SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages",

abstract = "This year's iteration of the SIGMORPHON Shared Task on morphological reinflection focuses on typological diversity and cross-lingual variation of morphosyntactic features. In terms of the task, we enrich UniMorph with new data for 32 languages from 13 language families, with most of them being under-resourced: Kunwinjku, Classical Syriac, Arabic (Modern Standard, Egyptian, Gulf), Hebrew, Amharic, Aymara, Magahi, Braj, Kurdish (Central, Northern, Southern), Polish, Karelian, Livvi, Ludic, Veps, V{\~o}ro, Evenki, Xibe, Tuvan, Sakha, Turkish, Indonesian, Kodi, Seneca, Ash{\'a}ninka, Yanesha, Chukchi, Itelmen, Eibela. We evaluate six systems on the new data and conduct an extensive error analysis of the systems' predictions. Transformer-based models generally demonstrate superior performance on the majority of languages, achieving textgreater90% accuracy on 65% of them. The languages on which systems yielded low accuracy are mainly under-resourced, with a limited amount of data. Most errors made by the systems are due to allomorphy, honorificity, and form variation. In addition, we observe that systems especially struggle to inflect multiword lemmas. The systems also produce misspelled forms or end up in repetitive loops (e.g., RNN-based models). Finally, we report a large drop in systems' performance on previously unseen lemmas.",

author = "Tiago Pimentel and Maria Ryskina and Mielke, {Sabrina J.} and Shijie Wu and Eleanor Chodroff and Brian Leonard and Garrett Nicolai and {Ghanggo Ate}, Yustinus and Salam Khalifa and Nizar Habash and Charbel El-Khaissi and Omer Goldman and Michael Gasser and William Lane and Matt Coler and Arturo Oncevay and {Montoya Samame}, {Jaime Rafael} and {Silva Villegas}, {Gema Celeste} and Adam Ek and Jean-Philippe Bernardy and Andrey Shcherbakov and Aziyana Bayyr-ool and Karina Sheifer and Sofya Ganieva and Matvey Plugaryov and Elena Klyachko and Ali Salehi and Andrew Krizhanovsky and Natalia Krizhanovsky and Clara Vania and Sardana Ivanova and Aelita Salchak and Christopher Straughn and Zoey Liu and Washington, {Jonathan North} and Duygu Ataman and Witold Kiera{\'s} and Marcin Woli{\'n}ski and Totok Suhardijanto and Niklas Stoehr and Zahroh Nuriah and Shyam Ratan and Tyers, {Francis M.} and Ponti, {Edoardo M.} and Grant Aiton and Hatcher, {Richard J.} and Emily Prud'hommeaux and Ritesh Kumar and Mans Hulden and Botond Barta and Dorina Lakatos and G{\'a}bor Szolnok and Judit {\'A}cs and Mohit Raj and David Yarowsky and Ryan Cotterell and Ben Ambridge and Ekaterina Vylomova",

year = "2021",

month = aug,

day = "5",

doi = "10.18653/v1/2021.sigmorphon-1.25",

language = "English",

pages = "229--259",

editor = "Garrett Nicolai and Kyle Gorman and Ryan Cotterell",

booktitle = "Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology",

publisher = "Association for Computational Linguistics",

note = "The 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, SIGMORPHON 2021 ; Conference date: 05-08-2021 Through 05-08-2021",

url = "https://sigmorphon.github.io/workshops/2021/",

}

Pimentel, T, Ryskina, M, Mielke, SJ, Wu, S, Chodroff, E, Leonard, B, Nicolai, G, Ghanggo Ate, Y, Khalifa, S, Habash, N, El-Khaissi, C, Goldman, O, Gasser, M, Lane, W, Coler, M, Oncevay, A, Montoya Samame, JR, Silva Villegas, GC, Ek, A, Bernardy, J-P, Shcherbakov, A, Bayyr-ool, A, Sheifer, K, Ganieva, S, Plugaryov, M, Klyachko, E, Salehi, A, Krizhanovsky, A, Krizhanovsky, N, Vania, C, Ivanova, S, Salchak, A, Straughn, C, Liu, Z, Washington, JN, Ataman, D, Kieraś, W, Woliński, M, Suhardijanto, T, Stoehr, N, Nuriah, Z, Ratan, S, Tyers, FM, Ponti, EM, Aiton, G, Hatcher, RJ, Prud'hommeaux, E, Kumar, R, Hulden, M, Barta, B, Lakatos, D, Szolnok, G, Ács, J, Raj, M, Yarowsky, D, Cotterell, R, Ambridge, B & Vylomova, E 2021, SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages. in G Nicolai, K Gorman & R Cotterell (eds), Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. Association for Computational Linguistics, Stroudsburg, PA, USA, pp. 229-259, The 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, 5/08/21. https://doi.org/10.18653/v1/2021.sigmorphon-1.25

SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages. / Pimentel, Tiago; Ryskina, Maria; Mielke, Sabrina J. et al.
Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. ed. / Garrett Nicolai; Kyle Gorman; Ryan Cotterell. Stroudsburg, PA, USA: Association for Computational Linguistics, 2021. p. 229-259.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages

AU - Pimentel, Tiago

AU - Ryskina, Maria

AU - Mielke, Sabrina J.

AU - Wu, Shijie

AU - Chodroff, Eleanor

AU - Leonard, Brian

AU - Nicolai, Garrett

AU - Ghanggo Ate, Yustinus

AU - Khalifa, Salam

AU - Habash, Nizar

AU - El-Khaissi, Charbel

AU - Goldman, Omer

AU - Gasser, Michael

AU - Lane, William

AU - Coler, Matt

AU - Oncevay, Arturo

AU - Montoya Samame, Jaime Rafael

AU - Silva Villegas, Gema Celeste

AU - Ek, Adam

AU - Bernardy, Jean-Philippe

AU - Shcherbakov, Andrey

AU - Bayyr-ool, Aziyana

AU - Sheifer, Karina

AU - Ganieva, Sofya

AU - Plugaryov, Matvey

AU - Klyachko, Elena

AU - Salehi, Ali

AU - Krizhanovsky, Andrew

AU - Krizhanovsky, Natalia

AU - Vania, Clara

AU - Ivanova, Sardana

AU - Salchak, Aelita

AU - Straughn, Christopher

AU - Liu, Zoey

AU - Washington, Jonathan North

AU - Ataman, Duygu

AU - Kieraś, Witold

AU - Woliński, Marcin

AU - Suhardijanto, Totok

AU - Stoehr, Niklas

AU - Nuriah, Zahroh

AU - Ratan, Shyam

AU - Tyers, Francis M.

AU - Ponti, Edoardo M.

AU - Aiton, Grant

AU - Hatcher, Richard J.

AU - Prud'hommeaux, Emily

AU - Kumar, Ritesh

AU - Hulden, Mans

AU - Barta, Botond

AU - Lakatos, Dorina

AU - Szolnok, Gábor

AU - Ács, Judit

AU - Raj, Mohit

AU - Yarowsky, David

AU - Cotterell, Ryan

AU - Ambridge, Ben

AU - Vylomova, Ekaterina

N1 - Conference code: 18

PY - 2021/8/5

Y1 - 2021/8/5

N2 - This year's iteration of the SIGMORPHON Shared Task on morphological reinflection focuses on typological diversity and cross-lingual variation of morphosyntactic features. In terms of the task, we enrich UniMorph with new data for 32 languages from 13 language families, with most of them being under-resourced: Kunwinjku, Classical Syriac, Arabic (Modern Standard, Egyptian, Gulf), Hebrew, Amharic, Aymara, Magahi, Braj, Kurdish (Central, Northern, Southern), Polish, Karelian, Livvi, Ludic, Veps, Võro, Evenki, Xibe, Tuvan, Sakha, Turkish, Indonesian, Kodi, Seneca, Asháninka, Yanesha, Chukchi, Itelmen, Eibela. We evaluate six systems on the new data and conduct an extensive error analysis of the systems' predictions. Transformer-based models generally demonstrate superior performance on the majority of languages, achieving textgreater90% accuracy on 65% of them. The languages on which systems yielded low accuracy are mainly under-resourced, with a limited amount of data. Most errors made by the systems are due to allomorphy, honorificity, and form variation. In addition, we observe that systems especially struggle to inflect multiword lemmas. The systems also produce misspelled forms or end up in repetitive loops (e.g., RNN-based models). Finally, we report a large drop in systems' performance on previously unseen lemmas.

AB - This year's iteration of the SIGMORPHON Shared Task on morphological reinflection focuses on typological diversity and cross-lingual variation of morphosyntactic features. In terms of the task, we enrich UniMorph with new data for 32 languages from 13 language families, with most of them being under-resourced: Kunwinjku, Classical Syriac, Arabic (Modern Standard, Egyptian, Gulf), Hebrew, Amharic, Aymara, Magahi, Braj, Kurdish (Central, Northern, Southern), Polish, Karelian, Livvi, Ludic, Veps, Võro, Evenki, Xibe, Tuvan, Sakha, Turkish, Indonesian, Kodi, Seneca, Asháninka, Yanesha, Chukchi, Itelmen, Eibela. We evaluate six systems on the new data and conduct an extensive error analysis of the systems' predictions. Transformer-based models generally demonstrate superior performance on the majority of languages, achieving textgreater90% accuracy on 65% of them. The languages on which systems yielded low accuracy are mainly under-resourced, with a limited amount of data. Most errors made by the systems are due to allomorphy, honorificity, and form variation. In addition, we observe that systems especially struggle to inflect multiword lemmas. The systems also produce misspelled forms or end up in repetitive loops (e.g., RNN-based models). Finally, we report a large drop in systems' performance on previously unseen lemmas.

U2 - 10.18653/v1/2021.sigmorphon-1.25

DO - 10.18653/v1/2021.sigmorphon-1.25

M3 - Conference contribution

SP - 229

EP - 259

BT - Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology

A2 - Nicolai, Garrett

A2 - Gorman, Kyle

A2 - Cotterell, Ryan

PB - Association for Computational Linguistics

CY - Stroudsburg, PA, USA

T2 - The 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology

Y2 - 5 August 2021 through 5 August 2021

ER -

Pimentel T, Ryskina M, Mielke SJ, Wu S, Chodroff E, Leonard B et al. SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages. In Nicolai G, Gorman K, Cotterell R, editors, Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. Stroudsburg, PA, USA: Association for Computational Linguistics. 2021. p. 229-259 doi: 10.18653/v1/2021.sigmorphon-1.25

SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages

Abstract

Workshop

Access to Document

Fingerprint

Cite this