Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems

In task-oriented dialogue (ToD), a user holds a conversation with an artificial agentwith the aim of completing a concrete task. Although this technology represents one ofthe central objectives of AI and has been the focus of ever more intense research anddevelopment efforts, it is currently limited to a few narrow domains (e.g., food ordering,ticket booking) and a handful of languages (e.g., English, Chinese). This work provides anextensive overview of existing methods and resources in multilingualToDas an entry pointto this exciting and emerging field. We find that the most critical factor preventing thecreation of truly multilingualToDsystems is the lack of datasets in most languages forboth training and evaluation. In fact, acquiring annotations or human feedback for eachcomponent of modular systems or for data-hungry end-to-end systems is expensive andtedious. Hence, state-of-the-art approaches to multilingualToDmostly rely on (zero- orfew-shot) cross-lingual transfer from resource-rich languages (almost exclusively English),either by means of (i) machine translation or (ii) multilingual representations. These approaches are currently viable only for typologically similar languages and languages with parallel / monolingual corpora available. On the other hand, their effectiveness beyond theseboundaries is doubtful or hard to assess due to the lack of linguistically diverse benchmarks(especially for natural language generation and end-to-end evaluation). To overcome this limitation, we draw parallels between components of the ToD pipeline and other NLP tasks,which can inspire solutions for learning in low-resource scenarios. Finally, we list additional challenges that multilinguality poses for related areas (such as speech, fluency in generated text, and human-centred evaluation), and indicate future directions that hold promise to further expand language coverage and dialogue capabilities of current ToD systems.