Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca

Pinzhen Chen, Shaoxiong Ji, Nikolay Bogoychev, Andrey Kutuzov, Barry Haddow, Kenneth Heafield

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Foundational large language models (LLMs) can be instruction-tuned to perform open-domain question answering, facilitating applications like chat assistants. While such efforts are often carried out in a single language, we empirically analyze cost-efficient strategies for multilingual scenarios. Our study employs the Alpaca dataset and machine translations of it to form multilingual data, which is then used to tune LLMs through either low-rank adaptation or full-parameter training. Under a controlled computation budget, comparisons show that multilingual tuning is on par or better than tuning a model for each language. Furthermore, multilingual tuning with downsampled data can be as powerful and more robust. Our findings serve as a guide for expanding language support through instruction tuning.
Original languageEnglish
Title of host publicationFindings of the Association for Computational Linguistics: EACL 2024
PublisherAssociation for Computational Linguistics
Pages1347–1356
Number of pages10
Publication statusPublished - 17 Mar 2024
EventThe 18th Conference of the European Chapter of the Association for Computational Linguistics - , Malta
Duration: 17 Mar 202422 Mar 2024
Conference number: 18
https://2024.eacl.org/

Conference

ConferenceThe 18th Conference of the European Chapter of the Association for Computational Linguistics
Abbreviated titleEACL 2024
Country/TerritoryMalta
Period17/03/2422/03/24
Internet address

Fingerprint

Dive into the research topics of 'Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca'. Together they form a unique fingerprint.

Cite this