HPLT's first release of data and models

Nikolay Arefyev; Mikko Aulamo; Pinzhen Chen; Ona De Gibert Bonet; Barry Haddow; Jindřich Helcl; Bhavitvya Malik; Gema Ramirez-Sánchez; Pavel Stepachev; Jörg Tiedemann; Dušan Variš; Jaume Zaragoza-Bernabeu

HPLT's first release of data and models

Nikolay Arefyev, Mikko Aulamo, Pinzhen Chen, Ona De Gibert Bonet, Barry Haddow, Jindřich Helcl, Bhavitvya Malik, Gema Ramirez-Sánchez, Pavel Stepachev, Jörg Tiedemann, Dušan Variš, Jaume Zaragoza-Bernabeu

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

The High Performance Language Technologies (HPLT) project is a 3-year EU-funded project that started in September 2022. It aims to deliver free, sustainable, and reusable datasets, models, and workflows at scale using high-performance computing. We describe the first results of the project. The data release includes monolingual data in 75 languages at 5.6T tokens and parallel data in 18 language pairs at 96M pairs, derived from 1.8 petabytes of web crawls. Building upon automated and transparent pipelines, the first machine translation (MT) models as well as large language models (LLMs) have been trained and released. Multiple data processing tools and pipelines have also been made public.

Original language	English
Title of host publication	Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 2)
Editors	Carolina Scarton, Charlotte Prescott, Chris Bayliss, Chris Oakley, Joanna Wright, Stuart Wrigley, Xingyi Song, Edward Gow-Smith, Mikel Forcada, Helena Moniz
Publisher	European Association for Machine Translation (EAMT)
Pages	53-54
Number of pages	2
Volume	2
ISBN (Electronic)	9781068690716
Publication status	Published - 27 Jun 2024
Event	The 25th Annual Conference of The European Association for Machine Translation - University of Sheffield, Sheffield, United Kingdom Duration: 24 Jun 2024 → 27 Jun 2024 Conference number: 25 https://eamt2024.sheffield.ac.uk/

Conference

Conference	The 25th Annual Conference of The European Association for Machine Translation
Abbreviated title	EAMT 2024
Country/Territory	United Kingdom
City	Sheffield
Period	24/06/24 → 27/06/24
Internet address	https://eamt2024.sheffield.ac.uk/

Access to Document

ArefyevEtalEAMT2024HPLTsFirstReleaseFinal published version, 92.2 KBLicence: Creative Commons: Attribution No Derivatives (CC-BY-ND)

https://aclanthology.org/2024.eamt-2.27Licence: Creative Commons: Attribution No Derivatives (CC-BY-ND)

Cite this

Arefyev, N., Aulamo, M., Chen, P., De Gibert Bonet, O., Haddow, B., Helcl, J., Malik, B., Ramirez-Sánchez, G., Stepachev, P., Tiedemann, J., Variš, D., & Zaragoza-Bernabeu, J. (2024). HPLT's first release of data and models. In C. Scarton, C. Prescott, C. Bayliss, C. Oakley, J. Wright, S. Wrigley, X. Song, E. Gow-Smith, M. Forcada, & H. Moniz (Eds.), Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 2) (Vol. 2, pp. 53-54). European Association for Machine Translation (EAMT). https://aclanthology.org/2024.eamt-2.27

Arefyev, Nikolay ; Aulamo, Mikko ; Chen, Pinzhen et al. / HPLT's first release of data and models. Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 2). editor / Carolina Scarton ; Charlotte Prescott ; Chris Bayliss ; Chris Oakley ; Joanna Wright ; Stuart Wrigley ; Xingyi Song ; Edward Gow-Smith ; Mikel Forcada ; Helena Moniz. Vol. 2 European Association for Machine Translation (EAMT), 2024. pp. 53-54

@inproceedings{c4848e0908a446b78fe86288412febaf,

title = "HPLT's first release of data and models",

abstract = "The High Performance Language Technologies (HPLT) project is a 3-year EU-funded project that started in September 2022. It aims to deliver free, sustainable, and reusable datasets, models, and workflows at scale using high-performance computing. We describe the first results of the project. The data release includes monolingual data in 75 languages at 5.6T tokens and parallel data in 18 language pairs at 96M pairs, derived from 1.8 petabytes of web crawls. Building upon automated and transparent pipelines, the first machine translation (MT) models as well as large language models (LLMs) have been trained and released. Multiple data processing tools and pipelines have also been made public.",

author = "Nikolay Arefyev and Mikko Aulamo and Pinzhen Chen and {De Gibert Bonet}, Ona and Barry Haddow and Jind{\v r}ich Helcl and Bhavitvya Malik and Gema Ramirez-S{\'a}nchez and Pavel Stepachev and J{\"o}rg Tiedemann and Du{\v s}an Vari{\v s} and Jaume Zaragoza-Bernabeu",

year = "2024",

month = jun,

day = "27",

language = "English",

volume = "2",

pages = "53--54",

editor = "Carolina Scarton and Charlotte Prescott and Chris Bayliss and Chris Oakley and Joanna Wright and Stuart Wrigley and Xingyi Song and Edward Gow-Smith and Mikel Forcada and Helena Moniz",

booktitle = "Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 2)",

publisher = "European Association for Machine Translation (EAMT)",

note = "The 25th Annual Conference of The European Association for Machine Translation, EAMT 2024 ; Conference date: 24-06-2024 Through 27-06-2024",

url = "https://eamt2024.sheffield.ac.uk/",

}

Arefyev, N, Aulamo, M, Chen, P, De Gibert Bonet, O, Haddow, B, Helcl, J, Malik, B, Ramirez-Sánchez, G, Stepachev, P, Tiedemann, J, Variš, D & Zaragoza-Bernabeu, J 2024, HPLT's first release of data and models. in C Scarton, C Prescott, C Bayliss, C Oakley, J Wright, S Wrigley, X Song, E Gow-Smith, M Forcada & H Moniz (eds), Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 2). vol. 2, European Association for Machine Translation (EAMT), pp. 53-54, The 25th Annual Conference of The European Association for Machine Translation, Sheffield, United Kingdom, 24/06/24. <https://aclanthology.org/2024.eamt-2.27>

HPLT's first release of data and models. / Arefyev, Nikolay; Aulamo, Mikko; Chen, Pinzhen et al.
Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 2). ed. / Carolina Scarton; Charlotte Prescott; Chris Bayliss; Chris Oakley; Joanna Wright; Stuart Wrigley; Xingyi Song; Edward Gow-Smith; Mikel Forcada; Helena Moniz. Vol. 2 European Association for Machine Translation (EAMT), 2024. p. 53-54.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - HPLT's first release of data and models

AU - Arefyev, Nikolay

AU - Aulamo, Mikko

AU - Chen, Pinzhen

AU - De Gibert Bonet, Ona

AU - Haddow, Barry

AU - Helcl, Jindřich

AU - Malik, Bhavitvya

AU - Ramirez-Sánchez, Gema

AU - Stepachev, Pavel

AU - Tiedemann, Jörg

AU - Variš, Dušan

AU - Zaragoza-Bernabeu, Jaume

N1 - Conference code: 25

PY - 2024/6/27

Y1 - 2024/6/27

N2 - The High Performance Language Technologies (HPLT) project is a 3-year EU-funded project that started in September 2022. It aims to deliver free, sustainable, and reusable datasets, models, and workflows at scale using high-performance computing. We describe the first results of the project. The data release includes monolingual data in 75 languages at 5.6T tokens and parallel data in 18 language pairs at 96M pairs, derived from 1.8 petabytes of web crawls. Building upon automated and transparent pipelines, the first machine translation (MT) models as well as large language models (LLMs) have been trained and released. Multiple data processing tools and pipelines have also been made public.

AB - The High Performance Language Technologies (HPLT) project is a 3-year EU-funded project that started in September 2022. It aims to deliver free, sustainable, and reusable datasets, models, and workflows at scale using high-performance computing. We describe the first results of the project. The data release includes monolingual data in 75 languages at 5.6T tokens and parallel data in 18 language pairs at 96M pairs, derived from 1.8 petabytes of web crawls. Building upon automated and transparent pipelines, the first machine translation (MT) models as well as large language models (LLMs) have been trained and released. Multiple data processing tools and pipelines have also been made public.

M3 - Conference contribution

VL - 2

SP - 53

EP - 54

BT - Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 2)

A2 - Scarton, Carolina

A2 - Prescott, Charlotte

A2 - Bayliss, Chris

A2 - Oakley, Chris

A2 - Wright, Joanna

A2 - Wrigley, Stuart

A2 - Song, Xingyi

A2 - Gow-Smith, Edward

A2 - Forcada, Mikel

A2 - Moniz, Helena

PB - European Association for Machine Translation (EAMT)

T2 - The 25th Annual Conference of The European Association for Machine Translation

Y2 - 24 June 2024 through 27 June 2024

ER -

Arefyev N, Aulamo M, Chen P, De Gibert Bonet O, Haddow B, Helcl J et al. HPLT's first release of data and models. In Scarton C, Prescott C, Bayliss C, Oakley C, Wright J, Wrigley S, Song X, Gow-Smith E, Forcada M, Moniz H, editors, Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 2). Vol. 2. European Association for Machine Translation (EAMT). 2024. p. 53-54

HPLT's first release of data and models

Abstract / Description of output

Conference

Access to Document

Fingerprint

Cite this