ParaCrawl: Web-Scale Acquisition of Parallel Corpora

Marta Bañón; Pinzhen Chen; Barry Haddow; Kenneth Heafield; Hieu Hoang; Miquel Esplà-Gomis; Mikel  Forcada; Amir  Kamran; Faheem Kirefu; Philipp Koehn; Sergio Ortiz-Rojas; Leopoldo Pla; Gema Ramírez-Sánchez; Elsa Sarrías; Marek Strelec; Brian Thompson; William Waites; Dion Wiggins; Jaume Zaragoza

doi:10.18653/v1/2020.acl-main.417

ParaCrawl: Web-Scale Acquisition of Parallel Corpora

Marta Bañón , Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz-Rojas, Leopoldo Pla, Gema Ramírez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, Jaume Zaragoza

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We report on methods to create the largest publicly available parallel corpora by crawling the web, using open source software. We empirically compare alternative methods and publish benchmark data sets for sentence alignment and sentence pair filtering. We also describe the parallel corpora released and evaluate their quality and their usefulness to create machine translation systems.

Original language	English
Title of host publication	Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Publisher	Association for Computational Linguistics (ACL)
Pages	4555–4567
Number of pages	13
ISBN (Electronic)	978-1-952148-25-5
DOIs	https://doi.org/10.18653/v1/2020.acl-main.417
Publication status	Published - 10 Jul 2020
Event	2020 Annual Conference of the Association for Computational Linguistics - Hyatt Regency Seattle, Virtual conference, United States Duration: 5 Jul 2020 → 10 Jul 2020 Conference number: 58 https://acl2020.org/

Conference

Conference	2020 Annual Conference of the Association for Computational Linguistics
Abbreviated title	ACL 2020
Country/Territory	United States
City	Virtual conference
Period	5/07/20 → 10/07/20
Internet address	https://acl2020.org/

Access to Document

10.18653/v1/2020.acl-main.417Licence: Creative Commons: Attribution (CC-BY)

ParaCrawl_BANON_DOA03042020_AFVAccepted author manuscript, 491 KBLicence: Creative Commons: Attribution (CC-BY)
ParaCrawl_BANON_DOA03042020_VOR_CC-BYFinal published version, 604 KBLicence: Creative Commons: Attribution (CC-BY)

https://www.aclweb.org/anthology/2020.acl-main.417Licence: Creative Commons: Attribution (CC-BY)

Paracrawl 3: Continued Web-Scale Provision of Parallel Corpora for European Languages
Koehn, P. (Principal Investigator), Heafield, K. (Co-investigator) & Waites, W. (Researcher)
EU government bodies
1/10/19 → 30/09/21
Project: Research

Cite this

Bañón , M., Chen, P., Haddow, B., Heafield, K., Hoang, H., Esplà-Gomis, M., Forcada, M., Kamran, A., Kirefu, F., Koehn, P., Ortiz-Rojas, S., Pla, L., Ramírez-Sánchez, G., Sarrías, E., Strelec, M., Thompson, B., Waites, W., Wiggins, D., & Zaragoza, J. (2020). ParaCrawl: Web-Scale Acquisition of Parallel Corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 4555–4567). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2020.acl-main.417

@inproceedings{aeb1138d856e477a9ea0f3ee5900cab1,

title = "ParaCrawl: Web-Scale Acquisition of Parallel Corpora",

abstract = "We report on methods to create the largest publicly available parallel corpora by crawling the web, using open source software. We empirically compare alternative methods and publish benchmark data sets for sentence alignment and sentence pair filtering. We also describe the parallel corpora released and evaluate their quality and their usefulness to create machine translation systems.",

author = "Marta Ba{\~n}{\'o}n and Pinzhen Chen and Barry Haddow and Kenneth Heafield and Hieu Hoang and Miquel Espl{\`a}-Gomis and Mikel Forcada and Amir Kamran and Faheem Kirefu and Philipp Koehn and Sergio Ortiz-Rojas and Leopoldo Pla and Gema Ram{\'i}rez-S{\'a}nchez and Elsa Sarr{\'i}as and Marek Strelec and Brian Thompson and William Waites and Dion Wiggins and Jaume Zaragoza",

year = "2020",

month = jul,

day = "10",

doi = "10.18653/v1/2020.acl-main.417",

language = "English",

pages = "4555–4567",

booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

note = "2020 Annual Conference of the Association for Computational Linguistics, ACL 2020 ; Conference date: 05-07-2020 Through 10-07-2020",

url = "https://acl2020.org/",

}

Bañón , M, Chen, P , Haddow, B, Heafield, K, Hoang, H, Esplà-Gomis, M, Forcada, M, Kamran, A, Kirefu, F, Koehn, P, Ortiz-Rojas, S, Pla, L, Ramírez-Sánchez, G, Sarrías, E, Strelec, M, Thompson, B, Waites, W, Wiggins, D & Zaragoza, J 2020, ParaCrawl: Web-Scale Acquisition of Parallel Corpora. in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics (ACL), pp. 4555–4567, 2020 Annual Conference of the Association for Computational Linguistics, Virtual conference, Washington, United States, 5/07/20. https://doi.org/10.18653/v1/2020.acl-main.417

TY - GEN

T1 - ParaCrawl: Web-Scale Acquisition of Parallel Corpora

AU - Bañón , Marta

AU - Chen, Pinzhen

AU - Haddow, Barry

AU - Heafield, Kenneth

AU - Hoang, Hieu

AU - Esplà-Gomis, Miquel

AU - Forcada, Mikel

AU - Kamran, Amir

AU - Kirefu, Faheem

AU - Koehn, Philipp

AU - Ortiz-Rojas, Sergio

AU - Pla, Leopoldo

AU - Ramírez-Sánchez, Gema

AU - Sarrías, Elsa

AU - Strelec, Marek

AU - Thompson, Brian

AU - Waites, William

AU - Wiggins, Dion

AU - Zaragoza, Jaume

N1 - Conference code: 58

PY - 2020/7/10

Y1 - 2020/7/10

N2 - We report on methods to create the largest publicly available parallel corpora by crawling the web, using open source software. We empirically compare alternative methods and publish benchmark data sets for sentence alignment and sentence pair filtering. We also describe the parallel corpora released and evaluate their quality and their usefulness to create machine translation systems.

AB - We report on methods to create the largest publicly available parallel corpora by crawling the web, using open source software. We empirically compare alternative methods and publish benchmark data sets for sentence alignment and sentence pair filtering. We also describe the parallel corpora released and evaluate their quality and their usefulness to create machine translation systems.

U2 - 10.18653/v1/2020.acl-main.417

DO - 10.18653/v1/2020.acl-main.417

M3 - Conference contribution

SP - 4555

EP - 4567

BT - Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

PB - Association for Computational Linguistics (ACL)

T2 - 2020 Annual Conference of the Association for Computational Linguistics

Y2 - 5 July 2020 through 10 July 2020

ER -

ParaCrawl: Web-Scale Acquisition of Parallel Corpora

Abstract

Conference

Access to Document

Fingerprint

Projects

Paracrawl 3: Continued Web-Scale Provision of Parallel Corpora for European Languages

Cite this