Quick and Reliable Document Alignment via TF/IDF-weighted Cosine Distance

Christian Buck; Philipp Koehn

doi:10.18653/v1/W16-2365

Quick and Reliable Document Alignment via TF/IDF-weighted Cosine Distance

Christian Buck, Philipp Koehn

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

This work describes our submission to the WMT16 Bilingual Document Alignment task. We show that a very simple distance metric, namely Cosine distance of tf/idf weighted document vectors provides a quick and reliable way to align documents. We compare many possible variants for constructing the document vectors. We also introduce a greedy algorithm that runs quicker and performs better in practice than the optimal solution to bipartite graph matching. Our approach shows competitive performance and can be improved even further through combination with URL based pair matching.

Original language	English
Title of host publication	Proceedings of the First Conference on Machine Translation, Volume 2: Shared Task Papers
Place of Publication	Berlin, Germany
Publisher	Association for Computational Linguistics
Pages	669-675
Number of pages	7
ISBN (Print)	978-1-945626-10-4
DOIs	https://doi.org/10.18653/v1/W16-2365
Publication status	Published - 12 Aug 2016
Event	First Conference on Machine Translation - Berlin, Germany Duration: 11 Aug 2016 → 12 Aug 2016 http://www.statmt.org/wmt16/

Conference

Conference	First Conference on Machine Translation
Abbreviated title	WMT16
Country/Territory	Germany
City	Berlin
Period	11/08/16 → 12/08/16
Internet address	http://www.statmt.org/wmt16/

Access to Document

10.18653/v1/W16-2365Licence: Creative Commons: Attribution (CC-BY)

W16-2365Accepted author manuscript, 402 KB

http://aclweb.org/anthology/W/W16/W16-2365.pdfLicence: Creative Commons: Attribution (CC-BY)

Cite this

@inproceedings{365275723b8840499358aea55248ef74,

title = "Quick and Reliable Document Alignment via TF/IDF-weighted Cosine Distance",

abstract = "This work describes our submission to the WMT16 Bilingual Document Alignment task. We show that a very simple distance metric, namely Cosine distance of tf/idf weighted document vectors provides a quick and reliable way to align documents. We compare many possible variants for constructing the document vectors. We also introduce a greedy algorithm that runs quicker and performs better in practice than the optimal solution to bipartite graph matching. Our approach shows competitive performance and can be improved even further through combination with URL based pair matching.",

author = "Christian Buck and Philipp Koehn",

year = "2016",

month = aug,

day = "12",

doi = "10.18653/v1/W16-2365",

language = "English",

isbn = "978-1-945626-10-4",

pages = "669--675",

booktitle = "Proceedings of the First Conference on Machine Translation, Volume 2: Shared Task Papers",

publisher = "Association for Computational Linguistics",

note = "First Conference on Machine Translation, WMT16 ; Conference date: 11-08-2016 Through 12-08-2016",

url = "http://www.statmt.org/wmt16/",

}

Buck, C & Koehn, P 2016, Quick and Reliable Document Alignment via TF/IDF-weighted Cosine Distance. in Proceedings of the First Conference on Machine Translation, Volume 2: Shared Task Papers. Association for Computational Linguistics, Berlin, Germany, pp. 669-675, First Conference on Machine Translation, Berlin, Germany, 11/08/16. https://doi.org/10.18653/v1/W16-2365

TY - GEN

T1 - Quick and Reliable Document Alignment via TF/IDF-weighted Cosine Distance

AU - Buck, Christian

AU - Koehn, Philipp

PY - 2016/8/12

Y1 - 2016/8/12

N2 - This work describes our submission to the WMT16 Bilingual Document Alignment task. We show that a very simple distance metric, namely Cosine distance of tf/idf weighted document vectors provides a quick and reliable way to align documents. We compare many possible variants for constructing the document vectors. We also introduce a greedy algorithm that runs quicker and performs better in practice than the optimal solution to bipartite graph matching. Our approach shows competitive performance and can be improved even further through combination with URL based pair matching.

AB - This work describes our submission to the WMT16 Bilingual Document Alignment task. We show that a very simple distance metric, namely Cosine distance of tf/idf weighted document vectors provides a quick and reliable way to align documents. We compare many possible variants for constructing the document vectors. We also introduce a greedy algorithm that runs quicker and performs better in practice than the optimal solution to bipartite graph matching. Our approach shows competitive performance and can be improved even further through combination with URL based pair matching.

U2 - 10.18653/v1/W16-2365

DO - 10.18653/v1/W16-2365

M3 - Conference contribution

SN - 978-1-945626-10-4

SP - 669

EP - 675

BT - Proceedings of the First Conference on Machine Translation, Volume 2: Shared Task Papers

PB - Association for Computational Linguistics

CY - Berlin, Germany

T2 - First Conference on Machine Translation

Y2 - 11 August 2016 through 12 August 2016

ER -

Quick and Reliable Document Alignment via TF/IDF-weighted Cosine Distance

Abstract

Conference

Access to Document

Fingerprint

Cite this