Efficient Machine Translation with Model Pruning and Quantization

Maximiliana Behnke; Nikolay Bogoychev; Alham Fikri Aji; Kenneth Heafield; Graeme Nail; Qianqian Zhu; Svetlana Tchistiakova; Jelmer van der Linde; Pinzhen Chen; Sidharth Kashyap; Roman Grundkiewicz

Efficient Machine Translation with Model Pruning and Quantization

Maximiliana Behnke, Nikolay Bogoychev, Alham Fikri Aji, Kenneth Heafield, Graeme Nail, Qianqian Zhu, Svetlana Tchistiakova, Jelmer van der Linde, Pinzhen Chen, Sidharth Kashyap, Roman Grundkiewicz

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

We participated in all tracks of the WMT 2021 efficient machine translation task: single-core CPU, multi-core CPU, and GPU hardware with throughput and latency conditions. Our submissions combine several efficiency strategies: knowledge distillation, a simpler simple recurrent unit (SSRU) decoder with one or two layers, lexical shortlists, smaller numerical formats, and pruning. For the CPU track, we used quantized 8-bit models. For the GPU track, we experimented with FP16 and 8-bit integers in tensorcores. Some of our submissions optimize for size via 4-bit log quantization and omitting a lexical shortlist. We have extended pruning to more parts of the network, emphasizing component- and block-level pruning that actually improves speed unlike coefficient-wise pruning.

Original language	English
Title of host publication	Proceedings of the Sixth Conference on Machine Translation
Place of Publication	Stroudsburg, PA, USA
Publisher	Association for Computational Linguistics
Pages	775-780
Number of pages	6
ISBN (Print)	978-1-954085-94-7
Publication status	Published - 10 Nov 2021
Event	EMNLP 2021 Sixth Conference on Machine Translation (WMT) - Punta Cana, Dominican Republic Duration: 10 Nov 2021 → 11 Nov 2021 Conference number: 6 https://www.statmt.org/wmt21/

Conference

Conference	EMNLP 2021 Sixth Conference on Machine Translation (WMT)
Abbreviated title	WMT 21
Country/Territory	Dominican Republic
City	Punta Cana
Period	10/11/21 → 11/11/21
Internet address	https://www.statmt.org/wmt21/

Access to Document

Efficient Machine_BEHNKE_DOA05092021_VOR_CC-BYFinal published version, 210 KBLicence: Creative Commons: Attribution (CC-BY)

https://aclanthology.org/2021.wmt-1.74Licence: Creative Commons: Attribution (CC-BY)

Cite this

Behnke, M., Bogoychev, N., Aji, A. F., Heafield, K., Nail, G., Zhu, Q., Tchistiakova, S., van der Linde, J., Chen, P., Kashyap, S., & Grundkiewicz, R. (2021). Efficient Machine Translation with Model Pruning and Quantization. In Proceedings of the Sixth Conference on Machine Translation (pp. 775-780). Association for Computational Linguistics. https://aclanthology.org/2021.wmt-1.74

@inproceedings{3a7287e8ee33432eacafa370ba76f8b5,

title = "Efficient Machine Translation with Model Pruning and Quantization",

abstract = "We participated in all tracks of the WMT 2021 efficient machine translation task: single-core CPU, multi-core CPU, and GPU hardware with throughput and latency conditions. Our submissions combine several efficiency strategies: knowledge distillation, a simpler simple recurrent unit (SSRU) decoder with one or two layers, lexical shortlists, smaller numerical formats, and pruning. For the CPU track, we used quantized 8-bit models. For the GPU track, we experimented with FP16 and 8-bit integers in tensorcores. Some of our submissions optimize for size via 4-bit log quantization and omitting a lexical shortlist. We have extended pruning to more parts of the network, emphasizing component- and block-level pruning that actually improves speed unlike coefficient-wise pruning.",

author = "Maximiliana Behnke and Nikolay Bogoychev and Aji, {Alham Fikri} and Kenneth Heafield and Graeme Nail and Qianqian Zhu and Svetlana Tchistiakova and {van der Linde}, Jelmer and Pinzhen Chen and Sidharth Kashyap and Roman Grundkiewicz",

year = "2021",

month = nov,

day = "10",

language = "English",

isbn = "978-1-954085-94-7",

pages = "775--780",

booktitle = "Proceedings of the Sixth Conference on Machine Translation",

publisher = "Association for Computational Linguistics",

note = "EMNLP 2021 Sixth Conference on Machine Translation (WMT), WMT 21 ; Conference date: 10-11-2021 Through 11-11-2021",

url = "https://www.statmt.org/wmt21/",

}

Behnke, M, Bogoychev, N, Aji, AF, Heafield, K , Nail, G, Zhu, Q, Tchistiakova, S, van der Linde, J, Chen, P, Kashyap, S & Grundkiewicz, R 2021, Efficient Machine Translation with Model Pruning and Quantization. in Proceedings of the Sixth Conference on Machine Translation. Association for Computational Linguistics, Stroudsburg, PA, USA, pp. 775-780, EMNLP 2021 Sixth Conference on Machine Translation (WMT), Punta Cana, Dominican Republic, 10/11/21. <https://aclanthology.org/2021.wmt-1.74>

TY - GEN

T1 - Efficient Machine Translation with Model Pruning and Quantization

AU - Behnke, Maximiliana

AU - Bogoychev, Nikolay

AU - Aji, Alham Fikri

AU - Heafield, Kenneth

AU - Nail, Graeme

AU - Zhu, Qianqian

AU - Tchistiakova, Svetlana

AU - van der Linde, Jelmer

AU - Chen, Pinzhen

AU - Kashyap, Sidharth

AU - Grundkiewicz, Roman

N1 - Conference code: 6

PY - 2021/11/10

Y1 - 2021/11/10

N2 - We participated in all tracks of the WMT 2021 efficient machine translation task: single-core CPU, multi-core CPU, and GPU hardware with throughput and latency conditions. Our submissions combine several efficiency strategies: knowledge distillation, a simpler simple recurrent unit (SSRU) decoder with one or two layers, lexical shortlists, smaller numerical formats, and pruning. For the CPU track, we used quantized 8-bit models. For the GPU track, we experimented with FP16 and 8-bit integers in tensorcores. Some of our submissions optimize for size via 4-bit log quantization and omitting a lexical shortlist. We have extended pruning to more parts of the network, emphasizing component- and block-level pruning that actually improves speed unlike coefficient-wise pruning.

AB - We participated in all tracks of the WMT 2021 efficient machine translation task: single-core CPU, multi-core CPU, and GPU hardware with throughput and latency conditions. Our submissions combine several efficiency strategies: knowledge distillation, a simpler simple recurrent unit (SSRU) decoder with one or two layers, lexical shortlists, smaller numerical formats, and pruning. For the CPU track, we used quantized 8-bit models. For the GPU track, we experimented with FP16 and 8-bit integers in tensorcores. Some of our submissions optimize for size via 4-bit log quantization and omitting a lexical shortlist. We have extended pruning to more parts of the network, emphasizing component- and block-level pruning that actually improves speed unlike coefficient-wise pruning.

M3 - Conference contribution

SN - 978-1-954085-94-7

SP - 775

EP - 780

BT - Proceedings of the Sixth Conference on Machine Translation

PB - Association for Computational Linguistics

CY - Stroudsburg, PA, USA

T2 - EMNLP 2021 Sixth Conference on Machine Translation (WMT)

Y2 - 10 November 2021 through 11 November 2021

ER -

Efficient Machine Translation with Model Pruning and Quantization

Abstract / Description of output

Conference

Access to Document

Fingerprint

Cite this