Evaluation of an InfiniBand Switch: Choose Latency or Bandwidth, but Not Both

M. R. Siavash Katebzadeh, Paolo Costa, Boris Grot

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Today’s cloud datacenters feature a large number of concurrently executing applications with diverse intra-datacenter latency and bandwidth requirements. To remove the network as a potential performance bottleneck, datacenter operators have begun deploying high-end HPC-grade networks, such as InfiniBand (IB), which offer fully offloaded network stacks, remote direct memory access (RDMA) capability, and non-discarding links. While known to provide both low latency and high bandwidth for a single application, it is not clear how well such networks accommodate a mix of latency and bandwidth-sensitive traffic that is likely in a real-world deployment.

As a step toward answering this question, we develop a performance measurement tool for RDMA-based networks, RPerf, that is capable of precisely measuring the IB switch performance without hardware support. Using RPerf, we benchmark a rack-scale IB cluster in isolated and mixedtraffic scenarios. Our key finding is that the evaluated switch can provide either low latency or high bandwidth, but not both simultaneously in a mixed-traffic scenario. We evaluate several options to improve the latency-bandwidth trade-off and demonstrate that none are ideal.
Original languageEnglish
Title of host publication2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Pages180-191
Number of pages12
ISBN (Electronic)978-1-7281-4798-7
ISBN (Print)978-1-7281-4799-4
DOIs
Publication statusPublished - 26 Oct 2020
Event2020 IEEE International Symposium on Performance Analysis of Systems and Software - Virtual conference, United States
Duration: 23 Aug 202026 Aug 2020
https://www.ispass.org/ispass2020/

Symposium

Symposium2020 IEEE International Symposium on Performance Analysis of Systems and Software
Abbreviated titleISPASS 2020
CountryUnited States
CityVirtual conference
Period23/08/2026/08/20
Internet address

Keywords

  • InfiniBand
  • Datacenter Networks
  • Quality-of-Service

Fingerprint

Dive into the research topics of 'Evaluation of an InfiniBand Switch: Choose Latency or Bandwidth, but Not Both'. Together they form a unique fingerprint.

Cite this