NetAgg: Using Middleboxes for Application-Specific On-Path Aggregation in Data Centres

Luo Mai, Lukas Rupprecht, Abdul Alim, Paolo Costa, Matteo Migliavacca, Peter Pietzuch, Alexander L. Wolf

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Data centre applications for batch processing (e.g. map/reduce frameworks) and online services (e.g. search engines) scale by distributing data and computation across many servers. They typically follow a partition/aggregation pattern: tasks are first partitioned across servers that process data locally, and then those partial results are aggregated. This data aggregation step, however, shifts the performance bottleneck to the network, which typically struggles to support many-to-few, high-bandwidth traffic between servers.

Instead of performing data aggregation at edge servers, we show that it can be done more efficiently along network paths. We describe NETAGG, a software platform that supports on-path aggregation for network-bound partition/aggregation applications. NETAGG exploits a middlebox-like design, in which dedicated servers (agg boxes) are connected by high-bandwidth links to network switches. Agg boxes execute aggregation functions provided by applications, which alleviates network hotspots because only a fraction of the incoming traffic is forwarded at each hop. NETAGG requires only minimal application changes: it uses shim layers on edge servers to redirect application traffic transparently to the agg boxes. Our experimental results show that NETAGG improves substantially the throughput of two sample applications, the Solr distributed search engine and the Hadoop batch processing framework. Its design allows for incremental deployment in existing data centres and incurs only a modest investment cost.
Original languageEnglish
Title of host publicationProceedings of the 10th ACM International on Conference on Emerging Networking Experiments and Technologies
Place of PublicationNew York, NY, USA
PublisherACM Association for Computing Machinery
Pages249–262
Number of pages13
ISBN (Print)9781450332798
DOIs
Publication statusPublished - 2 Dec 2014
Event10th International Conference on emerging Networking EXperiments and Technologies - Sydney, Australia
Duration: 2 Dec 20145 Dec 2014
https://conferences2.sigcomm.org/co-next/2014/

Publication series

NameCoNEXT ’14
PublisherAssociation for Computing Machinery

Conference

Conference10th International Conference on emerging Networking EXperiments and Technologies
Abbreviated titleCoNEXT 2014
Country/TerritoryAustralia
CitySydney
Period2/12/145/12/14
Internet address

Keywords / Materials (for Non-textual outputs)

  • on-path aggregation
  • data centres
  • in-network processing
  • middleboxes

Fingerprint

Dive into the research topics of 'NetAgg: Using Middleboxes for Application-Specific On-Path Aggregation in Data Centres'. Together they form a unique fingerprint.

Cite this