How to Win a Hot Dog Eating Contest: Distributed Incremental View Maintenance with Batch Updates

Milos Nikolic, Mohammad Dashti, Christoph Koch

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

In the quest for valuable information, modern big data applications continuously monitor streams of data. These applications demand low latency stream processing even when faced with high volume and velocity of incoming changes and the user's desire to ask complex queries. In this paper, we study low-latency incremental computation of complex SQL queries in both local and distributed streaming environments. We develop a technique for the efficient incrementalization of queries with nested aggregates for batch updates. We identify the cases in which batch processing can boost the performance of incremental view maintenance but also demonstrate that tuple-at-a-time processing often can achieve better performance in local mode. Batch updates are essential for enabling distributed incremental view maintenance and amortizing the cost of network communication and synchronization. We show how to derive incremental programs optimized for running on large-scale processing platforms. Our implementation of distributed incremental view maintenance can process tens of million of tuples with few-second latency using hundreds of nodes.
Original languageEnglish
Title of host publicationProceedings of the 2016 International Conference on Management of Data
Place of PublicationNew York, NY, USA
Number of pages16
ISBN (Print)978-1-4503-3531-7
Publication statusPublished - 26 Jun 2016
Event2016 ACM SIGMOD Conference on Management of Data - San Francisco, United States
Duration: 26 Jun 20161 Jul 2016

Publication series

NameSIGMOD '16


Conference2016 ACM SIGMOD Conference on Management of Data
Abbreviated titleSIGMOD 2016
Country/TerritoryUnited States
CitySan Francisco
Internet address

Keywords / Materials (for Non-textual outputs)

  • batch processing, distributed query processing, query compilation, query optimization, recursive incremental view maintenance, stream management system


Dive into the research topics of 'How to Win a Hot Dog Eating Contest: Distributed Incremental View Maintenance with Batch Updates'. Together they form a unique fingerprint.

Cite this