Chi: A Scalable and Programmable Control Plane for Distributed Stream Processing Systems

Luo Mai, Kai Zeng, Rahul Potharaju, Le Xu, Steve Suh, Shivaram Venkataraman, Paolo Costa, Terry Kim, Saravanan Muthukrishnan, Vamsi Kuppa, Sudheer Dhulipalla, Sriram Rao

Research output: Contribution to journalArticlepeer-review

Abstract

Stream-processing workloads and modern shared cluster environments exhibit high variability and unpredictability. Combined with the large parameter space and the diverse set of user SLOs, this makes modern streaming systems very challenging to statically configure and tune. To address these issues, in this paper we investigate a novel control-plane design, Chi, which supports continuous monitoring and feedback, and enables dynamic re-configuration. Chi leverages the key insight of embedding control-plane messages in the data-plane channels to achieve a low-latency and flexible control plane for stream-processing systems.

Chi introduces a new reactive programming model and design mechanisms to asynchronously execute control policies, thus avoiding global synchronization. We show how this allows us to easily implement a wide spectrum of control policies targeting different use cases observed in production. Large-scale experiments using production workloads from a popular cloud provider demonstrate the flexibility and efficiency of our approach.
Original languageEnglish
Pages (from-to)1303–1316
Number of pages14
JournalProceedings of the VLDB Endowment (PVLDB)
Volume11
Issue number10
DOIs
Publication statusPublished - 1 Jun 2018

Fingerprint

Dive into the research topics of 'Chi: A Scalable and Programmable Control Plane for Distributed Stream Processing Systems'. Together they form a unique fingerprint.

Cite this