Sieve: Actionable Insights from Monitored Metrics in Distributed Systems

Jörg Thalheim, Antonio Rodrigues, İstemi Ekin Akkuş, Pramod Bhatotia, Ruichuan Chen, Bimal Viswanath, Lei Jiao, Christof Fetzer

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Major cloud computing operators provide powerful monitoring tools to understand the current (and prior) state of the distributed systems deployed in their infrastructure. While such tools provide a detailed monitoring mechanism at scale, they also pose a significant challenge for the application developers/operators to transform the huge space of monitored metrics into useful insights. These insights are essential to build effective management tools for improving the efficiency, resiliency, and dependability of distributed systems.

This paper reports on our experience with building and deploying Sieve—a platform to derive actionable insights from monitored metrics in distributed systems. Sieve builds on two core components: a metrics reduction framework, and a metrics dependency extractor. More specifically, Sieve first reduces the dimensionality of metrics by automatically filtering out unimportant metrics by observing their signal over time. Afterwards, Sieve infers metrics dependencies between distributed components of the system using a predictive-causality model by testing for Granger Causality.

We implemented Sieve as a generic platform and deployed it for two microservices-based distributed systems: OpenStack and Share-Latex. Our experience shows that (1) Sieve can reduce the number of metrics by at least an order of magnitude (10 − 100×), while preserving the statistical equivalence to the total number of monitored metrics; (2) Sieve can dramatically improve existing monitoring infrastructures by reducing the associated overheads over the entire system stack (CPU—80%, storage—90%, and network—50%); (3) Lastly, Sieve can be effective to support a wide-range of workflows in distributed systems—we showcase two such workflows: Orchestration of autoscaling, and Root Cause Analysis (RCA).
Original languageEnglish
Title of host publicationACM/IFIP/USENIX Middleware 2017
Place of PublicationLas Vegas, Nevada
PublisherACM
Pages14-27
Number of pages14
ISBN (Electronic)978-1-4503-4720-4
DOIs
Publication statusPublished - 11 Dec 2017
Event18th ACM/IFIP/USENIX Middleware Conference - Las Vegas, United States
Duration: 11 Dec 201715 Dec 2017
http://2017.middleware-conference.org/

Conference

Conference18th ACM/IFIP/USENIX Middleware Conference
Abbreviated titleMiddleware 2017
CountryUnited States
CityLas Vegas
Period11/12/1715/12/17
Internet address

Fingerprint

Dive into the research topics of 'Sieve: Actionable Insights from Monitored Metrics in Distributed Systems'. Together they form a unique fingerprint.

Cite this