Orchestrating the Deployment of Computations in the Cloud with Conductor

Alexander Wieder, Pramod Bhatotia, Ansley Post, Rodrigo Rodrigues

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

When organizations move computation to the cloud, they must choose from a myriad of cloud services that can be used to outsource these jobs. The impact of this choice on price and performance is unclear, even for technical users. To further complicate this choice, factors like price fluctuations due to spot markets, or the cost of recovering from faults must also be factored in. In this paper, we present Conductor, a system that frees cloud customers from the burden of deciding which services to use when deploying MapReduce computations in the cloud. With Conductor, customers only specify goals, e.g., minimizing monetary cost or completion time, and the system automatically selects the best cloud services to use, deploys the computation according to that selection, and adapts to changing conditions at deployment time. The design of Conductor includes several novel features, such as a system to manage the deployment of cloud computations across different services, and a resource abstraction layer that provides a unified interface to these services, therefore hiding their low-level differences and simplifying the planning and deployment of the computation. We implemented Conductor and integrated it with the Hadoop framework. Our evaluation using Amazon Web Services shows that Conductor can find very subtle opportunities for cost savings while meeting deadline requirements, and that Conductor incurs a modest overhead due to planning computations and the resource abstraction layer.
Original languageEnglish
Title of host publicationPresented as part of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12)
Place of PublicationSan Jose, CA
PublisherUsenix
Pages367-381
Number of pages15
ISBN (Print)978-931971-92-8
Publication statusPublished - 2012

Fingerprint Dive into the research topics of 'Orchestrating the Deployment of Computations in the Cloud with Conductor'. Together they form a unique fingerprint.

Cite this