Asterism: Pegasus and dispel4py hybrid workflows for data-intensive science

Rosa Filgueira Vicente, Rafael Ferreira da Silva, Amrey Krause, Ewa Deelman, Malcolm Atkinson

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present Asterism, an open source data-intensive framework, which combines the strengths of traditional work-flow management systems with new parallel stream-based data flow systems to run data-intensive applications across
multiple heterogeneous resources, without users having to: re-formulate their methods according to different enactment engines; manage the data distribution across systems; parallelize their methods; co-place and schedule their methods
with computing resources; and store and transfer large/small volumes of data. We also present the Data-Intensive work-flows as a Service (DIaaS) model, which enables easy dataintensive work flow composition and deployment on clouds using containers. The feasibility of Asterism and DIaaS model have been evaluated using a real domain application on the NSF-Chameleon cloud. Experimental results shows how Asterism successfully and eciently exploits combinations of diverse computational platforms, whereas DIaaS delivers specialized software to execute data-intensive applications in a scalable, efficient, and robust way reducing the engineering time and computational cost.
Original languageEnglish
Title of host publicationDataCloud '16 Proceedings of the 7th International Workshop on Data-Intensive Computing in the Cloud
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Pages1-8
Number of pages8
ISBN (Print)978-1-5090-6158-7
DOIs
Publication statusPublished - 13 Nov 2016
Event7th International Workshop on Data-Intensive Computing in the Cloud - Salt Lake City, United States
Duration: 13 Nov 201618 Nov 2016
http://sc16.supercomputing.org/

Conference

Conference7th International Workshop on Data-Intensive Computing in the Cloud
Abbreviated titleSC16
CountryUnited States
CitySalt Lake City
Period13/11/1618/11/16
Internet address

Fingerprint

Dive into the research topics of 'Asterism: Pegasus and dispel4py hybrid workflows for data-intensive science'. Together they form a unique fingerprint.

Cite this