dispel4py: An User-friendly Framework for Describing eScience Applications

Rosa Filgueira, Amrey Krause, Malcolm Atkinson, Iraklis Klampanos, Alessandro Spinuso, Susana Sanchez-Exposito

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present dispel4py a versatile data-intensive kit presented as a standard Python library. It empowers scientists to experiment and test ideas using their familiar rapid-prototyping environment. It delivers mappings to diverse computing infrastructures, including cloud technologies, HPC architectures and specialised data-intensive machines, to move seamlessly into production with large-scale data loads. The mappings are fully automated, so that the encoded data analyses and data handling are completely unchanged. The underpinning model is lightweight composition of fine-grained operations on data, coupled together by data streams that use the lowest cost technology available. These fine-grained workflows are locally interpreted during development and mapped to multiple nodes and systems such as MPI and Storm for production. We explain why such an approach is becoming more essential in order that data-driven research can innovate rapidly and exploit the growing wealth of data while adapting to current technical trends. We show how provenance management is provided to improve understanding and reproducibility, and how a registry supports consistency and sharing. Three application domains are reported and measurements on multiple infrastructures show the optimisations achieved. Finally we present the next steps to achieve scalability and performance.
Original languageEnglish
Title of host publicationProceedings of 11th IEEE eScience 2015, Munich, Germany, September 1-4
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Pages454 - 464
Number of pages11
DOIs
Publication statusPublished - 30 Sep 2015

Fingerprint

Dive into the research topics of 'dispel4py: An User-friendly Framework for Describing eScience Applications'. Together they form a unique fingerprint.

Cite this