dispel4py: A Python Framework for Data-Intensive Scientific Computing

R. Filgueira, I. Klampanos, A. Krause, M. David, A. Moreno, M. Atkinson

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

This paper presents dispel4py, a new Python framework for describing abstract stream-based workflows for distributed data-intensive applications. The main aim of dispel4py is to enable scientists to focus on their computation instead of being distracted by details of the computing infrastructure they use. Therefore, special care has been taken to provide dispel4py with the ability to map abstract workflows to different enactment platforms dynamically, at run time. In this work we present four dispel4py mappings: Apache Storm, MPI, multi-threading and sequential. The results show that dispel4py is successful in enacting on different platforms, while also providing scalable performance.
Original languageEnglish
Title of host publicationData Intensive Scalable Computing Systems (DISCS), 2014 International Workshop on
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages8
Publication statusPublished - 1 Nov 2014

Keywords / Materials (for Non-textual outputs)

  • multi-threading
  • natural sciences computing
  • Apache Storm
  • MPI
  • Python framework
  • data-intensive scientific computing
  • dispel4py mappings
  • multithreading
  • stream-based workflows
  • Abstracts
  • Context
  • Java
  • Libraries
  • Noise
  • Storms
  • Topology
  • Data-intensive computing
  • e-Infrastructures
  • data streaming
  • scientific workflows
  • programming frameworks
  • Python


Dive into the research topics of 'dispel4py: A Python Framework for Data-Intensive Scientific Computing'. Together they form a unique fingerprint.

Cite this