Adaptive Optimizations for Stream-based Workflows

Liang Liang, Rosa Filgueira, Yan Yan

Research output: Chapter in Book/Report/Conference proceedingConference contribution


This work presents three new adaptive optimization techniques to maximize the performance of dispel4py workflows. dispel4py is a parallel Python-based stream-orientated
dataflow framework that acts as a bridge to existing parallel programming frameworks like MPI or Python multiprocessing. When a user runs a dispel4py workflow, the original framework performs a fixed workload distribution among the processes available for the run. This allocation does not take into account workflows’ features, which can cause scalability issues, specially for data-intensive scientific workflows. Therefore, our aim is to improve the performance of dispel4py workflows by testing different workload strategies that automatically adapt to workflows. For achieving this objective, we have implemented three new techniques, called Naive Assignment, Staging and Dynamic Scheduling. The evaluations show that our proposed techniques have significantly improved the performance of the original dispel4py framework.
Original languageEnglish
Title of host publication2020 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS)
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages8
ISBN (Electronic)978-0-7381-1040-0
ISBN (Print)978-0-7381-1041-7
Publication statusPublished - 4 Jan 2021
Event15th Workshop on Workflows in Support of Large-Scale Science - Virtual workshop
Duration: 11 Nov 202011 Nov 2020


Workshop15th Workshop on Workflows in Support of Large-Scale Science
Abbreviated titleWORKS 2020
CityVirtual workshop
Internet address


  • Scientific workflow
  • Stream-based workflow
  • Workflow optimization
  • dispel4py

Fingerprint Dive into the research topics of 'Adaptive Optimizations for Stream-based Workflows'. Together they form a unique fingerprint.

Cite this