Towards Optimising Distributed Data Streaming Graphs Using Parallel Streams

Chee Sun Liew, Malcolm P. Atkinson, Jano I. van Hemert, Liangxiu Han

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Modern scientific collaborations have opened up the opportunity of solving complex problems that involve multi-disciplinary expertise and large-scale computational experiments. These experiments usually involve large amounts of data that are located in distributed data repositories running various software systems, and managed by different organisations. A common strategy to make the experiments more manageable is executing the processing steps as a workflow. In this paper, we look into the implementation of fine-grained data-flow between computational elements in a scientific workflow as streams. We model the distributed computation as a directed acyclic graph where the nodes represent the processing elements that incrementally implement specific subtasks. The processing elements are connected in a pipelined streaming manner, which allows task executions to overlap. We further optimise the execution by splitting pipelines across processes and by introducing extra parallel streams. We identify performance metrics and design a measurement tool to evaluate each enactment. We conducted experiments to evaluate our optimisation strategies with a real world problem in the Life Sciences---EURExpress-II. The paper presents our distributed data-handling model, the optimisation and instrumentation strategies and the evaluation experiments. We demonstrate linear speed up and argue that this use of data-streaming to enable both overlapped pipeline and parallelised enactment is a generally applicable optimisation strategy.
Original languageEnglish
Title of host publicationProceedings of the 19th ACM International Symposium on High Performance Distributed Computing
Place of PublicationNew York, NY, USA
Number of pages12
Publication statusPublished - 2010

Publication series

NameHPDC '10


  • data-intensive computing, distributed computing, optimisation, parallel stream, scientific workflows

Fingerprint Dive into the research topics of 'Towards Optimising Distributed Data Streaming Graphs Using Parallel Streams'. Together they form a unique fingerprint.

Cite this