Driving asynchronous distributed tasks with events

Research output: Contribution to conferencePaperpeer-review

Abstract / Description of output

Open-source matters, not just to the current cohort of HPC users but also to potential new HPC communities, such as machine learning, themselves often rooted in open-source. Many of these potential new workloads are, by their very nature, far more asynchronous and unpredictable than traditional HPC codes and open-source solutions must be found to enable new communities of developers to easily take advantage of large scale parallel machines. Task-based models have the potential to help here, but many of these either entirely abstract the user from the distributed nature of their code, placing emphasis on the runtime to make important decisions concerning scheduling and locality, or require the programmer to explicitly combine their task based code with a distributed memory technology such as MPI, which adds considerable complexity. In this paper we describe a new approach where the programmer still splits their code up into distinct tasks, but is explicitly aware of the distributed nature of the machine and drives interactions between tasks via events. This provides the best of both worlds; the programmer is able to direct important aspects of parallelism whilst still being abstracted from the low level mechanism of how this parallelism is achieved. We demonstrate our approach via two use-cases, the Graph500 BFS benchmark and in-situ data analytics of MONC, an atmospheric model. For both applications we demonstrate considerably improved performance at large core counts and the result of this work is an approach and open-source library which is readily applicable to a wide range of codes.
Original languageEnglish
Number of pages10
Publication statusE-pub ahead of print - 12 Nov 2018
Event4th Workshop on Open Source Supercomputing - Dallas, United States
Duration: 12 Nov 201812 Nov 2018


Workshop4th Workshop on Open Source Supercomputing
Abbreviated titleOpenSuCo
Country/TerritoryUnited States
Internet address

Keywords / Materials (for Non-textual outputs)

  • arallel Programming Libraries
  • Open source
  • Event driven


Dive into the research topics of 'Driving asynchronous distributed tasks with events'. Together they form a unique fingerprint.

Cite this