Edinburgh Research Explorer

Driving asynchronous distributed tasks with events

Research output: Contribution to conferencePaper

Original languageEnglish
Number of pages10
Publication statusE-pub ahead of print - 12 Nov 2018
Event4th Workshop on Open Source Supercomputing - Dallas, United States
Duration: 12 Nov 201812 Nov 2018
http://www.opensuco.community/2018/04/09/opensuco-4-sc18/

Workshop

Workshop4th Workshop on Open Source Supercomputing
Abbreviated titleOpenSuCo
CountryUnited States
CityDallas
Period12/11/1812/11/18
Internet address

Abstract

Open-source matters, not just to the current cohort of HPC users but also to potential new HPC communities, such as machine learning, themselves often rooted in open-source. Many of these potential new workloads are, by their very nature, far more asynchronous and unpredictable than traditional HPC codes and open-source solutions must be found to enable new communities of developers to easily take advantage of large scale parallel machines. Task-based models have the potential to help here, but many of these either entirely abstract the user from the distributed nature of their code, placing emphasis on the runtime to make important decisions concerning scheduling and locality, or require the programmer to explicitly combine their task based code with a distributed memory technology such as MPI, which adds considerable complexity. In this paper we describe a new approach where the programmer still splits their code up into distinct tasks, but is explicitly aware of the distributed nature of the machine and drives interactions between tasks via events. This provides the best of both worlds; the programmer is able to direct important aspects of parallelism whilst still being abstracted from the low level mechanism of how this parallelism is achieved. We demonstrate our approach via two use-cases, the Graph500 BFS benchmark and in-situ data analytics of MONC, an atmospheric model. For both applications we demonstrate considerably improved performance at large core counts and the result of this work is an approach and open-source library which is readily applicable to a wide range of codes.

    Research areas

  • arallel Programming Libraries, Open source, Event driven

Event

4th Workshop on Open Source Supercomputing

12/11/1812/11/18

Dallas, United States

Event: Workshop

Download statistics

No data available

ID: 76722075