Edinburgh Research Explorer

In situ data analytics for highly scalable cloud modelling on Cray machines

Research output: Contribution to journalArticle

Related Edinburgh Organisations

Open Access permissions

Open

Documents

  • Download as Adobe PDF

    Accepted author manuscript, 351 KB, PDF document

    Licence: Creative Commons: Attribution Non-Commercial (CC-BY-NC)

https://onlinelibrary.wiley.com/doi/abs/10.1002/cpe.4331
Original languageEnglish
Article number4331
Number of pages14
JournalConcurrency and Computation: Practice and Experience
Volume30
Issue number1
Early online date26 Sep 2017
DOIs
Publication statusPublished - 10 Jan 2018
Event60th Meeting of the Cray-User-Group (CUG) - Redmond
Duration: 7 May 201711 May 2017

Abstract

MONC is a highly scalable modelling tool for the investigation of atmospheric flows, turbulence, and cloud microphysics. Typical simulations produce very large amounts of raw data, which must then be analysed for scientific investigation. For performance and scalability reasons, this analysis and subsequent writing to disk should be performed in situ on the data as it is generated; however, one does not wish to pause the computation whilst analysis is carried out. In this paper, we present the analytics approach of MONC, where cores of a node are shared between computation and data analytics. By asynchronously sending their data to an analytics core, the computational cores can run continuously without having to pause for data writing or analysis. We describe our IO server framework and analytics workflow, which is highly asynchronous, along with solutions to challenges that this approach raises and the performance implications of some common configuration choices. The result of this work is a highly scalable analytics approach, and we illustrate on up to 32 768 computational cores of a Cray XC30 that there is minimal performance impact on the runtime when enabling data analytics in MONC and also investigate the performance and suitability of our approach on the KNL.

    Research areas

  • data analysis, multithreading, numerical simulation, parallel processing, software performance, supercomputers

Event

60th Meeting of the Cray-User-Group (CUG)

7/05/1711/05/17

Redmond

Event: Conference

Download statistics

No data available

ID: 44424680