Supercomputing with MPI meets the Common Workflow Language standards: an experience report

Rupert W. Nash; Nick Brown; Michael R Crusoe; Max Kontak

doi:10.1109/WORKS51914.2020.00008

Supercomputing with MPI meets the Common Workflow Language standards: an experience report

Rupert W. Nash, Nick Brown, Michael R Crusoe, Max Kontak

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

Use of standards-based workflows is still somewhat unusual by high-performance computing users. In this paper we describe the experience of using the Common Workflow Language (CWL) standards to describe the execution, in parallel, of MPI-parallelised applications. In particular, we motivate and describe the simple extension to the specification which was required, as well as our implementation of this within the CWL reference runner. We discuss some of the unexpected benefits, such as simple use of HPC-oriented performance measurement tools, and CWL software requirements interfacing with HPC module systems. We close with a request for comment from the community on how these features could be adopted within versions of the CWL standards.

Original language	English
Title of host publication	2020 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS)
Publisher	Institute of Electrical and Electronics Engineers
Pages	17-24
Number of pages	8
ISBN (Electronic)	978-0-7381-1040-0
ISBN (Print)	978-0-7381-1041-7
DOIs	https://doi.org/10.1109/WORKS51914.2020.00008
Publication status	Published - 4 Jan 2021
Event	15th Workshop on Workflows in Support of Large-Scale Science - Virtual Duration: 11 Nov 2020 → 11 Nov 2020 https://works-workshop.org/

Workshop

Workshop	15th Workshop on Workflows in Support of Large-Scale Science
Abbreviated title	WORKS 2020
City	Virtual
Period	11/11/20 → 11/11/20
Internet address	https://works-workshop.org/

Access to Document

10.1109/WORKS51914.2020.00008Licence: All Rights Reserved

Supercomputing with MPI_NASH_DOA01102020_AFVAccepted author manuscript, 229 KBLicence: All Rights Reserved

https://arxiv.org/pdf/2010.00422.pdf
https://ieeexplore.ieee.org/document/9308116Licence: All Rights Reserved

Visual Exploration and Sampling Toolkit for Extreme Computing
Baxter, R.
EU government bodies
1/09/18 → 28/02/22
Project: Research
VESTEC: Visual Exploration and Sampling Toolkit for Extreme Computing
Baxter, R. & Brown, N.
1/09/18 → 31/08/21
Project: Research

Cite this

@inproceedings{91b7bbc936274faf89df68402f04d2aa,

title = "Supercomputing with MPI meets the Common Workflow Language standards: an experience report",

abstract = "Use of standards-based workflows is still somewhat unusual by high-performance computing users. In this paper we describe the experience of using the Common Workflow Language (CWL) standards to describe the execution, in parallel, of MPI-parallelised applications. In particular, we motivate and describe the simple extension to the specification which was required, as well as our implementation of this within the CWL reference runner. We discuss some of the unexpected benefits, such as simple use of HPC-oriented performance measurement tools, and CWL software requirements interfacing with HPC module systems. We close with a request for comment from the community on how these features could be adopted within versions of the CWL standards.",

author = "Nash, {Rupert W.} and Nick Brown and Crusoe, {Michael R} and Max Kontak",

year = "2021",

month = jan,

day = "4",

doi = "10.1109/WORKS51914.2020.00008",

language = "English",

isbn = "978-0-7381-1041-7",

pages = "17--24",

booktitle = "2020 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS)",

publisher = "Institute of Electrical and Electronics Engineers",

address = "United States",

note = "15th Workshop on Workflows in Support of Large-Scale Science, WORKS 2020 ; Conference date: 11-11-2020 Through 11-11-2020",

url = "https://works-workshop.org/",

}

Nash, RW , Brown, N, Crusoe, MR & Kontak, M 2021, Supercomputing with MPI meets the Common Workflow Language standards: an experience report. in 2020 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS). Institute of Electrical and Electronics Engineers, pp. 17-24, 15th Workshop on Workflows in Support of Large-Scale Science, Virtual, 11/11/20. https://doi.org/10.1109/WORKS51914.2020.00008

Supercomputing with MPI meets the Common Workflow Language standards: an experience report. / Nash, Rupert W.; Brown, Nick; Crusoe, Michael R et al.
2020 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS). Institute of Electrical and Electronics Engineers, 2021. p. 17-24.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Supercomputing with MPI meets the Common Workflow Language standards: an experience report

AU - Nash, Rupert W.

AU - Brown, Nick

AU - Crusoe, Michael R

AU - Kontak, Max

PY - 2021/1/4

Y1 - 2021/1/4

N2 - Use of standards-based workflows is still somewhat unusual by high-performance computing users. In this paper we describe the experience of using the Common Workflow Language (CWL) standards to describe the execution, in parallel, of MPI-parallelised applications. In particular, we motivate and describe the simple extension to the specification which was required, as well as our implementation of this within the CWL reference runner. We discuss some of the unexpected benefits, such as simple use of HPC-oriented performance measurement tools, and CWL software requirements interfacing with HPC module systems. We close with a request for comment from the community on how these features could be adopted within versions of the CWL standards.

AB - Use of standards-based workflows is still somewhat unusual by high-performance computing users. In this paper we describe the experience of using the Common Workflow Language (CWL) standards to describe the execution, in parallel, of MPI-parallelised applications. In particular, we motivate and describe the simple extension to the specification which was required, as well as our implementation of this within the CWL reference runner. We discuss some of the unexpected benefits, such as simple use of HPC-oriented performance measurement tools, and CWL software requirements interfacing with HPC module systems. We close with a request for comment from the community on how these features could be adopted within versions of the CWL standards.

U2 - 10.1109/WORKS51914.2020.00008

DO - 10.1109/WORKS51914.2020.00008

M3 - Conference contribution

SN - 978-0-7381-1041-7

SP - 17

EP - 24

BT - 2020 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS)

PB - Institute of Electrical and Electronics Engineers

T2 - 15th Workshop on Workflows in Support of Large-Scale Science

Y2 - 11 November 2020 through 11 November 2020

ER -

Supercomputing with MPI meets the Common Workflow Language standards: an experience report

Abstract / Description of output

Workshop

Access to Document

Fingerprint

Projects

Visual Exploration and Sampling Toolkit for Extreme Computing

VESTEC: Visual Exploration and Sampling Toolkit for Extreme Computing

Cite this