Achieving Efficient Strong Scaling with PETSc Using Hybrid MPI/OpenMP Optimisation

Michael Lange, Gerard Gorman, Michele Weiland, Lawrence Mitchell, James Southern

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

The increasing number of processing elements and decreasing memory to core ratio in modern high-performance platforms makes efficient strong scaling a key requirement for numerical algorithms. In order to achieve efficient scalability on massively parallel systems scientific software must evolve across the entire stack to exploit the multiple levels of parallelism exposed in modern architectures. In this paper we demonstrate the use of hybrid MPI/OpenMP parallelisation to optimise parallel sparse matrix-vector multiplication in PETSc, a widely used scientific library for the scalable solution of partial differential equations. Using large matrices generated by Fluidity, an open source CFD application code which uses PETSc as its linear solver engine, we evaluate the effect of explicit communication overlap using task-based parallelism and show how to further improve performance by explicitly load balancing threads within MPI processes. We demonstrate a significant speedup over the pure-MPI mode and efficient strong scaling of sparse matrix-vector multiplication on Fujitsu PRIMEHPC FX10 and Cray XE6 systems.
Original languageEnglish
Title of host publicationSupercomputing
Subtitle of host publication28th International Supercomputing Conference, ISC 2013, Leipzig, Germany, June 16-20, 2013. Proceedings.
PublisherSpringer Berlin Heidelberg
ISBN (Electronic)978-3-642-38750-0
ISBN (Print)978-3-642-38749-4
Publication statusPublished - 2013
EventISC'13 - Leipzig, Germany
Duration: 16 Jun 201320 Jun 2013

Publication series

NameLecture Notes in Computer Science
PublisherSpinger Berlin Heidelberg
ISSN (Print)0302-9743



Keywords / Materials (for Non-textual outputs)

  • PETSc
  • Hybrid MPI/OpenMP
  • strong scaling
  • task-based parallelism
  • hierarchical load balancing
  • sparse matrix-vector multiply


Dive into the research topics of 'Achieving Efficient Strong Scaling with PETSc Using Hybrid MPI/OpenMP Optimisation'. Together they form a unique fingerprint.

Cite this