Optimising Performance through Unbalanced Decompositions

When significant communication costs arise in the solution of multidimensional problems on parallel computers, optimal performance cannot always be achieved by perfectly balancing the computational load across cores. Modest sacrifices in the computational load balance may facilitate substantial overall performance improvements by achieving large savings in the costs associated with communications. This general approach is illustrated by application to GS2, an initial value gyrokinetic simulation code developed to study low-frequency turbulence in magnetized plasma. GS2 is parallelised using MPI with the simulation domain decomposed across tasks. The optimal domain decomposition is non-trivial, and is complicated by the fact that several domain decompositions are needed and that these do not all optimise at the chosen task count. Application to GS2, of the novel approach outlined in this paper, has improved performance by up to 17% for a representative simulation. Similar strategies may be beneficial in a broader class of problems.