Optimising Performance Through Unbalanced Decompositions

GS2 is an initial value gyrokinetic simulation code developed to study low-frequency turbulence in magnetized plasma. It is parallelised using MPI with the simulation domain decomposed across tasks. The optimal domain decomposition is non-trivial, and complicated by the different requirements of the linear and non-linear parts of the calculations. GS2 users currently choose a data layout, and are guided towards processor counts that are efficient for linear calculations. These choices can, however, lead to data decompositions that are relatively inefficient for the non-linear calculations. We have analysed the performance impact of the data decompositions on the non-linear calculation, and the communications required for those calculations. This has helped us to optimise the decomposition algorithm by using slightly imbalanced data layouts for the non-linear calculations whilst maintaining the existing decompositions for the linear calculations. With the imbalanced layouts we completely eliminate communications for parts of the non-linear simulation.