Scientific Area
Simulation of Turbulent Atmospheric and Oceanic Flows
Short Description
PALM [1] is a large-eddy simulation (LES) model for meteorological systems. It simulates turbulent atmospheric and oceanic flows by solving the Navier-Stokes equations on a discrete three-dimensional (3D) grid. To run on massively parallel computers, the model utilizes the Message Passing Interface (MPI) and a two-dimensional (2D) horizontal domain decomposition. Additionally the code includes OpenMP directives for exploiting shared-memory parallelism on current multi-core CPUs. The model is optimized for modern computer architectures with multiple cache levels and has been tested on more than 32,000 cores [1].
In the past, the developers had already ported PALM to GPUs, most recently in 2016 [2]. However the changes were never integrated into production code because of significant changes to core routines. So in this collaboration, we focused on porting the code with only minimal changes. To accomplish this goal, we used newer features of OpenACC 2.5 such as unstructured data movements.
For development, we designed a test case that simulates a convective atmosphere above a heated ground. In the porting process, we added directives to the subroutines one by one and checked their correctness with the test case. Afterwards we made sure that data stays on the GPU to avoid transfers as much as possible. This involved moving up the call stack, extending the scope of the data regions and making sure that data is still correctly exchanged via MPI.
Furthermore we optimized performance of selected subroutines as we found bottlenecks when running larger data sets. This included the use of cuFFT [3] and porting other parts of the simulation required for the FFT, such as transpose operations. In addition, we implemented a custom code path to find the maximum element and its position in a three-dimensional array. While this contradicts the initial goal of avoiding significant changes, we found it very worth-while from a performance perspective.
Eventually a first version of the code was merged into the main branch in December 2018. To avoid regressions, every new commit is now automatically tested on GPUs via the team's CI pipeline [4]. Whenever a problem is found, the system notifies the developers via email who can inspect the issue and fix the faulty commit.
Results
For the following performance results, we compare total runtimes to an execution on CPUs using an MPI version optimized for modern cache architectures. We build the binary with the Intel Compiler in version 19.0.1.144 and Intel MPI 2018.4.274 and bind each MPI process to a single core. The OpenACC version is compiled with version 19.7 of the PGI compiler and OpenMPI 3.1.3. Additionally we consider a version with support for CUDA-aware MPI that transfers data directly between two GPUs, possibly on different nodes. This benefits the performance of MPI_Alltoall communication used for FFT operations in the pressure solver.
The primary target GPUs were Nvidia Tesla P100 as available in the IT Center's CLAIX system. The first plot compares the total runtime for a representative data set on 1, 2, and 4 nodes of CLAIX-2016. Each node features 2 GPUs and 2x12 CPU cores (Intel Xeon E5-2650v4, Broadwell) which are used for execution with MPI. It can be seen that the OpenACC version with CUDA-aware MPI slightly outperforms the CPU configuration in this scenario.
The more recent CLAIX-2018 system has GPUs with the successor architecture Nvidia Volta (Nvidia Tesla V100). For a fair comparison, we also run the MPI configuration on 2x24 CPU cores (Intel Xeon Platinum 8160, Skylake) as found in that system. The plot shows that the GPU runtime of the same code improves by around 20% compared to CLAIX-2016. However the CPUs are faster by around 30% because their runtime reduces by a factor of up to 2x due to the increased number of cores.
To conclude, the ported application shows comparable performance on CPUs and GPUs. This enables the code to run on a wider range of architectures and systems. In particular, PALM is able to take advantage of current heterogeneous clusters equipped with GPUs. Due to the integration into the main branch, the port is sustainable and available to other users of the code. Furthermore the automated tests run on each commit and will catch problems early.