Scientific Area
Real-Time Singles and Coincidence Processing, Positron Emission Tomography
Short Description
For the deployment of novel PET scanners developed in the context of the HYPMED project [1], the institute for Experimental Molecular Imaging (ExMI) aims at utilizing software-based approaches for its singles and coincidence processing chain [2,3]. Contrary to current hardware-based implementations, this allows for easier prototyping and the retrieval of raw scanner data, which would otherwise be lost. However, achieving real-time processing throughput becomes a challenge with the increasing data streams of the scanners and the growing complexity of state-of-the-art positioning algorithms [4].
The application framework developed at the ExMI utilizes the Message Passing Interface (MPI) to allow for distributed processing of incoming scanner data. While throughputs of above 500 MB/s can already be reached with the current implementation, scalability to larger detector systems is desirable. As the application data input is limited by the physical connection between acquisition device and control PC, a throughput rate of 1-2 GB/s is targeted.
For performance analysis, hardware performance counter information collected during program execution gave insight into the dynamic application behaviour. We were able to optimize the locality of memory accesses since the collected metrics revealed high waiting times due to data translation lookaside buffer (TLB) cache misses. As data TLB cache entries map to 4 KB pages, memory accesses exceeded this range. By sorting a critical data structure before processing its entries, scalability and throughput rates could be improved significantly.
The performance counter data additionally showed low vectorization ratios in the kernel function. Although a memory layout of a structure of arrays was used inside the kernel loop, memory accesses proved to be irregular. Utilizing an array of structures instead led to scalar code execution but had a positive effect on execution time.
Results
Starting point of our performance evaluation was the original source code parallelized with MPI, utilizing a release build for compilation with the Intel 18.0 compiler and IntelMPI 2017.4. The stable-sort-timeline version contains optimizations aimed at improving data TLB cache locality, while vector-of-structs utilizes an array of structures to improve data locality within the kernel as described above.
The graphs below show the median throughput rate of 20 application runs on a single Intel Xeon Broadwell EP node with 24 cores, one node of the CLAIX2016 partition of the RWTH Compute Cluster. The stable-sort-timeline version speeds up execution by up to 97% compared to the initial release version. Additionally, parallel efficiency for 48 processes is increased from 41% to 58%. Further runtime improvements by the vector-of-structs version add up to an overall doubling of the throughput rate over the initial release version resulting in a peak throughput rate of 1,250 MB/s. Although this outcome satisfies the set goal of reaching a throughput rate above 1 GB/s, future collaboration efforts may focus on the analysis of other critical processing steps for achieving further performance improvements.