We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
Simplifying non-contiguous data transfer with MPI for Python.
- Authors
Nölp, Klaus; Oden, Lena
- Abstract
Python is becoming increasingly popular in scientific computing. The package MPI for Python (mpi4py) allows writing efficient parallel programs that scale across multiple nodes. However, it does not support non-contiguous data via slices, which is a well-known feature of NumPy. In this work, we therefore evaluate several methods to support the direct transfer of non-contiguous arrays in mpi4py. This significantly simplifies the code, while the performance basically stays the same. In a PingPong-, Stencil- and Lattice-Boltzmann-Benchmark, we compare the common manual copying, a NumPy-Copy design and a design that is based on MPI derived datatypes. In one case, the MPI derived datatype design could achieve a speedup of 15% in a Stencil-Benchmark on four compute nodes. Our designs are superior to naive manual copies, but for maximum performance manual copies with pre-allocated buffers or MPI persistent communication will be a better choice.
- Subjects
PYTHONS; SCIENTIFIC computing; PARALLEL programming
- Publication
Journal of Supercomputing, 2023, Vol 79, Issue 17, p20019
- ISSN
0920-8542
- Publication type
Article
- DOI
10.1007/s11227-023-05398-7