Parallel computing in Python: Current state and recent advances

Modern hardware is multi-core. It is crucial for Python to provide
high-performance parallelism. This talk will expose to both data-scientists and
library developers the current state of affairs and the recent advances for
parallel computing with Python. The goal is to help practitioners and
developers to make better decisions on this matter.

I will first cover how Python can interface with parallelism, from leveraging
external parallelism of C-extensions –especially the BLAS family– to Python's
multiprocessing and multithreading API. I will touch upon use cases, e.g single
vs multi machine, as well as and pros and cons of the various solutions for
each use case. Most of these considerations will be backed by benchmarks from
the scikit-learn machine
learning library.

From these low-level interfaces emerged higher-level parallel processing
libraries, such as concurrent.futures,
joblib and
loky (used by
dask and scikit-learn) These
libraries make it easy for Python programmers to use safe and reliable
parallelism in their code. They can even work in more exotic situations, such
as interactive sessions, in which Python’s native multiprocessing support tends
to fail. I will describe their purpose as well as the canonical use-cases they
address.

The last part of this talk will focus on the most recent advances in the Python
standard library, addressing one of the principal performance bottlenecks of
multi-core/multi-machine processing, which is data communication. We will
present a new
API
for shared-memory management between different Python processes, and
performance improvements for the serialization of large Python objects (PEP
574, pickle
extensions). These performance
improvements will be leveraged by distributed data science frameworks such as
dask, ray and
pyspark.