If you messed around with Python's command line options or read the official documentation, you might wonder what the -Xgil option or the PYTHON_GIL environment variable did to your scripts, and whether setting either affects performance. The hubbub on popular wheels such as pyo3, python-zstandard, numpy, uv, cffi, and cython supporting the free-threaded interpreter is no passing fad either. For Pythonistas that don't read PEPs in their spare time or contribute to the cpython project itself, an adventure that delves into a less known, yet jaw-dropping aspect of Python awaits!
Python's Global Interpreter Lock, which determines which single thread can execute native Python code and call C API functions, simplifies writing multithreaded code. However, sticking with this execution model leaves out extra performance afforded by modern multicore CPUs with hyperthreading, as automatic locking and unlocking of the GIL does not scale well with thread counts, especially in performance-sensitive workloads.
The newfangled free-threaded interpreter promises salvation when running either pure Python code or with compiled extensions. General multithreading rules apply (prefer thread-local variables, using locks to prevent simultaneous access of shared data), but when dealing with projects containing compiled extensions that directly or indirectly interface with Python's C API, more porting rules also apply.
Key porting tips, including projects using the Limited API, include: port native code away from C API functions that avoid borrowed references because they aren't thread-safe; modify unit tests to catch concurrency bugs arising from assuming the presence of the GIL; and extend CI coverage of Python interpreters both for testing and to build free-threaded compatible wheels.
Outline:
- Introduction (2-3 min.)
- What is the -Xgil option?
- What is the GIL?
- What is the free-threaded interpreter? (6-8 min.)
- Global Interpreter Lock: downsides of automatic serialization of parallel workloads
- How to try out the free-threaded interpreter
- Increased parallelism with the no-GIL interpreter with multi-core CPUs
- Porting tips (15-18 min)
- Adding a trove classifier in pyproject.toml
- Marking your extension module as supporting no-GIL
- Limited API (and PEP 803)
- Bumping key dependencies, including FFI wheels
- Using locks, mutexes, and atomics in native code to prevent concurrency bugs
- Including pytest-run-parallel to catch threading bugs
- Closing Remarks (2 min.)
- Q&A (2 min.)
