2024-09-27 –, 4F Track4
Python 3.12 introduced a new parallel execution model called "sub interpreters" that uses a per-interpreter GIL to unlock a new way of writing parallel code that's faster than multiprocessing. Python 3.13 introduces another new parallel execution model called "free-threading" (previously called no-gil) that replaces that optionally removes the GIL so that threads can truly be parallel. Does free-threading make sub interpreters redundant? We'll answer that question in this talk.
Python 3.12 introduced a new parallel execution model called "sub interpreters" that uses a per-interpreter GIL to unlock a new way of writing parallel code that's faster than multiprocessing. Python 3.13 introduces another new parallel execution model called "free-threading" (previously called no-gil) that replaces that optionally removes the GIL so that threads can truly be parallel. Does free-threading make sub interpreters redundant? We'll answer that question in this talk. We unpack what sub interpreters are, how to use them and their limitations. We'll explore free-threading, the optional flag to remove the GIL in 3.13. Finally we look at a use case for sub interpreters and free-threading together, a multi-worker, multi-thread ASGI web server. We benchmark this against traditional web server architecture with multiprocessing and threading. So fasten your seatbelts, fellow Pythonistas! Let’s ride the comet tail of parallelism, leaving behind the single-threaded past. 🚀✨
This is a rapidly changing area of Python and I have been contributing to the changes in the core interpreter.
This talk is based on research from 2022-23 on benchmarking the overhead of sub interpreters as well as PRs to improve the sub interpreter API.
I gave a version of this talk at PyCon US in 2024, but if accepted for PyCon JP plan to adapt it with all of the changes since May, of which there are many, including the introduction of a shared memory cache.
Knowledges and know-how the audience can get from your talk:The outcomes of this talk are:
- Attendee has a better understanding of what sub interpreters are
- Attendee has a better understanding of what free-threading is
- Attendee understands the pros and cons of both approaches compared with multiprocessing and traditional threading
- Basic understanding of parallelism
- Basic understanding of concurrency
Advanced
Language of presentation:English
Language of presentation material:Japanese
