2024-11-16 –, LT7
Language: English
Python is infamous for its slowless and the GIL problem. In Python 3.4, asyncio was introduced to allow non-blocking I/O, and concurrent.futures was introduced for an easier syntax to write parallel code. But still, Python is not super fast. In this talk, I will show case 10+ ways of generating data in parallel in Python and compare their performance. The unfortunate conclusion is that nothing is always the best, even with the free-threaded Python of 3.13, the silver bullet does not exist.
The talk is focus on CPython and experiment is done on Apple Silicon. It is to generate numpy array of random float as an example, and later to extend into Python lists of floats. I will compare the data generation throughput amongst using multiprocessing, threading, and concurrency.futures modules in Python as well as numba and joblib external libraries. The key result is to highlight the trade off between threading vs multiprocessing, in which if you want to use multiprocessing to work around the GIL, you pay the price of inter-process communication overhead. Even with free-threaded Python in 3.13 that you can avoid the GIL, you can't find a solution that is always better.
Adrian has been using Python for work for more than 20 years. Although not the only programming language to use, it is his favorite for quick experiments and prototyping. His interest is in mathematical modeling, number crunching, high performance computing, and in the last decade, machine learning and AI.