Itamar Turner-Trauring
Itamar Turner-Trauring is a consultant, and writes about Python performance at https://pythonspeed.com/. He helps companies maintain open source software and speed up their data processing code.
In his spare time he is a volunteer with Cambridge Bicycle Safety, and writes about Cambridge local politics at Let's Change Cambridge.
Session
12-10
14:15
40min
Processing large JSON files without running out of memory
Itamar Turner-Trauring
If you need to process a large JSON file in Python, it’s very easy to run out of memory while loading the data, leading to a super-slow run time or out-of-memory crashes. In this talk you'll learn:
- How to measure memory usage.
- Why loading JSON takes a lot of memory.
- Four different ways to reduce memory usage when loading large JSON files.
Abigail Adams