Readit Newsitamarst commented on Loading Pydantic models from JSON without running out of memory pythonspeed.com/articles/... · Posted by u/itamarst
itamarst commented on Loading Pydantic models from JSON without running out of memory pythonspeed.com/articles/... · Posted by u/itamarst
Having only recently encountered this, does anyone have any insight as to why it takes 2GB to handle a 100MB file?
This looks highly reminiscent (though not exactly the same, pedants) of why people used to get excited about using SAX instead of DOM for xml parsing.
I talk about this more explicitly in the PyCon talk (https://pythonspeed.com/pycon2025/slides/ - video soon) though that's not specifically about Pydantic, but basically:
1. Inefficient parser implementation. It's just... very easy to allocate way too much memory if you don't think about large-scale documents, and very difficult to measure. Common problem with many (but not all) JSON parsers.
2. CPython in-memory representation is large compared to compiled languages. So e.g. 4-digit integer is 5-6 bytes in JSON, 8 in Rust if you do i64, 25ish in CPython. An empty dictionary is 64 bytes.
itamarst commented on Loading Pydantic models from JSON without running out of memory pythonspeed.com/articles/... · Posted by u/itamarst
Maybe using mmap would also save some memory, I'm not quite sure if this can be implemented in Python.
Once you switch to ijson it will not save any memory, no, because ijson essentially uses zero memory for the parsing. You're just left with the in-memory representation.
itamarst commented on Loading Pydantic models from JSON without running out of memory pythonspeed.com/articles/... · Posted by u/itamarst
Or just dump pydantic and use msgspec instead: https://jcristharif.com/msgspec/
msgspec is much more memory efficient out of the box, yes. Also quite fast.
Our JSON parser, jiter (https://github.com/pydantic/jiter) already supports iterative parsing, so it's "just" a matter of solving the lifetimes in pydantic-core to validate as we parse.
This should make pydantic around 3x faster at parsing JSON and significantly reduce the memory overhead.