alt.hn

5/22/2025 at 6:06:37 PM

Loading Pydantic models from JSON without running out of memory

 https://pythonspeed.com/articles/pydantic-json-memory/

by itamarst

5/23/2025 at 10:56:26 AM

Pydantic author here. We have plans for an improvement to pydantic where JSON is parsed iteratively, which will make way for reading a file as we parse it. Details in https://github.com/pydantic/pydantic/issues/10032.

Our JSON parser, jiter (https://github.com/pydantic/jiter) already supports iterative parsing, so it's "just" a matter of solving the lifetimes in pydantic-core to validate as we parse.

This should make pydantic around 3x faster at parsing JSON and significantly reduce the memory overhead.

by scolvin

5/23/2025 at 11:12:44 AM

Pydantic is a life changing library, thanks so much for your work!

by Lucasoato

5/23/2025 at 1:16:51 PM

Seconded. Please keep up the awesome work!

by adeeshaek

5/22/2025 at 11:40:30 PM

Having only recently encountered this, does anyone have any insight as to why it takes 2GB to handle a 100MB file?

This looks highly reminiscent (though not exactly the same, pedants) of why people used to get excited about using SAX instead of DOM for xml parsing.

by fidotron

5/23/2025 at 12:31:20 AM

I talk about this more explicitly in the PyCon talk (https://pythonspeed.com/pycon2025/slides/ - video soon) though that's not specifically about Pydantic, but basically:

1. Inefficient parser implementation. It's just... very easy to allocate way too much memory if you don't think about large-scale documents, and very difficult to measure. Common problem with many (but not all) JSON parsers.

2. CPython in-memory representation is large compared to compiled languages. So e.g. 4-digit integer is 5-6 bytes in JSON, 8 in Rust if you do i64, 25ish in CPython. An empty dictionary is 64 bytes.

by itamarst

5/23/2025 at 2:07:03 AM

Funny to see awkward array in this context! (And... do people really store giant datasets in json?!?).

by cozzyd

5/23/2025 at 4:52:36 AM

Often the legacy of an engineer (or team) who "did what they had to do" to meet a deadline, and if they wanted to migrate to something better post-launch, weren't allowed to allocate time to go back and do so.

At least JSON or CSV is better than the ad hoc homegrown formats you found at medium-sized companies that came out of the 90's and 00's.

by chao-

5/23/2025 at 8:53:43 AM

Some people even use AI-generated JSON as a semantic layer over their SQL.

by ljm

5/23/2025 at 3:41:55 AM

My sweet summer child

by jfb

5/23/2025 at 10:11:27 AM

To take 2GB to parse a 100MB file, we increase file size 20x

Let's imagine the file is mostly full of single digit numbers with no spaces (so lists like 2,4,1,0,9,3...).

We need to spend 40 bytes storing a number.

Make a minimal sized class to store an integer:

    class JsonInt:
        x = 1
That object's size is already 48 bytes.

Usually we store floats from JSON, the size of 1 as a float in python is 24 bytes.

Now, you can get smaller, but as soon as you introduce any kind of class structure or not parsing numbers until they are used (in case you want people to be able to intrepret them as ints or floats), you blow through 20x memory size increase.

by CJefferson

5/23/2025 at 11:49:42 AM

> We need to spend 40 bytes storing a number.

But . . . why? Assuming they aren't BigInts or similar these are maximum 8 bytes of actual data. This overhead is ridiculous.

Using classes should enable you to be much smaller than the JSON representation, not larger. For example, V8 does it like https://v8.dev/docs/hidden-classes

> not parsing numbers until they are used

Doesn't this defeat the point of pydantic? It's supposed to be checking the model is valid as it's loaded using jiter. If the data is valid it can be loaded into an efficient representation, and if it's not the errors can be emitted during iterating over it.

by fidotron

5/23/2025 at 1:25:05 PM

"But . . . why?"

This is CPython. This is how it works. It's not particularly related to JSON. That sort of overhead is put on everything. It just hurts the most when the thing you're putting the overhead on is a single integer. It hurts less when you're doing it to, say, a multi-kilobyte string.

Even in your v8 example, that's a JIT optimization, not "how the language works". You break that optimization, which you can do at any moment with any change in your code base, you're back to similar sizes.

Boxing everything lets you easily implement the dynamic scripting language's way of treating everything as an Object of some sort, but it comes at a price. There's a reason dynamic scripting languages, even after the JIT has come through, are generally substantially slower languages. This isn't the only reason, but it's a significant part of it.

by jerf

5/23/2025 at 1:37:11 PM

> Even in your v8 example, that's a JIT optimization, not "how the language works". You break that optimization, which you can do at any moment with any change in your code base, you're back to similar sizes.

The whole point of the v8 optimization is it works in the face of prototype chains that merge etc. as you add new fields dynamically so if you change your code base it adapts.

by fidotron

5/22/2025 at 7:29:19 PM

My problem isn't running out of memory; it's loading in a complex model where the fields are BaseModels and unions of BaseModels multiple levels deep. It doesn't load it all the way and leaves some of the deeper parts as dictionaries. I need like almost a parser to search the space of different loads. Anyone have any ideas for software that does that?

by jmugan

5/22/2025 at 8:04:36 PM

The only reason I can think of for the behavior you are describing is if one of the unioned types at some level of the hierarchy is equivalent to Dict[str, Any]. My understanding is that Pydantic will explore every option provided recursively and raise a ValidationError if none match but will never just give up and hand you a partially validated object.

Are you able to share a snippet that reproduces what you're seeing?

by enragedcacti

5/22/2025 at 11:10:10 PM

That's an interesting idea. It's possible there's a Dict[str,Any] in there. And yeah, my assumption was that it tried everything recursively, but I just wasn't seeing that, and my LLM council said that it did not. But I'll check for a Dict[str,Any]. Unfortunately, I don't have a minimal example, but making one should be my next step.

by jmugan

5/22/2025 at 11:52:36 PM

One thing to watch out for while you debug is that the default 'smart' mode for union discrimination can be very unintuitive. As you can see in this example, an int vs a string can cause a different model to be chosen two layers up even though both are valid. You may have perfectly valid uses of Dict within your model that are being chosen in error because they result in less type coercion. left_to_right mode (or ideally discriminated unions if your data has easy discriminators) will be much more consistent.

    >>> class A(BaseModel):
    >>>     a: int
    >>> class B(BaseModel):
    >>>     b: A
    >>> class C(BaseModel):
    >>>     c: B | Dict[str, Any]

    >>> C.model_validate({'c':{'b':{'a':1}}})
    
    C(c=B(b=A(a=1)))

    >>> C.model_validate({'c':{'b':{'a':"1"}}})

    C(c={'b': {'a': '1'}})

    >>> class C(BaseModel):
    >>>     c: B | Dict[str, Any] = Field(union_mode='left_to_right')
    
    >>> C.model_validate({'c':{'b':{'a':"1"}}})

    C(c=B(b=A(a=1)))

by enragedcacti

5/22/2025 at 10:55:12 PM

going to shamelessly plug my own library here: https://github.com/mivanit/ZANJ

You can have nested dataclasses, as well as specify custom serializers/loaders for things which aren't natively supported by json.

by not_skynet

5/22/2025 at 11:11:36 PM

Ah, but I need something JSON-based.

by jmugan

5/23/2025 at 12:29:30 AM

It does allow dumping to/recovering from json, apologies if that isn't well documented.

Calling `x: str = json.dumps(MyClass(...).serialize())` will get you json you can recover to the original object, nested classes and custom types and all, with `MyClass.load(json.loads(x))`

by not_skynet

5/22/2025 at 8:08:41 PM

At some point, we have to admit we're asking too much from our tools.

I know nothing about your context, but in what context would a single model need to support so many permutations of a data structure? Just because software can, doesn't mean it should.

by cbcoutinho

5/22/2025 at 10:00:43 PM

Anything multi-tenant? There's a reason Salesforce is used for so many large organisations. The multi-nesting lets you account for all the descrepancies that come with scale.

Just tracking payments through multiple tax regions will explode the places where things need to be tweaked.

by shakna

5/23/2025 at 1:00:42 AM

Alternatively, if you had to go with json, you could consider using jsonl. I think I’d start by evaluating whether this is a good application for json. I tend to only want to use it for small files. Binary formats are usually much better in this scenario.

by deepsquirrelnet

5/22/2025 at 8:24:55 PM

i gave up on python dataclasses & json. Using protobufs object within the application itself. I also have a "...Mixin" class for almost every wire model, with extra methods

Automatic, statically typed deserialization is worth the trouble in my opinion

by dgan

5/22/2025 at 7:05:44 PM

So are there downsides to just always setting slots=True on all of my python data types?

by fjasdfas

5/22/2025 at 7:10:32 PM

You can't add extra attributes that weren't part of the original dataclass definition:

  >>> from dataclasses import dataclass
  >>> @dataclass
  ... class C: pass
  ... 
  >>> C().x = 1
  >>> @dataclass(slots=True)
  ... class D: pass
  ... 
  >>> D().x = 1
  Traceback (most recent call last):
    File "<python-input-4>", line 1, in <module>
      D().x = 1
      ^^^^^
  AttributeError: 'D' object has no attribute 'x' and no __dict__ for setting new attributes
Most of the time this is not a thing you actually need to do.

by itamarst

5/22/2025 at 8:05:36 PM

Also some of the introspection stops working e.g. vars().

If you're using dataclasses it's less of an issue because dataclasses.asdict.

by masklinn

5/22/2025 at 8:36:19 PM

I rarely need to dynamically add attributes myself on dataclasses like this but unfortunately this also means things like `@cached_property` won't work because it can't internally cache the method result anywhere.

by monomial

5/23/2025 at 6:38:35 AM

IIRC you can just include a __dict__ slot and @cached_property should start working again. I

by franga2000

5/22/2025 at 8:21:13 PM

Maybe using mmap would also save some memory, I'm not quite sure if this can be implemented in Python.

by zxilly

5/22/2025 at 8:42:14 PM

Once you switch to ijson it will not save any memory, no, because ijson essentially uses zero memory for the parsing. You're just left with the in-memory representation.

by itamarst

5/22/2025 at 6:57:17 PM

I'd like to see a comparison of ijson vs just `json.load(f)`. `ujson` would also be interesting to see.

by thisguy47

5/23/2025 at 4:07:37 AM

How does the speed of the dataclass version compare?

by kayson

5/22/2025 at 7:31:05 PM

Or just dump pydantic and use msgspec instead: https://jcristharif.com/msgspec/

by m_ke

5/22/2025 at 7:54:32 PM

A great feature of pydantic are the validation hooks that let you intercept serialization/deserialization of specific fields and augment behavior.

For example if you are querying a DB that returns a column as a JSON string, trivial with Pydantic to json parse the column are part of deser with an annotation.

Pydantic is definitely slower and not a 'zero cost abstraction', but you do get a lot for it.

by mbb70

5/22/2025 at 7:36:43 PM

msgspec is much more memory efficient out of the box, yes. Also quite fast.

by itamarst

5/23/2025 at 7:30:55 AM

Can it do incremental parsing? Cant tell from a brief look.

by aitchnyu

5/23/2025 at 3:36:15 PM

[dead]

by jtmcivor