mypy 0.990 is 1000x slow on pydantic codebase vs 0.982

[samuelcolvin]

Bug Report

Running mypy on the current pydantic codebase (specifically the code as of pydantic/pydantic#4735), give the following run times:

• running mypy pydantic with mypy 0.982, and no .mypy_cache: 8s
• running mypy pydantic with mypy 0.982, .mypy_cache present: 0.17s
• running mypy pydantic with mypy 0.990, and no .mypy_cache: 3m19s
• running mypy pydantic with mypy 0.990, .mypy_cache present: 3m6s

(1000x comes from (3 * 60 + 6) / 0.17)

There are quite a few errors, mostly due to no_implicit_optional=True, but still - this seems like a significant regression.

While mypy 0.990 is running, I see 100% usage on 1 CPU.

To Reproduce

Should be as simple as cloning pydantic, checkout that branch (uprev-mypy), install requirements from requirements/linting.txt, run mypy.

Your Environment

• Mypy version used: 0.990
• Mypy command-line flags: just mypy pydantic
• Mypy configuration options from mypy.ini (and other config files): see this bit of pyproject.toml
• Python version used:3.10.0 (just realised how old my version of 3.10 is, I've upgraded to 3.10.8, same result)

[samuelcolvin]

As with #14027, would it be worth adding pydantic to mypy_primer @hauntsaninja?

As with sqlalachemy, pydantic pushes python's type system fairly hard, in particular, the schema for pydantic-core, is defined as a recursive union of TypedDicts.

[ilevkivskyi]

Note that when an error is present in a file, mypy will not use cache for that file.

[ilevkivskyi]

Btw can you try --disable-recursive-aliases? Such big recursive types are known to be slow.

[samuelcolvin]

Yup, that makes a big different, time reduces to 9s.

[ilevkivskyi]

Also, just to be sure, can you also try --enable-recursive-aliases on 0.982?

[samuelcolvin]

Takes 261s as you predicted - seems that's the problem.

[ilevkivskyi]

OK, then this will be not easy to fix. The problem is that you have that big union of 40 items. Previously mypy didn't support recursive types, so most likely all recursive types were either plain Any or shallow TypedDicts with Any. Mypy used to give an error about this however. I guess this error was not shown because pydantic_core is checked as third party dependency? You can try --no-silence-site-packages when using --disable-recursive-aliases. (Btw everything on PYTHONPATH is considered third party, cc @JukkaL)

So now when recursive types are properly supported imagine you have a triple nested dictionary typed as CoreSchema. Mypy will need to try every item in the union until it finds a matching one. This will take on average 20 * 20 * 20 = 8000 type checks for a single dict. To make things worse, TypedDicts are structural, and some have overlapping keys so it may take a while to figure out that a given item doesn't actually match.

We will still try to profile this. Maybe there is some room for optimizations.

[samuelcolvin]

Thanks so much for the explanation.

Simple answer might be to move to pyright, which we already use to check pydantic-core and seems to do a reasonably good job (although that union found a bug in pyright too). (I don't mean this to sound hostile, mypy is amazing and I'd love to continue to use it, but I understand if this is a fundamental limitation of the library)

If it helps, all those TypedDicts have a type key which can (mostly) be used as a discriminator - that's how we validate pydantic-core schema using pydantic-core, see here. Using that discriminator, the number of types checked could be massively reduced, but I guess that's too specific for mypy to be able to pick up on.

[ilevkivskyi]

Btw if you fix the errors coming from no_implicit_optional=True the cache should became effective again. I guess even with recursive types enabled, a typical run should take just few seconds with warm cache.

[samuelcolvin]

ye, I thought that, but it's still not really acceptable for either CI or new contributors for mypy to take minutes.

I guess we'll have to use --disable-recursive-aliases.

[hauntsaninja]

Thanks, pydantic is already in mypy_primer :-) However, we don't currently monitor perf regressions in mypy_primer... it's a feature I've been thinking about though