ParamSpec too many arguments

[markedwards]

Bug Report

Passing list to a function which implements ParamSpec results in mypy throwing Too many arguments when trying to use the result.

To Reproduce

Example:

from typing import Callable, TypeVar

from typing_extensions import ParamSpec

P = ParamSpec('P')
R = TypeVar('R')


def wrapper(func: Callable[P, R]) -> Callable[P, R]:
    return func


wrapper(list)([1, 2, 3])  # "Too many arguments"

Expected Behavior

mypy should treat wrapper(list) exactly the same as list in this case.

Actual Behavior

"Too many arguments" is thrown when using the result of wrapper(list)

Your Environment

• Mypy version used: 0.950
• Mypy command-line flags:
• Mypy configuration options from mypy.ini (and other config files): follow_imports = silent, ignore_missing_imports = True, strict = True
• Python version used: 3.10.4
• Operating system and version: Debian bullseye

[JelleZijlstra]

Is it possible to simplify this to an example that's independent from the stubs for list (https://github.com/python/typeshed/blob/6e18441a2c1e5836eca4e6763a9590d2b9270378/stdlib/builtins.pyi#L864)?

I suspect it has something to do with the overloaded __init__; maybe mypy picks up only the first overload.

[AlexWaygood]

A workaround for this specific case is to use a TypeVar bound to Callable[..., Any] instead of ParamSpec:

from typing import Any, Callable, TypeVar

C = TypeVar('C', bound=Callable[..., Any])

def wrapper(func: C) -> C:
    return func

wrapper(list)([1, 2, 3])

(But this looks like a bug, for sure.)

[AlexWaygood]

Here's a self-contained repro that doesn't reference the typeshed stub for list:

from typing import Callable, TypeVar, Generic, overload, Iterable
from typing_extensions import ParamSpec

T = TypeVar("T")
R = TypeVar("R", covariant=True)
P = ParamSpec("P")

def wrapper(func: Callable[P, R]) -> Callable[P, R]:
    return func

class Foo(Generic[T]):
    @overload
    def __init__(self) -> None: ...
    @overload
    def __init__(self, __iterable: Iterable[T]) -> None: ...

wrapper(Foo)([1, 2, 3])  # error: Too many arguments

https://mypy-play.net/?mypy=latest&python=3.10&gist=1ef09e35aaa253c95d9165827c9add8f

[markedwards]

A workaround for this specific case is to use a TypeVar bound to Callable[..., Any] instead of ParamSpec:

Sure, I can avoid this by not using ParamSpec, but there are reasons I need ParamSpec. This is just a contrived example.

[AlexWaygood]

there are reasons I need ParamSpec. This is just a contrived example.

I assumed so; just thought I'd mention it anyway :)

[markedwards]

Incidentally, this works:

wrapper(cast(Callable[…, list[int]], list))([1, 2, 3])

Not sure if that clarifies anything though.

[AlexWaygood]

It looks like it's a bad interaction between overloads and ParamSpec that causes the bug.

[erictraut]

A ParamSpec "captures" the signature of a callable. In the case of an overload, there are multiple signatures, so the rules for capture get complex and murky. PEP 612 doesn't provide any guidance about how ParamSpec and overload are supposed to interact. When I implemented support for ParamSpec in pyright, I decided not to support this combination. Pyright therefore emits an error message when attempting to use them together.

wrapper(Foo) # ParamSpec cannot be used with overloaded function

It looks like mypy also doesn't support the use of ParamSpec with an overload, but it doesn't emit an error message. An explicit error message should be easy to add and would provide clarity.

If there is demand for using ParamSpec in conjunction with overloaded functions, I recommend that we have a discussion in one of the typing forums and collectively think through the rules, expected behaviors, and edge cases.

[markedwards]

If I’m using ParamSpec to parameterize arguments, then I probably don’t know in advance what kind of function I will be wrapping. Right? So it should be capable of representing all possibilities. Otherwise what’s the point of this?

[AlexWaygood]

FWIW, I think this is a reasonable feature request, though I can see that it might be... tricky to implement. I think Eric's input above is really valuable, but I'm not sure I see why this behaviour necessarily needs to be standardised across all type checkers in order for support to be added in mypy. I feel like it's probably in the purview of a type checker to optionally support additional features not explicitly covered by PEP 612.

But that's just my 2¢. Having a broader discussion, and standardising behaviour across type checkers, is obviously never a bad thing to do.

[markedwards]

There are other variations of this issue.

With tuple:

wrapper(tuple)([1])  # List item 0 has incompatible type "int"; expected "_T_co"

With iter:

wrapper(iter)([1, 2, 3])  # Argument 1 has incompatible type "List[int]"; expected "_SupportsIter[_SupportsNextT]"