Unknown type when using a cyclical reference to an inner type in stub.

[MHDante]

This is a particularly esoteric issue, but it's relevant in the context of Protobuf (a common generated code library).

While this issue involves four different repositories (Protobuf, mypy-protobuf, typeshed and pyright). The reason that I'm posting it here is that the error raised by this code does not happen with mypy. I believe this is because of a limitation in pyright's type resolution. However, if I'm incorrect, there is still some opportunity to update the implementation in the other two repos.

Here is some context about how the issue occurred.

1. The protobuf library uses getattr to create synthetic types. One of the types it creates is a named enumeration with integer values. Here's a simplified representation:

# recursive_definition.py

class EnumTypeWrapper(object):
    V = int
    def __init__(self, dict):
        self.value_dict = dict
        super().__init__()

    def __getattr__(self, name):
        return self.value_dict[name]

    def Name(self, number):
        for k, v in self.value_dict.items():
            if v == number:
                return k
        raise AttributeError()

Generated user code can then consume this type by creating instances of this class.

# colors_enum.py
from recursive_definition import EnumTypeWrapper

colors_dict = {
    "RED" :    0,
    "GREEN" :  1,
    "BLUE" :   2,
    }

Colors = EnumTypeWrapper(colors_dict)

2. typeshed adds a generic parameter to add a limitation to the parameters and return values of the class' functions.

# recursive_definition.pyi

from typing import Dict, Generic, TypeVar

_V = TypeVar("_V")

class _EnumTypeWrapper(Generic[_V]):
    def __init__(self, dict: Dict[str, _V]) -> None: ...
    def Name(self, number: _V) -> str: ...

class EnumTypeWrapper(_EnumTypeWrapper[int]): ...

3. Next, this is the point of contention. mypy-protobuf uses the generic parameter and NewType to generate (at edit-time) a stub with a subtype of EnumTypeWrapper. This allows the typechecker to coerce the class methods of the synthetic enum to be constrained to only its members. All the while allowing the actual values to remain as integers. The key in this "trick" is that the definition of the NewType is defined a class member of the sythetic enum type.

Why do this? @nipunn1313 explains here: nipunn1313/mypy-protobuf#169

Here is an example of a generated stub:

# colors_enum.pyi

from typing import NewType
from recursive_definition import _EnumTypeWrapper

class Colors(metaclass= _Colors):
    V = NewType('V', int)

class _Colors(_EnumTypeWrapper[Colors.V], type):
    RED = Colors.V(0)
    GREEN = Colors.V(1)
    BLUE= Colors.V(2)

This file produces the following output from pyright:

PS C:\Repos\pyright-test\proj1> pyright .\src\colors_enum.pyi
Loading configuration file at C:\Repos\pyright-test\pyrightconfig.json
Assuming Python platform Windows
stubPath C:\Repos\pyright-test\typings is not a valid directory.
Searching for source files
Found 1 source file
C:\Repos\pyright-test\proj1\src\colors_enum.pyi
  C:\Repos\pyright-test\proj1\src\colors_enum.pyi:9:39 - error: Cannot access member "V" for type "Type[Colors]"
    Member "V" is unknown (reportGeneralTypeIssues)
  C:\Repos\pyright-test\proj1\src\colors_enum.pyi:9:32 - error: Type of "V" is unknown (reportUnknownMemberType)
2 errors, 0 warnings, 0 infos
Completed in 0.71sec
PS C:\Repos\pyright-test\proj1>

It produces the following output from mypy:

PS C:\Repos\pyright-test\proj1> mypy .\src\colors_enum.pyi   
Success: no issues found in 1 source file

As you can see, analyzing this file raises an error not present in mypy.

The error is likely due to the recursive nature of the type definition. My question is about which implementation is canonical.

4. Finally, this is the usage of these constructs. This is the only code consumers are really concerned with.

# recursive_test.py
from recursive_definition import Colors

name = Colors.Name(5)
print(name)

This produces the following output in pyright:

PS C:\Repos\pyright-test\proj1> pyright src/recursive_test.py
Loading configuration file at C:\Repos\pyright-test\pyrightconfig.json
Assuming Python platform Windows
stubPath C:\Repos\pyright-test\typings is not a valid directory.
Searching for source files
Found 1 source file
C:\Repos\pyright-test\proj1\src\recursive_test.py
  C:\Repos\pyright-test\proj1\src\recursive_test.py:4:8 - error: Type of "Name" is partially unknown
    Type of "Name" is "(number: Unknown) -> str" (reportUnknownMemberType)
1 error, 0 warnings, 0 infos
Completed in 0.776sec

and in mypy:

PS C:\Repos\pyright-test\proj1> mypy .\src\recursive_test.py 
Success: no issues found in 1 source file

---

With all that said. Please let me know if this issue is something that will be addressed (even far in the future) in pyright. Otherwise, I will begin to advocate alternatives.

The only urgency is this open PR: protocolbuffers/protobuf#8182 . The only purpose in the creation of that PR is to complete the pattern mentioned in this issue. I will recommend waiting on merging that PR until this issue is addressed.

As usual, thanks for your efforts.

[erictraut]

I was recently looking at the problem with code generated by mypy-protobuf. Recent versions of this tool have started to introduce cyclical dependencies in the code.

As you know, Pyright is able to handle forward and backward references and will resolve them in whatever order is needed. It's also able to handle cyclical references in some cases. However, in this particular case, I don't see a viable way to break the cycle. The problem is here:

class Colors(metaclass= _Colors):
    V = NewType('V', int)

class _Colors(_EnumTypeWrapper[Colors.V], type):
    RED = Colors.V(0)
    GREEN = Colors.V(1)
    BLUE= Colors.V(2)

The generated code is using _Colors as a metaclass for Colors, but it's using Colors in the base class definition for _Colors. The behavior of accessing Colors.V from Colors depends potentially on the metaclass behavior, but the metaclass can't be defined without accessing Colors.V. It's a tight cycle, and I don't see any way to break it.

My plan was to reach out to the mypy-protobuf maintainers and brainstorm with them about how to generate different code that avoids this cycle.

I'm guessing that this happens to work within mypy because it's a multi-pass analyzer and probably just gives up after n passes of trying to resolve this dependency. Or it simply fails to detect that there is a dependent cycle and proceeds as though everything is OK.

[erictraut]

I've filed this issue in the mypy-protobuf github site: nipunn1313/mypy-protobuf#214.

I'm going to close this issue because I don't see anything more that we can do in pyright to address this issue.