pylint endless recursion with short example using setuptools.build_meta import

[kayhayen]

Bug description

""" Nuitka python -m build integration """

# This lines triggers the bug
from setuptools.build_meta import anything

The bug must be coming due to the module imported, if I change the name, no crash.

Command used

['/usr/bin/python3', 
'-m', 
'pylint', 
'--init-hook=import sys;sys.setrecursionlimit(1024*sys.getrecursionlimit())', 
'--disable=I0011,I0012,no-init,bad-whitespace,bad-continuation,E1103,W0632,W1504,C0123,C0411,C0413,R0204,similar-code,cyclic-import,duplicate-code,deprecated-module,deprecated-method,deprecated-argument,assignment-from-none,ungrouped-imports,no-else-return,c-extension-no-member,inconsistent-return-statements,raise-missing-from,import-outside-toplevel,useless-object-inheritance,useless-return,assignment-from-no-return,redundant-u-string-prefix,consider-using-f-string', 
'--enable=useless-suppression', 
'--msg-template="{path}:{line} {msg_id} {symbol} {obj} {msg}"', 
'--reports=no', 
'--persistent=no', 
'--method-rgx=[a-z_][a-zA-Z0-9_]{2,40}$', 
'--module-rgx=.*', 
'--function-rgx=.*', 
'--variable-rgx=.*', 
'--argument-rgx=.*', 
'--dummy-variables-rgx=_.*|trace_collection', 
'--ignored-argument-names=_.*|trace_collection', 
'--const-rgx=.*', 
'--max-line-length=125', 
'--no-docstring-rgx=.*', 
'--max-module-lines=5000', 
'--min-public-methods=0', 
'--max-public-methods=100', 
'--max-args=11', 
'--max-parents=13', 
'--max-statements=50', 
'--max-nested-blocks=10', 
'--max-bool-expr=10', 
'--score=no', 
'--rcfile=/dev/null', 
'--notes=', 
'nuitka/distutils/Build.py']

See above, I am using the init hook, because I have hit recursion limits in the past and there expanding them was helpful. With my command though it was a segfault, so it's clearly going endless.

Pylint output

Error, segfault from pylint. It sees if I remove my recursion limit init hook, I am getting this as an infinite trace:

    yield next(generator)
  File "/home/hayen/.local/lib/python3.7/site-packages/astroid/decorators.py", line 111, in wrapped
    for res in _func(node, context, **kwargs):
  File "/home/hayen/.local/lib/python3.7/site-packages/astroid/bases.py", line 156, in _infer_stmts
    for inf in stmt.infer(context=context):
  File "/home/hayen/.local/lib/python3.7/site-packages/astroid/nodes/node_ng.py", line 170, in infer
    for i, result in enumerate(generator):
  File "/home/hayen/.local/lib/python3.7/site-packages/astroid/decorators.py", line 142, in raise_if_nothing_inferred

Expected behavior

Do not crash.

Pylint version

pylint==2.12.1
astroid==2.9.0
setuptools 45.3.0 from /usr/local/lib/python3.7/dist-packages (Python 3.7)

OS / Environment

Debian Buster, Github Actions too, I don't think OS related

Additional dependencies

setuptools 45.3.0 code is what actually triggers it apparently

[DanielNoord]

I think I saw a similar stacktrace while trying to fix #5321. There might be a related issue here, although I'm not sure.

[DanielNoord]

I think this was fixed. On main:

""" Nuitka python -m build integration """

# This lines triggers the bug
from setuptools.build_meta import anything

No longer crashes either with:

pylint test.py
pylint test.py --init-hook="import sys;sys.setrecursionlimit(1024*sys.getrecursionlimit())" --persistent=no --rcfile=/dev/null

[Pierre-Sassoulas]

I can reproduce on main, even without installing setuptools 45.3.0 it seems.

[DanielNoord]

Oh wait, I didn't see the version requirement for setuptools. Never mind.

[DanielNoord]

I brought it down to the following issue:

import distutils

import six

def get_unpatched_class(cls):
    ...

def get_unpatched(item):
    lookup = (
        get_unpatched_class if isinstance(item, six.class_types) else
        lambda item: None
    )
    return lookup(item)

_Distribution = get_unpatched(distutils.core.Distribution)

class Distribution(_Distribution):
   
    def patch(cls):
        distutils.core.Distribution = cls

This resembles code that is part of setuptools.monkey on 45.3.0. This also crashes on 3.10.

I believe we get stuck on the inference of isinstance, at least that's what the stacktrace is pointing to. Some additional investigation will be needed though.

[DanielNoord]

New reproducing example:

class MyInnerClass:
    ...


class MySubClass:
    inner_class = MyInnerClass


class MyClass:
    sub_class = MySubClass()


def get_unpatched_class(cls):
    return cls


def get_unpatched(item):
    lookup = get_unpatched_class if isinstance(item, type) else lambda item: None
    return lookup(item)


_Child = get_unpatched(MyClass.sub_class.inner_class)


class Child(_Child):
    def patch(cls):
        MyClass.sub_class.inner_class = cls

Not quite sure how to resolve this. We get stuck on the last line as we're trying to infer what gets assigned to MyClass.sub_class.inner_class. We infer that it might be _Child, which is the result of a call to get_unpatched. We try to infer the call result of get_unpatched and we try to infer if the item in the isinstance is of type. To do so we infer item, which we know is MyClass.sub_class.inner_class. We know that MyClass.sub_class.inner_class can be cls, which can be _Child, etc...

Not sure where we should break this loop without causing issues for other valid recursive loops. Any help would be appreciated. (This is obviously an astroid issue).