[BUG] `ImportError` running entrypoint in setuptools 64.0.0+ when `PYTHONPATH` defined

[edmorley]

setuptools version

setuptools==64.0.0

Python version

Python 3.10

OS

Debian (official Python docker images)

Additional environment information

Last known good version: setuptools v63.4.3
First failing version: setuptools v64.0.0 (also confirmed failing with setuptools v65.0.2)

Description

Installing a local package (which has an entrypoint) as editable, now results in an ImportError when that entrypoint is run with PYTHONPATH set, when it worked before.

Example project repo:
https://github.com/edmorley/testcase-setuptools-pep660-pythonpath

Expected behavior

Either:

1. Existing package continues to work with setuptools v64's new PEP660 editable mode (if what the package is doing is reasonable)
2. Or, if a certain workflow/scenario is no longer supported, then ideally for setuptools to output an error or warning at build/install time, rather than there being a hard to diagnose ImportError at runtime.

How to Reproduce

1. git clone https://github.com/edmorley/testcase-setuptools-pep660-pythonpath && cd testcase-setuptools-pep660-pythonpath
2. docker build -t testcase . && docker run --rm -it -e PYTHONPATH=/src testcase hello

Notes:

• This testcase uses Python 3.10.6, pip 22.2.2, setuptools 65.0.2, wheel 0.37.1.
• Any of the following prevents the issue from occurring:
  • Downgrading to setuptools <64 (eg: 63.4.3) in the [build-system] requires of pyproject.toml.
  • Removing the -e PYTHONPATH=/src from the docker run command.
• However using SETUPTOOLS_ENABLE_FEATURE=legacy-editable doesn't seem to help (uncomment in Dockerfile to try it).

Output

Traceback (most recent call last):
  File "/usr/local/bin/hello", line 5, in <module>
    from my_package import hello
ImportError: cannot import name 'hello' from 'my_package' (unknown location)

[edmorley]

I'm presuming the issue here is that the new PEP660-style entrypoint wrapper script in bin/ does only a from my_package import hello, which now only looks in <repo>/my_package (due to the PYTHONPATH=/src).

Whereas before the wrapper script managed to find <repo>/my_package/my_package since it iterates through checking whether the entrypoint exists (I've only skimmed the code here; I could be wrong):

setuptools/setuptools/command/easy_install.py

Lines 2074 to 2081 in fd632df

	def importlib_load_entry_point(spec, group, name):
	dist_name, _, _ = spec.partition('==')
	matches = (
	entry_point
	for entry_point in distribution(dist_name).entry_points
	if entry_point.group == group and entry_point.name == name
	)
	return next(matches).load()

[edmorley]

• using SETUPTOOLS_ENABLE_FEATURE=legacy-editable doesn't seem to help

Ah this is because the docs were wrong - the feature is actually called SETUPTOOLS_ENABLE_FEATURES. (Note the trailing "S".)

I've opened a PR to fix the docs: #3538

[abravalheri]

Hi @edmorley thank you very much for reporting this problem, the very complete description and the very nice reproducer.

I think what is happening here is that when you set PYTHONPATH=/src, Python import system gets confused because it finds the /src/my_package folder before setuptools has the chance to tell Python to import /src/my_package/my_package.

Note that any folder can be a package in Python, it does not necessarily needs to contain a __init__.py file. So if you add /src to PYTHONPATH, all subdirectories will be considered packages, including the wrong /src/my_package directory.

If you change that directory name, everything works fine:

--- a/Dockerfile
+++ b/Dockerfile
@@ -2,9 +2,10 @@ FROM python:3.10.6

 RUN pip install pip==22.2.2 setuptools==65.0.2 wheel==0.37.1
 COPY . /src
-WORKDIR /src
+RUN mv /src/my_package /src/my_package_root
+WORKDIR /tmp

 # Uncomment to enable legacy mode (however it doesn't help with this issue)
 # ENV SETUPTOOLS_ENABLE_FEATURE=legacy-editable

-RUN pip install -e ./my_package
+RUN pip install -e /src/my_package_root

[abravalheri]

Ah this is because the docs were wrong - the feature is actually called SETUPTOOLS_ENABLE_FEATURES. (Note the trailing "S".)

I've opened a PR to fix the docs: #3538

Thank you very much!

[abravalheri]

I'm presuming the issue here is that the new PEP660-style entrypoint wrapper script in bin/ does only a from my_package import hello, which now only looks in /my_package (due to the PYTHONPATH=/src).
Whereas before the wrapper script managed to find /my_package/my_package since it iterates through checking whether the entrypoint exists (I've only skimmed the code here; I could be wrong):

Yes, that seems to be the case.

The wrapper script now is generated by pip (via distlib), not by setuptools itself.

[abravalheri]

I don't know if there is anything setuptools can do about this (the wrappers are created by pip).
To be sincere, it seems that everything else is operating how they should operate?

[edmorley]

Ah I didn't realise it was pip creating the bin wrapper script now.

Yeah I'm not sure there is much setuptools can do here either. I guess this is just another thing under the list of reasons why src/ based project layouts are just safer!

Part of the reason I opened this was the fact that the fallback method didn't work either, but it does now the docs are fixed. I've also since tried --config-settings editable_mode=compat and that work as a workaround too.

I also hadn't seen the long list of expected limitations here (I'd only looked at the changelog entries about the new feature):
https://setuptools.pypa.io/en/latest/userguide/development_mode.html#limitations

Given the long list of limitations/differences, it seems 1:1 parity is not intended, so this issue is just another of those. I don't know whether it's something that needs to be documented under the limitations list in the docs or not - I'll leave that for you to decide :-) (Either way, at least this GitHub issue might give someone else a clue when searching the repo.)

For context, the reason why PYTHONPATH is set is mostly historical in the Heroku Python buildpack - I've been wanting to stop setting it for a while (given that CWD is put on sys.path anyway), however I have a sneaking suspicion doing so will break a not insigificant number of end-user workflows (eg if they are not running the commands from their application root).

[abravalheri]

Whereas before the wrapper script managed to find /my_package/my_package since it iterates through checking whether the entrypoint exists (I've only skimmed the code here; I could be wrong):

Now that I am thinking, maybe there is a different, weird and accidental explanation related to the complex ways namespace packages work (by setting PYTHONPATH=/src the user is accidentally creating a namespace package) not to the script wrappers...

In that case, this is covered under the last item in the list of limitations (at least until we get guidance/public APIs from the Python core team, see https://github.com/python/cpython/issues/92054).

Given the long list of limitations/differences, it seems 1:1 parity is not intended,

If you consider pip install -e . AND pip install -e . --config-settings editable_mode=strict, all the previously supported use cases should also be covered in setuptools 64+, plus a bunch of other use cases that were not covered before. However it does require user opt-in via editable_mode= for the more complex use cases (accidental namespace packages or pkgutil/pkg_resources-style namespaces being one of them).

the reason why PYTHONPATH is set is mostly historical in the Heroku Python buildpack

I would suggest to use "regular installations" (in opposition to "editable installations") during deployments. Editable installations target a "staging environment" that work as a playground for the user, they don't offer the same guarantees... But I suppose that in Heroku this is ultimately a users' choice, right?

[edmorley]

But I suppose that in Heroku this is ultimately a users' choice, right?

Yeah it is unfortunately - welcome to my world! 😫 😆

I suspect the reason some users are doing this is so they can use the same requirements file for development as in production (but it's just a guess).