Support setuptools v64's PEP660 editable install mode

[mrcljx]

We were running into ModuleNotFoundError errors when trying
to run our application. After some investigation we noticed
that newer setuptools places new files site-packages when
using editable installs (I guess only in some situations):

mylib-0.0.0.dist-info
__editable__.mylib-0.0.0.pth
__editable___mylib_0_0_0_finder.py

The *.pth file in that case contains this stub:

import __editable___mylib_0_0_0_finder; __editable___mylib_0_0_0_finder.install()

while the py module is generated from a template and contains this line:

MAPPING = {'mylib': '/tmp/build_892299be/py-packages/mylib/src/mylib'}

This all seems to be related to setuptools 63 and might be mitigated
by the compat mode in 64. I assume we just started to see this
issue because of the upgrade from 60 to 63 yesterday.

This PR updates the buildpack to also patch the paths in those files.
Please let me know if you have any recommendations for tests to add.

[edmorley]

@mrcljx Hi! Thank you for the write up and opening a PR :-)

Would I be correct in thinking that the package being installed in editable mode has a pyproject.toml so is being installed via the new Pip build isolation mode? If so, I believe the cause of this is that in isolation mode, the latest version of setuptools is pulled in (via the build backend dependency ranges specified in pyproject.toml, unless they explicitly pin the setuptools version), which means it would be using a newer version than what the buildpack uses for the global install.

If that's the case, this may even be unrelated to #1344 and instead just have been caused by setuptools v64 having been released around the same time (11th Aug):
https://setuptools.pypa.io/en/latest/history.html#v64-0-0

Either way, it seems we'll need this change to adapt for setuptools's new editable mode. I'll see if I can come up with a test.

[mrcljx]

If that's the case, this may even be unrelated to #1344 and instead just have been caused by setuptools v64 having been released around the same time (11th Aug): setuptools.pypa.io/en/latest/history.html#v64-0-0

That would actually make a lot of sense as we started seeing failures last Thu/Fri (so around Aug 11) and not just yesterday - the patch here didn't seem to fully solve it yet for us either.

[mrcljx]

Thanks for the hint! I looked a bit further into it: so these finder files are really only created when setuptools comes to the conclusion that the project setup is not trivial. After deleting setup.py and fully embracing pyproject.toml the *.pth files actually became simple paths again (and no _finder.py files were emitted).

[edmorley]

so these finder files are really only created when setuptools comes to the conclusion that the project setup is not trivial

Yeah - looks like for this issue to occur, all of the following has to be true:

• the package is being installed as an editable install
• the package has a pyproject.toml with a build-system table defined with the setuptools version range set so that setuptools v64+ gets pulled in (which includes the new PEP660 editable install feature rather than legacy editable mode)
• the install is being performed in "lenient" editable mode (the current setuptools default), not "strict" editable mode
• the logic here decides the project layout is complex enough, that the Python finder scripts are also created, rather than just using the static .pth file

It seems like workarounds (until this is merged/released), are:

1. Adjust the package config so setuptools no longer deems the config as "complex" (eg by removing setup.py entirely, as you did)
2. Use "strict" editable install mode instead (I'm presuming; I've not tested)
3. Switch back to legacy editable behaviour via the env var SETUPTOOLS_ENABLE_FEATURE=legacy-editable
4. Use setuptools<64 in the build-system table in pyproject.toml to force the use of older setuptools

[edmorley]

@mrcljx Thank you for the PR! Our CI needs secrets to run (so doesn't work from forks), and I also needed to add tests and changelog entry - so I had to merge this via a separate PR, but I credited you in the PR description and via the Co-authored-by: commit metadata (which GitHub uses to populate the attribution on the commit: b2dfe73). I'll release this change shortly :-)

[mrcljx]

Thanks @edmorley - appreciate you taking over.