Skip to content

Update path rewriting to support setuptools v64's PEP660 editable install mode#1357

Merged
edmorley merged 4 commits intomainfrom
setuptools-pep660-compat
Aug 17, 2022
Merged

Update path rewriting to support setuptools v64's PEP660 editable install mode#1357
edmorley merged 4 commits intomainfrom
setuptools-pep660-compat

Conversation

@edmorley
Copy link
Copy Markdown
Member

@edmorley edmorley commented Aug 17, 2022

On Heroku, the application source directory exists at a different path at build time (/tmp/build_<hash>), than it does at runtime (/app). As such, the buildpack has to perform path rewriting via .profile.d scripts at runtime, to ensure any packaging related absolute paths in the build output are rewritten to reference the new path. (Thankfully this awful path rewriting will no longer be necessary in the future with CNBs.)

Previously the only files this path rewriting needed to update were the *.pth and *.egg-link files in site-packages created by setuptools when performing editable installs.

However setuptools v64 added support for PEP660 based editable install hooks:
https://setuptools.pypa.io/en/latest/history.html#v64-0-0
https://peps.python.org/pep-0660/

This feature is only used for projects that have a pyproject.toml, and for such projects, if the config is deemed complex enough, setuptools creates a new finder script in site-packages that dynamically handles package resolution. (Simpler configs get a static .pth file, which works fine with our existing path rewriting.)

This new file embeds the absolute path of the source directory at build time, so must be rewritten too. It has a filename of form: __editable___my_package_0_0_1_finder.py

As such, this PR adds support for rewriting these files, along with updated test fixtures to provide coverage of pyproject.toml based editable installs (alongside the existing setup.py based test fixture).

Whilst writing the new test, I encountered a difference in behaviour with setuptool's new editable install mode, which meant the fixtures had to be nested inside a packages/ directory in order to avoid an ImportError due to the fact that the Python buildpack currently sets PYTHONPATH=/app at runtime. See:
pypa/setuptools#3535

Note:

  • The Python buildpack doesn't yet globally install this newer setuptools v64 release, since by design it pins to a specific version to prevent upstream changes from breaking apps overnight. (The version was recently updated to 63.4.3 in Update pip to 22.2.2 and setuptools to 63.4.3 #1344.)
  • However, for packages that have a pyproject.toml pip uses the approach described in PEP518, which uses an isolated build environment. This environment typically pulls in latest setuptools (though this is controllable by the package owner via [build-system] in pyproject.toml), overriding our pinned global setuptools install.
  • As such, even though we're not using v64 globally, users can still be broken by the upstream release.

A big thanks to @mrcljx for the initial PR in #1355 on which this was based :-)

Closes #1355.
GUS-W-11608693.

@edmorley edmorley self-assigned this Aug 17, 2022
@edmorley edmorley marked this pull request as ready for review August 17, 2022 13:04
@edmorley edmorley requested a review from a team as a code owner August 17, 2022 13:04
@edmorley edmorley merged commit b2dfe73 into main Aug 17, 2022
@edmorley edmorley deleted the setuptools-pep660-compat branch August 17, 2022 13:14
@edmorley edmorley mentioned this pull request Sep 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants