Update path rewriting to support setuptools v64's PEP660 editable install mode (heroku#1357)

On Heroku, the application source directory exists at a different path at build time (`/tmp/build_<hash>`), than it does at runtime (`/app`). As such, the buildpack has to perform path rewriting via `.profile.d` scripts at runtime, to ensure any packaging related absolute paths in the build output are rewritten to reference the new path. (Thankfully this awful path rewriting will no longer be necessary in the future with CNBs.)

Previously the only files this path rewriting needed to update were the `*.pth` and `*.egg-link` files in `site-packages` created by setuptools when performing editable installs.

However setuptools v64 added support for PEP660 based editable install hooks:
https://setuptools.pypa.io/en/latest/history.html#v64-0-0
https://peps.python.org/pep-0660/

This feature is only used for projects that have a `pyproject.toml`, and for such projects, [if the config is deemed complex enough](https://github.com/pypa/setuptools/blob/d03da04e024ad4289342077eef6de40013630a44/setuptools/command/editable_wheel.py#L359-L368), setuptools creates a new [finder script](https://github.com/pypa/setuptools/blob/23d455c532fca91e6f00aa5950000739b058b6e5/setuptools/command/editable_wheel.py#L740-L809) in `site-packages` that dynamically handles package resolution. (Simpler configs get a static `.pth` file, which works fine with our existing path rewriting.)

This new file embeds the absolute path of the source directory at build time, so must be rewritten too. It has a filename of form: `__editable___my_package_0_0_1_finder.py`

As such, this PR adds support for rewriting these files, along with updated test fixtures to provide coverage of `pyproject.toml` based editable installs (alongside the existing `setup.py` based test fixture).

Whilst writing the new test, I encountered a difference in behaviour with setuptool's new editable install mode, which meant the fixtures had to be nested inside a `packages/` directory in order to avoid an `ImportError` due to the fact that the Python buildpack currently sets `PYTHONPATH=/app` at runtime. See:
pypa/setuptools#3535

Note:
- The Python buildpack doesn't yet globally install this newer setuptools v64 release, since by design it pins to a specific version to prevent upstream changes from breaking apps overnight. (The version was recently updated to 63.4.3 in heroku#1344.)
- However, for packages that have a `pyproject.toml` pip uses the approach described in PEP518, which uses an [isolated build environment](https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-isolation). This environment typically pulls in latest setuptools (though this is controllable by the package owner via `[build-system]` in `pyproject.toml`), overriding our pinned global setuptools install.
- As such, even though we're not using v64 globally, users can still be broken by the upstream release.

A big thanks to @mrcljx for the initial PR in heroku#1355 on which this was based :-)

Closes heroku#1355.
GUS-W-11608693.

Co-authored-by: Marcel Jackwerth <[email protected]>

Author: edmorley

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased
 
+- Ensure path rewriting works when using setuptools v64's new PEP660-based editable install mode ([#1357](https://github.com/heroku/heroku-buildpack-python/pull/1357)).
 - Display an EOL warning for Python 3.4, 3.5 and 3.6 ([#1356](https://github.com/heroku/heroku-buildpack-python/pull/1356)).
 - Improve the EOL warning for Python 2.7 ([#1356](https://github.com/heroku/heroku-buildpack-python/pull/1356)).
 - Display a deprecation warning for PyPy support ([#1356](https://github.com/heroku/heroku-buildpack-python/pull/1356)).

bin/compile

@@ -304,14 +304,14 @@ if [[ \$HOME != "/app" ]]; then
 fi
 EOT
 
-# At runtime, rewrite paths in editable package .egg-link and .pth files from the build time paths
+# At runtime, rewrite paths in editable package .egg-link, .pth and finder files from the build time paths
 # (such as `/tmp/build_<hash>`) back to `/app`. This is not done during the build itself, since later
 # buildpacks still need the build time paths. The `/lib*/*/` glob is to ensure it matches against both:
 # - lib/python3.NN/site-packages/ (CPython)
 # - lib-python/3/site-packages/ (PyPy)
 if [[ "${BUILD_DIR}" != "/app" ]]; then
   cat <<EOT >> "$PROFILE_PATH"
-find .heroku/python/lib*/*/site-packages/ -type f -and \( -name '*.egg-link' -or -name '*.pth' \) -exec sed -i -e 's#${BUILD_DIR}#/app#' {} \+
+find .heroku/python/lib*/*/site-packages/ -type f -and \( -name '*.egg-link' -or -name '*.pth' -or -name '__editable___*_finder.py' \) -exec sed -i -e 's#${BUILD_DIR}#/app#' {} \+
 EOT
 fi
 

spec/fixtures/requirements_editable/bin/compile

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 
-# This file is run by the inline buildpack, and tests that editable reqirements are
+# This file is run by the inline buildpack, and tests that editable requirements are
 # usable by buildpacks that run after the Python buildpack during the build.
 
 set -euo pipefail

spec/fixtures/requirements_editable/bin/test-entrypoints

@@ -2,12 +2,15 @@
 
 set -euo pipefail
 
-# List the filenames and contents of all .egg-link and .pth files in site-packages.
-find .heroku/python/lib*/*/site-packages/ -type f -and \( -name '*.egg-link' -or -name '*.pth' \) | sort | xargs -exec tail -n +1
+# List the filenames and contents of all .egg-link, .pth, and finder files in site-packages.
+find .heroku/python/lib*/*/site-packages/ -type f -and \( -name '*.egg-link' -or -name '*.pth' -or -name '__editable___*_finder.py' \) | sort | xargs -exec tail -n +1
 echo
 
-echo -n "Running entrypoint for the local package: "
-local_package
+echo -n "Running entrypoint for the pyproject.toml-based local package: "
+local_package_pyproject_toml
+
+echo -n "Running entrypoint for the setup.py-based local package: "
+local_package_setup_py
 
 echo -n "Running entrypoint for the VCS package: "
 gunicorn --version

spec/fixtures/requirements_editable/local_package/local_package/__init__.py

@@ -0,0 +1,2 @@
+def hello():
+    print("Hello pyproject.toml!")

spec/fixtures/requirements_editable/local_package/setup.cfg

@@ -0,0 +1,6 @@
+[project]
+name = "local_package_pyproject_toml"
+version = "0.0.1"
+
+[project.scripts]
+local_package_pyproject_toml = "local_package_pyproject_toml:hello"

spec/fixtures/requirements_editable/packages/local_package_pyproject_toml/local_package_pyproject_toml/__init__.py

@@ -0,0 +1,2 @@
+def hello():
+    print("Hello setup.py!")

spec/fixtures/requirements_editable/packages/local_package_pyproject_toml/pyproject.toml

@@ -0,0 +1,10 @@
+[metadata]
+name = local_package_setup_py
+version = 0.0.1
+
+[options]
+packages = local_package_setup_py
+
+[options.entry_points]
+console_scripts =
+    local_package_setup_py = local_package_setup_py:hello