Perform editable package .pth and .egg-link path rewriting at runtime

[edmorley]

Currently the build system performs builds in a different directory (/tmp/build_<hash>) to which the app will be run at runtime (/app). This means that any hardcoded paths in the slug must be rewritten by the buildpack, so the app still works after being moved.

One such case of hardcoded paths, are the .pth and .egg-link files that are created in the site-packages directory by Pip/setuptools for editable package installs (aka develop mode). The most common way someone might use editable mode is via -e <package specifier> entries in their requirements.txt file.

Until now, the Python buildpack rewrote paths inside these files during the compile itself, which meant the build-time paths were no longer present when subsequent buildpacks ran. This happened to work due to an interaction of legacy setuptools behaviour and a buildpack bug, but stops working in setuptools 47.2.0 or later - as described in #1006.

Longer term we would like to stop building in one location and running the app from another, so that the path rewriting isn't required at all. However that change breaks some other buildpacks so requires a long-term transition plan (internal notes).

In the meantime, this change moves path rewriting to a .profile.d/ script, so that it occurs at runtime, and so after all other buildpacks have run.

Additional test coverage of editable packages was added previously in #1251, and has confirmed that this new profile.d/ script approach will prevent the issues in #1006 when setuptools is updated in a future PR.

There is one subtle implication of moving this path rewriting, in that subsequent cached builds will no longer see the existing package as being already installed, so won't uninstall if before reinstalling it (as seen in the test log output change). However this is not believed to have any significant impact.

Fixes #1006. (And so unblocks updating to the newer setuptools required for #1248.)
GUS-W-7828034.

[dzuelke]

Nice one, @edmorley.

Two questions:

1. am I understanding the tests from Tests: Add additional coverage of editable package installation #1251 correctly: they're verifying that another buildpack ("inline", in that case) can successfully use the packages at build time?
2. Python doesn't resolve symlinks inside egg file loads, and that's why you can't rewrite the paths to /app/.heroku/python/ ahead of time? These symlinks will still be there when a subsequent buildpack runs, after all.

[edmorley]

@dzuelke Yes to (1).

For (2), the issue is that the .pth and .egg-link files don't just contain paths under .heroku/python/ but also to the app source too (in the case of local directory installs rather than VCS installs), and the app source isn't symlinked under /app. I'll cat the contents of the .pth and .egg-link files in the tests to make their contents easier to reason about (I'll add this in another PR, so the before/after can be seen in this PR once rebased).

[edmorley]

@dzuelke Additional test log output (that makes it easier to see what this PR does, and why it's needed) added in #1253 - please review and I'll rebase this PR on top.