Update pip to 22.0.4, setuptools to 60.10.0 and wheel to 0.37.1 (#1310)

edmorley · web-flow · commit 8a5c73effe3f · 2022-04-14T14:43:13.000+01:00
- Update pip from 21.3.1 to 22.0.4 for Python 3.7+ - Update setuptools from 57.5.0 to: - 59.6.0 for Python 3.6 - 60.10.0 for Python 3.7+ - Update wheel from 0.37.0 to 0.37.1 for Python 2.7 and Python 3.5+ The setuptools upgrade should resolve issues Pipenv users are seeing with conflicting setuptools versions causing errors like: ``` Traceback (most recent call last): ... File "/app/.heroku/python/lib/python3.10/site-packages/pipenv/environment.py", line 553, in get_distributions pkg_resources = self.safe_import("pkg_resources") File "/app/.heroku/python/lib/python3.10/site-packages/pipenv/environment.py", line 92, in safe_import six.moves.reload_module(module) File "/app/.heroku/python/lib/python3.10/importlib/__init__.py", line 169, in reload _bootstrap._exec(spec, module) File "<frozen importlib._bootstrap>", line 619, in _exec File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 74, in <module> from pkg_resources.extern.jaraco.text import ( ModuleNotFoundError: No module named 'pkg_resources.extern.jaraco' ``` However, the setuptools upgrade also drops support for the deprecated `2to3` feature, which may be used be a small number of legacy dependencies. For example: ``` Collecting Flask-OpenID==1.2.5 Downloading Flask-OpenID-1.2.5.tar.gz (43 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'error' error: subprocess-exited-with-error × python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [1 lines of output] error in Flask-OpenID setup command: use_2to3 is invalid. [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed ``` Affected apps should either: - Update to a newer version of affected dependencies. - Use a wheel rather than an sdist of those dependencies (wheels don't require the use of setuptools once created). For example by ensuring the Python version in use matches that of any published wheels (see the package's "Download files" section on pypi.org for which wheels exist), or by asking the package maintainer to upload wheels for an existing release. - Pin to the previous buildpack release (v209), using: https://devcenter.heroku.com/articles/buildpacks#buildpack-references Note: Pinning buildpack version should be used as a short-term workaround only, and after any affected dependencies are fixed, apps should switch back to the default stable-release buildpack alias of `heroku/python`. Release notes: https://pip.pypa.io/en/stable/news/#v22-0-4 https://setuptools.pypa.io/en/latest/history.html#v60-10-0 https://github.com/pypa/wheel/blob/0.37.1/docs/news.rst Commits: pypa/pip@21.3.1...22.0.4 pypa/setuptools@v57.5.0...v60.10.0 pypa/wheel@0.37.0...0.37.1 Note: The new version of pip includes new progress bars, which have had to be disabled using `--progress off` in order to reduce build log noise. GUS-W-10538676.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,6 +5,11 @@
 - Fix typo in the `BUILD_WITH_GEO_LIBRARIES` end-of-life error message ([#1307](https://github.com/heroku/heroku-buildpack-python/pull/1307)).
 - No longer set a fallback value for `$STACK`, since it is always set on Heroku ([#1308](https://github.com/heroku/heroku-buildpack-python/pull/1308)).
 - Adjust the configure options and packaging process for Python 3.7 releases, to enable loadable extensions in the `_sqlite` module, and to remove the `idle_test` module ([#1309](https://github.com/heroku/heroku-buildpack-python/pull/1309)). This change will only take effect as of the next Python 3.7 release (3.7.14).
+- Update pip from 21.3.1 to 22.0.4 for Python 3.7+ ([#1310](https://github.com/heroku/heroku-buildpack-python/pull/1310))
+- Update setuptools from 57.5.0 to: ([#1310](https://github.com/heroku/heroku-buildpack-python/pull/1310))
+  - 59.6.0 for Python 3.6
+  - 60.10.0 for Python 3.7+
+- Update wheel from 0.37.0 to 0.37.1 for Python 2.7 and Python 3.5+ ([#1310](https://github.com/heroku/heroku-buildpack-python/pull/1310))
 
 ## v209 (2022-03-24)
 
@@ -52,7 +57,7 @@
 - Update setuptools from 47.1.1 to: ([#1254](https://github.com/heroku/heroku-buildpack-python/pull/1254))
   - 50.3.2 for Python 3.5
   - 57.5.0 for Python 3.6+
-- Update wheel from 0.36.2 to 0.37.0 ([#1254](https://github.com/heroku/heroku-buildpack-python/pull/1254)).
+- Update wheel from 0.36.2 to 0.37.0 for Python 2.7 and Python 3.5+ ([#1254](https://github.com/heroku/heroku-buildpack-python/pull/1254)).
 - Perform editable package `.pth` and `.egg-link` path rewriting at runtime ([#1252](https://github.com/heroku/heroku-buildpack-python/pull/1252)).
 
 ## v200 (2021-10-04)
diff --git a/bin/steps/pip-install b/bin/steps/pip-install
@@ -41,7 +41,7 @@ if [ ! "$SKIP_PIP_INSTALL" ]; then
         exit 1
     fi
 
-    /app/.heroku/python/bin/pip install -r "$BUILD_DIR/requirements.txt" --exists-action=w --src=/app/.heroku/src --disable-pip-version-check --no-cache-dir 2>&1 | tee "$WARNINGS_LOG" | cleanup | indent
+    /app/.heroku/python/bin/pip install -r "$BUILD_DIR/requirements.txt" --exists-action=w --src=/app/.heroku/src --disable-pip-version-check --no-cache-dir --progress-bar off 2>&1 | tee "$WARNINGS_LOG" | cleanup | indent
     PIP_STATUS="${PIPESTATUS[0]}"
     set -e
 
diff --git a/bin/steps/python b/bin/steps/python
@@ -150,9 +150,9 @@ fi
 
 set -e
 
-PIP_VERSION='21.3.1'
-SETUPTOOLS_VERSION='57.5.0'
-WHEEL_VERSION='0.37.0'
+PIP_VERSION='22.0.4'
+SETUPTOOLS_VERSION='60.10.0'
+WHEEL_VERSION='0.37.1'
 
 if [[ "${PYTHON_VERSION}" == ${PY34}* ]]; then
   # Python 3.4 support was dropped in pip 19.2+, setuptools 44+ and wheel 0.34+.
@@ -167,6 +167,10 @@ elif [[ "${PYTHON_VERSION}" == ${PY35}* ]]; then
   # Python 3.5 support was dropped in pip 21+ and setuptools 51+.
   PIP_VERSION='20.3.4'
   SETUPTOOLS_VERSION='50.3.2'
+elif [[ "${PYTHON_VERSION}" == ${PY36}* || "${PYTHON_VERSION}" == ${PYPY36}* ]]; then
+  # Python 3.6 support was dropped in pip 22+ and setuptools 59.7.0+.
+  PIP_VERSION='21.3.1'
+  SETUPTOOLS_VERSION='59.6.0'
 fi
 
 puts-step "Installing pip ${PIP_VERSION}, setuptools ${SETUPTOOLS_VERSION} and wheel ${WHEEL_VERSION}"
diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb
@@ -22,7 +22,7 @@
           remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION}
           remote:        To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
           remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION}
-          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 22.0.4, setuptools 60.10.0 and wheel 0.37.1
           remote: -----> Installing SQLite3
           remote: -----> Installing requirements with pip
           remote:        Collecting urllib3
@@ -38,7 +38,7 @@
           remote:        To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
           remote: -----> No change in requirements detected, installing from cache
           remote: -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION}
-          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 22.0.4, setuptools 60.10.0 and wheel 0.37.1
           remote: -----> Installing SQLite3
           remote: -----> Installing requirements with pip
           remote: -----> Discovering process types
@@ -61,7 +61,7 @@
           remote:        To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
           remote: -----> Requirements file has been changed, clearing cached dependencies
           remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION}
-          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 22.0.4, setuptools 60.10.0 and wheel 0.37.1
           remote: -----> Installing SQLite3
           remote: -----> Installing requirements with pip
           remote:        Collecting urllib3
diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb
@@ -11,7 +11,7 @@
         remote: -----> Using Python version specified in Pipfile.lock
         remote: cp: cannot stat '/tmp/build_.*/requirements.txt': No such file or directory
         remote: -----> Installing python-#{python_version}
-        remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
+        remote: -----> Installing pip 22.0.4, setuptools 60.10.0 and wheel 0.37.1
         remote: -----> Installing dependencies with Pipenv 2020.11.15
         remote:        Installing dependencies from Pipfile.lock \\(.*\\)...
         remote: -----> Installing SQLite3
@@ -33,7 +33,7 @@
           remote:        To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
           remote: cp: cannot stat '/tmp/build_.*/requirements.txt': No such file or directory
           remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION}
-          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 22.0.4, setuptools 60.10.0 and wheel 0.37.1
           remote: -----> Installing dependencies with Pipenv 2020.11.15
           remote:        Installing dependencies from Pipfile...
           remote: -----> Installing SQLite3
@@ -53,7 +53,7 @@
           remote:        To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
           remote: cp: cannot stat '/tmp/build_.*/requirements.txt': No such file or directory
           remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION}
-          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 22.0.4, setuptools 60.10.0 and wheel 0.37.1
           remote: -----> Installing dependencies with Pipenv 2020.11.15
           remote:        Installing dependencies from Pipfile.lock \\(aad8b1\\)...
           remote: -----> Installing SQLite3
@@ -76,7 +76,7 @@
             remote:        Learn More: https://devcenter.heroku.com/articles/python-2-7-eol-faq
             remote: cp: cannot stat '/tmp/build_.*/requirements.txt': No such file or directory
             remote: -----> Installing python-#{LATEST_PYTHON_2_7}
-            remote: -----> Installing pip 20.3.4, setuptools 44.1.1 and wheel 0.37.0
+            remote: -----> Installing pip 20.3.4, setuptools 44.1.1 and wheel 0.37.1
             remote: -----> Installing dependencies with Pipenv 2020.11.15
             remote:        Installing dependencies from Pipfile.lock \\(b8efa9\\)...
             remote: -----> Installing SQLite3
@@ -114,7 +114,20 @@
   context 'with a Pipfile.lock containing python_version 3.6' do
     let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.6') }
 
-    include_examples 'builds using Pipenv with the requested Python version', LATEST_PYTHON_3_6
+    it 'builds with the latest Python 3.6' do
+      app.deploy do |app|
+        expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX))
+          remote: -----> Python app detected
+          remote: -----> Using Python version specified in Pipfile.lock
+          remote: cp: cannot stat '/tmp/build_.*/requirements.txt': No such file or directory
+          remote: -----> Installing python-#{LATEST_PYTHON_3_6}
+          remote: -----> Installing pip 21.3.1, setuptools 59.6.0 and wheel 0.37.1
+          remote: -----> Installing dependencies with Pipenv 2020.11.15
+          remote:        Installing dependencies from Pipfile.lock \\(.*\\)...
+          remote: -----> Installing SQLite3
+        REGEX
+      end
+    end
   end
 
   context 'with a Pipfile.lock containing python_version 3.7' do
@@ -153,7 +166,7 @@
           remote:        Learn More: https://devcenter.heroku.com/articles/python-runtimes
           remote: cp: cannot stat '/tmp/build_.*/requirements.txt': No such file or directory
           remote: -----> Installing python-3.10.0
-          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 22.0.4, setuptools 60.10.0 and wheel 0.37.1
           remote: -----> Installing dependencies with Pipenv 2020.11.15
           remote:        Installing dependencies from Pipfile.lock \\(99d8c9\\)...
           remote: -----> Installing SQLite3
@@ -203,7 +216,7 @@
           remote: -----> Using Python version specified in runtime.txt
           remote: cp: cannot stat '/tmp/build_.*/requirements.txt': No such file or directory
           remote: -----> Installing python-#{LATEST_PYTHON_3_10}
-          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 22.0.4, setuptools 60.10.0 and wheel 0.37.1
           remote: -----> Installing dependencies with Pipenv 2020.11.15
           remote:        Installing dependencies from Pipfile.lock \\(75eae0\\)...
           remote: -----> Installing SQLite3
@@ -221,7 +234,7 @@
           remote: -----> Python app detected
           remote: -----> Using Python version specified in Pipfile.lock
           remote: -----> Installing python-#{LATEST_PYTHON_3_10}
-          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 22.0.4, setuptools 60.10.0 and wheel 0.37.1
           remote: -----> Installing dependencies with Pipenv 2020.11.15
           remote:        Installing dependencies from Pipfile.lock (2d32e8)...
           remote: -----> Installing SQLite3
@@ -241,7 +254,7 @@
           remote:        To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
           remote: cp: cannot stat '/tmp/build_.*/requirements.txt': No such file or directory
           remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION}
-          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 22.0.4, setuptools 60.10.0 and wheel 0.37.1
           remote: -----> Installing dependencies with Pipenv 2020.11.15
           remote:        Your Pipfile.lock \\(aad8b1\\) is out of date. Expected: \\(2d32e8\\).
           remote:        \\[DeployException\\]: .*
diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb
@@ -9,7 +9,7 @@
         remote: -----> Python app detected
         remote: -----> Using Python version specified in runtime.txt
         remote: -----> Installing python-#{python_version}
-        remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
+        remote: -----> Installing pip 22.0.4, setuptools 60.10.0 and wheel 0.37.1
         remote: -----> Installing SQLite3
         remote: -----> Installing requirements with pip
         remote:        Collecting urllib3
@@ -89,7 +89,7 @@
             remote:  !     Python 2 has reached its community EOL. Upgrade your Python runtime to maintain a secure application as soon as possible.
             remote:        Learn More: https://devcenter.heroku.com/articles/python-2-7-eol-faq
             remote: -----> Installing python-#{LATEST_PYTHON_2_7}
-            remote: -----> Installing pip 20.3.4, setuptools 44.1.1 and wheel 0.37.0
+            remote: -----> Installing pip 20.3.4, setuptools 44.1.1 and wheel 0.37.1
             remote: -----> Installing SQLite3
             remote: -----> Installing requirements with pip
             remote:        Collecting urllib3
@@ -151,7 +151,7 @@
             remote: -----> Python app detected
             remote: -----> Using Python version specified in runtime.txt
             remote: -----> Installing python-#{LATEST_PYTHON_3_5}
-            remote: -----> Installing pip 20.3.4, setuptools 50.3.2 and wheel 0.37.0
+            remote: -----> Installing pip 20.3.4, setuptools 50.3.2 and wheel 0.37.1
             remote: -----> Installing SQLite3
             remote: -----> Installing requirements with pip
             remote:        Collecting urllib3
@@ -172,7 +172,20 @@
   context 'when runtime.txt contains python-3.6.15' do
     let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.6') }
 
-    include_examples 'builds with the requested Python version', LATEST_PYTHON_3_6
+    it 'builds with Python 3.6.15' do
+      app.deploy do |app|
+        expect(clean_output(app.output)).to include(<<~OUTPUT)
+          remote: -----> Python app detected
+          remote: -----> Using Python version specified in runtime.txt
+          remote: -----> Installing python-#{LATEST_PYTHON_3_6}
+          remote: -----> Installing pip 21.3.1, setuptools 59.6.0 and wheel 0.37.1
+          remote: -----> Installing SQLite3
+          remote: -----> Installing requirements with pip
+          remote:        Collecting urllib3
+        OUTPUT
+        expect(app.run('python -V')).to include("Python #{LATEST_PYTHON_3_6}")
+      end
+    end
   end
 
   context 'when runtime.txt contains python-3.7.13' do
@@ -208,7 +221,7 @@
           remote: -----> Python app detected
           remote: -----> Using Python version specified in runtime.txt
           remote: -----> Installing pypy2.7-#{LATEST_PYPY_2_7}
-          remote: -----> Installing pip 20.3.4, setuptools 44.1.1 and wheel 0.37.0
+          remote: -----> Installing pip 20.3.4, setuptools 44.1.1 and wheel 0.37.1
           remote: -----> Installing SQLite3
           remote: -----> Installing requirements with pip
           remote:        Collecting urllib3
@@ -227,7 +240,7 @@
           remote: -----> Python app detected
           remote: -----> Using Python version specified in runtime.txt
           remote: -----> Installing pypy3.6-#{LATEST_PYPY_3_6}
-          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 21.3.1, setuptools 59.6.0 and wheel 0.37.1
           remote: -----> Installing SQLite3
           remote: -----> Installing requirements with pip
           remote:        Collecting urllib3
@@ -270,7 +283,7 @@
           remote: -----> Python version has changed from python-#{LATEST_PYTHON_3_6} to python-#{LATEST_PYTHON_3_10}, clearing cache
           remote: -----> No change in requirements detected, installing from cache
           remote: -----> Installing python-#{LATEST_PYTHON_3_10}
-          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 22.0.4, setuptools 60.10.0 and wheel 0.37.1
           remote: -----> Installing SQLite3
           remote: -----> Installing requirements with pip
           remote:        Collecting urllib3
diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb
@@ -9,7 +9,7 @@
     # can successfully read the stack metadata written to the build cache in the past.
     # The buildpack version chosen is one which had an older default Python version, so
     # we can also prove that clearing the cache didn't lose the Python version metadata.
-    let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v189'] }
+    let(:buildpacks) { ['https://github.com/heroku/heroku-buildpack-python#v208'] }
     let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_unspecified', buildpacks: buildpacks) }
 
     it 'clears the cache before installing again whilst preserving the sticky Python version' do
@@ -22,14 +22,14 @@
         # TODO: The requirements output shouldn't say "installing from cache", since it's not.
         expect(clean_output(app.output)).to include(<<~OUTPUT)
           remote: -----> Python app detected
-          remote: -----> No Python version was specified. Using the same version as the last build: python-3.6.12
+          remote: -----> No Python version was specified. Using the same version as the last build: python-3.10.3
           remote:        To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
-          remote:  !     Python has released a security update! Please consider upgrading to python-#{LATEST_PYTHON_3_6}
+          remote:  !     Python has released a security update! Please consider upgrading to python-#{LATEST_PYTHON_3_10}
           remote:        Learn More: https://devcenter.heroku.com/articles/python-runtimes
           remote: -----> Stack has changed from heroku-18 to heroku-20, clearing cache
           remote: -----> No change in requirements detected, installing from cache
-          remote: -----> Installing python-3.6.12
-          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing python-3.10.3
+          remote: -----> Installing pip 22.0.4, setuptools 60.10.0 and wheel 0.37.1
           remote: -----> Installing SQLite3
           remote: -----> Installing requirements with pip
           remote:        Collecting urllib3
@@ -59,7 +59,7 @@
           remote: -----> Stack has changed from heroku-20 to heroku-18, clearing cache
           remote: -----> No change in requirements detected, installing from cache
           remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION}
-          remote: -----> Installing pip 21.3.1, setuptools 57.5.0 and wheel 0.37.0
+          remote: -----> Installing pip 22.0.4, setuptools 60.10.0 and wheel 0.37.1
           remote: -----> Installing SQLite3
           remote: -----> Installing requirements with pip
           remote:        Collecting urllib3
