Adjust curl timeouts, retries and logging

In #1439, the timeouts used with curl were set to higher values than
we needed for Heroku's build system, in order to prevent timeouts in
non-Heroku environments such as Dokku running on underpowered devices
(or devices with poor internet connections).

However, as seen in recent Honeycomb metrics, these higher timeouts
cause longer delays on Heroku before a curl retry if there is a
transient S3 connection issue - since the request will wait the full
10 seconds `connect_timeout` before retrying.

As such, the defaults have been lowered again to be optimised for
running on Heroku, and support for overriding the defaults via env var
added to still allow running in other environments.

Specifically:
- The default connection timeout has been lowered from 10 seconds to 3,
  but can now be overridden via the env var `CURL_CONNECT_TIMEOUT`
- The default total request timeout has been lowered from 120 seconds
  to 60, but can be overridden via `CURL_TIMEOUT`

These env vars cannot be set via Heroku config vars, and must instead
be set in the build system environment (or by an earlier buildpack).
The env var names match those used by the classic Ruby buildpack:
https://github.com/heroku/heroku-buildpack-ruby/blob/a6d124492aafe7f7ee0dd42d7a79284b6552d675/lib/language_pack/fetcher.rb#L47-L53

In addition, the usages of `--silent --show-error` have been replaced
with `--no-progress-meter`, which is otherwise identical apart from it
allowing warnings to be shown, such as the status messages output during
retry attempts (of form: "Will retry in 1 seconds. 3 retries left" etc).

Before:

```
-----> Installing Python 3.13.7
       curl: (6) Could not resolve host: heroku-buildpack-python.s3.us-east-1.amazonaws.com
       curl: (6) Could not resolve host: heroku-buildpack-python.s3.us-east-1.amazonaws.com
       ...
```

After:

```
-----> Installing Python 3.13.7
       curl: (7) Failed to connect to heroku-buildpack-python.s3.us-east-1.amazonaws.com port 443 after 3 ms: Couldn't connect to server
       Warning: Problem : connection refused. Will retry in 1 seconds. 5 retries left.
       curl: (7) Failed to connect to heroku-buildpack-python.s3.us-east-1.amazonaws.com port 443 after 4 ms: Couldn't connect to server
       Warning: Problem : connection refused. Will retry in 2 seconds. 4 retries left.
       ...
```

Lastly, the maximum number of retries has been increased slightly.

Note: The curl options used in `build_python_runtime.sh` have also been
adjusted, however, since these are only used by buildpack maintainers
(typically by running them on GitHub Actions), there is no need to those
timeouts configurable.

GUS-W-19414036.

Author: edmorley

CHANGELOG.md

@@ -3,6 +3,9 @@
 ## [Unreleased]
 
 - Updated uv from 0.8.9 to 0.8.13. ([#1880](https://github.com/heroku/heroku-buildpack-python/pull/1880))
+- Reduced default curl timeouts for faster retries of any transient connection issues on Heroku. ([#1884](https://github.com/heroku/heroku-buildpack-python/pull/1884))
+- Added support for overriding the default curl timeouts using `CURL_CONNECT_TIMEOUT` and `CURL_TIMEOUT`. These are intended for use in non-Heroku environments with slow connections, and so must be set via the build system rather than app config vars. ([#1884](https://github.com/heroku/heroku-buildpack-python/pull/1884))
+- Improved log output during curl retry attempts. ([#1884](https://github.com/heroku/heroku-buildpack-python/pull/1884))
 - Switched to Bash 5.0's `EPOCHREALTIME` for buildpack data store timing logic. ([#1881](https://github.com/heroku/heroku-buildpack-python/pull/1881))
 
 ## [v302] - 2025-08-21

builds/build_python_runtime.sh

@@ -1,5 +1,9 @@
 #!/usr/bin/env bash
 
+# This script is used by buildpack maintainers to compile new Python versions as they are released,
+# ready for upload to S3. (It isn't run as part of normal buildpack execution during app builds.)
+# See `builds/README.md` for how to invoke this via GitHub Actions.
+
 set -euo pipefail
 shopt -s inherit_errexit
 
@@ -71,8 +75,8 @@ set -o xtrace
 
 mkdir -p "${SRC_DIR}" "${INSTALL_DIR}" "${UPLOAD_DIR}"
 
-curl --fail --retry 3 --retry-connrefused --connect-timeout 10 --max-time 60 -o python.tgz "${SOURCE_URL}"
-curl --fail --retry 3 --retry-connrefused --connect-timeout 10 --max-time 60 -o python.tgz.asc "${SIGNATURE_URL}"
+curl --fail --retry 5 --retry-connrefused --connect-timeout 3 --max-time 30 -o python.tgz "${SOURCE_URL}"
+curl --fail --retry 5 --retry-connrefused --connect-timeout 3 --max-time 30 -o python.tgz.asc "${SIGNATURE_URL}"
 
 gpg --batch --verbose --recv-keys "${GPG_KEY_FINGERPRINT}"
 gpg --batch --verify python.tgz.asc python.tgz

lib/python.sh

@@ -37,21 +37,20 @@ function python::install() {
 		# shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled.
 		if ! {
 			{
-				# We set max-time for improved UX/metrics for hanging downloads compared to relying
-				# on the build system timeout. The Python archives are only ~10 MB so take < 1s to
-				# download on Heroku's build system, however, we use much higher timeouts so that
-				# the buildpack works in non-Heroku environments that may be far from `us-east-1`
-				# or have a slower connection. We don't use `--speed-limit` since it gives worse
-				# error messages when used with retries and piping to tar.
+				# We set max-time for improved UX/metrics for hanging downloads compared to relying on the build
+				# system timeout. We don't use `--speed-limit` since it gives worse error messages when used with
+				# retries and piping to tar. The Python archives are ~10 MB so only take ~1s to download on Heroku,
+				# so set low timeouts to reduce delays before retries. However, we allow customising the timeouts
+				# to support non-Heroku environments that may be far from `us-east-1` or have a slower connection.
+				# We use `--no-progress-meter` rather than `--silent` so that retry status messages are printed.
 				curl \
-					--connect-timeout 10 \
+					--connect-timeout "${CURL_CONNECT_TIMEOUT:-3}" \
 					--fail \
-					--max-time 120 \
-					--retry-max-time 120 \
-					--retry 3 \
+					--max-time "${CURL_TIMEOUT:-60}" \
+					--no-progress-meter \
+					--retry-max-time "${CURL_TIMEOUT:-60}" \
+					--retry 5 \
 					--retry-connrefused \
-					--show-error \
-					--silent \
 					"${python_url}" \
 					| tar \
 						--directory "${install_dir}" \

lib/uv.sh

@@ -30,22 +30,22 @@ function uv::install_uv() {
 		local uv_url="https://github.com/astral-sh/uv/releases/download/${UV_VERSION}/uv-${gnu_arch}-unknown-linux-gnu.tar.gz"
 
 		if ! {
-			# We set max-time for improved UX for hanging downloads compared to relying on the build system
-			# timeout. The uv archive is only ~15 MB so takes < 1s to download on Heroku's build system,
-			# however, we use much higher timeouts so that the buildpack works in non-Heroku or local
-			# environments that may have a slower connection. We don't use `--speed-limit` since it gives
-			# worse error messages when used with retries and piping to tar.
-			# We have to use `--strip-components` since the archive contents are nested under a subdirectory.
+			# We set max-time for improved UX/metrics for hanging downloads compared to relying on the build
+			# system timeout. We don't use `--speed-limit` since it gives worse error messages when used with
+			# retries and piping to tar. The Python archives are ~10 MB so only take ~1s to download on Heroku,
+			# so set low timeouts to reduce delays before retries. However, we allow customising the timeouts
+			# to support non-Heroku environments that may be far from `us-east-1` or have a slower connection.
+			# We use `--no-progress-meter` rather than `--silent` so that retry status messages are printed.
+			# We have to use `--strip-components` since the uv binary is nested under a subdirectory.
 			curl \
-				--connect-timeout 10 \
+				--connect-timeout "${CURL_CONNECT_TIMEOUT:-3}" \
 				--fail \
 				--location \
-				--max-time 120 \
-				--retry-max-time 120 \
-				--retry 3 \
+				--max-time "${CURL_TIMEOUT:-60}" \
+				--no-progress-meter \
+				--retry-max-time "${CURL_TIMEOUT:-60}" \
+				--retry 5 \
 				--retry-connrefused \
-				--show-error \
-				--silent \
 				"${uv_url}" \
 				| tar \
 					--directory "${uv_dir}" \