PETSc downloads broken, because the site only seems to accept certain User-Agent values in the header

[casparvl]

Issue

$ eb --fetch PETSc-3.20.3-foss-2023a.eb --force-download
Couldn't find file petsc-3.20.3.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): ...
/home/casparl/.local/easybuild/sources/petsc-3.20.3.tar.gz, https://web.cels.anl.gov/projects/petsc/download/release-snapshots/petsc-3.20.3.tar.gz, https://sources.easybuild.io/p/PETSc/petsc-3.20.3.tar.gz  (took 0 secs)

In the logs:

== 2025-06-17 11:24:31,727 filetools.py:903 INFO Attempt 1 of downloading https://web.cels.anl.gov/projects/petsc/download/release-snapshots/petsc-3.20.3.tar.gz to /home/casparl/.local/easybuild/sources/p/PETSc/petsc-3.20.3.tar.gz failed, trying again...
== 2025-06-17 11:24:31,727 filetools.py:908 INFO Downloading using requests package instead of urllib2
== 2025-06-17 11:24:32,037 filetools.py:886 WARNING URL https://web.cels.anl.gov/projects/petsc/download/release-snapshots/petsc-3.20.3.tar.gz was not found (HTTP response code 403), not trying again
== 2025-06-17 11:24:32,037 filetools.py:923 WARNING Download of https://web.cels.anl.gov/projects/petsc/download/release-snapshots/petsc-3.20.3.tar.gz to /home/casparl/.local/easybuild/sources/p/PETSc/petsc-3.20.3.tar.gz failed, done trying
== 2025-06-17 11:24:32,105 filetools.py:886 WARNING URL https://sources.easybuild.io/p/PETSc/petsc-3.20.3.tar.gz was not found (HTTP response code 404), not trying again
== 2025-06-17 11:24:32,106 filetools.py:923 WARNING Download of https://sources.easybuild.io/p/PETSc/petsc-3.20.3.tar.gz to /home/casparl/.local/easybuild/sources/p/PETSc/petsc-3.20.3.tar.gz failed, done trying

Cause

In EasyBuild

 # use custom HTTP header
    headers = {'User-Agent': 'EasyBuild', 'Accept': '*/*'}

Trying this interactively, I see:

>>> headers = {
...     "User-Agent": "EasyBuild",
...     "Accept": "*/*",
... }
>>> response = requests.get(url, headers=headers)
>>> response.status_code
403

However, when we pretend to be Wget:

>>> headers = {
...     "User-Agent": "Wget/1.21.1",
...     "Accept": "*/*",
... }
>>> response = requests.get(url, headers=headers)
>>> response.status_code
200

According to my AI friend, some sites only allow specific values for the User-Agent, and block the rest (e.g. to avoid scraping etc). We could make our header such that we pretend to be Wget. Note that we could even do that as a third option, if all else fails (currently option one is using urllib, option 2 is using requests).

[branfosj]

curl/8.7.1 also worked, but wget/1.21.1 and cURL/8.7.1 did not - so this is a case sensitive whitelist.

[casparvl]

Probably changing (in easybuild/tools/filetools.py)

        if not downloaded and attempt_cnt < max_attempts:
            _log.info("Attempt %d of downloading %s to %s failed, trying again..." % (attempt_cnt, url, path))
            if used_urllib is std_urllib and switch_to_requests:
                if not HAVE_REQUESTS:
                    raise EasyBuildError("SSL issues with urllib2. If you are using RHEL/CentOS 6.x please "
                                         "install the python-requests and pyOpenSSL RPM packages and try again.")
                _log.info("Downloading using requests package instead of urllib2")
                used_urllib = requests

            if wait:
                _log.info(f"Waiting for {wait_time} seconds before trying download of {url} again...")
                time.sleep(wait_time)
                # exponential backoff
                wait_time *= 2

Into

        if not downloaded and attempt_cnt < max_attempts:
            _log.info("Attempt %d of downloading %s to %s failed, trying again..." % (attempt_cnt, url, path))
            if used_urllib is std_urllib and switch_to_requests:
                if not HAVE_REQUESTS:
                    raise EasyBuildError("SSL issues with urllib2. If you are using RHEL/CentOS 6.x please "
                                         "install the python-requests and pyOpenSSL RPM packages and try again.")
                _log.info("Downloading using requests package instead of urllib2")
                used_urllib = requests

           if (switch_to_requests and attempt_cnt == 2) or attempt_cnt == 1:
                headers = {'User-Agent': 'Wget/1.0', 'Accept': '*/*'}
                url_req = std_urllib.Request(url, headers=headers)

            if wait:
                _log.info(f"Waiting for {wait_time} seconds before trying download of {url} again...")
                time.sleep(wait_time)
                # exponential backoff
                wait_time *= 2

Is sufficient. I don't have time to make a PR out of it and test things right now though...

[boegel]

I don't think we should be lying in the User-Agent field, we wouldn't be a good citizen if we start doing that.

While this problem is likely not specific to PETSc at all, maybe we can reach out to them and also accept EasyBuild as user agent for downloading?

[sassy-crick]

Update: That is still a problem and thus we should look into that. Reaching out is probably the best idea, so we are working with them. One advantage might be they could count how often it is downloaded using EasyBuild as well. :-)

Edit: I reached out to them via their email list and on their GitLab page

[sassy-crick]

The agent 'easybuild' was kindly added by the PETSC developers and I have updated the few old EasyConfig files which still had the old URL in them and were on GitHub. I think we can close that issue to avoid further confusion.