unable to sync python packages via proxy

[keilr]

• https://pypi.org/ and https://pythonhosted.org is reachable via proxy (tested via curl)
• syncing RPM repositories by using the same proxy works as expected

remote configuration:

$ .local/bin/pulp python remote list
[
  {
    "pulp_href": "/pulp/api/v3/remotes/python/python/c29cd792-70ee-4685-93a4-93406c0b9fb8/",
    "pulp_created": "2021-08-30T18:52:26.441534Z",
    "name": "pypi",
    "url": "https://pypi.org/",
    "ca_cert": null,
    "client_cert": null,
    "tls_validation": true,
    "proxy_url": "http://my-web-proxy.local:8080",
    "pulp_labels": {},
    "pulp_last_updated": "2021-08-30T19:24:11.665512Z",
    "download_concurrency": null,
    "max_retries": null,
    "policy": "on_demand",
    "total_timeout": null,
    "connect_timeout": null,
    "sock_connect_timeout": null,
    "sock_read_timeout": null,
    "headers": null,
    "rate_limit": null,
    "includes": [
      "ansible",
      "ansible-base",
      "ansible-lint"
    ],
    "excludes": [],
    "prereleases": false,
    "package_types": [],
    "keep_latest_packages": 0,
    "exclude_platforms": []
  }

error:

$ podman logs --follow pulp
pulp [e4be013a1fb34a829098328c8648769f]: bandersnatch.package:ERROR: Timeout error for ansible (0) not updating. Giving up.

Is it possible to enable verbose logging to see the URL that bandersnatch tries to fetch?

[gerrod3]

I think if you allow DEBUG log level you can see the urls bandersnatch tries to fetch (it's quite verbose). However, bandersnatch is only fetching from https://pypi.org since that is where the metadata is. Specifically for this remote bandersnatch will fetch the metadata at

• https://pypi.org/pypi/ansible/json/
• https://pypi.org/pypi/ansible-base/json/
• https://pypi.org/pypi/ansible-lint/json/

I know the proxy-url is passed to bandersnatch, which uses aiohttp to make requests, which uses urllib to get proxy info. pulp_python sets the environ variable http_proxy which is ultimately read by urllib to determine if a proxy is being used. https://github.com/pulp/pulp_python/blob/master/pulp_python/app/tasks/sync.py#L111

(Edit) After checking through aiohttp code I found that each request the client does checks the environ variable for proxy settings, so there is a bug in pulp_python. PR coming soon.

[gerrod3]

@keilr Do you need this fix back-ported to the version of pulp_python you are using? The next version release probably won't be for another 2 weeks minimum.

[keilr]

@gerrod3 thanks a lot for the fix! I'm using the official pulp/pulp:3.15 container image. Any chance to back-port it to this image?

[keilr]

@gerrod3 I just removed line 115+116 from /usr/local/lib/python3.8/site-packages/pulp_python/app/tasks/sync.py manually and tried to resync the repo. Unfortunately it still ignores the proxy settings. With the ss (or netstat) command I'm able to see, that the sync process still tries to reach pypi directly (151.101.* IP) and not via proxy.

[gerrod3]

@gerrod3 I just removed line 115+116 from /usr/local/lib/python3.8/site-packages/pulp_python/app/tasks/sync.py manually and tried to resync the repo. Unfortunately it still ignores the proxy settings. With the ss (or netstat) command I'm able to see, that the sync process still tries to reach pypi directly (151.101.* IP) and not via proxy.

Interesting, I checked to see if the proxy info was being passed to Bandersnatch, but didn't check if Bandersnatch was using the proxy correctly. My fix did ensure that the proxy was being passed so maybe there is a bug on their side or something else that I missed. I'll go mention this upstream and setup a proxy to test this properly to get it fix.

[keilr]

thanks for reopening the issue.

I set the environment variables http_proxy and https_proxy within the podman run command, but Bandersnatch still ignores it.

podman run --detach \
            --name pulp \
            ...omitted...
            -e http_proxy=http://my-web-proxy.local:8080 \
            -e https_proxy=http://my-web-proxy.local:8080 \
            pulp/pulp

commands to check if Bandersnatch is using the proxy or not:

• sudo ss -tpn | grep <IP of proxy> --> you shoud see connections with "CLOSE-WAIT" and/or "ESTAB" state.
• sudo ss -tpn | grep <IP of pypi> --> souldn't display any connection. If you see "SYN-SENT", Bandersnatch tries to reach pypi directly but FW denies the request. If you see "CLOSE-WAIT/ESTAB" then it was able to conenct to pypi, but didn't use the http proxy (no blocking FW in between).

Let me know if I can help you further with testing.

[gerrod3]

@keilr I've done some more testing and I have found that setting https_proxy is what is needed to get aiohttp to use the proxy to get https urls. If you could edit your installation on /usr/local/lib/python3.8/site-packages/pulp_python/app/tasks/sync.py at line 112 and change environ['http_proxy'] = ... to environ['https_proxy'] = ... and test that out for me. It's also possible that after changing the files you need to restart Pulp, the commands to do so for a running container should be

podman exec pulp bash -c "s6-svc -r /var/run/s6/services/pulpcore-api"
podman exec pulp bash -c "s6-svc -r /var/run/s6/services/pulpcore-content"
podman exec pulp bash -c "s6-svc -d /var/run/s6/services/pulpcore-resource-manager"
podman exec pulp bash -c "s6-svc -d /var/run/s6/services/pulpcore-worker@1"
podman exec pulp bash -c "s6-svc -d /var/run/s6/services/pulpcore-worker@2"
podman exec pulp bash -c "s6-svc -u /var/run/s6/services/new-pulpcore-resource-manager"
podman exec pulp bash -c "s6-svc -u /var/run/s6/services/new-pulpcore-worker@1"
podman exec pulp bash -c "s6-svc -u /var/run/s6/services/new-pulpcore-worker@2"

(Edit) Also, try making sure that only https_proxy is set and that http_proxy and HTTP_PROXY are both unset. My testing didn't show any difference in having both vars set, but just to be safe that aiohttp chooses the right one.