Replicate feature is not processing a pull-through cache of pypi

[PotentialIngenuity]

Version
core: 3.57.0
python: 3.12.1

Describe the bug
I am trying to use the /replicate/ endpoint to sync a pull-through cache to another pulp instance. After the replication is done I see no reference of any python resources in the logs or through the api.

pulp python distribution list
[
  {
    "pulp_href": "/pulp/api/v3/distributions/python/pypi/018dd189-f781-7ee0-a6de-b93d044dce0b/",
    "pulp_created": "2024-02-22T15:58:16.450965Z",
    "pulp_last_updated": "2024-02-22T15:58:20.737703Z",
    "base_path": "pypi-mirror",
    "base_url": "https://pulp-api/pypi/pypi-mirror/",
    "content_guard": null,
    "hidden": false,
    "pulp_labels": {
      "content_type": "pypi-mirror"
    },
    "name": "pypi-mirror",
    "repository": null,
    "publication": null,
    "allow_uploads": true,
    "remote": "/pulp/api/v3/remotes/python/python/018dcdde-5748-7239-8e91-79ee820fec0d/"
  }
]

[
  {
    "pulp_href": "/pulp/api/v3/remotes/python/python/018dcdde-5748-7239-8e91-79ee820fec0d/",
    "pulp_created": "2024-02-21T22:51:57.129136Z",
    "pulp_last_updated": "2024-02-21T22:51:57.129154Z",
    "name": "pypi-mirror",
    "url": "https://pypi.org",
    "ca_cert": null,
    "client_cert": null,
    "tls_validation": true,
    "proxy_url": null,
    "pulp_labels": {},
    "download_concurrency": null,
    "max_retries": null,
    "policy": "on_demand",
    "total_timeout": null,
    "connect_timeout": null,
    "sock_connect_timeout": null,
    "sock_read_timeout": null,
    "headers": null,
    "rate_limit": null,
    "hidden_fields": [
      {
        "name": "client_key",
        "is_set": false
      },
      {
        "name": "proxy_username",
        "is_set": false
      },
      {
        "name": "proxy_password",
        "is_set": false
      },
      {
        "name": "username",
        "is_set": false
      },
      {
        "name": "password",
        "is_set": false
      }
    ],
    "includes": [],
    "excludes": [],
    "prereleases": false,
    "package_types": [],
    "keep_latest_packages": 0,
    "exclude_platforms": []
  }
]

To Reproduce
Setup and pull-through and try to replicate it.

Expected behavior
All python resources are created on the 2nd pulp instance

Additional context

[lubosmj]

I guess the main issue here is that distributions which have pull-through caching enabled, contain an empty index, that is supposed to list all available packages. These packages are not a part of any repository. Therefore, once a replica hits the base-path of a distribution, it just gets nothing to work on and ignores it. Have you tried manually accessing the index via cURL or httpie?

cc: @gerrod3

[gerrod3]

Not being able to sync from a pull-through cache was an intentional design choice when I implemented pull-through caching. I expected that most people would set their pull-through to PyPI and thus syncing a pull-through cache would sync all of PyPI which would be terribly slow because of how large PyPI is. Maybe it would make sense to allow syncing from a pull-through if the user supplies an includes on their remote pointing to the pull-through, that way we know we aren't trying to sync all of PyPI.

As for the replicate feature I see two potential options for dealing with upstream pull-through distributions:

1. Simply ignore distributions that only have a remote and no backing repository/publication since we can't sync from them.
2. Recreate the pull-through distribution on the downstream without trying to sync from it. Now would it make sense for the downstream pull-through to point to the upstream pull-through or to point to the actual cache source? I don't know.

[ipanova]

probably option 1 makes more sense

[lubosmj]

Same for me. Replicating is about replicating content from Pulp, not entity objects.

[PotentialIngenuity]

What are my options right now to get the cached content to a secondary pulp instance? Does pulp_python have a copy function like rpm does? https://pulpproject.org/pulp_rpm/restapi/#tag/Rpm:-Copy

[gerrod3]

Well currently your cached content is probably just living as orphaned content if you haven't been adding them to a repository. I would get a list of them first, then add them to a repo using the modify endpoint. https://pulpproject.org/pulp_python/restapi/#tag/Repositories:-Python/operation/repositories_python_python_modify

ORPHANS=$(http https://pulp-api/pulp/api/v3/content/python/packages/?orphaned_for=0 | jq -jc '[.results[].pulp_href]')
http POST '$REPO_HREF'modify/ add_content_units:="$ORPHANS"

[PotentialIngenuity]

Thank you. That works well but the results are pagenated. Using the limit query doesnt seem to have any effect.

http --verify false GET https://localhost/pulp/api/v3/content/python/packages/ limit:=999999999

{
    "count": 335036,
    "next": "http://localhost/pulp/api/v3/content/python/packages/?limit=100&offset=100",
    "previous": null,
    "results": [
        {....

[lubosmj]

You have to append the query option to the URL, like so: https...packages/?offset=0'&'limit=9999.

[PotentialIngenuity]

Thank you. The replication doesnt pick up the correct url to sync from. It should be looking at the upstream url: https://repo.company.com/pypi/python-pypi-freeze

pulp [a0b73760ce164b47bc3cb0b7a696f924]: pulpcore.tasking.tasks:INFO: Starting task 019151cc-d244-75d6-aa21-d7c196ffeb6f
pulp [a0b73760ce164b47bc3cb0b7a696f924]: bandersnatch:INFO: Initialized release plugin blocklist_release, filtering []
pulp [a0b73760ce164b47bc3cb0b7a696f924]: bandersnatch.mirror:INFO: Syncing with https://pulp-api/pypi/python-pypi-freeze.
pulp [a0b73760ce164b47bc3cb0b7a696f924]: pulp_python.app.tasks.sync:INFO: Attempt 0 to get package list from https://pulp-api/pypi/python-pypi-freeze
pulp [a0b73760ce164b47bc3cb0b7a696f924]: pulp_python.app.tasks.sync:INFO: Syncing all packages.
pulp [a0b73760ce164b47bc3cb0b7a696f924]: pulp_python.app.tasks.sync:INFO: Attempt 1 to get package list from https://pulp-api/pypi/python-pypi-freeze
pulp [a0b73760ce164b47bc3cb0b7a696f924]: pulp_python.app.tasks.sync:INFO: Syncing all packages.
pulp [a0b73760ce164b47bc3cb0b7a696f924]: pulp_python.app.tasks.sync:INFO: Attempt 2 to get package list from https://pulp-api/pypi/python-pypi-freeze
pulp [a0b73760ce164b47bc3cb0b7a696f924]: pulp_python.app.tasks.sync:INFO: Syncing all packages.
pulp [a0b73760ce164b47bc3cb0b7a696f924]: pulp_python.app.tasks.sync:INFO: Failed to get package list using XMLRPC, trying parse simple page.
Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host pulp-api:443 ssl:default [Connect call failed ('IPv4', 443)])
pulp [a0b73760ce164b47bc3cb0b7a696f924]: backoff:INFO: Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host pulp-api:443 ssl:default [Connect call failed ('IPv4', 443)])
Backing off download_wrapper(...) for 1.9s (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host pulp-api:443 ssl:default [Connect call failed ('IPv6', 443, 0, 0)])
pulp [a0b73760ce164b47bc3cb0b7a696f924]: backoff:INFO: Backing off download_wrapper(...) for 1.9s (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host pulp-api:443 ssl:default [Connect call failed ('IPv6', 443, 0, 0)])
Backing off download_wrapper(...) for 3.7s (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host pulp-api:443 ssl:default [Connect call failed ('IPv6', 443, 0, 0)])
pulp [a0b73760ce164b47bc3cb0b7a696f924]: backoff:INFO: Backing off download_wrapper(...) for 3.7s (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host pulp-api:443 ssl:default [Connect call failed ('IPv6', 443, 0, 0)])
Giving up download_wrapper(...) after 4 tries (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host pulp-api:443 ssl:default [Connect call failed ('IPv4', 443)])
pulp [a0b73760ce164b47bc3cb0b7a696f924]: backoff:ERROR: Giving up download_wrapper(...) after 4 tries (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host pulp-api:443 ssl:default [Connect call failed ('IPv4', 443)])
pulp [a0b73760ce164b47bc3cb0b7a696f924]: pulpcore.tasking.tasks:INFO: Task 019151cc-d244-75d6-aa21-d7c196ffeb6f failed (Cannot connect to host pulp-api:443 ssl:default [Connect call failed ('IPv4', 443)])
pulp [a0b73760ce164b47bc3cb0b7a696f924]: pulpcore.tasking.tasks:INFO:   File "/usr/local/lib/python3.9/site-packages/pulpcore/tasking/tasks.py", line 75, in _execute_task
    result = func(*args, **kwargs)

  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/tasks/sync.py", line 61, in sync
    DeclarativeVersion(first_stage, repository, mirror).create()

  File "/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
    loop.run_until_complete(pipeline)

  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()

  File "/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 220, in create_pipeline
    await asyncio.gather(*futures)

  File "/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 41, in __call__
    await self.run()

  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/tasks/sync.py", line 152, in run
    await pmirror.synchronize(packages_to_sync)

  File "/usr/local/lib/python3.9/site-packages/bandersnatch/mirror.py", line 65, in synchronize
    await self.determine_packages_to_sync()

  File "/usr/local/lib/python3.9/site-packages/pulp_python/app/tasks/sync.py", line 227, in determine_packages_to_sync
    result = await downloader.run()

  File "/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py", line 269, in run
    return await download_wrapper()

  File "/usr/local/lib/python3.9/site-packages/backoff/_async.py", line 151, in retry
    ret = await target(*args, **kwargs)

  File "/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py", line 254, in download_wrapper
    return await self._run(extra_data=extra_data)

  File "/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py", line 287, in _run
    async with self.session.get(

  File "/usr/local/lib64/python3.9/site-packages/aiohttp/client.py", line 1197, in __aenter__
    self._resp = await self._coro

  File "/usr/local/lib64/python3.9/site-packages/aiohttp/client.py", line 581, in _request
    conn = await self._connector.connect(

  File "/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py", line 544, in connect
    proto = await self._create_connection(req, traces, timeout)

  File "/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py", line 944, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)

  File "/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py", line 1257, in _create_direct_connection
    raise last_exc

  File "/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py", line 1226, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(

  File "/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py", line 1033, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc

[ggainey]

Redacted ip4/6 addrs from prev comment - prob not a serious exposure, but seemed the polite thing to do :)

[PotentialIngenuity]

I made sure I have PYPI_API_HOSTNAME set but the replicate still uses pulp-api

[gerrod3]

On your upstream Pulp, when you list your distributions what is the host name in base_url? If it doesn’t match what you set PYPI_API_HOSTNAME then you need to restart your Pulp for the setting change to take effect.

Another possibility is that your downstream Pulp hasn’t updated the remotes url it is using to sync. Check the url field of the remotes on this system and see if the hostname is correct. I’m pretty sure the replicate task is supposed to update the remote automatically, but if it isn’t then you can delete the remotes and rerun the task and have it recreate them.