apt.cache.FetchFailedException - File not found - /cdrom/dists/resolute/Release

[vhaudiquet]

Running the replace-kernel action with a recent resolute image fails:

# python __main__.py ../../livecd.ubuntu-k3.iso k3l.iso --replace-kernel generic
set up loop device /dev/loop54 backing ../../livecd.ubuntu-k3.iso
found live iso9660 filesystem on /dev/loop54
running replace-kernel with arguments {'flavor': 'generic'}
  running unpack-initrd with arguments {}
  squashfs 'minimal' now mounted at 'new/minimal'
  ** updating apt lists... **
Ign file:/cdrom resolute InRelease
Err file:/cdrom resolute Release
  File not found - /cdrom/dists/resolute/Release (2: No such file or directory)
...
Fetched 18.4 MB in 0s (0 B/s)
umount: /tmp/tmp7nlcg70b/new/minimal: target is busy.
umount: /tmp/tmp7nlcg70b/old/iso: target is busy.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 565, in update
    res = self._cache.update(fetch_progress, slist, pulse_interval)
apt_pkg.Error: W:Download is performed unsandboxed as root as file '/tmp/tmp7nlcg70b/new/minimal/var/lib/apt/lists/partial/_cdrom_dists_resolute_InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied), W:Repositories should provide a clear-signed InRelease file, but none found at file:/cdrom/dists/resolute/InRelease., E:The repository 'file:/cdrom resolute Release' does not have a Release file.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vhaudiquet/Téléchargements/livefs-editor/livefs_edit/__main__.py", line 105, in <module>
    main(sys.argv[1:])
    ~~~~^^^^^^^^^^^^^^
  File "/home/vhaudiquet/Téléchargements/livefs-editor/livefs_edit/__main__.py", line 87, in main
    func(ctxt, **kw)
    ~~~~^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/dist-packages/livefs_edit/actions.py", line 57, in impl
    return func(ctxt, **kw)
  File "/usr/local/lib/python3.13/dist-packages/livefs_edit/actions.py", line 665, in replace_kernel
    cache.update(AcquireProgress())
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 567, in update
    raise FetchFailedException(e)
apt.cache.FetchFailedException: W:Download is performed unsandboxed as root as file '/tmp/tmp7nlcg70b/new/minimal/var/lib/apt/lists/partial/_cdrom_dists_resolute_InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied), W:Repositories should provide a clear-signed InRelease file, but none found at file:/cdrom/dists/resolute/InRelease., E:The repository 'file:/cdrom resolute Release' does not have a Release file.

I have locally fixed that by disabling cdrom sources before updating the cache in replace-kernel:
vhaudiquet@20d3c96

However that does not seem like a really good solution. I'm not sure why the file is not detected, because it is indeed present in the iso. Maybe something is wrong in the mounting or in the paths (maybe something is hardcoded to /cdrom, as this would be the case if we boot the iso, but here we mount it to a temporary path and so /cdrom does not exist).

This will need a bit more investigation.

[mwhudson]

we need to bind mount the iso into the rootfs at /cdrom i think

[vhaudiquet]

After some time spent debugging, it seems that we also should replace

cache.update(AcquireProgress())

with something like

ctxt.run(['chroot', base, 'apt-get', 'update'])

The issue seems to be that even if we bind-mount into /cdrom, we were doing apt update using the host python_apt library which was basically running on the host, and something prevents it, even with the right configuration, to be completely inside the chroot. While chrooting and executing apt-get update works after bind-mounting.

Does that make sense? I'm not completely sure of what is happening and how the whole codebase works but that is my current understanding.

EDIT: I'm leaving the end of my conversation with copilot here as it seems to sum up quite well the full problem

1. Root Cause: The Python apt library (apt_pkg) doesn't respect the Dir configuration for
   absolute file:// URIs in sources files. When the cdrom.sources file contains URIs:
   file:///cdrom, apt looks for /cdrom as an absolute path from the system root, not relative
   to the configured Dir.
2. The Solution:

• Mount the ISO content to {base}/cdrom where base is the overlay mount point
• Use chroot with apt-get update instead of Python's cache.update() for updating apt
  lists
• This way, from inside the chroot, /cdrom resolves to the correct bind mount
• Also configured APT::Sandbox::User = "" in the cache_for_dir function to disable apt
  sandboxing

3. Why this is the real fix: This matches the pattern already used in the codebase at
   line 752 where chroot is used with apt-get update to access bind-mounted repositories.
   It's the proper way to handle absolute file:// paths in a chrooted environment

I can try to clean it all up and prepare a PR if that seems like a good solution.

[mwhudson]

Oh right that makes sense yes. I'd forgotten that I'd tried to be fancy and use python-apt -- probably makes sense to just run chroot apt-get a lot instead.

[vhaudiquet]

Ah, but it is a bit tricky because then the code also uses python_apt to install the new kernel. So either we need to remove python_apt everywhere, or make it work with the cdrom thing. I will try to think of the best solution. I guess adding tests could also be a good idea, I will see if I have time to work on this.

[mwhudson]

There is another thing you could do, which would be to delete etc/apt/sources.list.d/cdrom.sources out of the rootfs -- which might actually make sense tbh, you're probably not using replace-kernel to fetch a package that's in the pool!

[vhaudiquet]

Yes, that is what I did locally with vhaudiquet@20d3c96
Maybe we can just merge that... although it really feels like a hack

[mwhudson]

yeah i think that probably makes sense.