Cloned VM fails to start with "Permission denied" on custom path (/opt) due

[WavesMan]

1. Summary

When cloning a running/stopped KVM virtual machine via the Cockpit web interface (or virsh), the new cloned VM fails to start. The error indicates a Permission denied when QEMU attempts to access the disk image located in a non-default directory (/opt/mirrors/...), despite correct file ownership and directory permissions. This suggests an AppArmor profile restriction that is not correctly applied or inherited during the cloning process.

2. Environment

• OS: Ubuntu 24.04 LTS (Noble Numbat)
• Kernel: Linux ubuntu 6.8.0-90-generic
• Libvirt Version: 10.0.0
• QEMU Version: 8.2.2
• Cockpit Version: 314-1
• Storage Path: Custom path /opt/mirrors/isos/ubuntu/ (Not the default /var/lib/libvirt/images)

3. Steps to Reproduce

1. Prepare a base VM (e.g., k8s-A-node1_Ubuntu2404) using a cloud image stored in /opt/mirrors/isos/ubuntu/noble-server-cloudimg-amd64_2404.img.
   • File permissions: -rw-rw-r-- 1 libvirt-qemu kvm ...
   • Directory permissions: drwxr-xr-x (all parent dirs including /opt).
2. Ensure the base VM starts successfully.
3. Use Cockpit Web Console -> Virtual Machines -> Select Base VM -> Clone.
4. Name the new VM (e.g., test) and proceed with cloning.
5. Attempt to start the newly cloned VM test via Cockpit or CLI (virsh start test).

4. Expected Behavior

The cloned VM should start successfully, accessing the disk image at /opt/mirrors/... just like the original VM.

5. Actual Behavior

The VM fails to start immediately with the following error:

error: Failed to start domain 'test'
error: internal error: process exited while connecting to monitor: 
2026-03-13T02:10:05.669349Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/opt/mirrors/isos/ubuntu/noble-server-cloudimg-amd64_2404.img","node-name":"libvirt-4-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/opt/mirrors/isos/ubuntu/noble-server-cloudimg-amd64_2404.img': Permission denied

6. Diagnostics & Troubleshooting Performed

We have verified the following, ruling out standard Linux permission issues:

• File Ownership: The image file is owned by libvirt-qemu:kvm.
• Directory Traversal: All parent directories (/opt, /opt/mirrors, etc.) have o+x (execute) permissions.
• AppArmor Attempts:
  • Added local overrides for /usr.sbin.libvirtd allowing access to /opt/mirrors/**.
  • Added local overrides for /usr.bin.qemu-system-x86_64 allowing access to /opt/mirrors/**.
  • Reloaded AppArmor (systemctl reload apparmor).
  • Result: The error persists specifically for the cloned VM, suggesting the dynamic AppArmor profile generated for the new VM instance might be ignoring the local overrides or resetting to a restrictive default.

7. Hypothesis

It appears that when Libvirt/Cockpit clones a VM, the generated AppArmor profile for the new domain (usually located in /etc/apparmor.d/libvirt/ or dynamically generated) does not correctly include the custom path allowances defined in /etc/apparmor.d/local/, or the cloning process resets the security label context in a way that blocks non-standard paths.

8. Request

• Is there a known issue with AppArmor profiles and cloned VMs using custom storage paths in Ubuntu 24.04?
• Should the cloning process automatically inherit the AppArmor exceptions of the source VM?
• What is the recommended persistent configuration to allow cloned VMs to access images in /opt without manually moving files?

9. Attachments (Logs)

• The XML configuration of the failed cloned VM (virsh dumpxml test)

XML configuration of the failed cloned VM

<domain type='kvm'>
  <name>test</name>
  <uuid>8873724c-3438-45bb-9f9e-73a37e2ad859</uuid>
  <metadata xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0" xmlns:cockpit_machines="https://github.com/cockpit-project/cockpit-machines">
    <libosinfo:libosinfo>
      <libosinfo:os id="http://ubuntu.com/ubuntu/24.04"/>
    </libosinfo:libosinfo>
    <cockpit_machines:data>
      <cockpit_machines:has_install_phase>false</cockpit_machines:has_install_phase>
      <cockpit_machines:install_source_type>cloud</cockpit_machines:install_source_type>
      <cockpit_machines:install_source>/opt/mirrors/isos/ubuntu/noble-server-cloudimg-amd64_2404.img</cockpit_machines:install_source>
      <cockpit_machines:os_variant>ubuntu24.04</cockpit_machines:os_variant>
    </cockpit_machines:data>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os firmware='efi'>
    <type arch='x86_64' machine='pc-q35-noble'>hvm</type>
    <firmware>
      <feature enabled='yes' name='enrolled-keys'/>
      <feature enabled='yes' name='secure-boot'/>
    </firmware>
    <loader readonly='yes' secure='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE_4M.ms.fd</loader>
    <nvram template='/usr/share/OVMF/OVMF_VARS_4M.ms.fd'>/var/lib/libvirt/qemu/nvram/test_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
    <smm state='on'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'/>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' discard='unmap'/>
      <source file='/var/lib/libvirt/images/ubuntu24.04-2026-3-13.1773367589-clone'/>
      <backingStore type='file'>
        <format type='qcow2'/>
        <source file='/var/lib/libvirt/images/ubuntu24.04-2026-3-13.1773362698'/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x18'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x19'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x1a'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    </controller>
    <controller type='pci' index='12' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='12' port='0x1b'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
    </controller>
    <controller type='pci' index='13' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='13' port='0x1c'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
    </controller>
    <controller type='pci' index='14' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='14' port='0x1d'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:b1:30:d6'/>
      <source bridge='k8s-br'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <tpm model='tpm-crb'>
      <backend type='emulator' version='2.0'/>
    </tpm>
    <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <graphics type='spice' autoport='yes' listen='::1'>
      <listen type='address' address='::1'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
    </sound>
    <audio id='1' type='spice'/>
    <video>
      <model type='virtio' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <watchdog model='itco' action='reset'/>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </rng>
  </devices>
</domain>

[39jf]

AppArmor/sVirt 对路径隔离*导致的问题，而不一定是 Cockpit 克隆逻辑
本身的独立缺陷。
问题本质
虽然克隆盘在 /var/lib/libvirt/images 下，但它的 qcow2 backing chain 仍然引用了 /opt/mirrors/... 的基础镜像。
启动时 QEMU 会去打开该 backing 文件，而 AppArmor 拒绝访问，于是报错：
Could not open '/opt/mirrors/...img': Permission denied
所以这不是普通 Linux 文件权限（owner/chmod）问题，而是 libvirt/QEMU 的受限策略问题。
关于“是否继承源 VM 例外规则”
克隆后的 VM 不会稳定地“继承”源 VM 的手工 AppArmor 放行规则。
libvirt 通常会按 domain/UUID 动态生成隔离配置。
Ubuntu 24.04 上的持久化建议
建议使用 AppArmor 的 local override（不要直接改 libvirt-<uuid> 这类动态文件）：

1. /etc/apparmor.d/local/abstractions/libvirt-qemu
2. /etc/apparmor.d/local/usr.lib.libvirt.virt-aa-helper
   示例最小规则：

/opt/mirrors/isos/ubuntu/** rk,
/opt/mirrors/ r,
/opt/mirrors/** r,
重载后生效：
sudo systemctl reload apparmor
sudo systemctl restart virtqemud.service || sudo systemctl restart libvirtd.service
验证步骤
qemu-img info --backing-chain /var/lib/libvirt/images/<clone-image>
UUID=$(virsh domuuid <vm-name>)
sudo grep -n "/opt/mirrors" /etc/apparmor.d/libvirt/libvirt-$UUID.files
virsh start <vm-name>
sudo journalctl -k -g 'apparmor="DENIED"' -n 100

[Venefilyn]

Translating your comment using DeepL for team members to read:

The issue stems from path isolation* in AppArmor/sVirt, rather than an inherent flaw in Cockpit’s cloning logic
itself.
The root of the problem
Although the cloned disk is located under /var/lib/libvirt/images, its qcow2 backing chain still references the base image at /opt/mirrors/....
At startup, QEMU attempts to open this backing file, but AppArmor denies access, resulting in the error:
Could not open ‘/opt/mirrors/...img’: Permission denied
Therefore, this is not a standard Linux file permission (owner/chmod) issue, but rather a restricted policy issue within libvirt/QEMU.
Regarding “Inheriting Source VM Exception Rules”
Cloned VMs do not reliably “inherit” the source VM’s manually configured AppArmor allow rules.
Libvirt typically generates isolation configurations dynamically based on domain/UUID.
Persistence Recommendations on Ubuntu 24.04
It is recommended to use AppArmor’s local override (do not directly modify dynamic files like libvirt-<uuid>):

1. /etc/apparmor.d/local/abstractions/libvirt-qemu
2. /etc/apparmor.d/local/usr.lib.libvirt.virt-aa-helper
   Example minimal rules:

/opt/mirrors/isos/ubuntu/** rk,
/opt/mirrors/ r,
/opt/mirrors/** r,
Apply changes after reloading:
sudo systemctl reload apparmor
sudo systemctl restart virtqemud.service || sudo systemctl restart libvirtd.service
Verification steps
qemu-img info --backing-chain /var/lib/libvirt/images/<clone-image>
UUID=$(virsh domuuid <vm-name>)
sudo grep -n “/opt/mirrors” /etc/apparmor.d/libvirt/libvirt-$UUID.files
virsh start <vm-name>
sudo journalctl -k -g ‘apparmor=“DENIED”’ -n 100

[WavesMan]

Status Update:

On Ubuntu 24.04, the standard AppArmor override method (editing /etc/apparmor.d/local/abstractions/libvirt-qemu) often fails for cloned VMs where the backing file is in a non-standard directory like /opt. Even with correct file ownership (libvirt-qemu:kvm) and directory permissions (+x), the "Permission denied" error persists.

The Root Cause:

The virt-aa-helper tool in Ubuntu 24.04 seems to have issues dynamically resolving the backing chain when it spans across different root-level directories. It fails to generate a valid AppArmor profile for the specific VM instance, causing QEMU to be blocked by the kernel.

Confirmed Solutions:

1. The "Escape Hatch" (Working but less secure):

   It is not recommended for use in a production environment

   Manually modify the VM XML to disable AppArmor for that specific domain:

   <seclabel type='none'/>

   This is currently the most reliable way to get cloned VMs running on 24.04 when using non-standard storage paths.

2. The "Global" Workaround (If you have many VMs):

   It is highly not recommended for use in a production environment

   Disable AppArmor security driver globally in /etc/libvirt/qemu.conf:

   security_driver = "none"
   

   (Restart libvirtd after change).

Better solutions still need to be sought

[WavesMan]

@39jf