CNI: Make firewalld rules --permanent

[Schrottfresse]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I configured CNI to use firewalld as firewall backend and added my podman network interface to the trusted zone. When I'm reloading firewalld (firewall-cmd --reload) all my podman related rules go missing. I'd like to have the ability to make the podman firewalld rules --permanent.
(I'm starting my containers as root.)

Steps to reproduce the issue:

1. Start container

2. firewall-cmd --reload

3. Rules are missing.

Describe the results you received:

# podman start pihole
5d95bb331f8d59c4d7d5da9af98570b3f82dd933aaf22126650e31741f225a6a
# podman ps
CONTAINER ID  IMAGE                                   COMMAND  CREATED            STATUS             PORTS  NAMES
5d95bb331f8d  docker.io/pihole/pihole:latest                   About an hour ago  Up 49 seconds ago         pihole
# firewall-cmd --zone=trusted --list-all
trusted (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces:
  sources: 172.16.21.4/32 fd00:c001::9e/128
  services:
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
# firewall-cmd --reload
success
# firewall-cmd --zone=trusted --list-all
trusted
  target: ACCEPT
  icmp-block-inversion: no
  interfaces:
  sources:
  services:
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

Describe the results you expected:

Rules don't go missing.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

# podman version
Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.12.12
OS/Arch:            linux/amd64

Output of podman info --debug:

# podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.12.12
  podman version: 1.6.4
host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.6-1.module_el8.1.0+272+3e64ee36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.6, commit: 7a4f0dd7b20a3d4bf9ef3e5cbfac05606b08eac0'
  Distribution:
    distribution: '"centos"'
    version: "8"
  MemFree: 32140668928
  MemTotal: 33369112576
  OCIRuntime:
    name: runc
    package: runc-1.0.0-64.rc9.module_el8.1.0+272+3e64ee36.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 16835932160
  SwapTotal: 16835932160
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: carter
  kernel: 4.18.0-147.5.1.el8_1.x86_64
  os: linux
  rootless: false
  uptime: 1h 35m 6.12s (Approximately 0.04 days)
registries:
  blocked: null
  insecure: null
  search:
  - registry.access.redhat.com
  - registry.fedoraproject.org
  - registry.centos.org
  - docker.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 2
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 3
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

# rpm -q podman
podman-1.6.4-2.module_el8.1.0+272+3e64ee36.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

physical server

[mheon]

I don't think we can do this because we want the rules to disappear after reboot.

We've talked about this before, and the best we think we can do is offer a command that forces network reconfiguration for all running containers, recreating the rules.

[baude]

@mheon doesnt this also make some assumptions about the network stack being exactly the same each time as well? Given that we have zero interaction with firewalld itself, this seems nearly impossible.

would it be possible to have something like a run-hook option where an executable could be run once the container state hit running or some such? of course, this could be done nicely with systemd as well and is maybe the more appropriate approach.

[mheon]

If we do it any later, we end up with a race where traffic after the container started but before the network started will be lost.

We can probably preserves IP address and MAC address on container recreation, even for non-static IP and MAC (we have a mechanism built-in already for doing this, originally for checkpoint/restore). I think we can probably duplicate the network configuration "well enough" with this (it works for checkpoint/restore on a different system, which seems promising)

[ttys3]

I've encountered with this problem too.

sudo podman run --name ngxdemo -p 8086:80 nginxdemos/hello

❯ sudo podman inspect ngxdemo | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "10.88.1.5",

❯ sudo firewall-cmd --zone=trusted --list-all 
trusted (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: 
  sources: 10.88.1.2/32 10.88.1.1/32 10.88.1.4/32 10.88.1.5/32
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

reload firewalld rules:

sudo firewall-cmd --reload

sources empty now:

❯ sudo firewall-cmd --zone=trusted --list-all                  
trusted
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

try restart one of the container:

❯ sudo podman restart ngxdemo
Error: cannot listen on the TCP port: listen tcp4 :8086: bind: address already in use

try restart again, this time it will success:

❯ sudo podman restart ngxdemo
c7a0efe0c7dc540717c301216f01fef7f9eba12315418c16ed554dd8733e2add

and got back:

❯ sudo podman inspect ngxdemo | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "10.88.1.6",

❯ sudo firewall-cmd --zone=trusted --list-all  
trusted (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: 
  sources: 10.88.1.4/32 10.88.1.1/32 10.88.1.6/32
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

---

ps: I even use firewalld as CNI firewall plugin backend, not the default iptables.

❯ cat /etc/cni/net.d/87-podman-bridge.conflist
{
    "cniVersion": "0.4.0",
    "name": "podman",
    "plugins": [
	{
            "type": "bridge",
            "bridge": "cni-podman0",
            "isGateway": true,
            "ipMasq": true,
            "ipam": {
		"type": "host-local",
		"routes": [
		    {
			"dst": "0.0.0.0/0"
		    }
		],
		"ranges": [
		    [
			{
			    "subnet": "10.88.0.0/16",
			    "gateway": "10.88.0.1"
			}
		    ]
		]
            }
	},
	{
            "type": "portmap",
            "capabilities": {
		"portMappings": true
            }
	},
	{
            "type": "firewall",
	    "backend": "firewalld"
	}
    ]
}

[abalage]

Although this is really annoying, as I recall Docker + firewalld worked the very same way.
I always had to restart docker daemon to recreate its iptables chains and rules after I did a 'firewall-cmd --reload'. I might recall that containers had to be redeployed as well.

For me what worked is to use a container specific OS (like OpenSUSE MicroOS, Fedora CoreOS) explicitly dedicated for containers and leave network and firewall management to podman alone. I can use firewall rules outside of this VM.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon Any movement on this one?

[mheon]

No. I do want to add a command to do this, but there is no way we'll have time until Podmanv2 is done.

[KlausThornProgrammfabrik]

To my understanding, firewalld and nftables were designed to solve this exact type of problem:

• does not assume complete control of firewall backend
• won’t delete firewall rules installed by other tools or users

... from: https://developers.redhat.com/blog/2018/08/10/firewalld-the-future-is-nftables/

I guess podman/CNI would have to use nftables and create own tables. And leave firewalld to the user's local customization, only. Then firewalld-cmd --reload will just affect the user's stuff.

Have you investigated this? Are there plans for this?

[dcode]

I just ran into this as well and thinking through it, it makes sense. I'm not familiar enough with the intimate details, but seems like CNI doesn't have any sort of daemon (sort of the point, I guess), would it be possible to use a dbus-activated service (via systemd) that could examine the state of containers and recreate the appropriate entries when firewalld reloads, e.g. when Reloaded signal is emitted on org.fedoraproject.FirewallD1

[sergeiwaigant]

Same here with Podman 1.6.4 on RHEL7.7 EUS.
I don't understand why this issue is not being tackled.
With this situation we would need to disable Firewalld at all to use Podman containers as services - or is there any other workaround to overcome that problem?

We are using systems maintained by central team with Puppet and they run firewalld --reload every 30 minutes, this is breaking the iptables and the services in the containers are not working anymore.