Container Rebalancing

[aluzzardi]

In case of node failure, containers should be rescheduled automatically by swarm.

Rebalancing means re-creating the container on another healthy node that fits the original constraints, affinities and requirements of such container.

• Only select containers should be re-scheduled. This strictly includes stateless containers with the go-ahead of the operator
  • Containers with volumes or any other explicit state should not be rescheduled
  • Users should be able to instruct swarm when creating a container whether it can be rescheduled or nor (-e reschedule:[always,never]?)
• Handle node resurrection
  • What happens when a failed node comes back to life? If the node has containers that have been rescheduled, we will end up with duplicates. Think of a stateless service that sends daily reports by e-mail: those messages will end up being sent twice. Ideas:
    • Swarm should kill duplicates (keep the last scheduled version)
    • Swarm should rewrite the restart policy of the container to never and start them manually (based on the restart policy). This way, duplicates would never be automatically started before being destroyed.
• Container IDs
  • Re-scheduling containers means they will change ID. In order to have a consistent ID exposed to the user, we need Virtual IDs. See Virtual Container IDs #600

[abronan]

Just a few thoughts:

Containers with volumes or any other explicit state should not be rescheduled

What if the container uses a distributed block storage (Ceph, GlusterFS, etc.), should it be excluded from rebalancing or could we find a way to rebalance on a host with a mount-point to the same distributed volume?

Swarm should kill duplicates (keep the last scheduled version)

Should we keep the latest version or the old one? What if the original scheduling of a container was using constraints (using a machine with an ssd storage) but we end up rebalancing on a node which does not have that kind of storage. It makes more sense to keep the older version once the host is back. In the case of a network partition, you would think that the host failed while it is indeed still alive, thus you can't rely on rewriting the restart policy.

[cultureulterior]

Speaking as someone investigating swarm, I don't see virtual container IDs necessary for this feature to be useful for me.

[abronan]

@cultureulterior It is not necessary for sure, but it is way more convenient in a huge cluster. It is easier to reason about your system with a single ID for a container no matter how many failures occurs in the system and how many times you container moves from a host to another.

[monokal]

I guess what you're asking for is mainly the functionality already implemented in Google's Kubernetes (Replication Controllers rescheduling Pods when a host dies), perhaps we can take some design decisions from there, or decide how far we want to abstract Swarm before just supporting Kubernetes?

[aluzzardi]

@cultureulterior So, virtual container IDs are an internal thing that won't be directly exposed to the user.

Regarding what to do with duplicates (kill first, kill last, do nothing), it should be user configurable through a reconciliation policy.

[shykes]

@aluzzardi you say here that "virtual ids won't be directly exposed to the user", but in #600 you say they will. Could you clarify?
Note, my preference is to expose them to the user and make the physical ids an advanced field.

[aluzzardi]

@shykes Short answer, will elaborate more:

They are exposed to the user as labels (com.docker.swarm.id).
One can manipulate a container using either its physical ID or virtual ID.
With moby/moby#12931 (or moby/moby#10255), the virtual ID could show up in docker ps.

However, when you do things such as docker create the phyID is returned rather than the virtual ID.

This is because:

• Feedback. I can gather the different pieces here for reference, but the overall feedback I got in GH was against that level of indirection
• Phy IDs are already globally unique
• Challenging UX. The phy ID leaks everywhere in Docker. We would have to catch it and rewrite it in many different places, if it's even possible. For instance, the container hostname is the physical short ID - it's very surprising to manipulate a virtual ID, attach to a container and see a random hostname out of nowhere.
• Not all containers have a Virtual ID - only those created through Swarm. We initially had a lot of code around to handle "backward" compatibility, but in the end decided against it. I think it's totally fine to let users manage their cluster the way they see fit - create stuff through swarm or create directly on the machine. This concept allows swarm and our users to make a clear distinction between the two.