Unable to remove a node from the cluster

[mountkin]

The Watch method of the discovery backend calls cluster.UpdateNodes method to update the node list according the return value of the Fetch method of the discovery service.
But it only inserts the nodes that aren't exist in the node list and doesn't touch the nodes that are "removed" from the discovery service.

If a node is removed from the discovery service, the swarm list command returns the correct result because it calls the Fetch method of the discovery service directly. But in fact the removed node is still in the node list of the cluster.

[vieux]

That's correct, right now, the node isn't removed, but flagged as unhealthy and is skipped by the scheduler.

We are not sure about how we want to handle this, because we don't want to loose information in case of the node comes back up.

Did you encounter a specific issue ?

[mountkin]

Hi, @vieux
As far as I can see the removed node is not flagged as unhealthy unless the refreshContainers method returns an error.
My case is that after I add a node to the cluster and then need to remove it for debugging, I don't want new containers to be launched on the removed node.

[mountkin]

What about add another command such as swarm leave --addr=1.2.3.4 or swarm remove --addr=1.2.3.4 that sends a "leave" message to the discovery service? The swarm daemon can remove the corresponding node when it receives the message.
Or perhaps we can send a specific signal to the swarm join process, when the process receives the signal it send the "leave" message to the discovery service and quit.

For tracking the information of the removed node when it comes back up, perhaps a persistent storage of the nodes information is required.

[vieux]

@mountkin I don't feel live a new command would be the solution. If you're using etcd or consul, there is a TTL so your node will disappear. This feature is coming to the hosted registry service.

[vieux]

@aluzzardi what do you think ?

[aluzzardi]

Eventually, we will need to handle the node removal case correctly.

Right now we don't do any house keeping even if the node gets removed through etcd / consul.

We need to find a way to make the distinction between node removal and unhealthy node.

Regarding the debugging, I don't think nodes should be removed straight from discovery as every implementation will act differently. Perhaps this should be a super set of the API (e.g. DELETE /v1/nodes/foo) and the swarm binary would provide easy access to it.

[tnachen]

Different schedulers also will have their own health check and tracking of different task status, such as Mesos.
I don't have a good picture how different schedulers are suppose to plug into swarm yet, but I think ideally is that there is both a way to notify swarm for actually node health, or let swarm's node detection and tracking to be pluggable so it can call out to existing services. In Mesos case it can simply contact the master to get all the information.

[vincentwoo]

👍 for node removal being a priority. If you have long running jobs, you need some way of signaling that a node is going to be removed but is still present while it is finishing its outstanding jobs.

[bryantidd]

what is the priority and timeline on node removal - housekeeping? It seems that this is integral to using Swarm. Otherwise it presents only a half solution for Docker cluster management. Would be happy to help test.

[mountkin]

The node list is synced from discovery services periodically.
If we simply remove it from swarm's cached node list, it'll be added again when the next UpdateNodes is triggered by discovery service.

I think the problem can be solved in two ways:

1. Add a new member to the Cluster struct to keep all the nodes that had been "removed". And modify the UpdateNodes function to exclude the nodes that are in the removed list.
   The list of removed nodes should be stored persistently.
   This will introduce two new APIs to swarm:
   • DELETE /nodes/:node_id which is responsible for add an entry to the removed list and delete the node from cluster.nodes.
   • POST /nodes/:node_id which is responsible for remove the node from the removed list.
2. The user removed the node from discovery service first, then call swarm's API to delete it from the cache.
   But this may not work for the discovery services that haven't exposed a DELETE API, such as nodes and token.

@vieux @aluzzardi What do you think?