Proposal: Serf-discovery mode

[chanwit]

Currently we need to hard-code a discovery URL for every swarm join to make the agent able to join a cluster. Hard-coded discovery makes this architecture not scaling very well.

This issue proposes a new architecture called self-discovery. It includes:

• IP scanning on a specific port to make an agent try its best to find a peer in the current network (but it's not the best idea anyway)
  • use the old previous connection information to find a peer
  • we should do only class-C scanning? is there any viable alternative?
  • should we use mdns, which requires multi-cast available on the network?
• peer-to-peer communication based on serf to form a local serf cluster
• a small modification to swarm manager acts as an aggregation point to tell the discovery service that we have N nodes in this local cluster.
  With this architecture, it allows other managers who are sharing the same discovery URL to form a larger swarm cluster.

This diagram describes the self-discovery architecture:

Step 1. swarm join --advertise=<ip:addr> self://:3388
        To make it as minimum configuration as possible, 
        we do heuristic-based discovery for finding peers.
        So IP address should not be required in the normal scenario.
                                                                                     
                 +---------+                                                         
                 |         |                                                         
     +----------->  agent  +---------+            +-----------+                      
     |           |         |         |    join    |           |                      
     |           +---------+         <------------+  manager  |                      
     |                               |            |           |                      
+---------+                     +----v----+       +-----------+                      
|         |                     |         |            |              +-------------+
|  agent  |    self-discovery   |  agent  |            |              |             |
|         |                     |         |            | expose nodes |  discovery  |
+---------+                     +---------+            +-------------->  service    |
     |                               |                                |             |
     |           +---------+         |                                +-------------+
     |           |         |         |                                               
     +----------->  agent  <---------+                                               
                 |         |                                                         
                 +---------+                                                         
                                                                                     
 Step 2. When calling `swarm manage --self-discovery-port=3388 zk://ip:addr`,                  
         the manager is also responsible for:                                        
         * joining a local serf cluster                                                   
         * exposing the local node list to the discovery URL
         * start the manager using the discovery URL
                                                                                     
In case of starting the manager as the first node,                                   
it also acts as the first peer waiting for other agents to join.                         

As discussed with @abronan on the IRC, @aluzzardi already has an idea to use serf for a similar architecture.
/cc @vieux @jimmyxian

WDYT?

[jimmyxian]

Cool!
But we need consider this case: Create two container on different two managers at the same time. We will meet Parallel scheduling problem.
Maybe two ways to solve:

• Using Something like Using Distributed Lock to solve Parallel scheduling.
• Using leader election. Only one leader at a time. If the request lands on a manager that is not currently the leader, it forward it to the right instance.

[chanwit]

@jimmyxian yep, leader election would be helping us.

[chanwit]

Maybe we can try mDNS mode in serf. It's known to work well for private networks (but not for cloud).
https://github.com/hashicorp/serf/blob/master/command/agent/mdns.go

[chanwit]

We will have this file: ~/.docker/swarm/.known_peers for storing a set of known peers to make bootstrapping faster for the first run. Let me know what do you think about this.

[abronan]

@chanwit Sounds good! 👍

IP scanning might not the best approach though, we can have a very simple UI like:

swarm join <flags>                       # this is the first node, serf port is implicit
swarm join <flags> --join-ip=10.0.0.7    # second node pointing to the first one
etc.

And this should be sufficient for most cases and the UI is not much more complex .

The flag is badly chosen because it stutters with the command name but we can find another one or use the existing notation for discovery services:

swarm join <flags> serf://<serf-ip1>,<serf-ip2>,<serf-ip3>

Also I see how the file could be useful to easily bootstrap a huge cluster quickly but it is unnecessary for now imho.

[abronan]

@jimmyxian Scheduling is an issue today even with a single Manager and multiple requests incoming. The purpose of a Serf backend for discovery would be to get rid of the store entirely though and keep the operational maintenance low, so Distributed Locks or mechanisms for Leader Election might not be available in this case.

My first shot for the parallel scheduler was to use Consul, Etcd and Zookeeper as a centralized ScheduleMap with a consistent view of the cluster and the resources available. The scheduling would be parallel by having cells in the ScheduleMap adopting multiple states (free, temporary reserved or locked). This way I was using only Compare and Swap operations and processes not fast enough to reserve a resource for a container (or possibly a group of containers) would retry until they had more chance to lock a resource slice.

I'm wondering if there is a better way to handle this though and separate this knowledge and atomic resource reservation mechanisms possibly directly inside the Engine, as the storage in this case is still a point of failure. Also, using Serf for discovery coupled to a Consul on the side for the Parallel Scheduler does not make much sense to me as the whole purpose of Serf is to have a loose infrastructure with failure detection and sharing information through gossiping (and also Consul is already backed by Serf so we'd be better using the node registration of Consul rather than discovering through Serf in this case). Having a Parallel Scheduler without this central piece can be challenging though. Still an open question 😄

[chanwit]

Waiting for moby/moby#16229

[vieux]

@chanwit I'm super excited by this PR :)

[chanwit]

@vieux should I go for it by now?
It seems there will be a similar mechanism inside the engine by 1.9.

[vieux]

@chanwit I think it's safer to wait until 1.9 PR is merged

[chanwit]

@vieux moby/moby#16229 is already get merged. I'll continue on this.