Show & Tell: Phase 9.1 — FederationGateway architecture (mTLS peer registry, gossip membership, broadcast fan-out)

[web3guru888]

Phase 9.1 — FederationGateway architecture

Phase 9.1 introduces the FederationGateway — the first component of ASI-Build's multi-agent federation layer. It gives every node a mTLS-authenticated peer registry, gossip-based membership, and a message-routing layer that later phases will build on.

---

Why federation now?

Phase 8 deliverable	Phase 9 enabler
Docker/Helm (8.4)	Run multiple nodes in same cluster
ExplainAPI (8.3)	Expose per-node introspection to operator
Sepolia CI (8.5)	CI proves contracts survive cross-node deploy
DecisionTracer (8.1)	Federate traces for distributed attribution

---

Component map

┌──────────────────────────────────────────────────────┐
│                   ASI-Build Node A                   │
│                                                      │
│  ┌─────────────────────────────────────────────┐    │
│  │          InMemoryFederationGateway           │    │
│  │                                             │    │
│  │  PeerRecord dict   PeerStatus dict          │    │
│  │  (asyncio.Lock)    (health FSM)             │    │
│  │                                             │    │
│  │  _health_poll_loop ──► /health ping         │    │
│  │  _evict_stale_loop ──► TTL eviction         │    │
│  │  _bootstrap_seeds  ──► gossip join          │    │
│  └──────────────┬──────────────────────────────┘    │
│                 │ mTLS / aiohttp                     │
└─────────────────┼────────────────────────────────────┘
                  │
    ┌─────────────▼─────────────┐
    │      ASI-Build Node B     │
    │  InMemoryFederationGateway│
    └───────────────────────────┘

---

PeerRecord design

@dataclass(frozen=True)
class PeerRecord:
    peer_id: str          # DID string, e.g. "did:key:z6Mk..."
    endpoint: str         # "https://peer-host:8443"
    public_key_pem: str   # PEM-encoded Ed25519 public key
    capabilities: FrozenSet[str] = field(default_factory=frozenset)
    joined_at_ms: int = 0

Why frozen? Same reason as CausalEdge and Episode — immutability ensures peer records can be safely shared across async tasks without a lock.

---

Peer status FSM

        register()
            │
            ▼
        HEALTHY ◄──────── health_ok (latency < 500ms)
            │
    latency > 500ms
            │
            ▼
        DEGRADED
            │
     3 consecutive failures
            │
            ▼
      UNREACHABLE
            │
    deregister() / TTL
            │
            ▼
        EVICTED

---

mTLS transport

In production, send_message() uses aiohttp with a per-session ssl.SSLContext:

ssl_ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ssl_ctx.load_cert_chain(cert_path, key_path)
ssl_ctx.load_verify_locations(ca_bundle)
ssl_ctx.verify_mode = ssl.CERT_REQUIRED

In tests, the transport is injected via _transport: Callable[[str, bytes], Awaitable[bytes]] — no TLS required, full determinism.

---

Gossip membership

On startup each node contacts its seed_peers list, downloads GET /federation/peers, and registers each returned PeerRecord. This bootstraps the membership table without a central coordinator.

Over time the health poller naturally converges the view: stale peers are evicted after ttl_ms (default 300 s) if not HEALTHY.

---

broadcast() fan-out

async def broadcast(self, payload: bytes, exclude: frozenset[str] = frozenset()):
    peers = [p for p in self._peers if p not in exclude]
    tasks = [self.send_message(pid, payload) for pid in peers]
    results = await asyncio.gather(*tasks, return_exceptions=True)
    return dict(zip(peers, results))

Errors are captured (not raised), so a single unreachable peer doesn't abort the broadcast.

---

Prometheus metrics

Metric	Type	Labels
federation_peers_total	Gauge	status
federation_messages_sent_total	Counter	peer_id
federation_messages_failed_total	Counter	peer_id, reason
federation_health_check_duration_seconds	Histogram	peer_id
federation_broadcast_fan_out_total	Counter	—

PromQL — detect isolated node:

federation_peers_total{status="HEALTHY"} == 0

PromQL — message failure rate per peer:

rate(federation_messages_failed_total[5m])
/ rate(federation_messages_sent_total[5m])

---

Phase 9 roadmap

Sub-phase	Component	Depends on
9.1	FederationGateway	Phase 8.4 Docker
9.2	FederatedBlackboard	9.1 gateway
9.3	FederatedTaskRouter	9.1 + 9.2
9.4	FederatedConsensus	9.3 router
9.5	FederationHealthMonitor	9.1–9.4

---

Open questions

1. DID method — did:key (local keygen) or did:web (DNS-resolvable)? did:key is simpler for CI but did:web is more production-realistic.
2. Gossip protocol — full push or pull? Push scales poorly above 100 nodes; pull adds latency for new peers.
3. Broadcast routing — should broadcast() be flood-fill or spanning-tree? Spanning-tree saves bandwidth but needs a root election.