Q&A: DistributedTemporalSync — vector-clock gossip, network partitions, and conflict resolution 🌐

[web3guru888]

Q&A: DistributedTemporalSync — implementation questions 🌐

Phase 18.3 · Issue #456 · Show & Tell #457

---

Q1: Why vector clocks instead of a simpler timestamp (e.g., wall-clock or Lamport scalar)?

A: Wall-clock timestamps require synchronised clocks across nodes (NTP drift can be milliseconds or worse), making causal ordering unreliable. A Lamport scalar gives a total order but cannot detect concurrent updates — you can't tell if two events are truly causally unrelated or just ordered by the scalar. Vector clocks preserve full causal structure: VectorClock.dominates() tells us precisely whether one agent's view is a causal superset of another's, which drives our conflict resolution logic correctly.

---

Q2: What happens during a network partition — agent A can't reach agent B for 60 seconds?

A: The broadcast_sync() method catches all exceptions and returns {peer_id: -1} for unreachable peers — degraded mode, not a crash. asi_temporal_sync_peers_reachable drops, triggering a Grafana alert. Meanwhile, A continues accumulating edges locally. When the partition heals, edges_since(peer_clock) returns the full 60-second backlog. To bound message size, a future max_delta_edges: int config cap (already on the SyncConfig) truncates the push to the most recent N edges — older edges get caught up on subsequent rounds.

---

Q3: Can _resolve_conflicts handle the case where neither clock dominates (true concurrency)?

A: The current LAST_WRITER_WINS branch returns an empty list when neither clock dominates — effectively discarding the remote edges. This is a known limitation documented in the implementation notes on issue #456. The correct fix is a _concurrent_merge() fallback that unions the edges (like MERGE_ALL) but only for the concurrent subset. This is deferred to Phase 18.5 (TemporalCoherenceArbiter) which will handle exactly these incoherence cases.

---

Q4: How does TemporalGraph.edges_since(clock) know which edges to return?

A: Each edge stored in the TemporalGraph must carry a source_agent_id: str and agent_clock_value: int field — the clock value of the authoring agent at edge creation time. edges_since(clock) filters to edges where edge.agent_clock_value > clock.clocks.get(edge.source_agent_id, 0). For performance, the graph maintains a _clock_index: dict[str, SortedList[int, EdgeId]] per agent, making lookup O(log n) instead of O(n).

---

Q5: Why is the asyncio.Lock held for the entire sync round (DIALING → VERIFYING)? Couldn't that block incoming pushes?

A: Yes — the coarse lock prevents concurrent outbound sync and inbound receive_push() from racing on the same TemporalGraph. For most deployments with sync_interval_s=5.0 and small deltas, the lock hold time is well under 100ms. For high-throughput scenarios (>10 peers, large deltas), the recommendation in issue #456 is per-peer locks: each peer gets its own asyncio.Lock, allowing parallel sync with different peers. receive_push() would acquire the graph write lock (a separate lock on TemporalGraph) independently.

---

Q6: How does DistributedTemporalSync interact with MemoryConsolidator (18.2)?

A: After apply_edges() completes, the newly merged edges are available to TemporalGraph.get_unconsolidated_traces() — the same interface MemoryConsolidator polls on its background sweep. No direct coupling is needed: the consolidator simply finds more traces on its next sweep. The verify_consistency() step (when verify_post_merge=True) ensures the graph is acyclic and causally consistent before consolidation can read it — preventing the consolidator from extracting patterns from a partially-applied delta.

---

Q7: How do I write tests for the gossip protocol without running real network sockets?

A: Inject an InMemoryPeerTransport:

class InMemoryPeerTransport:
    """Routes push/pull calls directly to peer agent instances."""
    def __init__(self, agents: dict[str, AsyncDistributedTemporalSync]) -> None:
        self._agents = agents

    async def fetch_clock(self, peer_id: str) -> VectorClock:
        return self._agents[peer_id].vector_clock

    async def push_edges(self, peer_id: str, edges: list, clock: VectorClock) -> None:
        await self._agents[peer_id].receive_push("self", edges, clock)

    async def pull_edges(self, peer_id: str, since: VectorClock) -> tuple[list, VectorClock]:
        peer = self._agents[peer_id]
        return peer._graph.edges_since(since), peer.vector_clock

This lets test_sync_with_peer_exchanges_edges run entirely in-memory, with two AsyncDistributedTemporalSync instances sharing an InMemoryPeerTransport dict — no asyncio server needed.

---

Grafana — Sync Topology Dashboard YAML:

panels:
  - title: "Sync Round Duration p99 (per peer)"
    type: timeseries
    targets:
      - expr: >
          histogram_quantile(0.99,
            rate(asi_temporal_sync_round_duration_seconds_bucket[5m])
          ) by (peer_id)
        legendFormat: "p99 / {{peer_id}}"
  - title: "Peers Reachable"
    type: stat
    targets:
      - expr: "asi_temporal_sync_peers_reachable"
        legendFormat: "live peers"
    thresholds:
      steps:
        - color: red
          value: 0
        - color: yellow
          value: 1
        - color: green
          value: 2