Q&A: SleepPhaseOrchestrator config — budget, circuit-breaker thresholds, custom hooks, and PromQL monitoring

[web3guru888]

Frequently Asked Questions about SleepPhaseOrchestrator

---

Q1: What wall_clock_budget_s should I set?

A: Start with the default of 30 s and observe asi_sleep_orchestrator_duration_seconds{quantile="0.99"}. For a typical 7-hook configuration on a CPU-only deployment:

Hardware	Typical p99	Recommended budget
GPU (A100)	2–5 s	15 s
GPU (RTX 3090)	4–8 s	20 s
CPU only (32-core)	8–20 s	40 s
CPU only (8-core)	15–40 s	60 s

Set wall_clock_budget_s to ~2× your observed p99. Monitor asi_sleep_budget_exceeded_total — if it increments, increase the budget or parallelize same-priority hooks.

---

Q2: What happens if a hook fails? Does the whole SLEEP_PHASE abort?

A: No. The orchestrator is fault-tolerant, not fail-fast. On hook failure:

1. hook.rollback(ctx) is called.
2. Already-succeeded hooks are rolled back in reverse priority order.
3. The orchestrator continues to the next hook.
4. The CircuitBreaker records the failure; after failure_threshold consecutive failures it opens and the hook is skipped until recovery_window_s elapses.

To get abort-on-first-failure behaviour, set failure_threshold=1.

---

Q3: How do I add a custom hook?

A: Implement the AdaptationHook Protocol:

class MyCustomHook:
    hook_name = "my_hook"
    priority = 55  # runs between ReplayBuffer (50) and HyperController (60)

    async def run(self, ctx: SleepContext) -> HookResult:
        t0 = time.monotonic()
        # ... do your work ...
        return HookResult(self.hook_name, success=True, duration_s=time.monotonic() - t0)

    async def rollback(self, ctx: SleepContext) -> None:
        pass  # implement if your hook has side effects

orchestrator.register_hook(MyCustomHook())

Because AdaptationHook is a @runtime_checkable Protocol, mypy will verify compliance at annotation sites without requiring inheritance.

---

Q4: Can I run hooks in parallel?

A: Not in the 7.5 release — hooks run sequentially in priority order. Parallel execution is an open question (see Discussion #273). The API is forward-compatible: same-priority hooks could be gathered in a future minor release without breaking existing code.

---

Q5: How do I disable a single hook without removing it?

A: Two options:

• Temporarily: call orchestrator.deregister_hook("hook_name") and orchestrator.register_hook(hook) to re-enable later.
• Permanently at build time: pass None for that component in build_sleep_orchestrator() — the factory simply omits it from the hook list.
• Via circuit breaker: artificially set failure_threshold=0 to keep the circuit permanently open (not recommended for production).

---

Q6: Is circuit-breaker state persisted across restarts?

A: No — in 7.5 circuit state is in-memory only. Use orchestrator.snapshot() / orchestrator.restore(snap) to serialise state before shutdown and reload it on startup. Persisting to Redis is straightforward:

snap = await orchestrator.snapshot()
await redis.set("sleep_orchestrator_snap", json.dumps(asdict(snap)))

---

Q7: How do I monitor orchestrator health with PromQL?

# Is any hook circuit-tripped in the last 5 min?
increase(asi_sleep_hook_circuit_trips_total[5m]) > 0

# Which hook is slowest (p95)?
histogram_quantile(0.95, rate(asi_sleep_hook_duration_seconds_bucket[10m]))

# Budget breaches per hour
increase(asi_sleep_budget_exceeded_total[1h])

# Fraction of cycles where replay buffer hook was skipped
rate(asi_sleep_hooks_skipped_total{hook="replay_buffer"}[10m])
/ rate(asi_sleep_orchestrator_duration_seconds_count[10m])