Q&A: Phase 10.3 PlanExecutor — concurrency tuning, retry configuration, failure modes, and Grafana monitoring

[web3guru888]

Q&A: Phase 10.3 PlanExecutor — concurrency tuning, retry configuration, failure modes, and Grafana monitoring

---

Q1: How do I tune max_concurrency?

A: Start with CPU count × 2 for I/O-bound tasks, then adjust based on asi_plan_executor_active_tasks gauge.

Deployment	Recommended max_concurrency	Rationale
Edge device (2-core)	2–4	Memory-constrained
Standard server (8-core)	8–16	Balanced I/O
High-throughput cluster	32–64	Network-bound dispatch

Monitoring: If asi_plan_executor_active_tasks consistently hits max_concurrency, increase the limit. If wave duration p95 is high but active tasks are low, the bottleneck is the dispatcher, not concurrency.

---

Q2: What happens if a sub-task fails and skip_on_failure=False?

A: The task is marked FAILED after exhausting max_retries. All downstream dependents (tasks that transitively depend on the failed task) are marked SKIPPED. PlanResult.failed > 0 triggers GoalRegistry.update_status(goal_id, GoalStatus.FAILED).

Wave 0: A → SUCCESS
Wave 1: B → FAILED (exhausted retries), C → SUCCESS (same wave, already in-flight)
Wave 2: D → SKIPPED (B is in failed_ids)
PlanResult: total=4, completed=2, failed=1, skipped=1

To allow partial completion despite failures, set skip_on_failure=True — failed tasks are treated as SKIPPED and do not block dependents.

---

Q3: How does retry backoff work?

A: PlanExecutor uses exponential backoff with jitter:

sleep = (retry_backoff_ms * (2 ** attempt) / 1000) + random.uniform(0, 0.05)

With the default retry_backoff_ms=500:

Retry attempt	Sleep time	Jitter
1st retry	500 ms	0–50 ms
2nd retry	1 000 ms	0–50 ms
3rd retry	2 000 ms	0–50 ms

Each retry is counted as asi_plan_executor_subtasks_total{state="retried"}. A successful retry emits asi_plan_executor_retry_attempts_total{result="success"}.

---

Q4: Can I implement a custom TaskDispatcher?

A: Yes — implement the TaskDispatcher Protocol:

from asi_build.cognition.goal_management import TaskDispatcher, SubTask, SubTaskResult, ExecutionState

class MyCustomDispatcher:
    async def dispatch(
        self, subtask: SubTask, context: dict
    ) -> SubTaskResult:
        # Your dispatch logic here
        result = await my_backend.execute(subtask.subtask_id)
        return SubTaskResult(
            subtask_id=subtask.subtask_id,
            state=ExecutionState.SUCCESS,
            output=result, error=None,
            attempts=1, duration_ms=42.0,
            completed_at_ms=int(time.time() * 1000),
        )

Register it:

from asi_build.cognition.goal_management import build_plan_executor
executor = build_plan_executor(dispatcher=MyCustomDispatcher())

The factory accepts an optional dispatcher argument; if omitted, it constructs a RouterTaskDispatcher using the default FederatedTaskRouter.

---

Q5: How does PlanExecutor integrate with GoalRegistry?

A: The 4-step integration flow in CognitiveCycle.run_once():

# 1. Gate: only proceed if a goal is ACTIVE
goal = await self.goal_registry.get_active_goal()
if not goal:
    return

# 2. Decompose: generate TaskGraph from goal
graph = await self.goal_decomposer.decompose(goal)

# 3. Execute: run all sub-tasks
result = await self.plan_executor.execute(graph, context=cycle_context)

# 4. Update: reflect outcome in GoalRegistry
new_status = GoalStatus.ACHIEVED if result.failed == 0 else GoalStatus.FAILED
await self.goal_registry.update_status(goal.goal_id, new_status)

This closes the Phase 10 autonomous loop: goals drive decomposition, decomposition drives execution, execution drives goal status updates.

---

Q6: What's the difference between FAILED and SKIPPED states?

A:

State	Meaning	Counted in
FAILED	Task was attempted, exhausted max_retries, never succeeded	PlanResult.failed, asi_plan_executor_subtasks_total{state="failed"}
SKIPPED	Task was never attempted — either a dependency failed or execution was cancelled	PlanResult.skipped, asi_plan_executor_subtasks_total{state="skipped"}

Key implication: SKIPPED subtasks do not consume retry budget and do not trigger RETRY_ATTEMPTS metrics. This makes it easy to distinguish "my task broke" from "my task never ran."

---

Q7: Grafana dashboard setup for PlanExecutor monitoring

A: Three recommended panels + one alert rule:

Panel 1 — Wave Duration Heatmap

# Heatmap visualization type
rate(asi_plan_executor_wave_duration_seconds_bucket[5m])
# Shows distribution of wave execution times; highlight slow waves

Panel 2 — SubTask State Breakdown (Pie Chart)

sum by (state) (
  increase(asi_plan_executor_subtasks_total[1h])
)
# Shows proportion of success / failed / skipped / retried

Panel 3 — Retry Success Rate (Gauge)

rate(asi_plan_executor_retry_attempts_total{result="success"}[5m])
/
(rate(asi_plan_executor_retry_attempts_total[5m]) + 1e-9)
# Ratio of retries that eventually succeeded

Alert Rule — Plan Failure Rate > 5%

alert: PlanExecutorHighFailureRate
expr: |
  rate(asi_plan_executor_subtasks_total{state="failed"}[5m])
  / rate(asi_plan_executor_subtasks_total[5m]) > 0.05
for: 2m
labels:
  severity: warning
annotations:
  summary: "PlanExecutor failure rate {{ $value | humanizePercentage }} exceeds 5%"