Show & Tell: Phase 11.4 InterpretabilityProbe — feature attribution and explanation generation for alignment decisions

[web3guru888]

What is InterpretabilityProbe?

Safety systems in autonomous agents are only as trustworthy as they are understandable. Phase 11.4 introduces InterpretabilityProbe — a component that produces post-hoc explanations for every alignment decision: why was this goal blocked? what features drove the alignment score down? which feedback signals most changed the reward model?

Without interpretability, the safety layer is a black box. With InterpretabilityProbe, operators can audit, debug, and build confidence in the system.

---

Component map

┌─────────────────────────────────────────────────────────────────┐
│                     CognitiveCycle                              │
│  ┌──────────────┐   ┌────────────────┐   ┌───────────────────┐ │
│  │ SafetyFilter  │──▶│ AlignmentMonitor│──▶│   ValueLearner    │ │
│  │  (verdict)   │   │  (score)       │   │  (weight update)  │ │
│  └──────┬───────┘   └───────┬────────┘   └─────────┬─────────┘ │
│         │                  │                       │           │
│         ▼                  ▼                       ▼           │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │               InterpretabilityProbe                      │  │
│  │                                                          │  │
│  │  ExplanationTarget: SAFETY_VERDICT / ALIGNMENT_SCORE /   │  │
│  │                     REWARD_WEIGHT                        │  │
│  │  AttributionMethod: PERMUTATION / INTEGRATED_GRAD /      │  │
│  │                     GRADIENT_SHAP / LIME_LOCAL           │  │
│  │                                                          │  │
│  │  Output: ProbeExplanation (attributions + counterfactual)│  │
│  │             ▼                                            │  │
│  │         Blackboard["explanation:{goal_id}"]              │  │
│  └──────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

---

Attribution methods

Method	Approach	Best for	Complexity
PERMUTATION	Zero-baseline ablation (zero out each feature, measure delta)	Rule-based models, fast sweep	O(n features)
INTEGRATED_GRAD	50-step Riemann sum over input manifold	Neural/gradient models	O(50n)
GRADIENT_SHAP	SHAP-style gradient × input	Complex non-linear models	O(n²)
LIME_LOCAL	Local linear approximation (n_lime_samples=100)	Any black-box model	O(100n)

Permutation attribution — zero-baseline ablation

async def _permutation_attribution(
    self,
    feature_vector: Sequence[float],
    decision_value: float,
    baseline_fn: Callable[[Sequence[float]], float],
) -> tuple[FeatureAttribution, ...]:
    raw: list[tuple[int, float]] = []
    for i in range(len(feature_vector)):
        ablated = list(feature_vector)
        ablated[i] = 0.0   # zero-baseline ablation
        delta = decision_value - baseline_fn(ablated)
        raw.append((i, delta))
    total = sum(abs(d) for _, d in raw) or 1.0
    return tuple(
        FeatureAttribution(
            feature_name=FEATURE_NAMES.get(i, f"feature_{i}"),
            attribution_score=delta,
            relative_importance=abs(delta) / total,
        )
        for i, delta in sorted(raw, key=lambda x: -abs(x[1]))[: self._config.top_k_features]
    )

---

Feature vector design

SafetyFilter verdict features (12 dimensions)

#	Feature	Description
0	goal_priority	normalised [0, 1]
1	task_count	log-normalised count
2–5	capability_bits	4-bit one-hot
6	resource_budget_cpu	declared CPU budget
7	resource_budget_mem	declared memory budget
8	constitutional_score	rolling alignment score
9	federation_trust_score	peer trust score
10	violation_rate_7d	rolling 7-day rate
11	escalation_history	log-normalised count

AlignmentMonitor score features (8 dimensions)

#	Feature	Description
0–4	dimension_id	5-bit one-hot
5	window_size_s	log-normalised
6	sample_count	log-normalised
7	recent_trend	slope of last 5 samples

---

Counterfactual generation

InterpretabilityProbe generates counterfactuals: the minimal set of feature changes that would reverse the decision.

def _build_counterfactual(
    self,
    attributions: tuple[FeatureAttribution, ...],
    decision_value: float,
) -> tuple[str, ...] | None:
    flips = [
        a.feature_name
        for a in sorted(attributions, key=lambda x: x.attribution_score)
        [: self._config.counterfactual_max_flips]
    ]
    return tuple(flips) if flips else None

Example: Goal blocked (score 0.21). Top negative features: violation_rate_7d (-0.38), federation_trust_score (-0.21). Counterfactual: "Reduce violation rate and improve federation trust to reverse this block."

---

Prometheus metrics

Metric	Type	Description
asi_probe_explanations_total{target,method}	Counter	explanations generated
asi_probe_cache_hits_total	Counter	cache hits
asi_probe_cache_size	Gauge	current cache entries
asi_probe_attribution_ms	Histogram	computation time
asi_probe_confidence_score{target}	Histogram	confidence [0, 1]

PromQL

# Cache hit rate
rate(asi_probe_cache_hits_total[5m]) / rate(asi_probe_explanations_total[5m])

# 95th-percentile attribution latency
histogram_quantile(0.95, rate(asi_probe_attribution_ms_bucket[5m]))

---

Open questions

1. Explanation persistence: Should ProbeExplanation objects be stored in the Blackboard permanently, or only surfaced on-demand via API? How long should explanations be retained?
2. Ground-truth validation: How do we validate that attributions are correct — that the features identified as important actually drive the model's decision?
3. Real-time streaming: Should explanations be pushed to a WebSocket stream so operators can inspect decisions as they happen, rather than querying post-hoc?

Full spec in Issue #346.