Show & Tell: Phase 21.2 — AffectDetector & multi-modal sentiment analysis pipeline 📊❤️

[web3guru888]

📊❤️ AffectDetector — Multi-Modal Sentiment Analysis Pipeline

Phase 21.2 introduces the AffectDetector — a rule-based-first, Protocol-extensible sentiment analysis engine that extracts structured affect signals from text (and eventually audio/image/video) to feed the EmotionModel (21.1).

Issue: #499 | Planning: #497

---

Architecture Overview

                    ┌─────────────────────────────────────────────┐
                    │              AffectDetector                  │
                    │         (LexiconAffectDetector impl)        │
                    │                                             │
  user_input ──────►│  ┌───────────┐   ┌───────────┐             │
                    │  │  sentence  │   │  lexicon  │             │
                    │  │  splitter  │──►│  lookup   │             │
                    │  └───────────┘   └─────┬─────┘             │
                    │                        │                    │
                    │              ┌─────────▼──────────┐        │
                    │              │  negation handler   │        │
                    │              │  (3-token window)   │        │
                    │              └─────────┬──────────┘        │
                    │                        │                    │
                    │              ┌─────────▼──────────┐        │
                    │              │ intensifier handler │        │
                    │              │ (multiplier table)  │        │
                    │              └─────────┬──────────┘        │
                    │                        │                    │
                    │              ┌─────────▼──────────┐        │
                    │              │   score aggregator  │        │
                    │              │ valence + arousal   │        │
                    │              └─────────┬──────────┘        │
                    │                        │                    │
                    │              ┌─────────▼──────────┐        │
                    │              │ polarity classifier │        │
                    │              │ POS/NEG/NEU/MIXED   │        │
                    │              └─────────┬──────────┘        │
                    │                        │                    │
                    │                  AffectSignal[]             │
                    └────────────────────┬────────────────────────┘
                                        │
                                        ▼
                              ┌──────────────────┐
                              │  EmotionModel     │
                              │  (21.1) .update() │
                              │  PADState delta   │
                              └──────────────────┘

---

Detection Pipeline — Step by Step

1. Sentence Splitting

Input: "I love this product. But the delivery was terrible!"
Output: [("I love this product.", 0, 21), ("But the delivery was terrible!", 22, 52)]

2. Token Scoring with Negation

Without negation:

"This is good" → tokens: [this, is, good]
                  scores: [0,    0,  3]     → valence ≈ +0.6

With negation (3-token window):

"This is not good at all" → tokens: [this, is, not, good, at, all]
                             scores: [0,    0,  0,   3,   0,  0]
                         negated →: [0,    0,  0,  -3,   0,  0]   → valence ≈ -0.6
                                              ↑ window starts ↑

3. Intensifier Amplification

"very good"      → 1.5 × 3 = 4.5  → valence ≈ +0.9
"extremely bad"  → 2.0 × -4 = -8  → valence ≈ -1.0 (clamped)
"slightly upset" → 0.5 × -2 = -1  → valence ≈ -0.2

4. Polarity Classification

Valence Range	Polarity	Condition
v > +0.05	POSITIVE	No strong negative tokens
v < −0.05	NEGATIVE	No strong positive tokens
−0.05 ≤ v ≤ +0.05	NEUTRAL	—
any	MIXED	Both strong pos AND strong neg tokens present

---

Multimodal Fusion

  ┌──────────┐     ┌──────────┐     ┌──────────┐     ┌──────────┐
  │   TEXT    │     │  AUDIO   │     │  IMAGE   │     │  VIDEO   │
  │  w=0.50  │     │  w=0.25  │     │  w=0.15  │     │  w=0.10  │
  │ v=+0.8   │     │ v=+0.6   │     │ v=+0.3   │     │ v=+0.5   │
  └────┬─────┘     └────┬─────┘     └────┬─────┘     └────┬─────┘
       │                │                │                │
       └───────┬────────┴────────┬───────┴────────┬───────┘
               │                 │                │
        ┌──────▼─────────────────▼────────────────▼──────┐
        │           Weighted Average Fusion               │
        │  v_fused = (0.50×0.8 + 0.25×0.6 + 0.15×0.3    │
        │            + 0.10×0.5) / (0.50+0.25+0.15+0.10) │
        │  v_fused = 0.645                                │
        └───────────────────┬────────────────────────────┘
                            │
                     ┌──────▼──────┐
                     │ AffectSignal│
                     │ MULTIMODAL  │
                     │ v = +0.645  │
                     └─────────────┘

Three fusion strategies are supported:

• weighted_average (default): modality weights from DetectorConfig
• max_confidence: picks the signal with highest confidence per window
• attention (future): learnable cross-modal attention mechanism

---

EmotionModel Integration

# AffectSignal → PADState conversion in EmotionOrchestrator (21.5):
for signal in await affect_detector.detect(user_input):
    pad_delta = PADState(
        pleasure=signal.valence,      # direct map
        arousal=signal.arousal,       # direct map
        dominance=0.0,                # supplied by EmpathyEngine (21.4)
    )
    emotion_model.update(pad_delta, weight=signal.confidence)

The confidence field naturally weights stronger signals more — high-confidence detections shift the emotional state more than uncertain ones.

---

Prometheus Metrics

Metric	Type	Purpose
affect_detections_total	Counter	Track detection volume by polarity × modality
affect_detection_latency_seconds	Histogram	Identify slow detection paths
affect_average_valence	Gauge	EWMA dashboard — emotional trend
affect_average_arousal	Gauge	EWMA dashboard — activation trend
affect_low_confidence_ratio	Gauge	Alert when lexicon coverage drops

# Grafana panel — Valence/Arousal time series
- title: "Affect Valence & Arousal (EWMA)"
  type: timeseries
  targets:
    - expr: affect_average_valence
      legendFormat: "Valence"
    - expr: affect_average_arousal
      legendFormat: "Arousal"

---

What's Next

• 21.3 MoodRegulator — smooths affect signals over time, prevents mood whiplash
• 21.4 EmpathyEngine — context-aware dominance extraction, perspective-taking
• 21.5 EmotionOrchestrator — fuses all emotion sub-systems into a single pipeline

---

Links: Issue #499 · Planning #497 · EmotionModel #498