Show & Tell: Phase 18.2 — MemoryConsolidator architecture

[web3guru888]

Phase 18.2 — MemoryConsolidator: offline episodic→semantic consolidation

What MemoryConsolidator Does (and Why)

MemoryConsolidator bridges the gap between raw episodic experience and stable semantic knowledge. Episodic traces in TemporalGraph accumulate rapidly during active cognition. Left unchecked this grows without bound and slows lookups.

The neuroscience analogy: the hippocampus rapidly encodes episodic memories with high fidelity. During rest (or low-load periods), hippocampal replay signals drive neocortical consolidation — repeated patterns are compressed into stable semantic representations. MemoryConsolidator mirrors this: it runs as a background asyncio.Task, sweeps unconsolidated EpisodicTrace objects, extracts centroid feature representations, and upserts SemanticPattern objects into WorldModel.

Benefits:

• Bounded episodic store: old consolidated traces are pruned after retention_days
• Enriched world model: WorldModel accumulates generalised patterns, not raw events
• Non-blocking: runs independently of CognitiveCycle hot path
• Configurable strategy: RECENCY, FREQUENCY, SURPRISE, or HYBRID scoring

---

ConsolidationStrategy Comparison Table

Strategy	Use Case	Score Formula	Gate Condition
RECENCY	Prevent indefinite backlog; oldest traces first	1 / (age_seconds + 1)	none
FREQUENCY	Compress repeated patterns into stable knowledge	trace.frequency / max_freq	frequency >= min_frequency_threshold
SURPRISE	Prioritise novel/unexpected events for fast learning	trace.surprise_score	surprise_score >= surprise_threshold
HYBRID	Balanced consolidation for production systems	0.3*recency + 0.4*frequency + 0.3*surprise	none

---

ASCII Data Flow Diagram

CognitiveCycle (hot path)
  EventSequencer --> TemporalGraph --> SurpriseDetector
                          |                  |
                  [EpisodicTrace written]  [surprise_score]
                          |                  |
              (background asyncio.Task - low priority)
                          v                  v
              MemoryConsolidator.sweep()
                  get_unconsolidated_traces(limit=N)
                          |
                  _score_trace(t, max_freq)
                    RECENCY:   1/(age_s+1)
                    FREQUENCY: freq/max_freq
                    SURPRISE:  surprise_score
                    HYBRID:    0.3R + 0.4F + 0.3S
                  sort DESC -> top-N
                          |
                  _consolidate_trace(trace)
                    get_event_features()
                    _compute_centroid()
                    build SemanticPattern
                    world_model.upsert_pattern()
                    emit metrics
                          |
                          v
              WorldModel (semantic)
                SemanticPattern store
                --> HorizonPlanner input

---

Background Loop Lifecycle

start()
  |
  v
_stop_event.clear()
asyncio.ensure_future(_run_consolidation_loop())
  |
  [loop iteration]
  |
  +-- stop_event set? --> EXIT
  |
  +-- No: call sweep()
          fetch candidates, score+sort
          _consolidate_trace() x N
          emit consolidation_sweep_total++
          |
          wait_for(stop_event, timeout=sweep_interval_s)
          TimeoutError (no stop) --> loop again
          stop_event set --> EXIT

stop()
  _stop_event.set()
  await _loop_task

---

WorldModel Integration

MemoryConsolidator calls world_model.upsert_pattern(pattern: SemanticPattern):

• Insert: if pattern_id is new, insert directly
• Merge: if pattern_id exists, merge abstraction dicts (new values win), take max(existing.confidence, new.confidence)
• Idempotent: same pattern_id + same abstraction — no-op
• Error: raises WorldModelError — caller logs and continues

WorldModel accumulates SemanticPattern objects over time, representing generalised regularities: e.g. "when module X is active and surprise > 0.8, event type Y follows 70% of the time."

---

HorizonPlanner Integration

HorizonPlanner (18.1) classifies goals into SHORT/MEDIUM/LONG horizons. With MemoryConsolidator online:

1. WorldModel.get_patterns(module=X) returns high-confidence SemanticPattern objects
2. HorizonPlanner uses pattern confidence to refine duration estimates
3. High-confidence semantic patterns lead to tighter deadline bounds and more accurate EDF ordering

This closes the hippocampal-neocortical loop: episodic experience → semantic compression → improved prospective planning.

---

Prometheus Metrics + Grafana

Metric	Type	PromQL Example
consolidation_sweep_total	Counter	rate(consolidation_sweep_total[5m]) — sweeps/sec
traces_consolidated_total	Counter	rate(traces_consolidated_total[5m]) — throughput
patterns_created_total	Counter	increase(patterns_created_total[1h]) — hourly growth
consolidation_sweep_duration_seconds	Histogram	histogram_quantile(0.95, consolidation_sweep_duration_seconds_bucket)
active_patterns_count	Gauge	active_patterns_count — current semantic store size

Grafana panel YAML:

panels:
  - title: Consolidation Rate (traces/sec)
    type: timeseries
    targets:
      - expr: rate(traces_consolidated_total[5m])
        legendFormat: traces consolidated/sec
      - expr: rate(consolidation_sweep_total[5m])
        legendFormat: sweeps/sec
    fieldConfig:
      defaults:
        unit: ops
  - title: Active Semantic Patterns
    type: stat
    targets:
      - expr: active_patterns_count
    fieldConfig:
      defaults:
        unit: short
        thresholds:
          steps:
            - color: green
              value: 0
            - color: yellow
              value: 500
            - color: red
              value: 2000

---

Open Questions for Contributors

1. Distributed consolidation: When multiple ASI-Build nodes run MemoryConsolidator independently, how do we prevent duplicate SemanticPattern creation for the same underlying traces? Should consolidation be elected to a single leader node, or should WorldModel handle merge conflicts?

2. Pattern overlap / conflict resolution: When two SemanticPattern objects have overlapping source_trace_ids, should they be merged, kept separate, or does one subsume the other? What conflict resolution semantics should world_model.upsert_pattern() implement?

3. Incremental vs batch consolidation: The current design consolidates max_traces_per_sweep traces per sweep as a batch. Would an incremental approach (one trace at a time, yield between each) be better for latency-sensitive deployments?