Phase 54.1 Spec & Technical Discussion: EnvironmentEncoder

[web3guru888]

EnvironmentEncoder — Spec & Technical Discussion

Technical discussion thread for Phase 54.1: EnvironmentEncoder (#1034).

Core Design Questions

1. Encoder architecture: Should we default to VAE, VQ-VAE, or JEPA-style contrastive encoding? DreamerV3 uses discrete categorical latents (32 categories × 32 classes) — should we adopt this as our baseline?

2. Multi-modal fusion strategy: Early fusion (concatenate before encoding), late fusion (encode separately then merge), or cross-attention fusion?

3. Latent dimensionality: What is the right trade-off between compression and predictive utility? Ha & Schmidhuber used 32-dim, DreamerV3 uses 32×32 discrete — how do we make this configurable?

4. Temporal context: GRU-based recurrent state (Dreamer) vs. attention-based context window (Transformer)? Trade-offs in memory, compute, and representational capacity.

5. Information bottleneck: How aggressively should we compress? Too much compression loses detail needed for planning; too little makes dynamics prediction harder.

Proposed API Surface

class EnvironmentEncoder:
    def encode(self, observation: Observation) -> LatentState
    def decode(self, latent: LatentState) -> Observation  # for visualization
    def get_latent_dim(self) -> int
    def reset_temporal_state(self) -> None

Share your thoughts, alternative designs, and implementation suggestions below.

Related: #1034 | #1033