Idea: migrate from pickle to safetensors / msgpack for checkpoint & key serialization

[egoran2]

Context

Across the codebase there are a number of pickle.load / pickle.loads / torch.load call sites that handle either checkpoint state or model weights:

• compute/fault_tolerance/checkpoint_manager.py — job state
• homomorphic/core/keys.py — key material (multiple call sites)
• homomorphic/ml/neural_networks.py — model data
• knowledge_management/learning/contextual_learner.py — learner state
• distributed_training/{blockchain,core}/... — federated model diffs

Pickle is fine when both producer and consumer are trusted, but the federated training surface in particular ingests serialized state from peers, which is a textbook arbitrary-code-execution vector.

Proposal

Standardize on:

• Tensor-only state → safetensors (HF's format — checked, no pickle, fast mmap)
• Mixed Python state (config dicts, RNG, stats) → msgpack or JSON with a small custom encoder
• Cryptographic key material → custom binary format with explicit version + HMAC (don't rely on pickle to roundtrip secrets)

Rolling this out incrementally — write side first, with read side keeping pickle compatibility — gives a clean migration path.

Questions

• Is anyone already prototyping this for the homomorphic module? The keys file feels like the highest-priority site (key serialization should not depend on Python's pickle semantics).
• Any reason safetensors wouldn't fit the federated diff path?

Happy to take a stab at a writeup / proof-of-concept if there's interest.