Problem
The catalog uses bitcode as a serialization format.
This was following the WAL's serialization format, where bitcode is used for speed due to the potential size and number of WAL files that may need to be loaded and deserialized on startup.
The catalog should not suffer from this problem (catalog files are fewer, and much smaller) and therefore does not need the speed bitcode.
Furthermore, bitcode is not self-describing, and makes additive changes to the types used by the catalog impossible without a version change to the file.
Proposed solution
- Use a different self-describing format for the catalog that is
serde compatible - since serde works best with JSON, I figure we might as well use that...
- All (yes, all) types used by the catalog need to be versioned, so that when deserializing the older format, i.e.,
bitcode, we deserialize into the older version; then convert to the newer version of the type. This can probably be done by copying the existing type definitions into a v1 module and a latest module, then writing conversions from the v1 types to the latest types; the latter will be what we use throughout the codebase.
- Have tests that check conversion from the
v1 format to the latest format.
Additional context
See Slack Thread.
Problem
The catalog uses
bitcodeas a serialization format.This was following the WAL's serialization format, where
bitcodeis used for speed due to the potential size and number of WAL files that may need to be loaded and deserialized on startup.The catalog should not suffer from this problem (catalog files are fewer, and much smaller) and therefore does not need the speed
bitcode.Furthermore,
bitcodeis not self-describing, and makes additive changes to the types used by the catalog impossible without a version change to the file.Proposed solution
serdecompatible - sinceserdeworks best with JSON, I figure we might as well use that...bitcode, we deserialize into the older version; then convert to the newer version of the type. This can probably be done by copying the existing type definitions into av1module and alatestmodule, then writing conversions from thev1types to thelatesttypes; the latter will be what we use throughout the codebase.v1format to thelatestformat.Additional context
See Slack Thread.