Skip to content

Use a different serialization format in catalog #26180

@hiltontj

Description

@hiltontj

Problem

The catalog uses bitcode as a serialization format.

This was following the WAL's serialization format, where bitcode is used for speed due to the potential size and number of WAL files that may need to be loaded and deserialized on startup.

The catalog should not suffer from this problem (catalog files are fewer, and much smaller) and therefore does not need the speed bitcode.

Furthermore, bitcode is not self-describing, and makes additive changes to the types used by the catalog impossible without a version change to the file.

Proposed solution

  • Use a different self-describing format for the catalog that is serde compatible - since serde works best with JSON, I figure we might as well use that...
  • All (yes, all) types used by the catalog need to be versioned, so that when deserializing the older format, i.e., bitcode, we deserialize into the older version; then convert to the newer version of the type. This can probably be done by copying the existing type definitions into a v1 module and a latest module, then writing conversions from the v1 types to the latest types; the latter will be what we use throughout the codebase.
  • Have tests that check conversion from the v1 format to the latest format.

Additional context

See Slack Thread.

Metadata

Metadata

Assignees

Labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions