Skip to content

Consistent handling of BINARY data types in materializations #4302

@mdibaiee

Description

@mdibaiee

So to sum up:

  1. We want to start using contentMediaType to know more details about the type of the field. For binaries, we would like to have application/octet-stream. For other types we can also utilise the same
  2. In materializations, for historical reasons, we are handling contentEncoding: base64 inconsistently: some connectors materialize to string, some to binary. We want to add a new feature flag (enabled by default) so that all connectors now materialize to binary. Existing tasks will have that feature flag set to off to retain backward compatibility.

Going forward that means materializations will materialize any field with either:

  1. contentEncoding: base64
  2. contentEncoding: base64 and contentMediaType: application/octet-stream

As BINARY columns in their destination

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions