For analytics use case we need to incrementally copy and export lineage_events to DWH. For this use case there is no good way to identify incrementally created events in database. The current event_time in lineage_events table is client generated and can be back dated.
Proposing to alter lineage_events table to add a new created_at timestamp column. Also index this new column.
More discussion to be had regarding backfill and default values of this column.
For analytics use case we need to incrementally copy and export lineage_events to DWH. For this use case there is no good way to identify incrementally created events in database. The current event_time in lineage_events table is client generated and can be back dated.
Proposing to alter lineage_events table to add a new created_at timestamp column. Also index this new column.
More discussion to be had regarding backfill and default values of this column.