Feature description
I'd like to have the staging-optimized replace strategy for the Clickhouse destination.
Are you a dlt user?
Yes, I run dlt in production.
Use case
We've been using DLT in production to move data from Google Drive to Clickhouse. Sometimes we get empty tables for a bit of time. I think it happens due to how insert-from-staging truncates the destination table, then inserts from the staging table. This leaves a period of time without data, because Clickhouse doesn't have transactions.
Proposed solution
I think staging-optimized should eliminate this downtime. The Postgres destination does this with a transaction. Clickhouse doesn't have transactions, but has the EXCHANGE TABLES statement for atomic swaps.
Related issues
No response
Feature description
I'd like to have the
staging-optimizedreplace strategy for the Clickhouse destination.Are you a dlt user?
Yes, I run dlt in production.
Use case
We've been using DLT in production to move data from Google Drive to Clickhouse. Sometimes we get empty tables for a bit of time. I think it happens due to how
insert-from-stagingtruncates the destination table, then inserts from the staging table. This leaves a period of time without data, because Clickhouse doesn't have transactions.Proposed solution
I think
staging-optimizedshould eliminate this downtime. The Postgres destination does this with a transaction. Clickhouse doesn't have transactions, but has theEXCHANGE TABLESstatement for atomic swaps.Related issues
No response