Active-Active Replication Feature

[lcwangchao]

Design Doc: #64294

Development Tasks

Feature Branches

Please merge the PRs to the feature branches:

• PD: https://github.com/tikv/pd/tree/feature/active-active
• TiDB: https://github.com/pingcap/tidb/tree/feature/active-active
• TiKV: https://github.com/tikv/tikv/tree/feature/active-active
• Client-go: https://github.com/tikv/client-go/tree/feature/active-active
• kvproto: https://github.com/pingcap/kvproto/tree/feature/active-active

1. M1 : Milestone 1 (12/15): E2E MVP
2. M2 : Milestone 2 (1/07): All Features

SoftDelete

TiDB

Parser & Optimizer

M1

• planner, executor: planner support the _tidb_commit_ts new hidden column | tidb-test=pr/2635 #64610
• planner, executor: support INSERT statements for softdelete table semantic in planner | tidb-test=feature/active-active tikv=feature/active-active #64892
• planner, expression: support hidden columns for SELECT statement and softdelete filters | tidb-test=feature/active-active tikv=feature/active-active #64918
• planner, executor, table: add tidb_translate_softdelete_sql support and fix the point get fast path in planner | tidb-test=feature/active-active tikv=feature/active-active  #64993
• planner: refactor buildUpdate and buildDelete to reduce code repetition and prepare for later changes | tidb-test=feature/active-active tikv=feature/active-active #65033
• planner: support DELETE statement rewrite for softdelete tables | tidb-test=feature/active-active tikv=feature/active-active #65013

M2

• *: add TIDB_SOFTDELETE_TABLE_STATS system table for softdelete tables | tidb-test=feature/active-active tikv=feature/active-active #65409
• planner: add the new tidb_translate_softdelete_sql variable into the plan cache key | tidb-test=feature/active-active tikv=feature/active-active #65053
• column: set correct flen and decimal for softdelete column | tidb-test=feature/active-active tikv=feature/active-active #65057
• planner: ban usage of _tidb_softdelete_time when tidb_translate_softdelete_sql is off | tidb-test=feature/active-active tikv=feature/active-active #65199
• *: add RECOVER VALUES statements for softdelete tables | tidb-test=feature/active-active tikv=feature/active-active #65389

DDL

M1

• model: add minimal info for active-active table #64512
• ddl, parser: add active acitve and softdelete options | tidb-test=feature/active-active tikv=feature/active-active  #64856
• ddl: refine syntax details | tidb-test=feature/active-active tikv=feature/active-active #64962

M2

• Create system table at bootstrap
• Add information_schema.tidb_softdelete_table_stats system table. *: add TIDB_SOFTDELETE_TABLE_STATS system table for softdelete tables | tidb-test=feature/active-active tikv=feature/active-active #65409
• Forbid hidden column in CREATE TABLE/ADD COLUMN
• new DDL ActionType must be determined
• disable unique index on softdelete table #64919 (comment)
• forbid creating indexes on _tidb_commit_ts
  • ddl: hide softdelete indexes in show create, forbidden constraints | tidb-test=feature/active-active tikv=feature/active-active #65354
• forbid creating unique indexes and constraits on _tidb_origin_ts and _tidb_softdelete_time
  • ddl: hide softdelete indexes in show create, forbidden constraints | tidb-test=feature/active-active tikv=feature/active-active #65354

M3:

• *: support softdelete cleanup | tidb-test=feature/active-active tikv=feature/active-active #65400 (comment)
• ttl: update grafana and logging, fix unstable test | tidb-test=feature/active-active tikv=feature/active-active #65537
• ttl: check ticdc progress table for consistency | tidb-test=feature/active-active #65638
• merge master branch for bootstrap refactor, cp master https://github.com/pingcap/tidb/pull/65400/changes#r2668084107 https://github.com/pingcap/tidb/pull/65409/changes#diff-01b1e4d5a7741315754aaa061f875bde2969c3f05c4670cc9623a74b3550fc2bR359
• Parsing bug

TiDB root@localhost:test> CREATE TABLE `t4_gi` (
                       ->   `id` int NOT NULL,
                       ->   PRIMARY KEY (`id`) /*T![clustered_index] CLUSTERED */,
                       ->   /*T![softdelete] KEY `i1` (`_tidb_softdelete_time`) GLOBAL */,
                       ->   KEY `i2` (`id`),
                       ->   KEY `i3` (`id`)
                       -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin
                       -> PARTITION BY HASH (`id`) PARTITIONS 4
                       -> SOFTDELETE_JOB_INTERVAL='24h';
(1064, 'You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use line 9 column 24 near "SOFTDELETE_JOB_INTERVAL=\'24h\'" ')
TiDB root@localhost:test> CREATE TABLE `t4_gi` (
                       ->   `id` int NOT NULL,
                       ->   PRIMARY KEY (`id`) /*T![clustered_index] CLUSTERED */
                       ->   /*T![softdelete] , KEY `i1` (`_tidb_softdelete_time`) GLOBAL */,
                       ->   KEY `i2` (`id`),
                       ->   KEY `i3` (`id`)
                       -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin
                       -> SOFTDELETE=RETENTION 7 DAY SOFTDELETE_JOB_INTERVAL='24h';
(1072, 'column does not exist: _tidb_softdelete_time')

Executors

M1

• InsertExec supports replacing softdelete rows instead of returning an error when a conflict occurs. #64701
  • executor: support soft delete table's insert executor | tidb-test=feature/active-active tikv=feature/active-active #64702
• executor: tiny fix for softdelete handling multiple unique keys | tidb-test=feature/active-active tikv=feature/active-active #64961

M2

• Load data support softdelete #65346
• pipelined dml support softdelete #65469
• The point-get with FOR UPDATE should lock the row even if it does not exist or is soft-deleted. #65171
• non-transaction dml support softdelet #65427
• softdelete for unique index is not correct #64960

M3

• executor/metrics: add softdelete implicit delete rows metric and expose stats in INSERT runtime | tidb-test=feature/active-active #65905
• metrics: Add new charts in Grafana for SoftDelete Implicit Delete Rows | tidb-test= fc7090b4ff2db9ac30fd22036c45f426053b8d28 #65923

Active-Active Support

PD

M1

• feat(active-to-active): impl unique tso index  tikv/pd#9982

proto

M1

• Support to get commit ts for Get / BatchGet kvproto#1373
  • add fields to get commit timestamp in Get or BatchGet kvproto#1371
  • Merge active-active kvproto to master

TiKV

M1

• *: support to return commit timestamp for Get / BatchGet requests tikv/tikv#19103
• coprocessor: support return _tidb_commit_ts column for TableScan coprocessor request tikv/tikv#19166

client-go

M1

• Support to return the commit timestamp for Get and BatchGet in the Transaction interface. tikv/client-go#1795
  • *: return commit timestamp for Get / BatchGet if needed tikv/client-go#1796
  • txnkv: Get always return nil value when not found tikv/client-go#1798
• Support to set the MinCommitTS manually for a transaction. #64797
• Replace TiDB deps as the feature branch
• Replace kvproto with master version

TiDB

Parser & Optimizer

M1

• Optimizer supports filtering new hidden columns.
• Optimizer supports outputting the _tidb_commit_ts hidden column and passing corresponding information to the executor.

M2

• planner: don't generate tiflash path when accessing _tidb_commit_ts column | tidb-test=feature/active-active tikv=feature/active-active #65507

M3

• executor: add metrics and logs for active-active execution | tidb-test=feature/active-active #65635

DDL

M1

• Add active_active options and AST in parser, including comment format in restore
  • ddl, parser: add active acitve and softdelete options | tidb-test=feature/active-active tikv=feature/active-active  #64856
  • ddl: refine syntax details | tidb-test=feature/active-active tikv=feature/active-active #64962

Executors & Transaction

M1

• tidb txn supports to set WithRequireCommitTS #64537
  • *: tidb txn supports to setting WithReturnCommitTS | tidb-test=957ed0b #64538
• DML should set the min-commit-ts for the current transaction when updating/deleting an old row that has _tidb_origin_ts not null. #64893
• PointGet / BatchPointGet support to return _tidb_commit_ts #64721
  • *: support _tidb_commit_ts for PointGet/BatchPointGet executor | tidb-test=feature/active-active tikv=feature/active-active #64720
• *: set kv.CommitWaitUntilTSO for current transaction when updating an old row with _tidb_origin_ts not null | tidb-test=feature/active-active tikv=feature/active-active #64901
• executor: return _tidb_commit_ts in 'select for update' | tidb-test=feature/active-active tikv=feature/active-active #65174

M2

• unistore support return _tidb_commit_ts column for TableScan coprocessor request #65106
• The pessimistic lock cache should contain _tidb_commit_ts #65268
• ddl: disallow creating softdelete table on cached table | tidb-test=feature/active-active #65443
• test: move a test case and cover _tidb_origin_ts for replace | tidb-test=feature/active-active tikv=feature/active-active #65238

M3

• *: fix _tidb_commit_ts bugs for cached table, temporary table, pessimistic lock and lock table | tidb-test=feature/active-active #65830

Observability

M2

• Metrics: number of rows that were discarded by TiCDC's conflict resolution #65269

TTL

• ttl: check ticdc progress table for consistency | tidb-test=feature/active-active #65638

CDC

M1

• Support sync in active-active mode to TiDB and sync active-active tables in non-active-active mode to all downstream ticdc#3299

M2

• Support LWW Mode SQL as LWW
• Create ticdcProgressTable to ensure data consistency
• Support active table syncing in non-active-active mode
• Support to collect metrics about how many rows skipped per changefeed / table

Others

Tool compatibility

M3

• BR
• Import into
• lightning

Master Cherry Pick

TiKV

• *: support returning commit ts for Get / BatchGet / Coprocessor tikv/tikv#19230
• coprocessor: fix panic when _tidb_commit_ts column is placed before PK handle columns tikv/tikv#19300
• *: support need_commit_ts for TiKV command batch_get_command tikv/tikv#19301
• cpor: check commit ts is present in RangesScanner when it is required tikv/tikv#19333

Client-go

• *: support WithReturnCommitTS and SetCommitWaitUntilTSO tikv/client-go#1831
• txnkv: Add a named error ErrCommitTSLag when waiting for commitWaitUntilTSO timeout tikv/client-go#1842
• txnkv: fix SetCommitWaitUntilTSOTimeout does not work in causal consistency tikv/client-go#1847
• txnkv: enhance commit ts check when WithReturnCommitTS is used (#1861) tikv/client-go#1864

TiDB

• model: add minimal info for active-active table #64512
• model: reserve active-active feature definitions #65295
• unistore: unistore supports to return commit timestamp #65282
• *: tidb txn supports to setting WithReturnCommitTS #65306
• planner, meta: code preparation for the _tidb_commit_ts new hidden column | tidb-test=pr/2663 #65620

Release-8.5

kvproto

• add fields to get commit timestamp in Get or BatchGet kvproto#1398

client-go

• support active-active tikv/client-go#1854
• *: cherry commit ts support to release-8.5 tikv/client-go#1850
• txnkv: enhance commit ts check when WithReturnCommitTS is used tikv/client-go#1861
• txnkv: support to SetCommitWaitUntilTSOTimeout and do not wait TSO tikv/client-go#1875

TiKV

• active active support tikv/tikv#19312
• *: support returning commit ts for Get / BatchGet / Coprocessor (#19230) tikv/tikv#19295

TiDB

• *: merge some basic enhancements to 8.5 to support get commit ts #65871
• ttl: disable async commit for softdelete / ttl jobs | tidb-test=release-8.5-20260121-v8.5.5 pd=release-8.5-20260121-v8.5.5 #67544