planner: support left outer join into anti semi join | tidb-test=pr/2649 by hawkingrei · Pull Request #64959 · pingcap/tidb

hawkingrei · 2025-12-10T11:55:42Z

What problem does this PR solve?

Issue Number: close #64329

Problem Summary: planner: support left outer join into anti semi join

What changed and how does it work?

This PR introduces two optimization patterns for transforming LEFT JOIN queries into NOT EXISTS subqueries when checking for missing matches.

Scenario 1: IS NULL on the Join Condition Column

In this scenario, the IS NULL filter is applied directly to the join key from the inner table.

Table Schema

CREATE TABLE Table_A (
    id INT,
    name VARCHAR(50)
);

CREATE TABLE Table_B (
    id INT, -- Join Key
    info VARCHAR(50)
);

Plan

-- The optimizer rewrites the LEFT JOIN ... WHERE ... IS NULL pattern into an efficient
-- Anti Semi Join to retrieve rows from Table A that have no match in Table B.
SELECT 
    Table_A.id, 
    Table_A.name
FROM 
    Table_A
LEFT JOIN 
    Table_B ON Table_A.id = Table_B.id
WHERE 
    Table_B.id IS NULL;

+--------------------------+-----------+---------------+--------------------------------------------------------------------------------------+
| id                       | task      | access object | operator info                                                                        |
+--------------------------+-----------+---------------+--------------------------------------------------------------------------------------+
| Projection               | root      |               | test.table_a.id, test.table_a.name                                                   |
| └─Selection              | root      |               | isnull(test.table_b.id)                                                              |
|   └─HashJoin             | root      |               | left outer join, left side:TableReader, equal:[eq(test.table_a.id, test.table_b.id)] |
|     ├─TableReader(Build) | root      |               | data:Selection                                                                       |
|     │ └─Selection        | cop[tikv] |               | not(isnull(test.table_b.id))                                                         |
|     │   └─TableFullScan  | cop[tikv] | table:B       | keep order:false, stats:pseudo                                                       |
|     └─TableReader(Probe) | root      |               | data:TableFullScan                                                                   |
|       └─TableFullScan    | cop[tikv] | table:A       | keep order:false, stats:pseudo                                                       |
+--------------------------+-----------+---------------+--------------------------------------------------------------------------------------+

=>

+----------------------+-----------+---------------+-------------------------------------------------------------------------------------+
| id                   | task      | access object | operator info                                                                       |
+----------------------+-----------+---------------+-------------------------------------------------------------------------------------+
| HashJoin             | root      |               | anti semi join, left side:TableReader, equal:[eq(test.table_a.id, test.table_b.id)] |
| ├─TableReader(Build) | root      |               | data:Selection                                                                      |
| │ └─Selection        | cop[tikv] |               | not(isnull(test.table_b.id))                                                        |
| │   └─TableFullScan  | cop[tikv] | table:B       | keep order:false, stats:pseudo                                                      |
| └─TableReader(Probe) | root      |               | data:TableFullScan                                                                  |
|   └─TableFullScan    | cop[tikv] | table:A       | keep order:false, stats:pseudo                                                      |
+----------------------+-----------+---------------+-------------------------------------------------------------------------------------+

Scenario 2: IS NULL on a Non-Join NOT NULL Column

If a column in the inner table is defined as NOT NULL in the schema, but is filtered as IS NULL after the join, it implies that the join failed to find a match. This allows us to convert the join into ANTI SEMI JOIN even if the column is not part of the join keys. （just guarantee that the condition only filter those null rows only generated from null supplied side）

Table Schema:

CREATE TABLE Table_A (
    id INT,
    name VARCHAR(50)
);

CREATE TABLE Table_B (
    id INT,
    status VARCHAR(20) NOT NULL -- Non-join column with NOT NULL constraint
);

Plan

-- Even though 'status' is not a join key, its NOT NULL constraint 
-- ensures that 'B.status IS NULL' only occurs when no match is found.
SELECT A.* FROM Table_A A 
LEFT JOIN Table_B B ON A.id = B.id 
WHERE B.status IS NULL;

+--------------------------+-----------+---------------+--------------------------------------------------------------------------------------+
| id                       | task      | access object | operator info                                                                        |
+--------------------------+-----------+---------------+--------------------------------------------------------------------------------------+
| Projection               | root      |               | test.table_a.id, test.table_a.name                                                   |
| └─Selection              | root      |               | isnull(test.table_b.status)                                                          |
|   └─HashJoin             | root      |               | left outer join, left side:TableReader, equal:[eq(test.table_a.id, test.table_b.id)] |
|     ├─TableReader(Build) | root      |               | data:Selection                                                                       |
|     │ └─Selection        | cop[tikv] |               | not(isnull(test.table_b.id))                                                         |
|     │   └─TableFullScan  | cop[tikv] | table:B       | keep order:false, stats:pseudo                                                       |
|     └─TableReader(Probe) | root      |               | data:TableFullScan                                                                   |
|       └─TableFullScan    | cop[tikv] | table:A       | keep order:false, stats:pseudo                                                       |
+--------------------------+-----------+---------------+--------------------------------------------------------------------------------------+

=>

+----------------------+-----------+---------------+-------------------------------------------------------------------------------------+
| id                   | task      | access object | operator info                                                                       |
+----------------------+-----------+---------------+-------------------------------------------------------------------------------------+
| HashJoin             | root      |               | anti semi join, left side:TableReader, equal:[eq(test.table_a.id, test.table_b.id)] |
| ├─TableReader(Build) | root      |               | data:Selection                                                                      |
| │ └─Selection        | cop[tikv] |               | not(isnull(test.table_b.id))                                                        |
| │   └─TableFullScan  | cop[tikv] | table:B       | keep order:false, stats:pseudo                                                      |
| └─TableReader(Probe) | root      |               | data:TableFullScan                                                                  |
|   └─TableFullScan    | cop[tikv] | table:A       | keep order:false, stats:pseudo                                                      |
+----------------------+-----------+---------------+-------------------------------------------------------------------------------------+

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

https://github.com/PingCAP-QE/planreplayertest/pull/31

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

hawkingrei · 2025-12-10T12:47:22Z

/retest

codecov · 2025-12-10T15:54:07Z

Codecov Report

❌ Patch coverage is 84.50704% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.9126%. Comparing base (4f1b985) to head (1be2f85).
⚠️ Report is 7 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #64959        +/-   ##
================================================
+ Coverage   77.8439%   78.9126%   +1.0686%     
================================================
  Files          1983       1912        -71     
  Lines        542787     527358     -15429     
================================================
- Hits         422527     416152      -6375     
+ Misses       118601     110316      -8285     
+ Partials       1659        890       -769

Flag	Coverage Δ
integration	`44.2898% <67.2985%> (-3.8982%)`	⬇️
tiprow_ft	`?`
unit	`76.8161% <79.3427%> (+0.3365%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`56.7974% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`48.7744% <ø> (-12.2722%)`	⬇️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

hawkingrei · 2025-12-11T01:26:30Z

/retest

hawkingrei · 2025-12-12T08:54:07Z

/retest

hawkingrei · 2025-12-12T12:14:01Z

/retest

hawkingrei · 2025-12-12T12:17:07Z

/retest

hawkingrei · 2025-12-12T12:20:31Z

/retest

hawkingrei · 2025-12-15T08:23:18Z

/retest

hawkingrei · 2025-12-15T13:29:14Z

/retest

hawkingrei · 2025-12-15T16:32:02Z

/retest

YangKeao · 2026-01-19T15:55:11Z

/retest

YangKeao · 2026-01-19T16:00:12Z

/retest

YangKeao · 2026-01-19T16:05:12Z

/retest

YangKeao · 2026-01-19T16:10:11Z

/retest

YangKeao · 2026-01-19T16:15:11Z

/retest

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

pkg/planner/core/rule/rule_outer_join_to_semi_join.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 2 comments.

pkg/planner/core/rule/rule_outer_join_to_semi_join.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

pkg/planner/core/rule/rule_outer_join_to_semi_join.go

hawkingrei · 2026-01-20T03:09:18Z

/retest

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.

pkg/planner/core/rule/rule_outer_join_to_semi_join.go

ti-chi-bot · 2026-01-20T07:06:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AilinKid, guo-shaoge, winoros

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [AilinKid,guo-shaoge,winoros]
~~pkg/planner/OWNERS~~ [AilinKid,winoros]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hawkingrei · 2026-01-20T07:38:20Z

/retest

hawkingrei · 2026-01-20T07:41:04Z

/unhold

hawkingrei · 2026-01-20T07:42:31Z

/hold

hawkingrei · 2026-01-20T07:46:38Z

/unhold

hawkingrei · 2026-01-20T08:44:43Z

/retest

hawkingrei · 2026-01-20T09:23:06Z

/retest

hawkingrei · 2026-01-20T09:45:45Z

/retest

YangKeao · 2026-01-20T10:10:41Z

/retest

YangKeao · 2026-01-20T10:55:41Z

/retest

YangKeao · 2026-01-20T13:10:41Z

/retest

ti-chi-bot bot added do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed do-not-merge/needs-tests-checked labels Dec 10, 2025

hawkingrei changed the title ~~planner: support left outer join into anti semi join~~ [WIP] planner: support left outer join into anti semi join Dec 10, 2025

ti-chi-bot bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 10, 2025

hawkingrei force-pushed the 64329 branch from b8e3014 to dc32fed Compare December 10, 2025 12:05

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 10, 2025

ti-chi-bot bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 10, 2025

hawkingrei changed the title ~~[WIP] planner: support left outer join into anti semi join~~ [WIP] planner: support left outer join into anti semi join | tidb-test=pr/2649 Dec 11, 2025

hawkingrei force-pushed the 64329 branch from afbfa65 to ad7cf38 Compare December 11, 2025 07:10

ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 11, 2025

hawkingrei force-pushed the 64329 branch from c8ea9e5 to c6f8595 Compare December 12, 2025 08:32

hawkingrei force-pushed the 64329 branch 2 times, most recently from ae46b9c to fc4d779 Compare December 12, 2025 11:34

hawkingrei force-pushed the 64329 branch from fc4d779 to 70280ea Compare December 15, 2025 07:42

hawkingrei added 3 commits January 20, 2026 00:39

fix some problems

9a866a5

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

refactor code

03b99cf

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

refactor code

5384b89

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

Copilot AI reviewed Jan 19, 2026

View reviewed changes

pkg/planner/core/rule/rule_outer_join_to_semi_join.go Outdated Show resolved Hide resolved

Update pkg/planner/core/rule/rule_outer_join_to_semi_join.go

1837b50

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI reviewed Jan 19, 2026

View reviewed changes

pkg/planner/core/rule/rule_outer_join_to_semi_join.go Outdated Show resolved Hide resolved

pkg/planner/core/rule/rule_outer_join_to_semi_join.go Show resolved Hide resolved

Update pkg/planner/core/rule/rule_outer_join_to_semi_join.go

d418d14

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI reviewed Jan 19, 2026

View reviewed changes

pkg/planner/core/rule/rule_outer_join_to_semi_join.go Show resolved Hide resolved

hawkingrei added 2 commits January 20, 2026 13:54

fix some problems

93a40ac

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

remove empty file

1be2f85

Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com>

Copilot AI reviewed Jan 20, 2026

View reviewed changes

AilinKid approved these changes Jan 20, 2026

View reviewed changes

guo-shaoge mentioned this pull request Mar 10, 2026

planner: enable cd-c algorithm by default for join reorder | tidb-test=pr/2676 #66349

Merged

13 tasks

Conversation

hawkingrei commented Dec 10, 2025 • edited by qw4990 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Scenario 1: IS NULL on the Join Condition Column

Table Schema

Plan

Scenario 2: IS NULL on a Non-Join NOT NULL Column

Table Schema:

Plan

Check List

Release note

Uh oh!

hawkingrei commented Dec 10, 2025

Uh oh!

codecov bot commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hawkingrei commented Dec 11, 2025

Uh oh!

hawkingrei commented Dec 12, 2025

Uh oh!

hawkingrei commented Dec 12, 2025

Uh oh!

hawkingrei commented Dec 12, 2025

Uh oh!

hawkingrei commented Dec 12, 2025

Uh oh!

hawkingrei commented Dec 15, 2025

Uh oh!

hawkingrei commented Dec 15, 2025

Uh oh!

hawkingrei commented Dec 15, 2025

Uh oh!

YangKeao commented Jan 19, 2026

Uh oh!

YangKeao commented Jan 19, 2026

Uh oh!

YangKeao commented Jan 19, 2026

Uh oh!

YangKeao commented Jan 19, 2026

Uh oh!

YangKeao commented Jan 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

hawkingrei commented Jan 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ti-chi-bot bot commented Jan 20, 2026

Uh oh!

hawkingrei commented Jan 20, 2026

Uh oh!

hawkingrei commented Jan 20, 2026

Uh oh!

hawkingrei commented Jan 20, 2026

Uh oh!

hawkingrei commented Dec 10, 2025 •

edited by qw4990

Loading

codecov bot commented Dec 10, 2025 •

edited

Loading