Skip to content

store/tikv: pipe the split and scatter task#23357

Closed
nolouch wants to merge 3 commits intopingcap:masterfrom
nolouch:refine-split
Closed

store/tikv: pipe the split and scatter task#23357
nolouch wants to merge 3 commits intopingcap:masterfrom
nolouch:refine-split

Conversation

@nolouch
Copy link
Copy Markdown
Member

@nolouch nolouch commented Mar 16, 2021

Signed-off-by: nolouch nolouch@gmail.com

What problem does this PR solve?

close #22969

Problem Summary:
Splitting and Scattering at the same time will cause multiple conflicts on the same region.
image

What is changed and how it works?

pipeline all splits task and scatter tasks

Release note

  • Make the time consuming of split table more stable
I have run multiple times  to split 4000 regions, and the time is stale about `1m30s`
2021/03/17 17:25:57 split single sql result: [map[SCATTER_FINISH_RATIO:1 TOTAL_SPLIT_REGION:2001]]
2021/03/17 17:25:57 split all sql run total cost: 1m28.908304306s
2021/03/17 17:25:57 split all sql run success!!!

Signed-off-by: nolouch <nolouch@gmail.com>
@ti-chi-bot
Copy link
Copy Markdown
Member

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

@ti-chi-bot ti-chi-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Mar 16, 2021
@github-actions github-actions bot added the sig/sql-infra SIG: SQL Infra label Mar 16, 2021
@nolouch nolouch requested a review from crazycs520 March 16, 2021 09:57
Comment on lines +209 to +213
for _, r := range spResp.Regions {
tid := int64(0)
if tableID != nil {
tid = *tableID
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for _, r := range spResp.Regions {
tid := int64(0)
if tableID != nil {
tid = *tableID
}
tid := int64(0)
if tableID != nil {
tid = *tableID
}
for _, r := range spResp.Regions {

Comment on lines +238 to +241
tid := int64(0)
if tableID != nil {
tid = *tableID
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
tid := int64(0)
if tableID != nil {
tid = *tableID
}

nolouch added 2 commits March 17, 2021 17:37
Signed-off-by: nolouch <nolouch@gmail.com>
Signed-off-by: nolouch <nolouch@gmail.com>
}
spResp := batchResp.resp.Resp.(*kvrpcpb.SplitRegionResponse)
for _, r := range spResp.Regions {
if err = s.scatterRegion(bo, r.Id, tableID); err == nil {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it's still using

_, err := s.pdClient.ScatterRegions(bo.ctx, []uint64{regionID}, opts...)
the batch API to scatter regions one by one. Which may make the scatter not work?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean #22788 .
OK, I'm off topic.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current region scatter may not work in a specific case which described in tikv/pd#3422 , I think it's not related to the api.

@ti-chi-bot
Copy link
Copy Markdown
Member

@nolouch: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 4, 2021
@nolouch nolouch closed this Jul 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-cherry-pick-release-5.0 needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. sig/sql-infra SIG: SQL Infra size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make the time consuming of split table more stable

5 participants