use the user provide process group for autotuning#1823
Merged
shunting314 merged 1 commit intomainfrom Apr 2, 2026
Merged
Conversation
shunting314
added a commit
that referenced
this pull request
Mar 26, 2026
stack-info: PR: #1823, branch: shunting314/stack/21
shunting314
force-pushed
the
shunting314/stack/21
branch
from
0fd4cbe to
8242314
Compare
This was referenced Mar 26, 2026
Merged
meta-cla
bot
added
the
CLA Signed
shunting314
added a commit
that referenced
this pull request
Mar 26, 2026
stack-info: PR: #1823, branch: shunting314/stack/21
shunting314
force-pushed
the
shunting314/stack/21
branch
from
8242314 to
cd1a8dd
Compare
shunting314
force-pushed
the
shunting314/stack/21
branch
from
cd1a8dd to
5a1af0f
Compare
shunting314
force-pushed
the
shunting314/stack/21
branch
from
5a1af0f to
41697a4
Compare
shunting314
force-pushed
the
shunting314/stack/21
branch
from
41697a4 to
4075b49
Compare
shunting314
added a commit
that referenced
this pull request
Mar 27, 2026
stack-info: PR: #1823, branch: shunting314/stack/21
shunting314
force-pushed
the
shunting314/stack/21
branch
from
39e0be8 to
161d0a0
Compare
shunting314
force-pushed
the
shunting314/stack/21
branch
from
161d0a0 to
7db7a82
Compare
shunting314
force-pushed
the
shunting314/stack/21
branch
from
7db7a82 to
4ec8d7c
Compare
This was referenced Apr 1, 2026
Open
shunting314
force-pushed
the
shunting314/stack/21
branch
from
4ec8d7c to
33392f1
Compare
stack-info: PR: #1823, branch: shunting314/stack/21
shunting314
force-pushed
the
shunting314/stack/21
branch
from
33392f1 to
53488e5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked PRs:
use the user provide process group for autotuning
Previously when we autotune a dist kernel, we assume the process group is dist.group.WORLD. It's often true but can be wrong for complex cases like multi-dimension parallelism. I added an example kernel for SP+TP.
With 4 GPUs, there will be 2 TP groups handling different part of the sequence. In this case, when we autotune, we should use the user specified process group for synchronization across ranks.