Fix multi-repo RPC deadlock via stderr redirect and dedicated GetObject pool#7036
Merged
jkschneider merged 1 commit intomainfrom Mar 18, 2026
Merged
Conversation
…ct pool Two independent deadlocks prevented `mod run` from completing across multiple C# repositories: 1. **Stderr pipe buffer deadlock**: On macOS the kernel pipe buffer for stderr is 64KB. When the dotnet subprocess writes enough to stderr to fill this buffer, it blocks on the next write(), which deadlocks any in-progress RPC response on stdout. Fix: redirect stderr at the OS level via ProcessBuilder.redirectError() — to the log file when configured, or to /dev/null otherwise. This eliminates the pipe entirely so the subprocess never blocks on stderr. 2. **GetObject thread pool starvation**: GetObject.Handler submitted background tree traversal tasks to ForkJoinPool.commonPool(), which is the same pool used by the CLI for repo-level fork-join work. When repo tasks saturated the pool (blocked on semaphores or waiting for RPC responses), GetObject producers couldn't start, deadlocking the batch protocol. Fix: use a dedicated cached thread pool for tree traversal so GetObject producers are never starved by unrelated work.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ProcessBuilder.redirectError()to prevent pipe-buffer deadlock on macOS (64KB kernel buffer). When a log path is configured, stderr goes to that file; otherwise it goes to/dev/null.GetObjecttree traversal instead ofForkJoinPool.commonPool(), which gets saturated by repo-level fork-join tasks and starves GetObject producers.stderrRedirectthrough C#, JavaScript, and Python RPC builders.These two deadlocks were independent — the stderr issue blocked the subprocess from writing RPC responses, while the pool starvation prevented Java from servicing
GetObjectcallbacks duringBatchVisit.Test plan
mod runwithUpgradeToDotNet10on 11 C# repos in parallel — completed in 6m 25s with no deadlock