Tracking Issue for Parallel Rustc Front-end

[SparrowLii]

This is a tracking issue for Parallel Rustc Front-end

This feature will improve compilation speed by parallelizing the process of rustc front end (before the codegen which has been parallelized).

The feature was previously called parallel queries (and its tracking issue can be found here), developed by the former parallel rustc working group. Thanks to their work, developers can currently use parallel rustc by setting parallel_compiler=true in config.toml when building the compiler.

The parallel rustc working group has rebooted (here's the associated MCP), which will make parallel front-end the default option in rustc. Here's the (draft) landing strategy for this feature.

Below is the task list.

Solve the problem of single-thread performance reduction

• Introduce DynSend and DynSync auto trait for parallel compiler #107586
• Use only one shard with a single thread #111755
• Make Sharded an enum and specialize it for the single thread case #114860
• Use conditional synchronization for Lock #111713
• Remove conditional use of Sharded from query state #114894
• Add optimized lock methods for Sharded and refactor Lock #115388
• make query caches use Sharded only for multiple threads #115162

Optimize multi-threading performance

• typeck in parallel #113519
• Increase parallelism in various locations #115003

Enable parallel compilation by default in nightly rustc

• test for parallel compiler by default #115235
• enable parallel rustc front end in nightly builds #117435

Bugs or test failures of parallel rustc

• get rid of tcx in deadlock handler when parallel compilation #98570
• Check that a query has not completed and is not executing before starting it #108845
• Deadlock when printing query stack -- fixed by Avoid calling queries during query stack printing #112708
• Exponential (?) time complexity in evaluate_trait_predicate_recursively in rustdoc when proving Send/Sync #106930
• ICE: parallel compiler: attempted to read from stolen value: rustc_middle::thir::Thir : -Zthir-unsafeck=yes #111520
• Fix waiting on a query that panicked #115198
• Add a CurrentGcx type to let the deadlock handler access TyCtxt #115220
• Add ParallelGuard type to handle unwinding in parallel sections #115144
• ICE: parallel compiler: deadlock detected #111521
• ICE: parallel compiler: no ImplicitCtxt stored in tls #111522
• ICE: parallel compiler: value must be in cache after waiting #111528
• UI test failures with parallel-compiler=true #75760
• parallel compiler: unconditional panic Resource temporarily unavailable when limiting max memory  #115021
• parallel compiler: assertion failed: found_cycle #115223
• other issues here

Testing

• Add parallel front-end test set to compiler-test #118698
• Enhance rustc-perf options to support multi-threading

Documentation

• Adjust documentation (doc-guide)

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

[the8472]

The core crate might be a good benchmark for this. You can immediately build it after rustc and it spends about 10% of its walltime in tokenization/parsing/macro-expansion and ~80% in the query parts. The fraction of codegen is small in comparison.

[tigerfsh]

It's been a long time since there was any update. Has all work been paused？

[SparrowLii]

Them main work now is to fix deadlock and ice bugs in the issue list

I recently submitted a PR for an enhanced compiler test to reproduce the issue and successfully located three deadlock bugs. Unfortunately, it takes timeto figure out the cause of these bugs, which is what is making the updates slow.

I expect it will take another 1 to 2 months to fix the remaining bugs, after which the stabilization will begin.