[epic] Use custom CUDA stream for the entire codebase.

[trivialfis]

This will be a long refactoring task. The objective is to enable the use of a custom CUDA stream to improve control over asynchronous memory allocation and to enable stream-specific device.

We have support for device ordinal cuda:1. This has been a pain point for XGBoost, yet it's a widely used feature. In CUDA 13, streams are implicitly attached to the device during creation. As a result, if we can use a custom stream, we can remove the C API guard and avoid initializing the CUDA context.

Plan:

• Provide optional context parameter in all storage class, including:
  • DeviceUVector
  • HostDeviceVector
  • Tensor
  • TemporaryArray
• Use stream-oriented memory allocation in the booster class.
• Use stream-oriented memory allocation in the DMatrix classes.
• Provide synchronization between the DMatrix and the booster.
• Remove the set device when a device ordinal is not provided.
• Remove C API guard. Verify that XGBoost doesn't initialize the CUDA context when CUDA is not used.

PRs:

• Optional context for the host device vector. #12121

Related:

• GPU memory usage when only using CPU. #12116
• xgboost4j-spark-gpu fails to run on multi-GPU server with GPUs in exclusive process mode and spark-rapids plugin #11884

[RAMitchell]

I noticed that host device vector is quite fragile. In some places if it's not configured correctly we get crashes. Would be very nice If we can improve this.

[trivialfis]

@RAMitchell Could you please elaborate on the crash? Thus far, the biggest issue I have with the HDV is its mutable state, which can lead to race condition.

[RAMitchell]

From memory I think it was something like setting the device preparing for the first time to be used on the gpu.

[trivialfis]

That's probably the race condition issue. I don't have a good solution at the moment.