Support CUDA 13.1

[cslominski]

Description

Hi,

I am receiving errors when compiling HOOMD-blue 6.1.1 with CUDA 13.1 on Ubuntu 24.04. I would greatly appreciate it if you could help resolve this for me.

Thanks,
Charlie

Motivation and context

FAILED: hoomd/CMakeFiles/_hoomd.dir/HOOMDVersion.cc.o
In file included from hoomd-blue/hoomd/extern/HIP/include/hip/hip_runtime_api.h:344,
from hoomd-blue/hoomd/extern/HIP/include/hip/nvcc_detail/hip_runtime.h:28,
from hoomd-blue/hoomd/extern/HIP/include/hip/hip_runtime.h:58,
from hoomd-blue/hoomd/HOOMDVersion.cc:10:
hoomd-blue/hoomd/extern/HIP/include/hip/nvcc_detail/hip_runtime_api.h: In function ‘hipError_t hipChooseDevice(int*, const hipDeviceProp_t*)’:
hoomd-blue/hoomd/extern/HIP/include/hip/nvcc_detail/hip_runtime_api.h:556:12: error: ‘struct cudaDeviceProp’ has no member named ‘clockRate’
556 | cdprop.clockRate = prop->clockRate;
| ^~~~~~~~~
hoomd-blue/hoomd/extern/HIP/include/hip/nvcc_detail/hip_runtime_api.h:561:12: error: ‘struct cudaDeviceProp’ has no member named ‘computeMode’
561 | cdprop.computeMode = prop->computeMode;
| ^~~~~~~~~~~
hoomd-blue/hoomd/extern/HIP/include/hip/nvcc_detail/hip_runtime_api.h:563:12: error: ‘struct cudaDeviceProp’ has no member named ‘memoryClockRate’
563 | cdprop.memoryClockRate = prop->memoryClockRate;
| ^~~~~~~~~~~~~~~

[joaander]

#2178 will add CUDA 13 support, but it is blocked because hipCUB does not yet support CUDA 13. Please install CUDA 12 to build HOOMD-blue from source.

[cslominski]

Hi Joshua,

Thanks for the reply.

I was flushed into the CUDA 13 upgrade because my CUDA 12 installation broke Ubuntu updates. I am not doing anything too fancy, so others may encounter this issue. I believe the break is due to Nvidia changing their apt packaging structure in recent updates.

I will have to revert and cross my fingers that I can skirt around this issue for some time, and potentially stop rolling updates altogether. It would be a much better state if HOOMD-blue supported CUDA 13, which has been out for over six months now.

Best,
Charlie

[joaander]

Yes, I do not understand why AMD has not yet added support for CUDA 13. There is nothing I can do about it except wait for upstream support.

[cslominski]

The hipCUB used in HOOMD-blue appears to be forked from a deprecated repository. Have you tried the newer hipCUB from rocm-libraries?

[joaander]

The hipCUB used in HOOMD-blue appears to be forked from a deprecated repository. Have you tried the newer hipCUB from rocm-libraries?

Yes, please look at #2178 which I already mentioned.

[cslominski]

OK I see the discussion now -- what a mess! You might have to rescue the project from this corporate feud.

The issue I encountered was a conflict between the current Ubuntu kernel (6.17.0-14-generic) and the drivers in the CUDA 12.9.1 local .deb installation. If anyone runs into this, using the apt packages for the drivers works. In particular, installing nvidia-headless-590-open and nvidia-utils-590 after installing CUDA resolved the problem for me.

[joaander]

CUDA drivers are backward compatible. You can install the latest drivers and an older version of the CUDA toolkit. Don't install the meta package that includes both the drivers and toolkit. If I recall correctly, I have all the toolkit versions (12.2+) and the latest drivers for testing #2178 locally.

I tried the latest rocm development commits in #2178. They fail with thousands of compiler errors in rocm headers. We may never get another working version of HIP for NVIDIA HPC. At the same time, we don't have the resources to write our own translation layer to insulate us from HIP. We started discussing that back in 2021: #1063.

If anyone can find a way around the compile errors in #2178, I would appreciate the help. I honestly have no idea if support for CUDA 13 has been added or not. CUDA is not mentioned in the release notes or documentation, and rarely mentioned in issues/prs. There are obviously no unit tests to ensure that HIP interoperates with CUDA.

[cslominski]

Thanks for all this! I can't take it on right now but I might be able to look in the somewhat near future. Sounds difficult though.

[joaander]

This is what I did:

sudo apt install cuda-drivers-580
sudo apt install cuda-toolkit-12-9

[cslominski]

I meant root causing the compile errors sounds difficult. Thanks though for the above too.

[joaander]

I meant root causing the compile errors sounds difficult. Thanks though for the above too.
Sorry, I misunderstood.

It is indeed very difficult to debug the HIP code. For starters, I don't understand the HIP release process. The most recent release appears to be 7.2.0: https://rocm.docs.amd.com/projects/HIP/en/latest/index.html Yet the repository has tags for therock-7.11 and many other versions between 7.2 and 7.11. I can't find a change log anywhere so there I don't know what to expect when trying different versions.

I've tested the 7.2.0 release on #2178 and it worked --- but only for CUDA 12.9. I attempted to patch it to make it work for CUDA 12.4 -- 12.9. It failed for CUDA 13 in hipCUB. The 7.11 tag and current HEAD versions all fail with thousands of error: 'hipCUDAErrorTohipError' was not declared in this scope; The function appears to be in the same header that is causing all the errors.

At this point, we wait until "TheRock" is in production (if ever).

[mphoward]

I did some digging through the AMD documentation, and they claim we should be able to smell what The Rock is cooking by mid-2026:

https://rocm.docs.amd.com/en/7.9.0-preview/about/release-notes.html#preview-stream-note

It seems that AMD intentionally introduced a discontinuity in their versioning. ROCm 7.2 is the current production release, while ROCm 7.9 and up seem to be for this new development. I have no idea why they chose to do this rather than make pre-release candidates of ROCm 8 or if the mid-2026 timeline is realistic because their progress and goals are opaque to me.

Clearly HIP is becoming a major issue for HOOMD. Following our discussion in #1063 and #2178, I will prioritize getting hipper to work with our minimal CUDA and ROCm dependencies. I think I can start to poke at it next week.

@joaander do you have access to systems with AMD GPUs for testing? I don't have any here, but I have allocations on Delta. If you've had luck using Delta's AMD nodes, I'd appreciate an example of a minimum build environment.

[joaander]

I have access to NCSA Delta, but have not tried their single AMD GPU node. I also have access to Frontier and am happy to build and run tests there as needed. I will port the CI changes I made in #2178 to test compilation only with hipper targeting HIP. I don't have CI infrastructure with AMD GPUs for automatic testing.

[mphoward]

@joaander could you create a branch with the minimal CI to build HOOMD with CUDA or HIP? I have had some success setting up hipper to build with different version of ROCm via the AMD repositories:

mphowardlab/hipper#19

I'd like to do a minimal bit of modernizations to hipper, then start to work on porting chunks of HOOMD to figure out which APIs (if any) we need are missing from hipper. I can work on making the hipper API complete at a later time.