Make CICE processor decomposition configurable via Spack

[Whyborn]

The current SPRs for CICE4 and CICE5 hard-code the processor decomposition and grid for the build. This is immediately problematic for users who which to debug their applications, as debuggers often have limits on the number of processes a task can have (e.g. Linaro Forge on NCI Gadi has a maximum of 256 processors for DDT and Map). It also makes build optimisation a pain, as every build has to modify the SPR. The configuration should be settable via the spack.yaml.

[Whyborn]

This issue will be addressed after the refactoring done in addressing #330

[manodeep]

Noting that this feature is necessary for scaling studies - where the total number of nodes used is varied systematically, and consequently requires a varying number of cores assigned to CICE (depending on the exact layouts obtained for UM and MOM)

[anton-seaice]

Currently, compiling with one task attempts to use the serial code from the cice repositories (which compiles without mpi calls).

@manodeep has found this doesn't work.

My view is we are unlikely to need this feature, so we can remove it (so COMMDIR is always mpi)

https://github.com/ACCESS-NRI/spack-packages/blob/f4d3c99e762493fb4df0f812059386bb8b51a11c/packages/cice5/spack-build.sh#L129

[manodeep]

Currently, compiling with one task attempts to use the serial code from the cice repositories (which compiles without mpi calls).

@manodeep has found this doesn't work.

My view is we are unlikely to need this feature, so we can remove it (so COMMDIR is always mpi)

spack-packages/packages/cice5/spack-build.sh

Line 129 in f4d3c99
setenv COMMDIR mpi

Compilation succeeds with the patch to always setting COMMDIR to mpi (regardless of NTASKS). Previously, I was getting a compilation error for single-core CICE5:

     255    mpifort -c -r8 -i4 -w -fpe0 -ftz -convert big_endian -assume byterecl -check noarg_temp_created -fp-model precise -fp-model source -align all -g3 -O2 -axCORE-AVX2 -debug all -check none -traceback -assume buffered_io  -I. -I/scratch/tm70/ms2335/tmp/spack-stage/spack-stage-cice5-git.access-esm1.6-2025.09.000_access-esm1.6-vqgd4vetgpeyxiuguwk6fb6gjwtiscdn/spack-src/drivers/access -I/scratch/tm70/ms2335/tmp/spack-stage/spack-stage-cice5-git.access-esm1.6-2025.09.000_access-esm1.6-vqgd4vetgpeyxiuguwk6fb6gjwtiscdn/spack-src/source -I/scratch/tm70/ms2335/tmp/spack-stage/spack-stage-cice5-git.access-esm1.6-2025.09.000_access-esm1.6-vqgd4vetgpeyxiuguwk6fb6gjwtiscdn/spack-src/io_netcdf -I/scratch/tm70/ms2335/tmp/spack-stage/spack-stage-cice5-git.access-esm1.6-2025.09.000_access-esm1.6-vqgd4vetgpeyxiuguwk6fb6gjwtiscdn/spack-src/csm_share -I. -I/g/data/tm70/ms2335/spack/0.22/release/linux-rocky8-x86_64_v4/oneapi-2025.2.0/oasis3-mct-stable-buvghbdy2t2kgw2mxowzytf7rqu3hobc/include/psmile.MPI1 -I/g/data/tm70/ms2335/spack/0.22/release/linux-rocky8-x86_64_v4/oneapi-2025.2.0/oasis3-mct-stable-buvghbdy2t2kgw2mxowzytf7rqu3hobc/include -I/g/data/tm70/ms2335/spack/0.22/release/linux-rocky8-x86_64_v4/oneapi-2025.2.0/netcdf-fortran-4.6.1-qzbm7itcq2tr42lv4jyu6nmxupqcpy3s/include cpl_parameters.f90
  >> 256    /scratch/tm70/ms2335/tmp/spack-stage/spack-stage-cice5-git.access-esm1.6-2025.09.000_access-esm1.6-vqgd4vetgpeyxiuguwk6fb6gjwtiscdn/spack-src/drivers/access/cpl_parameters.F90(198): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [ICE_EXIT]
     257    use ice_exit, only: abort_ice
     258    ----^
  >> 259    /scratch/tm70/ms2335/tmp/spack-stage/spack-stage-cice5-git.access-esm1.6-2025.09.000_access-esm1.6-vqgd4vetgpeyxiuguwk6fb6gjwtiscdn/spack-src/drivers/access/cpl_parameters.F90(200): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [ICE_COMMUNICATE]
     260    use ice_communicate, only: my_task, master_task
     261    ----^
  >> 262    /scratch/tm70/ms2335/tmp/spack-stage/spack-stage-cice5-git.access-esm1.6-2025.09.000_access-esm1.6-vqgd4vetgpeyxiuguwk6fb6gjwtiscdn/spack-src/drivers/access/cpl_parameters.F90(212): error #6406: Conflicting attributes or multiple declaration of name.   [MY_TASK]
     263    if (my_task == master_task) then
     264    ----^

Side-note: @Whyborn I am building CICE5 (slenderX1 layout) for ncores in [1, 2, .., 23, 24] with oneAPI@2025.2.0, and OpenMPI@5.0.8 - so you might be able to use those if you wanted.

[Whyborn]

I don't need it for my own work, just for the debugging documentation.

[anton-seaice]

The alternative approach would be to make this runtime configurable in CICE, I think the change is

https://github.com/CICE-Consortium/CICE/pull/194/files

it's not clear to me how hard that would be to apply to cice5 (e.g. it was implemented after CICE split in CICE and Icepack) , nor whether that PR is complete

[manodeep]

That looks like a fair bit of a change to get rid of the compile-time constraint for ncores.

@anton-seaice CICE6 allows ncores to be specified at runtime right ?

[anton-seaice]

Yeah , I think I agree

Correct - the change I linked is included in CICE6

[anton-seaice]

@manodeep has also found that

https://github.com/ACCESS-NRI/spack-packages/blob/079f8b9ffd4bcfc06f98dcd37dd5529ad1847d42/packages/cice5/spack-build.sh#L89-L90

should round up (rather than down)

[harshula]

Hi @anton-seaice ,

IIRC, that's based on:
https://github.com/ACCESS-NRI/cice5/blob/master/bld/config.nci.auscom.360x300
https://github.com/ACCESS-NRI/cice5/blob/master/bld/config.nci.auscom.1440x1080
https://github.com/ACCESS-NRI/cice5/blob/master/bld/config.nci.auscom.3600x2700
where the dividend is a multiple of the divisor. Hence, rounding is not applicable.

It might be best to add some validation of parameters to the add_target() member function:
https://github.com/ACCESS-NRI/spack-packages/blob/079f8b9ffd4bcfc06f98dcd37dd5529ad1847d42/packages/cice5/package.py#L88-L91
i.e. Make sure the grid is a multiple of blocks. We could split the parameters grid into grid_x and grid_y and blocks into blocks_x and blocks_y to do the validation and then construct the string.

[anton-seaice]

where the dividend is a multiple of the divisor. Hence, rounding is not applicable.

It's not a requirement that the dividend is a multiple of the divisor. In this case, having a larger nxblock would be ok

e.g. 13 tasks should be possible, with nxblock of 28. but 13*28 = 364. 364 is > nxglob (360). cice would then internally just mask the extra 4 columns.

i.e. Make sure the grid is a multiple of blocks. We could split the parameters grid into grid_x and grid_y and blocks into blocks_x and blocks_y to do the validation and then construct the string.

In the more general sense, this isn't a requirement.

There needs to be enough blocks to cover the whole grid, depending on how they are distributed (which is configured at runtime). I think trying to validate the choices would be very hard as there are too manual variables (nxblock, nyblock, ntask, nblocks), it depends on runtime configuration and would be duplicating the model code doing the validation.

from the manual:

Although we recommend that the user choose the
local domains so that the global domain is evenly divided, if this is not possible then the furthest east and/or
north blocks will contain nonphysical points (“padding”). These points are excluded from the computation
domain and have little effect on model performance.

This is covered without much clarity in Section 4.2.1 of the manual - see https://github.com/ACCESS-NRI/cice5/blob/master/cicedoc/cicedoc.pdf

[harshula]

Hi @anton-seaice , If it is not a requirement, the rounding should be handled by the CICE5 source package itself.

[anton-seaice]

We decided to add a variable (block_size) in the spack package recipe to calculate the correct block size (with the rounding as needed) , and then pass that as an argument to spack_build.sh