Skip to content

parallel test race condition between ref_ctest/ref_ctest64 and ncdump_tst_output #3272

@edhartnett

Description

@edhartnett

When running tests in parallel (ctest -j N or make -j check), ncdump_tst_output intermittently fails, which then causes ncdump_tst_nccopy3 to fail as well.

The root cause is a file race condition. Both the standalone tests ref_ctest / ref_ctest64 and the shell script tst_output.sh write to ctest0.nc (and ctest0_64.nc) in the same build directory. With parallel execution, these can run concurrently, corrupting each other's output.

tst_output.sh calls ref_ctest internally and then runs ncgen -lc -o ctest0.nc, followed by a diff against a reference file. If the standalone ref_ctest test is simultaneously writing ctest0.nc, the diff or a subsequent step fails.

ncdump_tst_nccopy3 already declares a dependency on ncdump_tst_output, so it correctly waits — but when tst_output fails partway through, the files tst_nccopy3 needs (e.g. tst_output_c0tmp.nc) are never created, producing a "No such file or directory" error.

Observed CI output

 13/224 Test  #20: ncdump_tst_output .........................***Failed    0.86 sec
...
220/224 Test  #21: ncdump_tst_nccopy3 ........................***Failed    0.07 sec

*** Testing nccopy tst_output_c0tmp.nc nccopy3_copy_of_tst_output_c0tmp.nc ...
No such file or directory
Location: file ncdump/nccopy.c; fcn copy line 2011

Fix

Declare that tst_output depends on ref_ctest and ref_ctest64 in both build systems so they cannot run concurrently.

CMake (ncdump/CMakeLists.txt):

set_tests_properties(ncdump_tst_output PROPERTIES DEPENDS "ref_ctest;ref_ctest64")

Autotools (ncdump/Makefile.am):

# tst_output.sh runs ref_ctest and ref_ctest64 internally and
# writes to the same files (ctest0.nc, ctest0_64.nc).
tst_output.log: ref_ctest.log ref_ctest64.log

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions