'ncdap_tst_remote3' test failure in Linux

[abhibaruah]

NetCDF version: 4.8.1
HDF5 version: 1.10.8
OS: Debian 10

Recently, we changed the upstream dependency of netcdf from hdf5 1.8.12 to 1.10.8.
After this, on building netcdf, the test 'ncdap_tst_remote3' fails with a 'Permission Denied' error message.

The test log \netcdf\Testing\Temporary\LastTest.log records the following:

179/199 Testing: ncdap_tst_remote3
179/199 Test: ncdap_tst_remote3
Command: "/usr/local/bin/bash" "-c" "export srcdir=/mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test;export TOPSRCDIR=/mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf;/mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh "
Directory: /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test
"ncdap_tst_remote3" start time: Mar 09 12:51 EST
Output:
----------------------------------------------------------
/usr/local/bin/bash: /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh: Permission denied
<end of output>
Test time =   0.02 sec
----------------------------------------------------------
Test Failed.
"ncdap_tst_remote3" end time: Mar 09 12:51 EST
"ncdap_tst_remote3" time elapsed: 00:00:00

Is this a known issue or a server issue? Or can anything be done on the software side to fix this test failure?

[WardF]

If you run ls -alh /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh, what do you see? This seems like a pretty straightforward linux filesystem permissions issue, and it doesn't appear to be reflecting anything specific to the release.

[abhibaruah]

Hello Ward,
Thank you for looking into this. I agree that the error message is not much helpful.
Here is the output that I get on executing the ls -alh command

-rw-r--r--+ 1 abaruah users 928 Mar 3 16:04 /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh

Thanks

[WardF]

Hi @abhibaruah it looks like the 'execute' bit is not set on this file. I'm not sure how that would have happened, although it might be something to do with the underlying filesystem, or if this is a network share. Try setting the executable bit with the following command:

$ chmod 755 /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh

After running this command, run ls -alh /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh again, and we should see that the executable bit is set.

[abhibaruah]

Hello Ward,
After I executed the chmod command, I did see the permissions change.

-rwxr-xr-x+ 1 abaruah users 928 Mar 3 16:04 /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh*

But, when I build netcdf again after that, the permissions on tst_remote3.sh revert back to what I showed initially, and I see the same 'Permission denied' error.

The ls -alh return this now:

-rw-r--r--+ 1 abaruah users 928 Mar 3 16:04 /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh

Meanwhile, I am also trying to see if anything in our makefile or build harness is messing up the permissions.

Thanks,
Abhi

[abhibaruah]

Hello @WardF,
I was able to resolve the 'Permission Denied' error. However now, the same test fails with the following log:

179/199 Testing: ncdap_tst_remote3
179/199 Test: ncdap_tst_remote3
Command: "/usr/local/bin/bash" "-c" "export srcdir=/mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test;export TOPSRCDIR=/mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf;/mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test/tst_remote3.sh "
Directory: /mathworks/devel/sandbox/abaruah/3p-tmw/3p/derived/glnxa64/netcdf/ncdap_test
"ncdap_tst_remote3" start time: Mar 11 13:02 EST
Output:
----------------------------------------------------------
XXXXXXXXXXXXXXXXXXXX
YYYYYYYYYYYYYYYYYYYY
*** Testing DAP to netCDF-3 translation using remote server 
        Base URL: https://remotetest.unidata.ucar.edu/dts
        Client Parameters: [log][netcdf3]
        Caching: on
    Note: The remote tests may be slow or even fail if the server is overloaded
*** Testing: test.01 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.01
*** Testing: test.02 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.02
*** Testing: test.04 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.04
*** Testing: test.05 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.05
*** Testing: test.07a ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07a
*** Testing: test.07 ; url=https://remotetest.unidata.ucar.edu/dts/test.07
*** Testing: test.21 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.21
*** Testing: test.50 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.50
*** Testing: test.53 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.53
*** Testing: test.55 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.55
*** Testing: test.56 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.56
*** Testing: test.57 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.57
*** Testing: test.66 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.66
*** Testing: test.67 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.67
41c41
<     0.589788025031098, 0.581683089463883, 0.573519986072457, 
---
>     0.589788025031098, 0.581683089463884, 0.573519986072457, 
*** FAIL:  test.67
*** Testing: test.68 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.68
*** Testing: test.69 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.69
*** Testing: test.01.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.01?f64
*** Testing: test.02.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.02?b[1:2:10]
*** Testing: test.03.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.03?i32[0:1][1:2][0:2]
*** Testing: test.04.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.04?types.i32
*** Testing: test.05.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.05?types.floats.f32
*** Testing: test.07.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?person.age
*** Testing: test.07.3 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?person
*** Testing: test.07.4 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?types.f32
*** FAILED: 23/24 ; 0 expected failures ; 1 unexpected failures
<end of output>
Test time =  11.60 sec
----------------------------------------------------------
Test Failed.
"ncdap_tst_remote3" end time: Mar 11 13:02 EST
"ncdap_tst_remote3" time elapsed: 00:00:11
----------------------------------------------------------

[DennisHeimbigner]

<     0.589788025031098, 0.581683089463883, 0.573519986072457, 
---
>     0.589788025031098, 0.581683089463884, 0.573519986072457, 

The problem is the middle value differs in the last digit: 3 vs 4.
This occasionally occurs depending on the platform.

[abhibaruah]

Thank you for taking a look Dennis.
Is there a way to fix this, or can this be treated as an outlier and filtered for now?

[DennisHeimbigner]

If the data is being accesses with ncdump, then you might be able to use the -p flag.

[DennisHeimbigner]

Aince it is clear that there is no essential problem, I recommend that
you build netcdf-c with the --disable-dap-remote-tests option.

[abhibaruah]

Thank you Dennis.
I have filtered out the corresponding test by commenting out the line
add_sh_test(ncdap tst_remote3)
in ncdap_test/CMakeLists.txt

[ArchangeGabriel]

Just wanted to say that I’m seeing the same failure on Arch Linux (while it wasn’t showing before):

199/236 Testing: ncdap_tst_remote3
199/236 Test: ncdap_tst_remote3
Command: "/usr/bin/bash" "-c" "export srcdir=/build/netcdf-openmpi/src/netcdf-c-4.8.1/ncdap_test;export TOPSRCDIR=/build/netcdf-openmpi/src/netcdf-c-4.8.1;/build/netcdf-openmpi/src/build/ncdap_test/tst_remote3.sh "
Directory: /build/netcdf-openmpi/src/build/ncdap_test
"ncdap_tst_remote3" start time: Mar 20 22:45 UTC
Output:
----------------------------------------------------------
*** Testing DAP to netCDF-3 translation using remote server 
        Base URL: https://remotetest.unidata.ucar.edu/dts
        Client Parameters: [log][netcdf3]
        Caching: on
    Note: The remote tests may be slow or even fail if the server is overloaded
*** Testing: test.01 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.01
*** Testing: test.02 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.02
*** Testing: test.04 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.04
*** Testing: test.05 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.05
*** Testing: test.07a ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07a
*** Testing: test.07 ; url=https://remotetest.unidata.ucar.edu/dts/test.07
*** Testing: test.21 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.21
*** Testing: test.50 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.50
*** Testing: test.53 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.53
*** Testing: test.55 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.55
*** Testing: test.56 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.56
*** Testing: test.57 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.57
*** Testing: test.66 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.66
*** Testing: test.67 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.67
41c41
<     0.589788025031098, 0.581683089463883, 0.573519986072457, 
---
>     0.589788025031098, 0.581683089463884, 0.573519986072457, 
*** FAIL:  test.67
*** Testing: test.68 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.68
*** Testing: test.69 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.69
*** Testing: test.01.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.01?f64
*** Testing: test.02.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.02?b[1:2:10]
*** Testing: test.03.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.03?i32[0:1][1:2][0:2]
*** Testing: test.04.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.04?types.i32
*** Testing: test.05.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.05?types.floats.f32
*** Testing: test.07.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?person.age
*** Testing: test.07.3 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?person
*** Testing: test.07.4 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?types.f32
*** FAILED: 23/24 ; 0 expected failures ; 1 unexpected failures
<end of output>
Test time =  26.18 sec
----------------------------------------------------------
Test Failed.
"ncdap_tst_remote3" end time: Mar 20 22:45 UTC
"ncdap_tst_remote3" time elapsed: 00:00:26
----------------------------------------------------------

[Dave-Allured]

The test.67 error is a duplicate of #2188.

[serbinsh]

I am not sure if this is the same or related but I am also running into a testing error on 4.8.1, but only the DAP test

I am compiling with HDF5 1.12.1 with gcc 8.5.0-4 (system gcc on RHEL 8.5)

All other tests passed.

FAIL: tst_remote3.sh
============================================================================
Testsuite summary for netCDF 4.8.1
============================================================================
# TOTAL: 16
# PASS:  15
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See ncdap_test/test-suite.log
Please report to support-netcdf@unidata.ucar.edu
============================================================================
make[5]: *** [Makefile:1090: test-suite.log] Error 1
make[5]: Leaving directory '/data/software/src/netcdf/netcdf-c-4.8.1/ncdap_test'
make[4]: *** [Makefile:1198: check-TESTS] Error 2
make[4]: Leaving directory '/data/software/src/netcdf/netcdf-c-4.8.1/ncdap_test'
make[3]: *** [Makefile:1346: check-am] Error 2
make[3]: Leaving directory '/data/software/src/netcdf/netcdf-c-4.8.1/ncdap_test'
make[2]: *** [Makefile:977: check-recursive] Error 1
make[2]: Leaving directory '/data/software/src/netcdf/netcdf-c-4.8.1/ncdap_test'
make[1]: *** [Makefile:1348: check] Error 2
make[1]: Leaving directory '/data/software/src/netcdf/netcdf-c-4.8.1/ncdap_test'
make: *** [Makefile:729: check-recursive] Error 1

=============================================
   netCDF 4.8.1: ncdap_test/test-suite.log
=============================================

# TOTAL: 16
# PASS:  15
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: tst_remote3
=================

*** Testing DAP to netCDF-3 translation using remote server
        Base URL: https://remotetest.unidata.ucar.edu/dts
        Client Parameters: [log][netcdf3]
        Caching: on
    Note: The remote tests may be slow or even fail if the server is overloaded
*** Testing: test.01 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.01
*** Testing: test.02 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.02
*** Testing: test.04 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.04
*** Testing: test.05 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.05
*** Testing: test.07a ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07a
*** Testing: test.07 ; url=https://remotetest.unidata.ucar.edu/dts/test.07
*** Testing: test.21 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.21
*** Testing: test.50 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.50
*** Testing: test.53 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.53
*** Testing: test.55 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.55
*** Testing: test.56 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.56
*** Testing: test.57 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.57
*** Testing: test.66 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.66
*** Testing: test.67 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.67
41c41
<     0.589788025031098, 0.581683089463883, 0.573519986072457,
---
>     0.589788025031098, 0.581683089463884, 0.573519986072457,
*** FAIL:  test.67
*** Testing: test.68 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.68
*** Testing: test.69 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.69
*** Testing: test.01.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.01?f64
*** Testing: test.02.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.02?b[1:2:10]
*** Testing: test.03.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.03?i32[0:1][1:2][0:2]
*** Testing: test.04.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.04?types.i32
*** Testing: test.05.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.05?types.floats.f32
*** Testing: test.07.1 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?person.age
*** Testing: test.07.3 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?person
*** Testing: test.07.4 ; url=[cache][prefetch]https://remotetest.unidata.ucar.edu/dts/test.07?types.f32
*** FAILED: 23/24 ; 0 expected failures ; 1 unexpected failures
FAIL tst_remote3.sh (exit status: 1)