IRI/LDEO OPeNDAP endpoint for NOAA ESSRT errors in NetCDF >= 4.7.3

[ellenjohnson]

Hi NetCDF developers!
A MATLAB customer reported the following NOAA ESSRT OPeNDAP endpoint fails with a NetCDF error as of MATLAB R2020b. This is the version in which we upgraded from NetCDF version 4.6.1 to 4.7.3. I confirmed that we can read the data fine in MATLAB R2020a (NetCDF 4.6.1), but we get the error in MATLAB versions 20b and 21a (NetCDF 4.7.3). It also errors in our current development branch using NetCDF 4.7.4.

I can reproduce this using ncdump in 4.7.4 on Linux:

Dataset:
http://iridl.ldeo.columbia.edu/SOURCES/.NOAA/.NCDC/.ERSST/.version5/.sst/

OPeNDAP URL:
http://iridl.ldeo.columbia.edu/SOURCES/.NOAA/.NCDC/.ERSST/.version5/.sst/X/%2880W%29%280%29RANGEEDGES/T/%28Jul%201993%29%28Jun%202019%29RANGEEDGES/Y/%2820S%29%2880N%29RANGEEDGES/T/%28Jul-Jun%29seasonalAverage/dods

Using:
ncdump -h "http://iridl.ldeo.columbia.edu/SOURCES/.NOAA/.NCDC/.ERSST/.version5/.sst/X/%2880W%29%280%29RANGEEDGES/T/%28Jul%201993%29%28Jun%202019%29RANGEEDGES/Y/%2820S%29%2880N%29RANGEEDGES/T/%28Jul-Jun%29seasonalAverage/dods"

Results in:
syntax error, unexpected WORD_WORD, expecting '=' context: Dataset { Grid { ARRAY: Float32 T[ Float32 T^[T = 2008]; = 2008]; MAPS: Float32 T[T = 2008]; } T;} T; Illegal dimension declaration context: Dataset { Grid { ARRAY: Float32 T[ Float32 T^[T = 2008]; = 2008]; MAPS: Float32 T[T = 2008]; } T;} T; ./ncdump: http://iridl.ldeo.columbia.edu/SOURCES/.NOAA/.NCDC/.ERSST/.version5/.sst/X/%2880W%29%280%29RANGEEDGES/T/%28Jul%201993%29%28Jun%202019%29RANGEEDGES/Y/%2820S%29%2880N%29RANGEEDGES/T/%28Jul-Jun%29seasonalAverage/dods: http://iridl.ldeo.columbia.edu/SOURCES/.NOAA/.NCDC/.ERSST/.version5/.sst/X/%2880W%29%280%29RANGEEDGES/T/%28Jul%201993%29%28Jun%202019%29RANGEEDGES/Y/%2820S%29%2880N%29RANGEEDGES/T/%28Jul-Jun%29seasonalAverage/dods: NetCDF: Malformed or inaccessible DAP DDS

This looks kinda similar to #569 but looks like that was fixed in 4.6.0.

Hopefully this is an easy fix? Fingers crossed!

[DennisHeimbigner]

This URL worked in the current master, so something we did already apparently fixed the problem. Attached is the output I got with ncdump. If you can, you might verify.
ireo.txt

[ellenjohnson]

Thanks Dennis. Sorry for the delayed response. I'll add your info about it working in master branch to our bug ticket for whomever on our team will be taking up the bug. Thanks!

[DennisHeimbigner]

I can track down the fix if you need it, although it may take a while.

[ellenjohnson]

I would greatly appreciate it, otherwise it will be a lot of detective work for us to sift through the issues/PRs/commits, etc.

[DennisHeimbigner]

I forgot to ask; does it fail with 4.8.0?

[DennisHeimbigner]

It appears to me that it works with netcdf-c version 4.8.0. Any chance you can upgrade
to that version?

[ellenjohnson]

Hi Dennis - Thank for checking! We do have plans to upgrade to 4.8.0 but even so there will be a delay in customers getting it because of our dev cycle. I was hoping maybe the fix would be a small patch we could put on top of our 4.7.4 version, like only affecting several files. But you said it would be non-trivial to down the exact change that fixed this? Can you confirm that it won't be easy to find it? Thanks!

[DennisHeimbigner]

So it turns out that the problem is that in 4.7.4, you need to not use the %xx encodings.
This command, for example, works for me (note the that the url needs to be single or double quoted.
../ncdump/ncdump -h 'http://iridl.ldeo.columbia.edu/SOURCES/.NOAA/.NCDC/.ERSST/.version5/.sst/X/(80W)(0)RANGEEDGES/T/(Jul 1993)(Jun 2019)RANGEEDGES/Y/(20S)(80N)RANGEEDGES/T/(Jul-Jun)seasonalAverage/dods'

[ellenjohnson]

Thanks -- indeed this also works in MATLAB (without the %xx URL encodings)!
Do you know if '%xx' URL encodings are allowed in an OPeNDAP endpoint? I can't recall if I've seen them in DAP URLs before.
I'm trying to figure out if the Columbia University website is listing an incorrect DAP URL (with the URL encodings), or if they are allowed. Not sure of why it works in 4.8.0 unless you guys either added logic to handle an edge case of a DAP endpoint containing URL encodings, or if it was never working in netcdf and you guys fixed a bug?

[DennisHeimbigner]

Sort of. There is a change in 4.8.0 (see this PR #1884) specifically because of the problem that different serves handle escaping of URL differently.

[ellenjohnson]

Thanks for the PR link. And thanks for making my day with your "sort of" comment (made me laugh). We'll provide the workaround to our customer (to use the link with URL encodings replaced with character equivalents) until we release new MATLAB with 4.8.0.