'ncdump -t' does not work with a units attribute of type variable length string.

[srherbener]

The upcoming release of netcdf-c v4.9.1 will introduce the capability to read fixed length strings for attribute and variable values. This is a great addition and thanks for putting it in!

It would be equally great to add the capability to write fixed length strings for both attributes and variables. I work on the JEDI project at JCSDA, and we are using netcdf extensively for storing model and observational data.

An example use case is that we store datetime information as a 64-bit integer offset from a specified "epoch" datetime. The epoch is recorded in a string attribute on the datetime variable in the ISO-8601 format. This was done to take advantage of the ncdump -t feature that writes out datetime information in human readable format. The issue is that ncdump -t wants a fixed length string for the epoch attribute but we cannot write it out as such using the netcdf API (only as a variable length string). So we have to put in a workaround to change that attribute to a fixed length string after writing the file.

I'm happy to provide more information as needed for this issue.

Thanks for your help with this!

[Dave-Allured]

... ncdump -t wants a fixed length string for the epoch attribute ...

My understanding is that all netcdf4 releases that supported ncdump -t, also accepted both character array and variable length string for the time "units" attribute. I assume that is what you mean by "the epoch attribute". If not, please explain.

Can you show an example of ncdump -t that only accepts a fixed length string for units, and no other type of string? Which version of netcdf is this?

---

EDIT Dec. 23: My assumption was wrong. Variable length string is not currently supported with ncdump -t in all netcdf-C library versions through current release 4.9.0. See conversation below.

[srherbener]

@Dave-Allured thanks for your response.

We are using netcdf-c v4.8.1 and I am indeed talking about the time "units" attribute. I didn't test using a character array for the units attribute and I suspect that that will work correctly. I admit that I might be on the wrong track in my interpretation of what is happening so there might be something the we are not doing on our end that is causing the result I'm seeing.

Here is a tar file with two examples:
units.attr.examples.tgz

• vlen.nc4 has a time variable (MetaData/dateTime) with an attribute named "units" which is of a type variable length string.
• flen.nc4 is the same except the "units" attribute is a fixed length string.

Here are two h5dump snippets showing the nature of the units attributes, and why I am saying what I listed above.

# "h5dump vlen.nc4" shows this:

...
         ATTRIBUTE "units" {
            DATATYPE  H5T_STRING {
               STRSIZE H5T_VARIABLE;
               STRPAD H5T_STR_NULLTERM;
               CSET H5T_CSET_UTF8;
               CTYPE H5T_C_S1;
            }
            DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
            DATA {
            (0): "seconds since 2018-04-14T21:00:00Z"
            }
         }
...

# and "h5dump flen.nc4" shows this:

...
         ATTRIBUTE "units" {
            DATATYPE  H5T_STRING {
               STRSIZE 34;
               STRPAD H5T_STR_NULLTERM;
               CSET H5T_CSET_ASCII;
               CTYPE H5T_C_S1;
            }
            DATASPACE  SCALAR
            DATA {
            (0): "seconds since 2018-04-14T21:00:00Z"
            }
         }
...

Note the STRSIZE H5T_VARIABLE; for the vlen.nc4 case and the STRSIZE 34; for the flen.nc4 case.

When I run ncdump -t on these two files, I see this:

# ncdump -t vlen.nc4
...
group: MetaData {
  variables:
  	int64 dateTime(nlocs) ;
  		dateTime:_FillValue = -9223372036854775801LL ;
  		string dateTime:units = "seconds since 2018-04-14T21:00:00Z" ;
  data:

   dateTime = 1800, 5400, 9000, 12600 ;
  } // group MetaData
...

# ncdump -t flen.nc4
...
group: MetaData {
  variables:
  	int64 dateTime(nlocs) ;
  		dateTime:_FillValue = -9223372036854775801LL ; // "S�"
  		dateTime:units = "seconds since 2018-04-14T21:00:00Z" ;
  data:

   dateTime = "2018-04-14 21:30", "2018-04-14 22:30", "2018-04-14 23:30",
      "2018-04-15 00:30" ;
  } // group MetaData
...

Note you only get the human readable times with my flen.nc4 file. Again, I could be constructing my vlen.nc4 file incorrectly, but what I'm after is seeing the human readable times for both the vlen.nc4 and flen.nc4 files. (I think the // "S�" annotation, plus a warning message I didn't attach, is due to the extremely large negative fill value that was chosen, where the code wasn't successful at converter the fill value to human readable form.)

Any help you can provide would be greatly appreciated. Thanks!

[Dave-Allured]

Interesting. I can reproduce both of your ncdump -t outputs with netcdf 4.9.0 on a Mac laptop. Let me look into this a little deeper.

The warning message and garbled annotation //S? are a known issue which is off topic for the current issue. But I will explain anyway. That is "CDMS error: Error on time conversion", right? You are correct, that is a side effect of that atrocious fill value.

It would take a small amount of code work to clean up that garbled annotation. But everybody has been too busy to deal with something this minor. In addition, there are at least two obvious user workarounds to stop the warning and normalize the annotation. Therefore, for that much, I strongly recommend user-based fixes. _FillValue attributes in cases like this are unnecessary and should be deleted before creating any time variable that is fully populated. If you must have a _FillValue, please pick one that is within a few millenia of the current epoch, not halfway to the edge of the universe. ;-) I rather like whatever number translates to 9999-1-1.

I thought this CDMS issue had been discussed some time recently, but I can't find it right now.

[Dave-Allured]

Got it. For the time units string, type character is supported for ncdump -t, type string is NOT currently supported and is marked TODO.

Version 4.9.0, this file currently the same as the devel branch:

netcdf-c/ncdump/nctime0.c

Line 207 in 0fc2c81

if(nc_status == NC_NOERR && uatt.type == NC_CHAR) { /* TODO: NC_STRING? */

So the support for variable length strings was never added. There is no good reason to leave it this way. It would probably be simple for someone who knows their way around the innards better than I do.

For a quick and permanent user fix, change time:units from string to character, and do something about that fill value. You are most of the way there in your second case flen.nc4, which works fine for me except for the CDMS warning.

Alternatively, fix up the argument passing in that function, and submit a PR for the next release.

[srherbener]

@Dave-Allured you are correct about the CDMS error. Here is the message I left out:
CDMS error: Error on time conversion: invalid hour = -2561996247984714.500000.

Thank you for the details about what is going on. I have a lot of familiarity with netcdf and hdf5 API's and features but no familiarity with the underlying code bases (ie, implementations). How much effort do you think it would be for me to come up to speed on the netcdf code to make a contribution like you are suggesting? Is there documentation for a contributing developer for example?

Thanks!

[Dave-Allured]

@srherbener, hmmm, I actually mentioned two different things to fix. The missing support for traditional variable-length strings should be simple, if you know C and Github.

This section of source file nctime0.c is written just like a normal user application written in C. The protocol to read a 1-D string of Netcdf type character is already present. You would just need to learn the protocol to read a scalar variable-length string of Netcdf type string, and add that to this ncdump support function with a conditional to switch based on the type of attribute. The calendar attribute is part of this function and should be updated the same way, if you have the time for it.

If you can figure out the C part, I would be glad to assist with the Github PR if you like. There may be a contributors' guide, but I don't know where it is right now.

The other project is that CDMS messaging, which would be more involved. I did look into that a while back, and I could tell you roughly what needs to be done. I would guess you are probably not interested in that one.

[srherbener]

@Dave-Allured thanks for the clarification. We are currently in the throes of getting a release out the door. We are getting close, but it will probably occur toward the latter half of next week. Then I'm taking the week off between Christmas and New Year's.

I'm definitely interested in contributing, but I likely won't be able to get engaged with this until the first week of January. I'd like to start with the nctime0.c issue first, and perhaps take a look at the CDMS messaging issue after that.

[Dave-Allured]

If you would like to expedite this, then please implement my earlier suggestion. You will not need to get into the Netcdf code base in any way, or make any changes to it. This suggestion will ensure that your files will be fully compatible with all recent versions of ncdump -t.

Change your output data files so that time:units attributes are only a 1-D string of Netcdf type character. Also, omit the _FillValue attribute on time variables, unless you have a special case where they are necessary.

[srherbener]

Just a point of clarification that this issue isn't holding up our current release. We are okay for this release with ncdump -t not working, but we do want to get this resolved for the next release (March, 2023). Thanks!

[srherbener]

I decided to change the title of this issue since it is really about the use case I initially gave as an example.

[Dave-Allured]

@srherbener, thanks for taking care of this long standing deficiency!

[srherbener]

No problem. I'm glad to help out!