Conventions for string and character array encoding

[rsignell-usgs]

As discussed here Unidata/netcdf4-python#654 (comment), there is a need for conventions to specify the encoding of strings and character arrays in netcdf.

There is also a need to specify whether char arrays in NetCDF3 contain strings or character arrays.

@BobSimons addressed these issues in an enhancement to CF conventions that would specify charset for NetCDF3 and _Encoding for NetCDF4, and the Unidata gang (@DennisHeimbigner, @WardF, @ethanrd and @cwardgar) agreed with the concept, but suggested this be handled in the NUG and we came up with this slightly different proposal that would still accomplish Bob's goals of making it easy for software to figure out what is stuffed in those char or string arrays!

Proposal:

• Use _CharType variable attribute with allowed values ['STRING', 'CHAR_ARRAY'] to specify if a char array variable should be interpreted as a string or as an array of individual characters. If _CharType is missing, default is 'STRING'.
• Use _Encoding variable attribute with allowed values ['ISO-8859-1', 'ISO-8859-15', 'UTF-8'] to specify the encoding. If _Encoding is missing for _CharType='STRING', default is 'UTF-8'. If _Encoding is missing for _CharType='CHAR_ARRAY', default is 'ISO-8859-15'.

[ethanrd]

Should an _Encoding attribute on a 'char' typed variable be restricted to a 7- or 8-bit encoding?

[ethanrd]

As @DennisHeimbigner mentions here Unidata/netcdf4-python#654 (comment), this proposal does deals only with char or String typed variables, not char or String typed attributes.

[jswhit]

Why wouldn't _Encoding apply to attributes as well as variable data?

[DennisHeimbigner]

Because netcdf does not support attributes for attributes. We would need to come up with some kind of convention for this: a second attribute that could be interpreted as applying to the string/char attribute. Alternate is to define a global encoding for all attributes. =Dennis Heimbigner Unidata

…

On 5/3/2017 8:01 AM, Jeff Whitaker wrote: Why wouldn't |_Encoding| apply to attributes as well as variable data? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#402 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA3P2yhdO2_-QxIAH57sQUlvJMbiB3I-ks5r2IjKgaJpZM4NOoaO>.

[rsignell-usgs]

@DennisHeimbigner, I guessing you can answer @ethanrd question:

Should an _Encoding attribute on a char typed variable be restricted to a 7- or 8-bit encoding?

[DennisHeimbigner]

Should an _Encoding attribute on a char typed variable be restricted to a 7- or 8-bit encoding?

If an _Encoding is specified, then that technically determines 7 vs 8 bit. E.g. Ascii is 7 bit, but
ISO-Latin-8859-1 is 8-bit. The tricky case is when something like utf-8 encoding is specified.
Technically, the single character subset of utf-8 is 7-bit ascii. But, it is clear that some users
treat an array of chars as a string, in which case any legal utf-8 char bit pattern should be legal.
IMO, we should always treat char as essentially equivalent to unsigned byte so that a char
can hold any 8-bit bit pattern so that e.g. _Encoding = "iso-latin-8859-1" is legal and does not
lose information.

[ethanrd]

I suggested always indicating the encoding/charset with the same attribute (_Encoding), whether string or character, thinking it simplified things. Now that restrictions on allowed encodings has come up, I'm seeing the wisdom of @BobSimons original proposal to the CF list with one attribute that gives a string encoding for use when interpreting strings and one that gives the character set for use when interpreting individual characters. (It avoids the need for different restrictions on the value of _Encoding depending on the situation.)

So, as an alternate to the above proposal, I'll restate Bob's proposal here with a change or two given the target is the NUG rather than CF:

• Use the _CharSet variable attribute to indicate that a char array should be interpreted as individual 8-bit characters. The value of the attribute gives the 8-bit character set to use when interpreting the 8-bit characters. (E.g., 'ISO-8859-15'.)

• Use the _Encoding variable attribute to indicate which character encoding should be used to interpret a string variable. Used with a char array, the attribute indicates that it should be interpreted as a string (or an array of strings).

---

Reviewing Bob's original proposal brought up a number of questions on how the netCDF-4 and HDF5 libraries handle string encoding (if they enforce the encoding or not, etc.). I'm still digging and will report back when I get somewhere.

Also, there was some question in the CF discussion on whether an explicit indicator was needed to differentiate between whether a char array should be interpreted as individual 8-bit characters or as a string(s). Since the current proposal is suggesting a change to the NUG, I'm not sure if this question will or should play out the same as in a CF discussion.

[rsignell-usgs]

@lesserwhirls, do you have any thoughts here? @ethanrd are you still looking at this, or can we propose the above changes to NUG?

[DennisHeimbigner]

I do not understand the need for the _CharSet attribute. The type of the variable (char vs String)
and the _Encoding attribute seem to me to encompass _CharSet. That is
_Encoding for char type == CharSet

[rsignell-usgs]

@DennisHeimbigner, the problem is that while netcdf4 has char or string, netcdf3 has only char. So we don't know whether the netcdf3 char holds a string or an array of 8 bit characters.

[BobSimons]

@DennisHeimbigner, this is an alternative proposal.
In this proposal _CharSet and _Encoding apply to different situations, have different options, and are used differently (mandatory vs optional):

_CharSet would be for char variables when they are to be interpreted as individual chars.
The options are ISO-8859-1 and ISO-8859-15.
_CharSet is mandatory if the chars should be interpreted as individual chars.

_Encoding would be for String variables (e.g., in nc4) and char variables in nc3 which should be interpreted as Strings.
The options are ISO-8859-1, ISO-8859-15, and UTF-8. [Different!]
_Encoding is optional. The default is UTF-8. [Different!]

A further advantage is that only one attribute is needed per variable, not two.

Think of it from a software reader's point of view:

• Is there a _CharSet attribute? Then these are chars and I now know the charset.
• Is there an _Encoding attribute? Then these are strings and I now know the encoding.
• Is there neither? Then these are UTF-8 strings.

[DennisHeimbigner]

I think the term "mandatory" is being misused here since a default is defined.
But the real issue in Bob's proposal has to do with whether a character typed
variable (or attribute?) is to be treated as if it was a surrogate for the lack of
a String type in netcdf-3 and do we want a special attribute to mark that case.
Personally undecided on that issue.

[DennisHeimbigner]

One other question. If we had an attribute to indicate that a char array
should be treated like a string, do we want to limit the use of that
attribute to netcdf-3 only. Since netcdf-4 has string type, that attribute
is technically not needed.

[ethanrd]

@BobSimons Given the backward compatibility issues, I'm not sure the NUG should specify how character arrays are interpreted when the proposed attributes are not used. At least not at the level of a MUST.

[rsignell-usgs]

The proposed default behavior is to assume that a netcdf3 char array is a string.
With Bob's proposal, if a_CharSet attribute is found, we know it's not a string.