String, char, unsigned integers, and character encodings.

[JohnLCaron]

From: John Caron

Background:

In the classic model, data using the "byte" data type are interpreted as signed when converting. However, the byte data type is sometimes used for unsigned data. Unidata introduced the "_Unsigned" attribute to allow the user to specify this. Not all libraries look for this attribute.

Sometimes the "char" data type is intended to mean unsigned byte data. More typically it is used for encoding text data, but the character encoding is undefined. Probably "printable ASCII" is a reasonable assumption. Char data are fixed length arrays only, and one must specify the length using a global, shared dimension, which is unneeded and clutters the dimension namespace.

The NetCDF-4 enhanced model adds Strings and unsigned integer types, so we have the opportunity to clarify. Lots of work on character encodings have been done in the last 20 years with Unicode, and we should leverage that. UTF8 is a variable length encoding of Unicode that has ASCII as a subset, allows any language to be encoded, and has become the dominant encoding on the web. NetCDF libraries assume Strings are UTF8 encoded. If your text is ASCII, you are using UTF8 already.

Also see:

CDL Data Types

Developing Conventions for NetCDF-4 : Use of Strings

Proposal:

1. Use the unsigned or signed integer data types when your data is unsigned or signed, respectively.
2. Do not use _Unsigned attribute.
3. Use the String data type for text data, encoded in UTF-8. Any language (aka character set) is allowable.
4. The char data type is deprecated. If you must use it, use it only for ASCII text data.

[rsignell-usgs]

This has been a major pain in the neck for years, and this proposal sounds like just what the doctor ordered.

[chris-little]

I support all four points of John's proposal, leaving aside any quibbles over glyph/encoding/repertoire/character set/etc terminology. When the defining document is written you will probably need references to ISO10646 (Unicode), ISO646 (ASCII) and possibly the US version: ANSI_X3.4-1968 aka US-ASCII

[JonathanGregory]

Dear John

Thanks for this. These new facilities are certainly attractive and will clarify things.

Regarding signed and unsigned, I agree it is sensible to choose the one which is appropriate. I see the ubyte is unsigned, but http://www.unidata.ucar.edu/software/netcdf/docs/cdl_data_types.html does not say that byte is signed. Should be be (a) recommended to use byte for signed 8-bit data, (b) assumed that byte is signed (like short and int) when reading? As far as CF-2 is concerned, we can include your recommendation to use signed and unsigned types as appropriate, but I don't think any automatic checking can be done, can it, because the checker could not know whether the user had made the right choice for their data.

How do you indicate in CDL that a text attribute (such as the history etc. which you mention) is of string type rather than char type? There are many text attributes in CF e.g. standard_name. Do you think these should be strings rather than char?

I presume that in CDL you can declare "string array(dimension)" in CDL - is that right? I agree that this would be neater than a 2D char array. I would be happy to see this adopted in CF-2 but we would have to check carefully in the standard document to see if it causes any problems, and some careful wording would be needed. The main use of string '''arrays''' in variables (rather than text attributes) is for string-valued auxiliary coordinate variables, I think. If we made this change, the CF checker could issue a warning for the use of a 2D char array where a string array could be used. (It gives warnings for recommendations that have not been followed.)

Is there any way to check your final point? I mean, can you tell if a char variable is being used for data other than ASCII text? I suppose if it exceeds 127b (unsigned!) it is not ASCII - anything else?

Best wishes

Jonathan

PS. It seems that in this browser (Firefox 15.0) the Preview doesn't work. I hope this comes out all right!

[bennoblumenthal]

Up to now, we have used valid-max and valid-min to distinguish between signed and unsigned data, i.e. netcdf byte data is ambiguous, it is not signed. There may be a slight violation of CF in that the values of valid-max and valid-min are not bytes but integers, but I think that is not particularly confusing.

Please don't break this.

[JohnLCaron]

Hi Jonathan:

Regarding signed and unsigned, I agree it is sensible to choose the one which is appropriate. I see the ubyte is unsigned, but http://www.unidata.ucar.edu/software/netcdf/docs/cdl_data_types.html does not say that byte is signed. Should be be (a) recommended to use byte for signed 8-bit data, (b) assumed that byte is signed (like short and int) when reading?

I think that the answer is that byte is signed. I will check with Russ about clarifying the document.

As far as CF-2 is concerned, we can include your recommendation to use signed and unsigned types as appropriate, but I don't think any automatic checking can be done, can it, because the checker could not know whether the user had made the right choice for their data.

agree, theres no way to tell if the stored data values actually match the type declaration.

How do you indicate in CDL that a text attribute (such as the history etc. which you mention) is of string type rather than char type? There are many text attributes in CF e.g. standard_name. Do you think these should be strings rather than char?

i think ncgen now stores string attributes as strings, rather than chars, for netcdf4. I will double check that. programmatically you can do either char or strings, but strings are the right way because the encoding is defined (namely UTF-8).

I presume that in CDL you can declare "string array(dimension)" in CDL - is that right?

yes, one can have multidimensional arrays of strings.

I agree that this would be neater than a 2D char array. I would be happy to see this adopted in CF-2 but we would have to check carefully in the standard document to see if it causes any problems, and some careful wording would be needed. The main use of string '''arrays''' in variables (rather than text attributes) is for string-valued auxiliary coordinate variables, I think. If we made this change, the CF checker could issue a warning for the use of a 2D char array where a string array could be used. (It gives warnings for recommendations that have not been followed.)

agree

Is there any way to check your final point? I mean, can you tell if a char variable is being used for data other than ASCII text? I suppose if it exceeds 127b (unsigned!) it is not ASCII - anything else?

Im not sure of the exact definition of ASCII. Probably we could use some help (Chris Little are you there?) in deciding exactly what we should recommend if you have to use char data. Not sure what the use case is, for writing new files. For backwards compatibility, you would just stick to CF-1.X.

Note that not all sequences of bytes are valid UTF-8, and that could be checked for string type data.

thanks for your comments, as usual, covering some corners i hadn't noticed.

John

[JohnLCaron]

Hi Benno:

Up to now, we have used valid-max and valid-min to distinguish between signed and unsigned data, i.e. netcdf byte data is ambiguous, it is not signed. There may be a slight violation of CF in that the values of valid-max and valid-min are not bytes but integers, but I think that is not particularly confusing. Please don't break this.

I dont see this implemented anywhere in the Java or C libraries. Did you mean that your code implements that?

In any case, if you are writing new files in netCDF-4, using CF-2, you would not want to do that. You would want to use signed or unsigned datatypes to indicate signed or unsigned data.

Regards,
John

[JohnLCaron]

signedness of the byte data type.

In C library for netcdf-3, a byte is signed for the purposes of type conversion (eg asking for data as a float). The _Unsigned attribute is not yet implemented in the C library, as it turns out to be complicated. However, we have a JIRA ticket (https://bugtracking.unidata.ucar.edu/browse/NCF-274) on it, so we hope to do it eventually.

We will fix http://www.unidata.ucar.edu/software/netcdf/docs/cdl_data_types.html to make that clearer.

[bennoblumenthal]

Note that the ticket https://bugtracking.unidata.ucar.edu/browse/NCF-274
quotes the user guide as recommending valid-min and valid-max -- so yes my
code implements it, but I did not make it up: it is pretty obvious usage
of CF attributes.

My point is pretty much implicit in John's two responses: netcdf-3 handles
byte sign as attribute(s), so if you/we want to have byte sign as a pair of
datatypes in netcdf-4, anything that converts between netcdf-3 and netcdf-4
(including the netcdf library!) better do it correctly. That seems to
imply awareness of valid-min/valid-max CF attributes, which is a road not
traveled by the netcdf library. Alternatively, the library can have its
own attribute and we can implement conversion of attributes between
conventions, my favorite topic [?](namespaces, anyone?)

On Mon, Oct 27, 2014 at 5:55 PM, John Caron notifications@github.com
wrote:

signedness of the byte data type.

In C library for netcdf-3, a byte is signed for the purposes of type
conversion (eg asking for data as a float). The _Unsigned attribute is not
yet implemented in the C library, as it turns out to be complicated.
However, we have a JIRA ticket (
https://bugtracking.unidata.ucar.edu/browse/NCF-274) on it, so we hope to
do it eventually.

We will fix
http://www.unidata.ucar.edu/software/netcdf/docs/cdl_data_types.html to
make that clearer.

—
Reply to this email directly or view it on GitHub
#3 (comment).

Dr. M. Benno Blumenthal benno@iri.columbia.edu
International Research Institute for climate and society
The Earth Institute at Columbia University
Lamont Campus, Palisades NY 10964-8000 (845) 680-4450

[JohnLCaron]

Hmm, youre right, I never noticed (or probably just forgot) about that, thanks for the reminder. I agree that any conversion program had better understand the signed conventions of the source file.

Having the signed/unsigned types will solve this sort of confusion.

[steingod]

This discussion is interesting and the proposal is supported. Especially the UTF-8 binding is highly welcome as this issue has caused many problems working with strings (e.g. station or PI names) in data sets.

[chris-little]

A little UTF-8 tutorial: all ASCII characters are represented by octets with the high order bit set to zero. 'Plain' 8-bit ASCII is UTF-8. If the high bit is set to 1, the next few octets (2,3,4 or 5) are used to define one of the Unicode basic 64000 characters, and the next high order bits define how many octets are used for that character.

This means that there is not a simple mapping of n characters to n octets, but for western alphabets, there will be probably be significantly less than 2n octets.

If there are a lot of Chinese/Japanese/Korean characters, there could be up to 6n octets needed, but for this use case, it would be better to use UTF-16, and most of the n characters would be represented by 2n octets.

This sugggests the question: what will CF2 do if the characters are not in UTF-8? UTF-16 is the obvious case to address. I recommend not allowing any other 8 bit encodings, such as Windows, ISO2022, Mac, etc.

HTH, Chris

[mikeggrant-pml]

I support John's suggestion of only allowing UTF-8 and definitely agree with Chris on excluding other 8 bit encodings. While UTF-16 might have some limited advantages in particular cases [http://en.wikipedia.org/wiki/UTF-8#Compared_to_UTF-16], the extra hassle of two encodings isn't worth it. It's highly likely that the largest contributer to the size of a netcdf file will be the data rather than the text, so a few bytes saved in some languages won't make any significant difference.

[stephenpascoe]

I think UTF-8 is an extremely sensible idea for String types which are variable length by default. It might be worth specifically saying char[] is ASCII not UTF-8. It will be more awkward to deal with variable length octet encoding for a fixed-width type like char[].