Localized metadata in NetCDF files

[turnbullerin]

Hi Everyone!

So I work for the Government of Canada and I am working on defining the required metadata fields for us to publish data in NetCDF format. We'll be moving a lot of data into this format, so we are trying to make sure we get the format right the first time. The CF conventions are our starting point for metadata attributes.

As the data will be officially published by the Government of Canada eventually, we will have to make sure the metadata is available in both English and French. If the data contains English or French text (not from a controlled list), it needs to be translated too. I haven't found any efforts towards creating a convention for bilingual (or multilingual) metadata and data in NetCDF formats, so I wanted to reach out here to see if anyone has been working on this so we could collaborate on it.

My initial thought is that the metadata should be included in such a way as to make it easy to programmatically extract each language separately. This would allow applications that use NetCDF files (or tools that draw on the CF conventions like ERDDAP) to display the available language options and let the user select which one they would like to see without additional clutter. It should also be included in a way that does not impact existing applications to ensure compatibility.

Of note though is that some data comes from controlled lists where the values have meaning beyond the English meaning. This data probably shouldn't be translated as it would lose its meaning. For many controlled lists, applications can use their own lookup tables to translate the display if they want, and bigger vocabulary lists (like GCMD keywords) can have translations available on the web.

ISO-19115 handles this by defining "locales" (a mix of a mandatory ISO 639 language code, optional ISO 3166 country code, and optional IANA character set) and using PT_FreeText to define one value per locale for different text fields. I like this approach and I think it can translate fairly cleanly to NetCDF attributes. To align with ISO-19115, I would propose two global attributes, one called locale_default and one called locale_others (I kept the word 'locale' in front instead of at the end like in ISO-19115 since this groups similar attributes and I see this is what CF has usually done). The locale_others could use a prefix system (like what keywords_vocabulary uses) to separate different values. I would propose using the typical standards used in the HTTP protocol for separating the language, country, and encoding, e.g. language-COUNTRY;encoding. Maybe encoding and country are not necessary, I'm not sure, I just know ISO included them.

I would then propose using the prefixes from locale_others as suffixes on existing attribute names to represent the value of that attribute in another locale.

For example, this would give us the following global attributes if we wanted to include English (Canada), French (Canada), and Spanish (Mexico) in our locales and translate the title:

  :locale_default = 'en-CA;utf-8';
  :locale_others = 'fra:fr-CA;utf-8 esp:es-MX;utf-8';
  :title = 'English Title';
  :title_fra = 'Titre française';
  :title_esp = 'Título en español';

I was torn if the default locale should define a prefix too, if it did, it would let one use the non-suffixed attribute name for a combination of languages as the default (for applications that don't support localization); for example:

  :locale_default = 'eng:en-CA;utf-8';
  :locale_others = 'fra:fr-CA;utf-8 esp:es-MX;utf-8';
  :title = 'English Title | Titre française';
  :title_eng = 'English Title'
  :title_fra = 'Titre française';
  :title_esp = 'Título en español';

But then this seems like an inaccurate use of locale_default since the default is actually a combo. Maybe English should be added to locale_others in this case and locale_default changed to something like und;utf-8 or even just use the delimiter like [eng] | [fra] to show the format.

I haven't run into a data variable that needs translating yet, but if so, my thought was to define an attribute on the data variable that would allow an application to identify all the related localized variables (i.e. same data, different locale) and which variable goes with which locale. Something like

  var_name_en:locale = ':var_name';      # locale identified in locale_default
  var_name_fr:locale = 'fra:var_name';   # locale identified in locale_others

Thoughts, feedback, any other suggestions are very welcome!

[czender]

Interesting idea. If you'd like more input/discussion, this could form the basis for a breakout at the upcoming 2023 CF Workshop. ..

[turnbullerin]

oh thats cool - I can't find any info on that yet, I guess more info will be coming later?

[ethanrd]

Hi Erin - The dates for the 2023 CF Workshop (virtual) were just announced (issue #243). There has also been a call for breakout session proposals (issue #233). Further information will be broadcast here as well so everyone watching this repo will get the updates. A web page for the workshop will be added to the CF meetings page in the next month or two.

[Zeitsperre]

Hi @turnbullerin and others,

I wanted to echo my interest in seeing a metadata translation convention come about from the CF Conventions. My team and I have been developing some implementations of metadata translations to better support our French-speaking users, as well as open the possibility of supporting other language translations for climate metadata.

One of our major open source projects for calculating climate indicators (xclim) has an internationalization module built into it for conditionally providing translated fields based on the ISO 639 Language code found within the running environment's locale or set explicitly. For more information, here is some documentation that better describes our approach:

• https://xclim.readthedocs.io/en/stable/internationalization.html
• https://xclim.readthedocs.io/en/stable/notebooks/customize.html#Adding-translated-metadata

We would love to take part in this discussion if there happens to be a session in October.

Best,

[Dave-Allured]

Erin, I like the general direction of your localization proposal. I would like to suggest a simplified strategy. I do not see a need for those global attributes or the level of indirection represented in them. In short, I suggest simply add ISO-19115 suffixes to standard CF attribute names, as needed. Here are a few more details.

• The general form would be attribute_name.lang-country.

• lang-country is the two-part locale, exactly as prescribed by ISO-19115.

• country is optional, just like you described.

• Never use the third element, IANA charset. The entire netCDF name space is fixed on UTF-8, so charset here is unnecessary.

• The entire ISO suffix is optional. With that, you are left with just three possible basic forms, and no other complications. The ISO suffixes can now be easily be machine-recognized by aware software:

  • attribute_name
  • attribute_name.lang
  • attribute_name.lang-country

More details:

• Define English as the core language of CF controlled vocabulary.
• Define English as the universal default locale. This is basically saying, keep the status quo as the default.
• When localization is desired, add the lang suffix.
• Avoid country except when the data publisher deems it necessary.
• attribute_name and attribute_name.en are fully redundant and should not both be applied together.
• This localization strategy is intended equally for both CF-controlled and user-defined attribute names.
• The primary application is to provide bilingual support. For example, under this scheme you might have long_name.en and long_name.fr on the same data variable.

The choice of the primary delimiter will be controversial. I like period "." for visual flow and general precedent in language design. Some will hold out for underscore as the CF precedent. I think underscore is overused in CF. In particular, the ISO suffix deserves some kind of special character to stand out as a modifier.

The general use of special characters such as "." and "-" is part of proposal cf-convention/cf-conventions#237.

[turnbullerin]

@Dave-Allured

Thanks for your feedback!

I think there is value in the two attributes.

Defining English (and which English, eng-US, eng-CA, eng-UK, etc.) as the universal default is very Anglo-centric. There is a clear use case for datasets produced in other countries to have a primary language that is not English, and documenting it is valuable to inform locale-aware applications processing CF-compliant files. Not everyone will want to provide an English version of every string. So having an attribute that defines the default locale of the text strings in the file is still useful I feel, but perhaps we could define the default if not present as "eng" (no country specified) so that it can be omitted in many cases.

For the other locales, I think it helps applications and humans reading the metadata to know what languages are in the file. If we did not list them, applications would need to be aware of all ISO-639 codes and check each attribute if it exists with any mix of country/language code suffix to build a list of all languages that exist in the metadata. Having a single attribute list them all has a lot of value in my opinion. In unilingual datasets, it can of course be omitted.

This also raises the question on if we should use ISO 639-1 or ISO 639-2/T or ISO 639-3. ISO 19115 allows users to specify the vocabulary that codes are taken from, but if we were to specify one I would recommend ISO 639-2/T for language and ISO 3166 alpha-3 for country (this aligns with the North American Profile of ISO-19115). Alternatively, we could just specify the delimiter and let people override the vocabulary for language and country codes in attributes if they want.

I am torn on the delimiter - I see the value in what you propose, but I would not want to delay this issue if #237 is not adopted quickly and I foresee some technical issues adopting it even if it is agreed to (for example, the Python NetCDF4 library supports attributes as Python variables on the dataset or variable objects [and thus are restricted to [A-Za-z0-9_]; allowing arbitrary names would require them to make a significant change before the standard could be adopted; see https://unidata.github.io/netcdf4-python/#attributes-in-a-netcdf-file).

I do like the idea of standardizing the suffixes though and if we can agree on a format, I support that wholeheartedly. I would propose _xxxYYY where xxx is the lower-case ISO 639-2/T code and YYY is the ISO 3166 alpha-3 country code. If #237 is adopted, .xxx-YYY is also a good solution I think. We could include both for compatibility with applications and libraries that won't support #237 right away if adopted.

Also, I fully agree on UTF-8. It supports all natural languages as far as I know, so there should be no issue with using it as the default encoding. However, I do note that the NetCDF standard allows for other character sets - I guess we are then just saying that all text data must be in UTF-8 (i.e. _Encoding="utf-8")?

In terms of display, I agree with you that locale-aware applications (given a country and language code they should display in) should use the attributes in the following order:

1. attribute_langCOUNTRY
2. attribute_lang
3. attribute

[Dave-Allured]

Erin, thank you for your very thoughtful reply.

Anglo-centric: Yes I was thinking about that when I wrote down my initial thoughts, but I decided to test the waters. I am glad to have triggered that direct conversation. English is a dominant language in the science and business worlds. However, this CF enhancement is a great opportunity for constructs to level the playing field, within the technical context of file metadata.

I agree immediately to the value of a global attribute that sets the default language for the current data file, such that all string attributes with no suffix are interpreted in the specified language. I leave the name of such attribute up to you and others. Yes, keep the default as English if the global attribute is not included.

[larsbarring]

I think adding support for multiple languages to selected CF attribute values would be a great addition. As I have absolutely zero insight into the technical aspects please bear with me if I am asking a stupid question: If this functionality is implemented without an universal default language does it mean that all string valued attributes are expected to follow a specified locale? If so, how would CF attributes that only can take values from a controlled vocabulary be treated, e.g. units, standard_name, cell_methods, axis, calendar?

Thanks,
Lars

[Dave-Allured]

List of languages present: It really is no problem to scan a file's metadata, pull off all the language specifiers, and sort them into an organized inventory. This is the kind of thing that can be programmed once, added in to a convenience library, and then used by everybody. If you have a redundant inventory attribute, you immediately have issues with maintenance and mismatches. Such issues will persist forever.

[Dave-Allured]

ISO vocabulary: It would be really nice if CF could settle on single universal choices for thelang and country vocabs. I really like the cadence of [dot] [two] [dash] [three], and no extra steps for alternative vocabularies. Failing that, I would suggest defer to an ISO 19115 self-identifying scheme if there is such a thing. I suppose there could be a vocabulary identifier global attribute, but I would like to avoid that if possible.

[turnbullerin]

@larsbarring I think we would apply this only to natural language attributes, not to those taking their values from a controlled vocabulary.

So title, summary, acknowledgement, etc. are translated; units, standard_name, cell_methods, etc. are not.

Perhaps some form of identification of those would be useful?

[turnbullerin]

@Dave-Allured

I think identifying what is and is not a language specifier might be challenging. Assuming attribute_xxx[YYY] as an algorithm, I would write this:

1. Look at every attribute name.
2. If it has an underscore, take the text from the last underscore to the end of the string and continue. Otherwise next attribute name (not a locale).
3. If it is not either 3 or 6 letters long and either all lower case (if 3) or first three lower case and last three upper case (if 6), then continue (not a locale)
4. Check that the first three are a defined ISO 639-2/T code and the last three (if present) are a defined ISO 3166 alpha-3 code (requires a list of all valid codes that needs to be updated as ISO makes changes to those vocabularies). If not, continue (not a locale)
5. Assemble and deduplicate the results

Versus, with an attribute, it is:

1. Read the attribute and split it by spaces.

I think, while it can be done, having an attribute with all languages in the file greatly simplifies the code for understanding which languages are present (which is the point of some of the metadata, like we could calculate geospatial_max_lon and geospatial_min_lon but we have those for convenience). It also ensures attributes which happen to look like valid localized attributes are not actually treated as such.

[Dave-Allured]

Identifying: Yeah. ;-) Add this to my list of reasons for dot notation.

Suffixes ... We could include both ...

I see great value in settling on a single, optimal syntax up front, and not providing alternative syntaxes. I also value adopting an exact syntax from ISO 19115, rather than having a new CF creation. You already see my preference for dot and dash, and my reasons. I think it is worth holding out for the optimal syntax. I see a growing interest in character set expansion for CF.

The classic netCDF API's included special character handling from the moment of their creation. Python can adapt.

I like 2-letter ISO 639 language codes, but 3-letter will be okay too. Choose one. I defer to your greater expertise on the various ISO flavors. I am not well studied there.