Standard Name (or other mechanism) to indicate that the variable value is a URL to other/external data

[DocOtak]

Topic for discussion

In the discrete sample data (niskin bottles) that my group manages, we have the situation where some data will never be included inside the data files themselves. This is often for programmatic reasons, i.e. agreement between participants that certain data will be in various other domain specific repositories. Quite often however, this lack of inclusion is due to the data themselves not fitting into a netCDF model well. Things like images from e.g. a FlowCam. Where this gets interesting from my group's data management perspective is that we often know exactly which sample (niskin) the external data came from. So we are able to be more specific when it comes to how to find the external datasets. I did some small tests with one of our files by inserting the URL of the landing page for the data files for that specific niskin bottle closure. When visualizing this test file in ODV, it would display a clickable URL in the sidebar for the exact bottle sample I had selected. This complexity is also why simply storing the URI in a variable attribute did not work.

Seeing how successful and powerful this was, my thoughts turned to what metadata should be attached to the variable itself to indicate that it contains URIs to external other datasets. Naturally my first idea was to ask for some standard name that indicates this e.g. external_dataset_uri (not an actual proposal). While related, I didn't think of this as being appropriate for external_variables since these are whole datasets I want to reference and they probably aren't even netCDF files. Though this external URI reference could maybe help improve the situation of "CF does not provide conventions for identifying the files concerned".

I'm also starting a discussion rather than a specific proposal for a standard name as this didn't feel like it fit the purpose of the standard names, that is, I don't need to track a comparable quantity.

[JonathanGregory]

Thanks for the comment, @DocOtak. Please could you explain a bit more, because I didn't understand it entirely. Do you mean you want to store the URL of the dataset in a variable (char or string-valued, I guess) in the file(s) of the dataset? Why wouldn't it work to put it in a global (or variable) attribute? Jonathan

[ChrisBarker-NOAA]

I had the same thought -- this seems like something to put in a global attribute.

Though we may want to standardize something about that attribute -- a standard attribute name, for instance.

Also -- any limitations on putting an array of strings in one attribute? that would make sense if there were more than one url.

However, as I say that, if you have more that one url, you may want to provide more info about each one, which might call for a variable, and then you could have more information about each url.

However again -- maybe we could standardize the form in each string, e.g.:

// global attributes:
		string :external_references = "source_data: http://some.domain.edu/nifty_experiment_data/data/index.html", "project_background: http://some.domain.edu/nifty_experiment_data/index.html" ;

Or is the idea that you want a URI that is specific not to that Dataset, but to a particular variable? In which case, I suppose the same scheme could be used, but put the attribute on a varaible.

[DocOtak]

Hi @JonathanGregory and @ChrisBarker-NOAA

There are a few reasons I have a strong desire to put the URI in the data part of a char or string dtype variable.

• In our "worst" case scenario, the URI is different for every single sample. This can be a few thousand URIs.
• My group has programmatic requirements to track where samples are going, this is easily done in a variable with coordinates. In our case, auxiliary coordinates are used.
• We want the ability to filter the stations and even vertical points based if the URI is present. See the attached screen shot where I was able to filter a few thousand bottle samples to the 18 or so that actually have HPLC data for this expedition. This was done in ODV without modification to the program/software.
• In a program like ODV, these values are made clickable as a convenience for our users, this works right now and doesn't actually require any sort of standardization effort within e.g. CF.
• Some of our cruise data records are not for one port to port expedition, but often the concatenation of two or more expeditions. With differing links (sually due to different funding or institution requirements). Having these URIs exist as variable data solves a problem of combining this information in a way that attributes don't. That is, I can either just extend the existing dimension, or concatenate along a new dimension.

Or is the idea that you want a URI that is specific not to that Dataset, but to a particular variable? In which case, I suppose the same scheme could be used, but put the attribute on a varaible.

Even more specific than this, each data point in one of our variables is basically itself a whole other dataset with a unique URI. In this case it's genomic sequence data. IIRC I could link to either a specific dataset (i.e. the sequence from one niskin sample) or a collection of all the datasets this program has ever done, but not to e.g. a specific subset matches the data from my expedition.

So my idea is to generalize this approach for our Profile DSG data (we are using the incomplete multidimensional array representation):

• Applies/unique to a the whole data file -> URI in a scalar variable
• Applies/unique to each profile individually -> URI in a 1D variable with shared profile dimension
• Applies/unique to each data point -> URI in a 2D variable with shared profile and vertical level dimension

Only in the first case would storing in any sort of attribute make sense to me.

[larsbarring]

I think this is a highly interesting extension of the concept of standard names to discuss. From my previous work the different versions of the standard name table I recall that there are (were) some standard names that did not have a canonical unit, which may be used as an indication of a standard name for something else than a typical geophysical quantity. In the current version I think those either are deprecated or now have canonical unit 1. By quickly (and not too carefully) scanning the standard name table filtered on unit 1, I found a few possible candidates for precedence:

• various *_quality_flag_*, which may likely be mostly used for numbers (on ordinal or nominal scale) representing flag_values/flag_meanings, even though that is not specifically stated in the descriptions that I had a quick glance at.
• realization, where the description does not say anything about the data type, which thus may be just numbers, or perhaps strings identifying e.g. CMIP models or SSP scenarios.
• cloud_type, where the description explicitly mentions string representation of cloud type.

A key consideration is probably to find a standard name that reasonably well describes the intended content without neither becoming to generic so that it is used as a catch-all for any kind of external data, nor too restricted to only your specific use-case.

[JonathanGregory]

@DocOtak, thanks for further explanation. I hadn't appreciated before that you meant metadata that is specific to individual data values. I agree that you need a variable to contain a field of metadata, and it could be string-valued. I think that's fine. area_type and region are standard names for string-valued quantities, as well as ones Lars mentioned. Somewhat similar to this, I recall a discussion (long ago) about the possibility of allowing some global or variable attribute names, such as source or institution, to be used as standard names when they varied over elements of a data variable.

A variable of strings could get quite large. That's often the motivation for encoding it as integers using flag_values and flag_meanings attributes. Perhaps the size of the file isn't a problem in this case, and it wouldn't help anyway if the values are all unique.

How do you associate the data variable with the variable containing URLs? The latter could be an auxiliary coordinate variable with all the dimensions of the data variable, but its purpose isn't really coordinates, it seems to me. It doesn't locate the data or provide some other geoscientific quantity on which the data depends. If the intention is per-cell metadata, I think it's an ancillary variable in CF terms.

Cheers, Jonathan

[DocOtak]

I wasn't too concerned about the ability to have string/char array type data variables in a CF file, there was lots of precedent for that. Especially in e.g. the taxon names such as biological_taxon_lsid which do not have a canonical unit at all.

A variable of strings could get quite large.

In netCDF4, storing things in car arrays and being very clever about compression and chunking can reduce the storage space significantly. Particularly with the char dimension, if e.g. all my values start with https:// and I have a chunk of size 8, I can end up with an entire chunk containing the same string repeated over and over. I've seen this technique result in a netCDF that was 5% the size of the original input text file, which was 50% smaller than a simple gzip of the input text as well.

I recall a discussion (long ago) about the possibility of allowing some global or variable attribute names, such as source or institution, to be used as standard names when they varied over elements of a data variable.

This is indeed interesting and something that my group basically needs.

How do you associate the data variable with the variable containing URLs? The latter could be an auxiliary coordinate variable with all the dimensions of the data variable, but its purpose isn't really coordinates, it seems to me. It doesn't locate the data or provide some other geoscientific quantity on which the data depends.

In my use case/test files, there is no "data variable", the variable containing the URIs is the "data" and has the same auxiliary coordinates as the rest of the data variables that have numeric values. It also needs to have its own ancillary variable that has a "status_flag". I should probably state that the data we are managing here are all observations and not model outputs or other calculated values (with some caveats).

I want to be clear here that I'm not immediately advocating for a standard_name to be the mechanism to indicate that the data values are URIs. For example, if we extend the cf_role attribute to have a value such as "external_link", this could then free the standard_name attribute describe what that external link is to or frees the standard name attribute to be omitted entirely because the data value isn't a comparable quantity. This is why I started a discussion rather than jumped to a standard name proposal.