Best practices for missing values (aka _FillValues) within VLENs?

[czender]

Does Unidata have guidelines to represent missing values within
VLENs, rather than the VLENs themselves?
My understanding is that VLEN _FillValues must themselves be VLENs.
This is fine. I want to understand the best practices for indicating
and treating missing values within each VLEN ragged array.
Here I use _FillValue for the attribute name, and "missing value"
for values to "ignore" within a VLEN.

The semantics are confusing. I hope my description is clear.
As I see it, VLENs are incomplete without a well-understood way of
specifying a "missing value" for elements within the VLEN.
Hopefully this is well-trod ground and I can learn from previous
internal Unidata discussions on this topic. Otherwise I can revise
this and post to the netCDF group for discussion.

There are at least three options:

1. Treat element values that equal the base-type NC_FILL_* as missing.
   This allows a disconnect between the _FillValue (a VLEN) and the
   value treated as "missing" for data within a VLEN.
   This means NC_FILL_* for the VLEN base type is always missing,
   regardless of whether the user wants that. It prevents users from
   assigning a custom value for missing data within a VLEN.
2. Treat the first element of the VLEN _FillValue as missing.
   This connects the VLEN _FillValue to the "missing value" for
   elements within a VLEN. The user controls _FillValue, and can set
   it to any desired value, e.g., _FillValue={-999.0}.
3. Use _FillValue for the VLEN, and missing_value for elements within
   the VLEN. Both are independently configurable.
4. Other/Combinations of the above.

Recommendations?

[WardF]

Interesting question; it hasn't come up in my time here yet, so I'll have to consult the documentation. @DennisHeimbigner any thoughts?

[WardF]

You have asked an excellent question and we are still reviewing it.

[DennisHeimbigner]

Charlie-

The short answer is that there is no special
policy about VLEN and fillvalue because none is needed.

But I need to make sure I understand the question.

There are two possible places where VLEN and fillvalue might
interact.

First, suppose I have the following definitions.

netcdf ... {
dimensions: d1 = 1;
types: int(*) istar_t;
variables: istart_t v(d1);
...
}

I can set a _FillValue for v in the usual way.
I might do some code like this
where ncid is that of the file and varid is the id of variable v.

{
    int seq[3] = {1,2,3};
    nc_vlen_t fillval;
    fillval.len = 3;
    fillval.p = (void*)seq;
    nc_def_var_fill(ncid,varid,NC_FILL,&fillval);
}

BTW, the default fill value for a vlen is a zero-length vlen: {0,NULL}.

But I suspect this is not what you are referring to.

The important thing to note is that the data associated
with the VLEN instance (pointed to by nc_vlen_t.p)
is a sequence, not an array. That means that there can be no
"holes" in the sequence. This is because there is no way in the
netcdf API to read or write a single element in the sequence.
One can only read or write the whole sequence as one object
[IMO, which is why using UNLIMITED is better].

This means that there can be no concept of a missing-value that
needs a fillvalue. All of the values in the sequence are defined
as having whatever value was put there by the user's program. If
the user chooses some value to represent a missing value, then
that is their choice and the netcdf library has no knowledge of it.

Possibly you are being misled by the two functions
nc_get_vlen_element and nc_put_vlen_element in netcdf.h. These
functions are technically there only to support access to VLENs
by Fortran code. It appears the documentation for these functions
is incorrect or at best misleading. They do not allow the
read/write of a specific element in a VLEN sequence.
I will submit an issue to correct the documentation.

Let me know if I am not addressing your question.

[czender]

Thank you, Dennis.
I understand that there is no policy needed by the netCDF library.
However, users will need a best practice to indicate which values
in a VLEN sequence are to be arithmetically ignored. Consider storing
all temperature measurements from a set of N drifting buoys in
a VLEN array of size N. Each sequence has its own number of
measurements. A VLEN array is presumably a suitable storage
strategy for ragged data like this. Each sequence can
have gaps with bad data that should be ignored. What is the
best practice for indicating the value that indicates bad data?
For all atomic NC_TYPEs we do this with _FillValue, but that
does not work directly for elements of VLEN sequence.

The best practice is something that NCO, not netCDF, needs to implement in order
to produce meaningful averages etc. of VLEN sequences. I think the lack of a
best practice will lead to confusion and continued non-adoption of VLENs.
This may be an OK outcome, but the first satellite data (S5P) to use VLEN is now
using it for radiance data and we need some best practices to handle the data.
I can define a best practice by implementing it in NCO. This is Unidata's chance
to voice its preference. Voicing no preference is also fine. I just want there to
be a consensus among those who are affected by this issue.

Some would say using "missing_value" to indicate ignorable elements.
That would open another can of worms, though at least it
would be defensible. Currently I favor using the first element
of any user-defined _FillValue to indicate data to arithmetically
ignore in a VLEN sequence.

[DennisHeimbigner]

I see, and as a best practice it is important for users.
My immediate reaction is to define the preferred fill value
for a VLEN sequence of values is to use the default fillvalue
for the basetype of the VLEN. However, the problem is what to
do if the user specifies an explicit fill value for some variable whose
type is a VLEN. The problem is that there is defined way to specify
a fillvalue for a type (as opposed to a variable). So there is no way to
set a fill value for the basetype of the vlen.
This can only be defined by some convention, as you note.

You propose this:

Currently I favor using the first element of any user-defined _FillValue to indicate data to
arithmetically ignore in a VLEN sequence.
but I am not sure I understand it. Can you give an example?

[czender]

_FillValue={1,bp} where bp points to the basetype sequence whose first value would be used, e.g., 1.0e36 for floating point types. I could also be convinced that using the default NC_FILL for the basetype should be treated prima facie as indicating to ignore such data. However, I prefer placing the choice and responsibility of the indicator on the dataset producer, to avoid implicit assumptions.

[DennisHeimbigner]

If I understand this correctly, then your solution rings a bell.
It is in fact in line with how
we specify fill values for variables whose base type is compound.
That is, the attribute _FillValue must be of the same compound type
and a single instance. The values of the fields of the single instance are used to
(recursively) define the fill value for the fields of the compound type.
I will need to do some research to see if this approach for compound types
is well described in the documentation and if there is some corresponding
documentation for VLENs. I suspect neither compound nor VLENfill values
are described (at least in detail). Note that your solution also should work
recursively if the VLEN base type is a compound or VLEN type itself.
In sum, this seems like the right solution and is consistent with existing
(probably undocumented) practice for compound types.

[czender]

Thank you for the extensive feedback and explanation of _FillValue for compound. Based on that, NCO will follow the method I propose above to indicate ignorable elements for VLEN. As you say, it is in the same spirit as the _FillValue solution for compound.