Add data storage expiry date

[mortenwh]

B2: "As the head of IT, I want to know the lifespan of different data sets so that lifecycle management is maintained. I want data that MET no longer needs to be removed. So that we keep down the costs and maintenance needs for MET, and contribute to green IT."

This could be solved by adding a field "expiry date" or similar to the storage_information field.

[steingod]

What makes me a bit concerned here is that IT potentially automatically removes data.
So perhaps storage_expiry_date, make this optional and make sure no one is automatically deleting something?

[mortenwh]

storage_expiry_date sounds good. I understand your concern here, so this needs to be highlighted somehow..

[steingod]

Sounds good to me. Do we need more in the storage_information field?

[ferrighi]

Since this will be optional, would we assume that the lack of it means that there is no expiration date? Or should this be used also to proactively state that the data will never expire? We can think of using this as date, but allowing also controlled vocabulary, as "Not applicable" for example.

[steingod]

This is a good question. I would say missing is no expiration, but do see the point of explicitly specifying this. However I would start with a specific date for now (and foreseeable future) and rather have default behaviour if missing as no expiry date.

[ferrighi]

We go for a single field of type "date" (YYYY-MM-DD). Can you provide a definition you want to have?

Something like "The date after which the dataset is no longer actively maintained." ?

I will also add "When this information is not present, it means that the dataset does not have an storage expiration date, it is therefore actively maintained and should not be deleted".

[steingod]

This is good, concerning the explanations I would go along the lines below.

<storage_information>
  <file_name>
    osisaf_radiative_flux_24h_hl_polstere-050_multi_202003051200.nc
  </file_name>
  <file_location>/home/steingod/Desktop</file_location>
  <file_format>NetCDF-CF</file_format>
  <file_size unit="GB">0.12</file_size>
  <checksum type="sha512sum">
ad33563f9ab3a6f2ba74ebd72cb1f45fda00d55121a29a29230147e3472ef267c50882b0e0b21ef9ee5ed5cc25e454167cd19a1818f1e13bc044b6fc3ef8f285
  </checksum>
  <storage_expiry_date>2027-07-13</storage_expiry_date>
</storage_information>

Adding the following to the element description
...
Sub elements include:

• ...
• storage_expiry_date: The date when the dataset reaches end of life cycle management. After this date the dataset can be deleted. When this information is not present, it is implied that the dataset does not have an storage expiration date, and is in active life cycle management and cannot be deleted.

[lhmarsden]

In NBS, for some products are store CF-NetCDF files temporarily as well as the SAFE files. The CF-NetCDF files are deleted after some period. It would be good if we could communicate this to the users via the MMD.

The storage information looks like this, and only relates to the SAFE file:

  <mmd:storage_information>
    <mmd:file_name>S2A_MSIL2A_20260101T105501_N0511_R051_T32VNK_20260101T145209</mmd:file_name>
    <mmd:file_format>SAFE</mmd:file_format>
    <mmd:file_size unit="MB">284.06</mmd:file_size>
    <mmd:checksum type="md5sum">35081df47c783397b9389a35b3b6dd3f</mmd:checksum>
  </mmd:storage_information>

Then the only mention of the NetCDF is in the data_access elements, for better or worse...

  <mmd:data_access>
    <mmd:type>HTTP</mmd:type>
    <mmd:description>Direct access to the full data file.</mmd:description>
    <mmd:resource>https://nbstds.met.no/thredds/fileServer/nbsArchive/S2A/2026/01/01/S2A_MSIL1C_20260101T105501_N0511_R051_T32VMK_20260101T132330.zip</mmd:resource>
  </mmd:data_access>
  <mmd:related_dataset relation_type="parent">e4a167dd-3951-565a-907c-b632e536f5e1</mmd:related_dataset>
  <mmd:data_access>
    <mmd:type>OPeNDAP</mmd:type>
    <mmd:description>Open-source Project for a Network Data Access Protocol</mmd:description>
    <mmd:resource>https://nbstds.met.no/thredds/dodsC/NBS/S2A/2026/01/01/S2A_MSIL1C_20260101T105501_N0511_R051_T32VMK_20260101T132330.nc</mmd:resource>
  </mmd:data_access>
  <mmd:data_access>
    <mmd:type>HTTP</mmd:type>
    <mmd:description>Direct access to the full data file. May require authentication, but should point directly to the data file or a catalogue containing the data.</mmd:description>
    <mmd:resource>https://nbstds.met.no/thredds/fileServer/NBS/S2A/2026/01/01/S2A_MSIL1C_20260101T105501_N0511_R051_T32VMK_20260101T132330.nc</mmd:resource>
  </mmd:data_access>
  <mmd:data_access>
    <mmd:type>OGC WMS</mmd:type>
    <mmd:description>OGC Web Mapping Service, URI to GetCapabilities Document.</mmd:description>
    <mmd:resource>https://adc-wms.met.no/get_wms/aaff52c3-0da0-4626-86f9-c10d61664bf9/wms</mmd:resource>
  </mmd:data_access>

For this to work in NBS, we would either need to

1. Allow multiple instances of the storage_information element
2. Encode something like a valid_until attribute within the data access element - either as a new element or as an attribute to the resource element perhaps

[steingod]

If so I would opt for

1. allowing multiple instances of storage_information (but keep that internal and not expose externally)
2. add some attribute indicating file type in data_access and convey this to users, I am not sure we should add valid_until for data_access, when data is removed, MMD needs to be updated.

[lhmarsden]

Just checking; Could anyone foresee a situation where we might want to remove data access (e.g. WMS) after some time period but keep other data accesses (e.g. OPeNDAP) available?