Collection of Collections

[cmheazel]

Can a collection contain other collections? The GFM says yes, but that may be too much complexity to address in simple core.

[cportele]

See also opengeospatial/ogcapi-features#90 and opengeospatial/ogcapi-features#76

[cportele]

If we are talking about collections of collections we should also make sure we define the distinction between dataset (as in DCAT, see https://www.w3.org/TR/vocab-dcat/#vocabulary-overview) and our feature/coverage/etc collection resources. Datasets have a specific role (see schema.org, the Google Dataset Search, the Data on the Web Best Practices) and we should fit into this picture.

In WFS 3.0 Core we support the simplest case, "every WFS instance provides access to a single dataset" (see e.g. the Scope statement).

In the strawman OGC API resource diagram in which I have added the feature/process/coverage/tile/style resources that I found in drafts I have also included one approach how to support multiple dataset as an input for discussion.

[nmtoken]

For coverages we have collection of coverages, is that a similar concept?

[jerstlouis]

I believe it would be extremely useful to define a proper optional collection of collections conformance class as part of OGC API - Common (Collections).

See also discussions on recursive datasets and large number of collections in #17, #69 and #111 .

[cmheazel]

A comment received from Uwe Voges:
I was wondering how the use case will be covered that there may be hierarchies of collections (see also STAC) ?
This will include the case that some hierarchy levels may be searchable for items (records) while others may not ...
Example:

Meteosat Catalogue (Searchable for all Meteosat products)
MTG (Searchable just for MTG products)
MSG (Searchable just for MSG)
Metop Catalogue (Searchable for all Metop products)
EPS (Searchable for EPS products)
EPS-SG (Searchable for EPS-SG products)
Sentinel Catalogue (not searchable)
Sentinel3 Catalogue (Searchable for Sentinel3 products)

Ok, one idea could be to provide different API´s with an own landing page etc for every hierarchyLevel....

Another option could be to have the /collections resource which may not even return links to single collections (/collection) but also links to other hierarchy levels of collections (/collections) (rel="collections" like the rel "data" in the landing page).

I tried ad hoc a JSON encoding of the example above which could be provided by the link from the landing page (rel="data") (just a few metadata elements considered here):

{
"collections": [
{
"id": "msgCat",
"type": "catalogue",
"title": "Meteosat Catalogue",
"links": [
{
"href": "http://.../meteosatCat/collections",
"rel": "collections",
"title": "At this endpoint you can retrieve hypermedia controls that allow you to navigate sub-collections, e.g. MTG, MSG. Here you can retrieve descriptions of these sub-catalogues as well as hypermedia controls that allow you to query the sub-catalogues"
},
{
"href": "http://.../meteosatCat/collections/meteosat",
"rel": "collection",
"title": "At this endpoint you can retrieve a description of this catalogue as well as hypermedia controls that allow you to query the catalogue."
}
]
},
{
"id": "metopCat",
"type": "catalogue",
"title": "Metop Catalogue",
"links": [
{
"href": "http://.../metopCat/collections",
"rel": "collections",
"title": "At this endpoint you can retrieve hypermedia controls that allow you to navigate sub-collections, e.g. EPS-SG. Here you can retrieve descriptions of these sub-catalogues as well as hypermedia controls that allow you to query the sub-catalogues"
},
{
"href": "http://.../metopCat/collections/metop",
"rel": "collection",
"title": "At this endpoint you can retrieve a description of this catalogue as well as hypermedia controls that allow you to query the metop catalogue."
}
]
},
{
"id": "sentinel3Cat",
"type": "catalogue",
"title": "Sentinel3 Catalogue",
"links": [
{
"href": "http://.../sentinelCat/collections",
"rel": "collections",
"title": "At this endpoint you can retrieve hypermedia controls that allow you to navigate sub-collections, e.g. Sentinel3. Here you can retrieve descriptions of these sub-catalogues as well as hypermedia controls that allow you to query the sub-catalogues"
}
]
}
]
}

[cmheazel]

Copied from the Wiki at the request of Dave Blodgett

One of the most contentious issues we face in API development is consensus on standard resource paths. API-Features uses /collections/{collectionId}/items. Other SWGs plan to use /collections/{collectionId}/coverage, /collections/{collectionId}/maps, etc. Which is correct?

There appears to be consensus on /collections/{collectionId}. At least for those resources which are typically distributed as collections. The final part of the path tells you something about the collection. A taxonomy of these path fragments could look something like this:

/coverage -> Measurements organized in a regular or irregular grid

/map -> 2D Rendered information displayable for human visual consumption

/tile -> A subset of a dataset based on area of interest and level of detail

/items -> Any

Wait a minute -- "Any"!!

What use is a path fragment which can be anything?

Actually, it’s the most useful of all.

OGC Web Services are built around a taxonomy of data types; maps, features, coverages, etc. It’s a data provider centric view of the world. But what if we took a user-centric view of APIs. What if collections are built around the information that a user needs to do their job?

Consider a first responder. They are going to need a building model, routes of ingress and egress, photographs, overhead imagery, utilities, an elevation model, and all sorts of other supporting data. Everything they need to respond to an incident at a specific location. And they are going to want it in one neat little package, a collection. So how is our first responder going to use “/collections/110_Maple_Avenue/”? /maps, /coverage, /routes all have value. But how do you access the collection contents as a whole? /items.

/items allows us to locate and access any content in the collection regardless of type. So the fundamental access path for any collection is /collections/{collectionId}/items. This does not mean that the other paths don’t have value, just at a different level.

We can view /coverage, /maps, etc. as profiles of /items. They both restrict and extend /items. They restrict in that they only provide access to a specific type of item in the collection. They extend in that they bring to the user a whole collection of tools which are only useful for that type of resource. So “/collections/110_Maple_Avenue/coverage” brings to the first responder all of the sub-setting and processing tools which come with coverage data. But only for that subset of the collection which is coverage data.

So what does /items actually return? A Feature collection.

Now that the outrage has had a chance to settle,

We tend to think of Features as vector data. But at its most fundamental level, a Feature is an empty container. It has neither type, identity, nor properties. So a Feature can grow up to be just about anything. Which is exactly the sort of construct we are looking for.

Under this approach, API-Common would define the bbox, limit, and datetime parameters and how they would be applied to a generic data item. Other API standards would specialize those capabilities for the unique properties of the resources they address.

[rob-metalinkage]

when a problem is very complex in one level of abstraction - as I feel this one is - it can help to change the domain of discourse to a different level (e.g. fourier transforms between time and frequency domains)

In this case I think the problem is too narrow a focus, and hence different solutions for smuggling additional information into the places you can see..

At a higher level of abstraction I believe you can look at "context" for metadata - and understand there is always an element of "out-of-band" metadata - the question is what is necessary and what is acceptable.

In the "classic" SOA a client may go to a catalog and discover a service end-point and out-of-band metadata about the service type - at which point they use this context to formulate a query (get capabilities) about additional metadata the server will self-report.

The same pattern holds true for data descriptions. To what extent is the data responsible for self-reporting. In general, as little as possible, otherwise you need to find and access and interrogate too much before you can select useful data.

So much of the context comes down to some form of "out-of-band" description - some form of catalog. This can vary from a hyper-local (hardcoded in a client) catalog, to service self-description to aggregated catalogs with well-known service interfaces. Its still pushing the data description problem to some out-of-band metadata construct.

So now the question is really one of minimising impedance when a client accesses such out-of-band sources of context - i.e. the interoperability of the description. This includes a description of how the data model of the data maps onto the data meta-model of the service (in this case collections/items/features)

As @cmheazel suggests - you have all the mechanisms you need - its a choice of how you use them, and the point here IMHO is understanding how to map interoperable metadata forms onto the two different models supported by the service: container based addressing and reporting of metadata (capabilities). We can push all this to the issue of publishing Feature Types with sufficient richness and interoperability that such mappings can be immediately inferred - but its probably worth developing test cases and examples for what ever pattern you choose.

[dblodgett-usgs]

I thought I responded to this but my comment is missing.

@cmheazel : When I called out the wiki post, and said "collection" of collection issues, I thought it was a pun!

I linked to the collection TAG trying to make the suggestion that I thought your wiki post was a good frame for bringing some closure to a lot of similar issues related to using collections. Not just this one. But I'll take it. We need to get this one settled!

[jerstlouis]

The approach currently implemented in our client & service:

• /collections returns a list of a all collections

• /collections/{collectionId} returns the resource for that collection (containing the links to the rel items, tiles etc.)

• As an extension, a special character within the {collectionId} is understood to be a hierarchy delimiter, and the service supports returning a partial list of collection if the client requests /collections/{partial-collectionId}.

This works nicely with multi-layer vector tiles as well, in which case a {partial-collectionId} can be both a collection linking to multi-layer /tiles end-point, as well as list the individual collections.

At the moment that special character happens to be a '/', and the invalidity of this can be debated, but in practice it has caused no interoperability problem with our service during 2+ years of TIEs.
This could be changed to use another character and specify that character with a property, e.g. "hierarchySeparator" : "-".

Features also state that an OGC API end-point should contain a single dataset, and that hierarchies of multiple datasets should be implemented entirely at another level (i.e. higher than the landing page).

I propose that combined hierarchies of datasets and collections could be implemented as an integrated optional capability, while still being able to mark which level constitutes a dataset. There could also potentially be landing pages at different levels...

[dblodgett-usgs]

@jerstlouis -- I'm having a hard time connecting the first part of your comment to the last. Maybe some more for instance API paths would help?

Are you saying we might have a hierarchy of OGC-API landing pages?

What API path would dereference to a dataset (landing page) and what API path would dereference to a collection that is subordinate to that dataset?

I think you are saying that an integrated dataset-landing page might be something like:
/collections/demo/ would behave as an integrated collection with endpoints like:
/collections/demo/tiles and /collections/demo/items?

Subordinate to that integrated collection could then be the members at API paths like:
/collections/demo-roads/ which would dereference to a dataset (landing page) for demo-roads?

Is this what you are getting at or am I misunderstanding?

[jerstlouis]

@dblodgett-usgs The first part of my comment we have implemented and used in many TIEs (but currently with slashes rather than dashes).
e.g.

• http://maps.ecere.com/geoapi/collections
• http://maps.ecere.com/geoapi/collections/vtp
• http://maps.ecere.com/geoapi/collections/vtp/Daraa2
• http://maps.ecere.com/geoapi/collections/vtp/Daraa2/AgricultureSrf

When requesting the JSON representation of these, the whole list of collections is returned, but the HTML representation only shows the immediate hierarchy level.

The latter part of my comment about the landing page at each level, I guess would imply combining those list of collections with the elements normally found in a landing page (e.g. having an api description, in addition to a list of collections and/or tiles & items)