add DRU replace operation precisions

[fmigneault]

Context / Justification

Certain databases support "upsert" (update or insert). This change allows PUT to behave the same way, so this pattern can be easily integrated. Furthermore, certain backends DO NOT want to remove older processes for traceability, integrity, provenance, etc., or literally cannot do it (e.g.: blockchain). Therefore, the "replace" operation requirements are slightly loosened with a permission to replace only the "processID" reference rather than the entire "process definition" to offer flexibility in the underlying behaviour. Some recommendations are added to guide implementers toward best practices.

Changes

• add PUT replace "unsert" details
• add missing/alternate replace HTTP codes and body refs to OpenAPI
  (notably, adds the precision of "process summary" to align with the behaviour of "process deploy" response)
• add replace version link recommendation (RFC 5829)
  moved/backup to potential "versioning" part for future revision

References

• Fixes Allow Replace HTTP 201 response #573

[pvretano]

@fmigneault whoa! Version navigation!?

I thought you were going to make a small change to say that 201 can be returned in the case where the replace happens to create a new instance of a process because the server happens to support versioning but then also mention that this standard does not cover versioned processes. In other words accommodate the fact that your server returns 201 on a replace in some cases (so that a client can be prepared to accept that) but go no further.

As it stands I would not support merging this PR because these changes here also mean that a bunch of changes need to be made to Part 1 to describe how all operations are expected to behave when processes are versioned.

Even making those changes, that would constitute a HUGE technical change which means we would need to delay the RFC submission until we can get more than one implementation and test in code sprints or test beds.

[jerstlouis]

@pvretano It's all only recommendations though, right (except for the requirement about the content for a 200,201,202 response instead of 204, which says nothing about versioning)? The version navigation is just the RFC for link relations to use if the server does implement versionining? It doesn't seem like such a big technical change to me, and focused on accommodating that 201 response (with related recommendations).

[jerstlouis]

This is identified as a permission (firt part uses MAY) but includes a recommendation (SHOULD) inside for second part.

[fmigneault]

Is there actually something against that term? How to indicate something optional, but if you do, it should be done that way ideally? "SHOULD" is still optional / not prescriptive (unlike SHALL).

[fmigneault]

Otherwise transform it as a recommendation?

[jerstlouis]

@fmigneault Usually we try to strictly use SHALL in requirements, MAY in permission, and SHOULD in recommendations. One way I've done it e.g. in the Coverages issue we've been working on together lately is by having a separate recommendation in the context of (immediately following) the permission.

[fmigneault]

One issue is that I don't find it too appropriate to make it a recommendation and use SHOULD for the first part, since supporting "PUT as insert" is really an extra to the more classical HTTP 200 of inplace replacement.

[pvretano]

@jerstlouis aside form the fact that we should not be making any technical changes at this point ... especially ones that we have not really tested, @fmigneault has a normative reference to RFC 5829! That implies things about versioning that would need to be explained in Part 1 as well.

[jerstlouis]

@pvretano Normally should things referenced in recommendations only be included in the normative references, or can they just be in the bibliography?

[pvretano]

@jerstlouis if a requirement references the normative reference then it needs to be in the references. Otherwise bibliography.

@fmigneault have you consulted OGC API - Features - Part 4: Create, Replace, Update and Delete? That standard covers the case of creating a new feature instance using PUT and we should be doing it the same way here. Part 4, however, does not cover versionsing so its content may or may not be relevant here ... see @jerstlouis ... versioning has all kinds of hidden ramifications! ;)

[pvretano]

@jerstlouis does your server support versioning processes?

[pvretano]

@gfenoy does your server support versioning processes? Just trying to get a sense of who has an implementation of this.

[jerstlouis]

@pvretano My server doesn't even support DRU :)

if a requirement references the normative reference

The PR is only referencing the RFC in recommendations, so could the RFC mention be moved the bibliography, and would that address your concerns?

What I'm trying to say is that the versioning mention in this PR is just some good general recommendations about how one might integrate versioning, without any normative impact other than allowing 200,201,202 HTTP responses when deploying the process, in my interpretation/understanding.

[fmigneault]

@pvretano
I did not look at features, but I took the typical method of representing CRUD operations and, if versioning is involved, to provide IANA relation types that describe them. This is just a recommendation however, so users are still free to omit them entirely or employ another strategy if the want versioning. The RFC is provided mostly to give a context of what we are talking about.

The RFC should already be in the bibliography, or do you mean the "annex_bibliography" ?

[jerstlouis]

@fmigneault The RFC mention should probably be removed from the Section 3 "Normative References" (only for things referenced in requirements), and added to the annex_bibliography.

[pvretano]

@jerstlouis @fmigneault I think you guys are missing the point entirely. Simply mentioning versioning invites all kinds of questions that we are not answering right now. It is not enough to say "employ another strategy if the[y] want versioning". You need to explain how all operations behave in the content of versioning. So, does the version process appear at GET /processes? For @fmigneault the answer is yes. For someone else if might be no because versioning is simply a means of maintaining life-cycle history.

Yes, I know, it is all seems logical and straight forward but the moment you introduce the concept of versioning then you need to fully explain it everywhere!

@fmigneault can you make this change without needing to add the context of versioning? If yes, then great! If no, then we may have an issue here.

[jerstlouis]

@pvretano My assumption was that you would not get separate processes for each versions, but multiple links for each process for the different versions. Is that how it works @fmigneault ? So in that sense it would have minimum impact on Part 1 operations (other than additional links, that would likely also support execution as part of that "versioning extension").

But I get that versioning opens up a can of worms and that it might be best to tweak the requirements to support the use case without implying the versioning context specifically.

[pvretano]

@jerstlouis that would be my assumption too but we can't expect implementers to assume ... we need to be clear. I would say that versioning would require its own part to describe all the implications on all the other parts. We took a stab at versioning for features in the WFS standard and it was a mess. I am trying to avoid that here.

[fmigneault]

@jerstlouis
The way they are implemented on my end is that /processes will list only the un-versioned "latest" processes by default, and I have another flag ?revisions=true to list them all explicitly as processID:version. Any place that uses {processID} can use the explicit processID:version instead to describe/execute/undeploy that one specifically. In by DB backend, they are actual instances that live separately. The API acts by default as if versions do not exist. I consider that my own implementation detail, so it is not indicated in the proposed recommendations.

[jerstlouis]

@fmigneault

Maybe you could remove all recommendations, the RFC reference, and the by some versioning mechanism, and that would address the concerns, and this could be saved for a potential future "Versioning" part?

[fmigneault]

@pvretano
Yes. I can remove the /rec/deploy-replace-undeploy/replace-response-links entirely. It is supplementary to the general "PUT as insert" which could be addressed in another part separately.

[pvretano]

@fmigneault thanks! Apologies for being such a stickler about this. As I mentioned above we went down the versioning road with WFS and it was not a good experience. I am trying to avoid that here ...

[fmigneault]

@pvretano No problem :) I agree that more clarifications are needed and a dedicated part for it would be better.

[gfenoy]

@pvretano, our implementation does not support versioning yet.

I agree with the proposed direction to include this in an extension.