api: add possibility to delete NAMESPACES

[mobuchowski]

Building on top of possibility added in #2032 - add endpoint to soft delete everything in particular dataset - both jobs and datasets.

[mobuchowski]

Few thoughts about deleting namespaces:

Deleting namespace can be thought as a two operations from UI side.

• All jobs and datasets belonging to the namespace are deleted.
• After delete we can’t see the deleted namespace in Marquez namespace dropdown.

That leaves us with few challenges if we want to have both.

• How would deleted namespace interact with receiving new event for the same namespace?
  • Should the namespace be back in namespace list?
    • This would require event to check and clear potential flag on namespaces.
  • Should we “undelete” the namespace with all the resources?
    • Naive is_hidden flag would work exactly like that. Maybe better idea is having all is_hidden flags - on jobs, datasets and namespaces set. If event clears namespace flag, it does not clear all flags for jobs and namespaces.
    • Having flags require us to change a lot of logic related to namespaces. There is a lot throwIfNotExists(namespaceName) logic on API requests.
    • Alternative to flags could be logic that shows on the UI only namespaces with some existing jobs or datasets - checking count on every GET /namespace call. That is not consistent with default namespace though.
  • Should we only show the jobs/datasets that were created by events received after deletion?
    • In this case, should there be any undelete logic for the whole namespace?
• Should behavior of deleted namespace, and namespace from which all the datasets and jobs were deleted be different?
• Theoretically simplest approach, hard delete namespaces would require multiple cascades - everything references namespaces.

[howardyoo]

My two-cents:

• Should the namespace be back in namespace list? --> Yes
• Should we “undelete” the namespace with all the resources --> Yes
• Should we only show the jobs/datasets that were created by events received after deletion? --> Yes

[mobuchowski]

Should we “undelete” the namespace with all the resources --> Yes
Should we only show the jobs/datasets that were created by events received after deletion? --> Yes

But those conflict 😉

[howardyoo]

Should we “undelete” the namespace with all the resources --> Yes
Should we only show the jobs/datasets that were created by events received after deletion? --> Yes

But those conflict 😉

So, when we're undeleting, we should keep track of some sort of 'deleted' version, such that only the 'latest' deleted jobs / datasets should be undeleted. How about that?

[julienledem]

I agree that the namespace should be marked soft deleted as well. It does not show up in the list of existing namespaces but we can retrieve it.

I think we should adopt the following behavior: Anywhere we would have created a resource (Job, Dataset, Namespace) that didn't exist yet and there was previously an existing but (soft) deleted one we should just undelete it and update it.
Example:

• when receiving an OL event for a job or dataset that don't exist yet.
• when creating a Dataset or Job in a namespace that doesn't exist yet (from OL event).

In this context a NAMESPACE is either:

• a datasource, the namespace for datasets. This means we deleted a database instance for example and all datasets it contains get (soft) deleted. When new events come in we have to assume it is an empty instance with the same name. If the same datasets exist in it, they will eventually be observed.
• a job scheduler, the namespace for jobs. This means we deleted a scheduler instance and all jobs it runs get soft deleted. If a new scheduler comes in with the same name, the jobs will be observed again.

In both cases, if new events come in with those namespaces that have been deleted previously, we should, behave like we would for a new job or dataset. The namespace gets created automatically (or in this case we just remove the soft delete).

As a follow up, do we need a parameter on GET that allows seeing soft deletes?
GET /api/v1/namespaces/foo/datasets?include_deleted=true

[mobuchowski]

This bug blocks it: #2179

Need to fix it first before doing this.