Operator doesn't handle errors well

[Smiley73]

What happens?

During the testing of the new examples, a couple of problems were observed:

• The operator sometimes gets stuck and doesn't start new jobs if the previous didn't complete successfully. A restart is required.
• When there are errors with the repo (no access, wrong branch, missing files, missing template image) and the config doesn't get applied, the GitOpsConfig CRs often can't be deleted afterwards. The Finalizer has to be disabled to remove it.
• Behavior affects other namespaces also.

Not sure if these are related. I'll keep documenting more behaviors as I do more testing.

What were you expecting to happen?

The Operator gracefully handles any issues, doesn't get "stuck" , and the CR can always be deleted without having to "patch" the resource.

Steps to reproduce:

• Use the examples and modify one of the CRs to not being able to pull the repo. Leave the trigger as change.
• Apply the CR and let it spin.
• Try deleting the CR
• Try applying an other CR in a different namespace

[Smiley73]

Here's some sample code for the finalizer: https://github.com/hossainemruz/k8s-initializer-finalizer-practice

[akavel]

With regards to "errors with the repo": I think fixing this part of the above issue is not sustainable with the current approach, which basically requires the .yaml files to be fetched correctly in order to be able to delete them, i.e. to even know what to delete:

eunomia/template-processors/base/bin/resourceManager.sh

Lines 32 to 36 in 12c2afb

	for file in $(find $MANIFEST_DIR -iregex '.*\.yaml'); do
	cat $file | yq 'select(.kind == "GitOpsConfig")' | kube delete -f - --wait=true
	done
	set +u
	kube delete -R -f $MANIFEST_DIR

So, when the git repo can't even be fetched, the current approach just doesn't have anything to grab onto.

My suggestion is to revisit this after #24. I believe adding labels to any pods created with eunomia should allow us to just confidently purge anything even remotely connected to a particular label when performing a delete. This would allow us to ignore any problems with git, or whatever else.

This might be somewhat tricky with regards to ResourceHandlingMode = "None" and ResourceDeletionMode = "None". However, I would suggest that we introduce a completely separate path for handling delete in template-processors, that would not require going through gitPull.sh etc. at all. Any "special" templating engine, which requires Mode = "None", would then have to override a separate "deleteResources.sh" script too.

@seanmalloy @Smiley73 what do you think? Does it make sense to you, or did I miss something? I'm still only learning eunomia...

[seanmalloy]

@akavel i'm find with waiting on this fix until #24 is implemented.