Submitted count is incorrect when aborting a run

[bentsherman]

When a run is aborted (e.g. cancelling a run from Seqera Platform), all submitted/running tasks are killed, marked as "aborted", and the workflow stats are updated accordingly:

nextflow/modules/nextflow/src/main/groovy/nextflow/trace/WorkflowStats.groovy

Lines 348 to 358 in 99f9edf

	void markCompleted(TaskRun task, TraceRecord trace) {
	ProgressRecord state = getOrCreateRecord(task.processor)
	state.taskName = task.name
	state.hash = task.hashLog
	state.running --
	state.loadCpus -= task.getConfig().getCpus()
	state.loadMemory -= (task.getConfig().getMemory()?.toBytes() ?: 0)
	this.runningCount --
	this.loadCpus -= task.getConfig().getCpus()
	this.loadMemory -= (task.getConfig().getMemory()?.toBytes() ?: 0)

However, markCompleted always decrements the running count regardless of whether the completed task was running or submitted. If there are submitted tasks that were aborted, the submitted count will be too high and the running count will be too low (or even negative).

I see two solutions:

1. Include the previous state in the call to WorkflowStats::markCompleted() so that it knows which count to decrement
2. Add a function markAborted() and call it before updating the task state in TaskPollingMonitor, so that it can see the previous state.

[swingingsimian]

Along side this workflow level oddity, we are also seeing odd WorkflowProgress counts in the processes list. The same issue is prevalent with submitted, but also running count of -4?

processes:[TraceProgressDetail(index:1, name:sayHello, pending:0, submitted:4, running:-4, succeeded:0, cached:0, failed:0, aborted:4, stored:0, ignored:0, retries:0, terminated:false, loadCpus:-4, loadMemory:0, peakRunning:0, peakCpus:0, peakMemory:0, ...)]

TraceProgressDetail is our incoming request model for the nextflow ProgressRecord.

[bentsherman]

This is a side effect of decrementing the running count instead of submitted count:

If there are submitted tasks that were aborted, the submitted count will be too high and the running count will be too low (or even negative).

[swingingsimian]

@bentsherman FYI I just observed a similar effect with pending tasks, although extremely hard to replicate in a live environment as they are very short lived.