Feature Request: Have an ERS cooldown period a la RecoveryPeriodBlockSeconds of classical orchestrator

[ejortegau]

Feature Description

Under some scenarios (e.g. client app overloading a shard's primary), vtorc can detect a DeadPrimary when the host is not really dead, just overloaded. This triggers an ERS, after which the newly elected primary is also overloaded, which leads to another DeadPrimary, which leads to another ERS... in other words, we have a cascading failure that can remove capacity from the shard.

Classical orchestrator had RecoveryPeriodBlockSeconds to avoid this. We propose having the same for vtorc/ERS, with suitable changes:

• Since vtorc instances do not have a shared state, they need to figure out whether an ERS has taken place less than the cooldown time ago externally. We propose leveraging the reparenting journal for that.
• Given the above, we might as well put the logic outside of vtorc so it works not only for it but also for any externally triggered ERS.
• EmergencyReparentShardRequest would include an extra field that specifies an CoolDown.
• The ERS code would:
  • Fetch the last ERS reparenting journal entry of each tablet in the shard.
  • Find the most recent one.
  • Compare that with now - CoolDown and decide whether the ERS can proceed or not.

Use Case(s)

Avoid cascading failure scenarios lie the one described above.

[mhamza15]

I've found a need for this as well, but more specifically a cooldown after an attempted reparent, whether successful or not. The goal is to avoid a thundering herd of PRS/ERS, especially after a failure. Additionally, it can prevent attempting a reparent when buffering is still cooling down due to FailoverTooRecent.

I was thinking of storing a last attempted reparent time in the topology, like PrimaryTermStartTime is stored currently. Would "attempted reparent" cover your case as well? Or is there a specific need for an option for "last successful reparent time"? (In the successful case, we might be able to just use PrimaryTermStartTime instead of relying on the reparent journal? I'm not sure if there are edge cases where that can be out of sync and cause issues.)

[ejortegau]

Hi, @mhamza15:

Thanks for getting back to me on the FR. My comments/answers follow

Would "attempted reparent" cover your case as well?

That would depend:

• You mention PRS as well. I don't think we should block an ERS just because we recently ran a PRS (regardless of whether that failed or succeeded). At least for us, it is perfectly valid to try to PRS first and resort to ERS if that fails, within a relatively short amount of time.
• If it was only time of last attempted ERS, it might work.

I have this on the pipe for our fork which implements the idea referenced in the original RFC, any feedback on it would be welcome.