feat: retry with exponential backoff for DynamoDb interaction

[dispanser]

Description

We use an external crate, backoff, to retry DynamoDb read and write operations when the error in the response is ProvisionedThroughPutExceeded, indicating an overload of DynamoDb wrt to the configured read and write capacity.

[dispanser]

I decided against using the existing retry logic from crates/deltalake-core/src/data_catalog/client/retry.rs that was suggested by @roeap:

• with the current crate structure, it's not possible to access from deltalake-aws crate
• there's not much that can be directly reused, most of the logic resides in impl RetryExt for reqwest::RequestBuilder
• the model (extension trait RetryExt) isn't really viable because we're retrying four different operations, so we'd be duplicating the actual retry handling (loop, considering max_retries, actual sleeping, etc) multiple times
• I think the approach taking by backoff, marking errors as either transient or permanent, is self-documenting and easy to understand

@rtyler: We're currently only retrying ProvisionedThroughputExceeded, but for most of these error enums there's also the RequestLimitExceeded variant, documented via:

Throughput exceeds the current throughput quota for your account. Please contact AWS Support at AWS Support to request a quota increase

In theory, this indicates that the error is also transient, but I'm not an AWS expert enough to understand if it's sensible to handle this as well.

[rtyler]

I think pulling the backoff crate into deltalake-aws is reasonable. The nice thing about the subcrates is that this new dependency only affects AWS users 😄

[rtyler]

The max elapsed interval of 60 I am assuming is just a number that you've picked?

I think this is something that will likely need to be configurable since different systems would have a different tolerance here. Based on how retry is being invoked I don't see a clear path to pulling configuration down into this call 🤔

[dispanser]

Yes, these numbers would ideally be configurable. It seems other aspects of delta-rs, including the locking provider choice itself, are being configured via environment variables, so I guess that's the way to go.

My question would mostly be if we want all five of possible parameters to be configurable:

• max elapsed time
• max interval
• multiplier
• initial interval (using defaults in this PR)
• randomization factor (using defaults in this PR)

The most obvious one is probably the one you singled out, max elapsed time, but there's probably a user out there for any of these :).

[rtyler]

@dispanser I agree there are folks likely that would want to tune every single one of these...maybe.

I think making an environment variable for max elapsed time is the only one really important to implement to merge this. The others people may want, or not, but they should make themselves known in the future 😄

[dispanser]

I added an environment variable for the max elapsed time setting.

I'd address the diverging docs in a separate PR, they are already way off anyways :)

[dispanser]

I think the failing test (macos, restore_by_datetime) is unrelated.

[rtyler]

@dispanser please rebase this on the latest main with the mega refactor 😄

[dispanser]

@rtyler rebase is done, all tests pass locally.