尝试列出大目录时 429 Too Many Requests

[shniubobo]

问题描述

有许多类似 issue 提到过 429 相关的问题，比如 #121 和 #805，但是并没有从根本上解决问题。

通过 WebDAV 列出目录时整个目录都在一个请求里返回，对应向阿里云盘发出的多个 HTTP 请求，所以客户端无法做 rate limit，只能靠 aliyundrive-webdav 自己限制。然而目前 aliyundrive-webdav 仅进行 3 次重试，且重试间隔极短（如所附日志所示）。另外重试的次数和间隔也均为 hard-coded，无法通过命令行调整：

aliyundrive-webdav/src/drive/mod.rs

Lines 83 to 86 in 24925dd

	let retry_policy = ExponentialBackoff::builder()
	.backoff_exponent(2)
	.retry_bounds(Duration::from_millis(100), Duration::from_secs(5))
	.build_with_max_retries(3);

希望能做出以下改进：

• 添加命令行选项，比如 --retries 和 --retries-interval
• 增加 rate limit 机制，可以尝试这个库，或参考这个 issue
• 识别阿里云盘返回的 header 以调整重试时间

重现步骤

1. 启动 aliyundrive-webdav
2. 用任意 WebDAV 客户端尝试列出一个大目录

版本

2.3.3

运行平台

Windows

日志

[...]
WARN reqwest_retry::middleware: Retry attempt #0. Sleeping 128.045104ms before the next attempt
WARN reqwest_retry::middleware: Retry attempt #1. Sleeping 110.082827ms before the next attempt
WARN reqwest_retry::middleware: Retry attempt #2. Sleeping 677.208296ms before the next attempt
WARN reqwest_retry::middleware: Retry attempt #0. Sleeping 145.568725ms before the next attempt
WARN reqwest_retry::middleware: Retry attempt #0. Sleeping 241.640289ms before the next attempt
WARN reqwest_retry::middleware: Retry attempt #0. Sleeping 195.823185ms before the next attempt
WARN reqwest_retry::middleware: Retry attempt #2. Sleeping 786.743604ms before the next attempt
WARN reqwest_retry::middleware: Retry attempt #1. Sleeping 374.866109ms before the next attempt
WARN reqwest_retry::middleware: Retry attempt #1. Sleeping 513.848439ms before the next attempt
WARN reqwest_retry::middleware: Retry attempt #0. Sleeping 199.09798ms before the next attempt
WARN reqwest_retry::middleware: Retry attempt #0. Sleeping 37.960208ms before the next attempt
WARN reqwest_retry::middleware: Retry attempt #2. Sleeping 192.46117ms before the next attempt
ERROR aliyundrive_webdav::vfs: list_files_and_cache failed path=[REDACTED] error=HTTP status client error (429 Too Many Requests) for url (https://openapi.aliyundrive.com/adrive/v1.0/openFile/list)
ERROR dav_server::handle_props: read_dir error GeneralFailure
[...]

[chenqimiao]

其实最好的方式还是做好分片了，单个文件夹拆分成多个文件夹