enhance: introduce intermediate checkpoint state to minimize recovery time

[tinswzy]

In the current recovery flow using MinIO as the storage backend, fencing and finalization require scanning and processing all blocks of a segment to ensure there are no holes in the block sequence and to reconstruct the sparse index.

When a segment contains close to 1000 blocks and object storage operations are relatively slow (e.g. some cloud object stores), the recovery cost can be significant:

• Each S3 read costs ~20–80 ms
• 1000 blocks ⇒ 20–80 seconds just for fence & finalize
• This makes recovery time scale linearly with block count
• Worst case: recovery becomes the dominant latency during restart or failover

This is especially problematic for:

• Cloud object storage (S3 / OSS / GCS)
• Large or long-lived segments
• Frequent restarts or crash recovery scenarios

[tinswzy]

Now the numbers are apples-to-apples：

Recovery at 35ms is right in the ~10-block ballpark (19.9ms baseline for 10 blocks, plus the overhead of reading and parsing checkpoint.blk itself). It's 48x faster than a full 909-block scan at 1.67s.
If it is object storage on the cloud, this optimization effect is considerable.

[xiaofan-luan]

can we do parallel read?
say read 5 blobs concurrently to reduce failure recovery time.

[tinswzy]

can we do parallel read? say read 5 blobs concurrently to reduce failure recovery time.

Sure, but the optimization time obtained by increasing concurrency still has a linear relationship with the total number of blks. checkpoint can ensure that recovery is completed within a time window of a constant coefficient, which is independent of the total number of blks in the segment.