Skip to content

[Feature]: Replace crossStorage with smarter auto-detection and clearer copyMode config #993

@huanghaoyuanhhy

Description

@huanghaoyuanhhy

Part of #637 (Overly Complex Configuration File)

Is your feature request related to a problem? Please describe.

The current minio.crossStorage configuration is confusing and hard to understand for users:

  1. The name describes the scenario, not the behavior. crossStorage sounds like "are you doing cross-storage backup?" but it actually controls how data is copied — via the storage system's native COPY API (false) or proxied through the backup server process (true). Users have to read a dedicated FAQ just to understand what it means (docs/user_guide/cross_storage.md).

  2. Manual configuration is needed for same-type-different-instance scenarios. Auto-detection (core/backup/task.go:111) only compares Provider — if both are minio but at different addresses (Minio A → Minio B), the user must manually set crossStorage: true. This is the most confusing case: "both are Minio, why do I need crossStorage?"

  3. Wrong config section. crossStorage is under the minio section, but it's not a MinIO setting — it's a backup behavior setting.

  4. The FAQ in the docs is itself evidence of a naming failure. If a config option needs a dedicated FAQ entry explaining "what does this mean", the name isn't working.

Describe the solution you'd like.

This is a breaking change proposal:

1. Smarter auto-detection (P0) — eliminate manual configuration

Extend auto-detection to compare address and port in addition to provider:

// current: only compares provider
if backupProvider != milvusProvider { copyByServer = true }

// proposed: also compare address/port
if backupProvider != milvusProvider ||
   backupAddress != milvusAddress ||
   backupPort != milvusPort {
    copyByServer = true
}

This covers the Minio A → Minio B scenario automatically. Users almost never need to set this manually.

2. Rename the config (P1) — name should describe behavior, not scenario

Replace minio.crossStorage with an enum-based copyMode:

backup:
  copyMode: "auto"     # auto-detect (default, recommended)
  # copyMode: "direct" # force storage COPY API
  # copyMode: "proxy"  # force server-side proxy transfer
  • auto: current implicit behavior, picks the right mode automatically
  • direct: uses storage's native COPY API (fast, requires both ends accessible to each other)
  • proxy: data flows through backup server (slower, works across any storage combination)

3. Move config from minio to backup section (P2)

crossStorage is not a storage configuration — it's a backup operation behavior. It belongs under the backup section.

4. Log the actual copy mode at task start (P3)

INFO  copy mode: proxy (auto-detected: backup and milvus storage have different addresses)
INFO  copy mode: direct (same storage, using native COPY API)

Describe an alternate solution.

A minimal non-breaking approach: keep crossStorage as-is but improve auto-detection (P0) so users rarely encounter it. Add deprecation warnings pointing to the new config name.

Anything else? (Additional Context)

Code references

  • Config definition: internal/cfg/cfg.go:234,277
  • Auto-detection: core/backup/task.go:110-113, core/check/task.go:139-150
  • Copier selection: internal/storage/copier.go:126-137
  • Restore skip logic: core/restore/coll_task.go:336-343
  • Backup copy: core/backup/coll_dml_task.go:474-483
  • User docs: docs/user_guide/cross_storage.md

Migration path

  • Support both old (minio.crossStorage) and new config for one release cycle with a deprecation warning
  • Remove the old config in the following release

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions