Add bulk dataset storage migration

[davelopez]

Resolves #22404

Summary

This branch introduces end-to-end bulk storage migration for history items.

Users can now:

• Start a bulk Move operation for selected history datasets and collections.
• Choose a target storage location.
• Preview eligibility before execution, including:
  • eligible vs ineligible items,
  • estimated transfer size,
  • quota delta per source/target,
  • warnings (for example, moving from private to shareable storage).
• Execute the operation and track progress through a dedicated run view (in-progress and finished runs, per-run details, per-item outcomes).
• Receive completion notifications (optional at submission time, with category-level notification preferences).

Notes:

• Only "Move" operations are supported for now.
• Runs include detailed terminal outcomes (succeeded, failed, skipped) with reason codes.

Screenshots

New bulk selection option (requires celery and selectable object stores)

For potentially long operations, you can opt to receive a notification/email.

Select the destination storage location for the selected datasets and collections

You can preview the estimated impact of the operation

When a background operation is running, you will see this indicator

Here, you can inspect the latest operations you have run in the current history

When some datasets are skipped or fail, you can see the reason for each of them in the details

You can also access the results from the notification received after the operation is completed

Most Relevant Backend Changes

• Added persistent tracking models for storage operations:
  • snapshot table (preview snapshot and expiry),
  • run table (operation lifecycle and aggregate counters),
  • run item table (per-dataset state/reason/message/bytes).
• Added a new migration to create these tables and relationships.
• Added history content API endpoints for:
  • previewing a storage operation,
  • executing from a preview snapshot,
  • retrieving run summary,
  • retrieving paginated/searchable run items.
• Extended storage operation schemas with typed request/response models, run states, item states, and failure reason codes.
• Implemented backend execution flow for bulk storage operations, including async run tracking and robust item-level result reporting.
• Integrated storage-operation completion notifications, including:
  • new notification category and payload shape,
  • email templates for completion messaging.

Testing

Coverage added/updated in this branch includes:

• Integration tests for bulk storage operations, covering:
  • preview eligibility behavior,
  • invalid/edge reason codes,
  • execution lifecycle and terminal run states,
  • skip_ineligible behavior,
  • per-item status/results,
  • run-items filtering/search,
  • snapshot/history mismatch errors,
  • collection (HDCA) expansion/execution,
  • dataset-not-found between preview and execute.
• Unit tests for notification manager behavior for storage operation completion notifications.
• Frontend/store/query/component tests updated for the new storage operation flows and notification handling.

TODO

• Automatically prune the database tables for storage operations (snapshot, run, run item) after a reasonable retention period

How to test the changes?

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:

  • This is a basic object_store_conf.yml. You should replace the files_dir with something that works for you and play around with different device and quota, etc.

    type: distributed
    backends:
        - id: default
          type: disk
          device: device1
          weight: 1
          allow_selection: true
          name: Default Galaxy Storage
          description: >
              Some description for Default Galaxy Storage
          files_dir: /home/dlopez/sandbox/data-gx/dev/objects/default
          badges:
              - type: slower
              - type: more_stable
              - type: backed_up
                message: >
                    Backed up to Galaxy's institutional long term tape drive nightly. More information about our tape drive can be
                    found on our [Archive Tier Storage](https://www.msi.umn.edu/content/archive-tier-storage) page.
        - id: second_tier
          type: disk
          device: device1
          weight: 0
          allow_selection: true
          name: Second Tier Storage
          description: >
              This object store is connected in the same disk as the default object store.
          quota:
              source: second_tier
          files_dir: /home/dlopez/sandbox/data-gx/dev/objects/second_tier
          badges:
              - type: slower
              - type: more_stable
              - type: backed_up
                message: >
                    Backed up to Galaxy's institutional long term tape drive nightly. More information about our tape drive can be
                    found on our [Archive Tier Storage](https://www.msi.umn.edu/content/archive-tier-storage) page.
        - id: scratch
          type: disk
          device: device2
          weight: 0
          allow_selection: true
          private: true
          name: Scratch Storage
          description: >
              This object store is connected to institutional scratch storage. This disk space is not backed up and private to
              your user, and datasets belonging to this storage will be automatically deleted after one month.
          quota:
              source: scratch
          files_dir: /home/dlopez/sandbox/data-gx/dev/objects/temp
          badges:
              - type: faster
              - type: less_stable
              - type: not_backed_up
              - type: short_term
                message: The data stored here is purged after a month.
          object_expires_after_days: 30
    

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.