Skip to content

Commit 20e3aa3

Browse files
committed
Document --max-shard-size-bytes support for shards larger than 80 GiB
- Update is-migration-assistant-right-for-you.md to note larger shards can be configured and link to configuration options - Add 'Configuring large shard support' section to configuration-options.md with --max-shard-size-bytes usage and disk space requirements - Add troubleshooting entry to backfill.md for shards that appear stuck due to exceeding the default size limit Signed-off-by: Brian Presley <bjpres@amazon.com>
1 parent ca18d5d commit 20e3aa3

File tree

3 files changed

+17
-1
lines changed

3 files changed

+17
-1
lines changed

_migration-assistant/is-migration-assistant-right-for-you.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -161,7 +161,7 @@ To use `Reindex-from-Snapshot` (RFS), ensure the following:
161161
- If you choose to bring your own snapshot (that is, one not created by Migration Assistant), the following settings must be applied when creating the snapshot:
162162
- `include_global_state: true` – Ensures that global cluster state is included.
163163
- `compress: false` – Disables metadata compression, which is required for compatibility with RFS.
164-
- Shards of up to **80 GiB** are supported by default. Larger shard sizes can be configured, **except in AWS GovCloud (US)**, where 80 GiB is the maximum.
164+
- Shards of up to **80 GiB** are supported by default. Larger shard sizes can be configured. For details, see [Backfill migration using RFS]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/configuration-options/#backfill-migration-using-rfs). **In AWS GovCloud (US)**, 80 GiB is the maximum supported shard size.
165165
- In OpenSearch 2.9 and later, snapshots of indexes that use the zstd or zstd_no_dict codecs are not supported. If you need to migrate these indexes using `Reindex-from-Snapshot`, you must first reindex them on the source cluster using either `default` or `best_compression` before creating a new snapshot for use with RFS.
166166

167167
### Capture and Replay

_migration-assistant/migration-phases/backfill.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,12 @@ Migration Assistant creates an Amazon CloudWatch dashboard, named `MigrationAssi
124124

125125
You can find the backfill dashboard in the CloudWatch console based on the AWS Region in which you have deployed Migration Assistant. The metric graphs for your target cluster will be blank until you select the OpenSearch domain you're migrating to from the dropdown menu at the top of the dashboard.
126126

127+
## Troubleshooting
128+
129+
### Shards appear stuck with no errors
130+
131+
If `console backfill status --deep-check` shows shards that remain in progress indefinitely with no errors in the logs, the shard may exceed the default **80 GiB** size limit. Shards larger than this limit are silently rejected by RFS workers and will never complete. To resolve this, increase the `--max-shard-size-bytes` value in your deployment configuration. For details, see [Configuring large shard support]({{site.url}}{{site.baseurl}}/migration-assistant/migration-phases/deploy/configuration-options/#configuring-large-shard-support).
132+
127133
## Validating the backfill
128134

129135
After the backfill is complete and the workers have stopped, examine the contents of your cluster using the [Refresh API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/refresh/) and the [Flush API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/flush/). The following example uses the console CLI with the Refresh API to check the backfill status:

_migration-assistant/migration-phases/deploy/configuration-options.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,16 @@ The RFS configuration uses the following options. All options are optional.
8080

8181
To view all available arguments for `reindexFromSnapshotExtraArgs`, see [Snapshot migrations README](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments). At a minimum, no extra arguments may be needed.
8282

83+
### Configuring large shard support
84+
85+
By default, RFS supports shards of up to **80 GiB**. To migrate larger shards, pass the `--max-shard-size-bytes` flag through `reindexFromSnapshotExtraArgs`. For example, to support shards up to 200 GiB:
86+
87+
```json
88+
"reindexFromSnapshotExtraArgs": "--max-shard-size-bytes 200000000000"
89+
```
90+
91+
Ensure that your worker nodes have sufficient local disk space, because RFS requires approximately **2x the shard size** in local storage to unpack and process the Lucene index. For more information about available RFS arguments, see the [DocumentsFromSnapshotMigration README](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments).
92+
8393
## Live capture migration with C&R
8494

8595
The following sample CDK performs a live capture migration with C&R:

0 commit comments

Comments
 (0)