--- title: "Migrate from S3 to VAST S3" slug: "migrate-from-s3-to-vast-s3-34" updated: 2026-10-24T13:46:10Z published: 2026-10-23T15:23:22Z canonical: "kb.vastdata.com/migrate-from-s3-to-vast-s3-34" --- > ## Documentation Index > Fetch the complete documentation index at: https://kb.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # Migrate from S3 to VAST S3 ## Overview SyncEngine features an S3 connector that lets you copy objects from any S3-compatible store (AWS S3, MinIO, IBM COS and other) into a VAST S3 bucket. The following capabilities are supported for S3-to-VAST S3 migrations: - To manage the destination: - [Auto-Create Destination Bucket](/documentation/docs/migrate-from-s3-to-vast-s3-34#autocreate-destination-bucket) - To determine when data synchronization occurs: - [Periodic Resync](/documentation/docs/migrate-from-s3-to-vast-s3-34#periodic-resync) - [Event-Driven Resync](/documentation/docs/migrate-from-s3-to-vast-s3-34#eventdriven-resync) - To manage which objects are being synchronized: - [Skip Existing Objects](/documentation/docs/migrate-from-s3-to-vast-s3-34#skip-existing-objects) - [Include List](/documentation/docs/migrate-from-s3-to-vast-s3-34#include-list) - [Mirror Deletions](/documentation/docs/migrate-from-s3-to-vast-s3-34#mirror-deletions) - To determine what is synchronized per object: - [Copy Object Tags](/documentation/docs/migrate-from-s3-to-vast-s3-34#copy-object-tags) - [Preserve object last-modified date and time](/documentation/docs/migrate-from-s3-to-vast-s3-34#preserve-object-lastmodified-date-and-time) - To configure SyncEngine processes: - [Source S3 Rate Limits](/documentation/docs/migrate-from-s3-to-vast-s3-34#source-s3-rate-limits) Complete these steps to set up migration from an S3 store to a VAST S3 bucket: 1. Ensure that the [prerequisites](/documentation/docs/migrate-from-s3-to-vast-s3-34#before-you-begin) are met. Note that for S3-to-S3 migration, it is recommended that you manually create the destination bucket on the VAST cluster prior to starting the migration job. 2. [Create](/documentation/docs/migrate-from-s3-to-vast-s3-34#create-s3-connector) a connector for the source S3 store. 3. [Create](/documentation/docs/migrate-from-s3-to-vast-s3-34#create-vast-s3-connector) a connector for VAST S3. 4. [Create](/documentation/docs/migrate-from-s3-to-vast-s3-34#create-migration) a migration from the source S3 store to the destination VAST S3 bucket. 5. [Start](/documentation/docs/migrate-from-s3-to-vast-s3-34#start-migration) the migration. > **Note:** This step is only required when the migration does not have periodic resynchronization enabled. 6. [Monitor](/documentation/docs/migrate-from-s3-to-vast-s3-34#monitor-migration) migration progress and completion. 7. If necessary, [troubleshoot](/documentation/docs/migrate-from-s3-to-vast-s3-34#troubleshoot). The following limitations apply: - S3 objects can only be migrated to a VAST S3 bucket. No other destination types are supported. - Only general-purpose S3 buckets are supported as the migration source. S3 Express One Zone directory buckets are not supported. - One source bucket is allowed per migration. If you need to migrate multiple buckets, multiple migrations are to be set up. - S3 ACLs are not preserved. For objects in the destination VAST S3 bucket, access control is to be implemented through identity or bucket policies. - S3 versioning is not carried over to the destination bucket. - When using an include list, the migration must be configured to specify the source and destination paths as bucket names only, without any object prefixes. - Event-driven resynchronization cannot be used together with an include list. ## Features ### Auto-Create Destination Bucket For S3 to VAST S3 migrations, it is recommended to create the destination bucket **manually** prior to running the migration. If the destination bucket is not created, SyncEngine will create one when you submit the migration job. ### Periodic Resync In addition to starting a migration once it is created, SyncEngine can be configured to automatically run a resynchronization at user-supplied intervals. - In SyncEngine UI, set the **SyncMode** to **Automatic** and enter an interval (in hours) in the **Sync Interval** field. - In SyncEngine CLI, include the `--sync-mode automatic=true` and `--sync-interval <interval>` options when submitting the `migration create` command. For migrations set to **Automatic** synchronization mode, the first synchronization starts immediately after the migration is created, and the subsequent synchronizations are performed periodically with the specified interval. A subsequent synchronization cannot be triggered manually. To be able to manually start a synchronization on demand, set **SyncMode** to **Manual**. ### Event-Driven Resync SyncEngine can run incremental resynchronization between the source and destination buckets based on source bucket events published on a Simple Queue Service (SQS) queue. > **Note:** This feature is available for AWS S3 source buckets only. To configure incremental synchronization triggered by SQS queue events: - Specify the SQS queue URL and ARN when creating a migration: - In SyncEngine UI, fill in the **SQS Queue URL** and **SQS Queue ARN** fields in migration settings. - In SyncEngine CLI, include the `--event-queue-url <URL>` and `--event-queue-arn <ARN>` options when submitting the `migration create` command. - Ensure that the SQS queue is configured to receive event notifications from the source bucket. - Ensure that the SyncEngine worker is allowed to perform `sqs:ReceiveMessage` and `sqs:DeleteMessage` actions for the SQS queue. > **Note:** Pruned or missed queue events result in objects not being synchronized until the next full scan of the source bucket. The following limitation applies: - Event-driven incremental synchronization cannot be used together with an include list. ### Skip Existing Objects By default, if the destination VAST S3 bucket contains an object with the same key as the object being migrated, SyncEngine overwrites the existing object with the new data from the source S3 store. You can configure SyncEngine to skip objects that already exist in the destination bucket so that they do not get overwritten from the source. To do so: - In SyncEngine UI, select the **Skip object keys that already exist at destination** option in migration settings. - In SyncEngine CLI, include the `--ignore-existing` option on the `migration create` command. ### Include List You can supply a list of specific object keys to be migrated. If an include list is uploaded, SyncEngine does not scan the source bucket for objects subject to migration; it only copies the objects specified in the list. Each object key in an include list must be listed on a separate line in the format: ``` ,, ``` For example: ``` https://s3.amazonaws.com,my-bucket,path/to/file1.txt https://s3.amazonaws.com,my-bucket,path/to/file2.txt https://s3.amazonaws.com,my-bucket,path/to/file3.txt ... ``` Lines starting with `#` are ignored. To supply an include list: - In SyncEngine UI, upload the include list file in the **Inclusion list** field when creating a migration. - In SyncEngine CLI: - Include the `--inclusion-file <file path>` option on the `migration create` command, or - Run this command after the migration is created: ``` migration inclusion-file upload --migration-id= ``` The following limitations apply: - When using an include list, the migration must be configured to specify the source and destination paths as bucket names only, without any object prefixes. - An include list cannot be used together with event-driven resynchronization. ### Mirror Deletions You can determine how SyncEngine handles the items in the source store that have been deleted since the previous migration run. By default, SyncEngine does not delete such items from the destination. If you want to have them deleted, configure the migration to mirror source deletions at the destination: - In SyncEngine UI, select the **Mirror source deletions at destination** option when creating a migration. - In SyncEngine CLI, include the `--enable-deletes=true` option when submitting the `migration create` command. > **Note:** Deletion of items at the destination cannot be undone by disabling this setting. ### Copy Object Tags By default, SyncEngine copies object tags from the source bucket. You can disable this feature in case it impacts performance. - In SyncEngine UI, unselect the **Copy S3 object tags** option when creating a migration. - In SyncEngine CLI, include the `--copy-tags=false` option when submitting the `migration create` command. Copying of object tags requires that the `s3:GetObjectTagging` action is allowed for the worker on the source bucket. ### Preserve Objects' Last-Modified Date and Time When used with VAST Cluster 5.5.0 or later, SyncEngine can preserve the `Last-Modified` date and time of objects being migrated so that after the migration, this attribute still points to the last modification made by a user to the object in the source bucket. If this feature is disabled, the `Last-Modified` attribute is set to the date and time when the object was copied to the destination VAST S3 bucket. When migrating to a VAST cluster version prior to 5.5.0, the `Last-Modified` attribute cannot be preserved. Use the following options to enable or disable preserving of objects' `Last-Modified` date and time: - In SyncEngine UI, select or unselect the **Stamp source last-modified via request header** checkbox when creating a migration. - In SyncEngine CLI, include the `--src-last-modified-header={true|false}` option when submitting the `migration create` command. When preserving of source `Last-Modified` date and time is enabled and copying of S3 object tags is disabled, objects that have the same `Last-Modified` timestamp in source and destination buckets are skipped during automatic resynchronization, which helps improve performance. ### Source S3 Rate Limits You can set request rate limits to prevent SyncEngine from causing excessive load on the source S3 store: - Cap the overall number of requests (per second) that SyncEngine sends to the source API. - Restrict throughput when getting objects from the source S3 store. Each of the limits applies to the aggregate amount of requests or throughput by all workers used by the connector. Each limit is enforced independently of the other limit. On reaching a limit, workers wait and retry so that synchronization continues after the pause. By default, no request rate limits are set. Set rate limits if during a representative migration, vendor 429/503/SlowDown errors appear in the worker logs, and the Grafana **Connector Rate Limits** panel (or the Prometheus metrics `connector_rate_limit_waits_total{connector_id, axis}`) shows the wait counter growing. If no throttling and no vendor errors occur, you do not need to set rate limits. Consider setting rate links when you expect many workers to scan or copy the same prefix. For recommended request rates for AWS S3, see [AWS S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html). Typical rate limits you may need when migrating from AWS S3 to VAST S3 are 1000–3000 API calls per second, with the throughput based on your link capacity. To set S3 rate limits: - In SyncEngine UI, use the **API calls per second** and **Bytes per second** fields in the S3 connector settings. > **Note:** Connector rate limits can only be set in SyncEngine UI. Avoid configuring multiple connectors for the same source S3 store account with rate limits that, when combined, exceed the maximum allowed request rate for the source S3 store API. Use a single connector instead. ## Before You Begin Ensure that the prerequisites are met: ### Source S3 Store - S3 store endpoint URL, region (required for AWS S3 buckets only), source bucket name and, optionally, object prefix. - Access keys of the user with read access (ListBucket, GetObject, GetObjectTagging) to the bucket to be migrated. - If you are going to use [event-driven resynchronization](/documentation/docs/migrate-from-s3-to-vast-s3-34#eventdriven-resync), an SQS queue configured to receive event notifications from the source bucket. ### VAST Cluster #### Networking - VAST cluster's management IP. - A DNS-resolvable name of the tenant's virtual IP pool to access the VAST S3 endpoint, or a virtual IPv4 address from that pool. - Network connectivity from workers to both source S3 endpoint and destination VAST S3 endpoint. #### VAST Tenant and Admin Credentials - If your VAST cluster has multiple VAST tenant configured: - To migrate to one of the tenants: - The name of the VAST tenant where the destination VAST S3 bucket will reside - Username and password for the VAST tenant admin user - To migrate to the default tenant: - Username and password for the VAST cluster admin user - If your VAST cluster is not configured for multi-tenancy: - Username and password for the VAST cluster admin user #### Create User Create a user with S3 access keys allowing to create a bucket and put objects to it. 1. Create a local user on the VAST provider attached to the tenant where the destination VAST S3 bucket will reside. Ensure that the user has the *Allow Bucket Creation* option enabled. 2. Generate S3 access key for the user. 3. Create an identity policy to grant the user full access to the destination bucket (`"Action": "s3:*", "Effect": "Allow"`). 4. Assign the identity policy to the user. Note the username and UID of the created user. #### Create View Policy Create a view policy that will control access to the destination VAST S3 view. In the view policy: - Select the **S3 Native** security flavor. - Set **Name length limit** to **Native Protocol Limit**. - Set **Allowed Characters** to **Native Protocol Limit**. #### Create View Create an Element Store view exposing the destination VAST S3 bucket. In the view settings: - Select the **S3 Bucket** protocol. - Select the view policy you created. - Select the user you created as the bucket owner. ### SyncEngine - The SyncEngine control plane and at least one worker are up and running. - At least one worker label of a deployed worker is known. (The worker label is set when you deploy the worker.) - The SyncEngine worker must have `s3:ListBucket` and `s3:GetObject` actions allowed for the source bucket. If [copying of object tags](/documentation/docs/migrate-from-s3-to-vast-s3-34#copy-object-tags) is enabled, the `s3:GetObjectTagging` action must also be allowed. If you are going to use [event-driven synchronization](/documentation/docs/migrate-from-s3-to-vast-s3-34#eventdriven-resync), the worker must also have `sqs:ReceiveMessage` and `sqs:DeleteMessage` actions allowed for the SQS queue. ## Create S3 Connector An S3 connector is used to connect to the S3 store that holds the data to be migrated. ### SyncEngine UI 1. In the **Connectors** page, click **New Connector**. A wizard dialog appears to walk you through the connector creation steps. 2. In the **Type & name** step: - Select the **S3** connector type. - In the **Connector name** field, enter a unique name for the connector. The name can contain up to 63 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-) and underscores (_). 3. In the **Credentials** step, provide information for SyncEngine to be able to retrieve the data from the source bucket: | Field | Description | | --- | --- | | **Access Key** | S3 access key of the user with read access to the source bucket. | | **Secret Key** | Private key of the user with read access to the source bucket. | | **Region** | If the source bucket is on AWS S3, the region of the source bucket. Default is `us-east-1`. | | **Endpoint URL** | The URL of the S3 endpoint where the source bucket resides. For AWS S3, leave this field blank. | 4. In the **Compatibility & Timeouts** step: - Under **Bucket & Metadata Compatibility**, make settings to align metadata conventions between the two source and destination buckets. | Field | Description | | --- | --- | | **Normalize metadata keys to lowercase** | When enabled, SyncEngine converts metadata keys to lower case (making them compatible with AWS S3). By default, no normalization is performed. | | **Case-sensitive bucket names** | When enabled, SyncEngine allows for bucket names that contain both lowercase and uppercase characters. Select this option when migrating from a non-AWS S3 store. | - Under **Timeout & Retry Settings**, optionally fine-tune migration task timeouts: | Field | Description | | --- | --- | | **Read Timeout** | Socket read timeout, in seconds. Valid values: 60–3600. Default is 300. | | **Connect Timeout** | S3 endpoint connection timeout, in seconds. Valid values: 5–300. Default is 60. | | **Max Retries** | The maximum allowed number of retries by the VAST S3 client. Valid values: 1–100. Default is 3. | | **Retry Mode** | Select one of VAST S3 client retry modes: **Adaptive** (default) for rate-limiting of requests when throttled, **Standard** for exponential backoff with jitter, **Legacy** for simple exponential backoff. | 5. In the **SSL** step, use the **SSL Certificate Content** field to enter the TLS certificate (in PEM format) to secure the S3 connection. If left blank, default CA certificates are used. 6. In the **Rate limits** step, set optional [request rate limits](/documentation/docs/migrate-from-s3-to-vast-s3-34#source-s3-rate-limits) for the connector: | Field | Description | | --- | --- | | **API calls per second** | Maximum allowed number of API calls per second. Leave blank for unlimited (default). | | **Bytes per second** | Maximum allowed download throughput. Leave blank for unlimited (default). | 7. In the **Review** step, review the settings you've made. If you want to make changes, click **Back**. Otherwise, click **Create Connector**. The connector is created and displayed in the **Connectors** page. ### SyncEngine CLI The following command creates an S3 connector named `src-S3` to migrate from an AWS S3 bucket in region `us-west-2`: ``` syncengine-cli connector create \ --type=s3 \ --name=src-s3 \ --access-key= \ --secret-key= \ --region=us-west-2 \ ``` Optional flags: ``` --endpoint # Required for non-AWS S3 stores --lowercase-metadata-keys={true|false} --case-sensitive-bucket-names={true|false} --read-timeout --client-connect-timeout --client-retries-max-attempts --client-retries-mode --ssl-cert-file ``` The output should be similar to the following: ``` Successfully created connector 'src-s3' with ID 15 ``` > **Tip:** Note the connector ID in the output. ## Create VAST S3 Connector A VAST S3 connector is used to connect to the VAST S3 server on the VAST cluster to which you want to migrate the data. ### SyncEngine UI 1. In the **Connectors** page, click **New Connector**. A wizard dialog appears to walk you through the connector creation steps. 2. In the **Type & name** step: - Select the **Vast S3** connector type. - In the **Connector name** field, enter a unique name for the connector. The name can contain up to 63 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-) and underscores (_). 3. In the **VMS cluster** step, enter VAST cluster's VMS IP and admin credentials: | Field | Description | | --- | --- | | **Management IP** | VAST cluster's management IPv4 address or DNS-resolvable hostname (up to 253 characters). | | **Username** and **Password** | Username and password to log in to the VMS. If the destination bucket is on one of VAST cluster tenants, enter credentials of the admin user for that tenant. If the destination bucket is on the default tenant, enter credentials of the VAST cluster admin user. | | **Tenant name** | The name of the VAST cluster tenant where the destination bucket resides. If not specified, the default tenant is used. | 4. In the **S3 endpoint** step, configure an S3 endpoint on the VAST cluster. | Field | Description | | --- | --- | | **VIP** | A DNS-resolvable name of the tenant's virtual IP pool to access the VAST S3 endpoint, or a virtual IPv4 address from that pool. A DNS-resolvable name is recommended. | | **Access Key** | Enter the S3 access key for the bucket owner user. | | **Secret Key** | Enter the private key for the bucket owner user. | | **S3 Username** | Enter the username for the bucket owner user. | | **S3 UID** | The UID of the bucket owner user. | | **Endpoint URL** | The URL to access the S3 endpoint. This field is optional. If not specified, the URL is derived from the **VIP** field. | | **Use secure S3 connection (HTTPS)** | Select to use HTTPS. If selected, you'll need to provide a TLS certificate in a next step. If not selected (default), HTTP is used. | 5. In the **Bucket & advanced** step, optionally configure advanced settings for the destination S3 bucket: | Field | Description | | --- | --- | | **Case-sensitive bucket names** | When enabled, SyncEngine allows for bucket names that contain both lowercase and uppercase characters. | | **Checksum Algorithm** | Checksum algorithm for S3 PUT operations: `CRC32C` (default), `CRC64NVME` or `SHA256`. Note that `CRC64NVME` and `SHA256` require VAST Cluster 5.5.0 or later. SyncEngine will verify if your VAST cluster supports checksum verification for the selected algorithm. | | **Read Timeout** | Socket read timeout, in seconds. Valid values: 60–3600. Default is 300. | | **Connect Timeout** | S3 endpoint connection timeout, in seconds. Valid values: 5–300. Default is 60. | | **Max Retries** | The maximum allowed number of retries by the VAST S3 client. Valid values: 1–100. Default is 3. | | **Retry Mode** | Select one of VAST S3 client retry modes: **Adaptive** (default) for rate-limiting of requests when throttled, **Standard** for exponential backoff with jitter, **Legacy** for simple exponential backoff. | 6. In the **Rate limits** step, leave the defaults unchanged ('0', no limits set). These fields are reserved for future use. 7. In the **SSL** step, use the **SSL Certificate Content** field to enter the TLS certificate (in PEM format) to secure the S3 connection. If left blank, default CA certificates are used. 8. In the **Review** step, review the settings you've made. If you want to make changes, click **Back**. Otherwise, click **Create Connector**. The connector is created and displayed in the **Connectors** page. ### SyncEngine CLI The following command creates a destination VAST S3 connector named `dest-vast-s3` to connect to the S3 endpoint at `<VAST S3 VIP>` on a VAST cluster with a management IP of `<VMS IP>`: ``` syncengine-cli connector create \ --type=vast_s3 \ --name=dest-vast-s3 \ --mgmt-ip= \ --username=admin \ --password= \ --vip= \ --access-key= \ --secret-key= ``` > **Note:** To auto-generate S3 access keys, omit the `--access-key` and `--secret-key` options from the command. Optional flags when creating a connector of the `vast_s3` type: ``` --s3-username # Default: SYNCENGINE_USER --s3-uid # Default: 270270270 --tenant-name --secure-s3={true|false} --checksum-algorithm {CRC32C|CRC64NVME|SHA256} --case-sensitive-bucket-names={true|false} --read-timeout --client-connect-timeout --client-retries-max-attempts --client-retries-mode --ssl-cert-file ``` The output should be similar to the following: ``` Successfully created connector 'dest-vast-s3' with ID 37 ``` > **Tip:** Note the connector ID in the output. ## Create Migration Create a migration to move the data from your source S3 store to an S3 bucket on the VAST cluster. ### SyncEngine UI 1. In the **Migrations** page, click **New Migration**. A wizard dialog appears to walk you through the migration creation steps. 2. In the **Source** step, select the S3 connector that will be used to connect to the source S3 store. 3. In the **Destination** step, select the VAST S3 connector that will be used to connect to the destination VAST cluster. 4. In the **Paths** step: - Under **Migration Label**, specify a SyncEngine worker label for the migration. The label is used to route migration jobs to the appropriate SyncEngine worker. One worker can be associated with one or more migration labels. Workers deployed with a certain label handle jobs for all migrations that have that label. You can create a new label or use an existing one. - To create a new label, select **Create new label** and enter a text string in the **New Label** field. The label can contain up to 128 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-) and underscores (_). - To use an existing label, select **Use existing label** and enter the label string in the **Existing Label** field. - Under **Paths**, specify the paths to the migration source and destination. - Under **Source Bucket / Prefix**, enter the source bucket name and, optionally, an object key prefix (used to migrate only objects with the prefix specified), for example: `bucket-name/object-prefix/`. - In the **Destination Bucket / Prefix**, enter a name for the destination S3 bucket on the VAST cluster and, optionally, an object key prefix to be assigned to the migrated objects. SyncEngine will automatically create the bucket if it does not exist. Note that you do not need to specify `s3://` in the fields. If an object prefix is specified, also include the trailing slash. If you specify the bucket name only, no trailing slash is needed. If you are going to use an [include list](/documentation/docs/migrate-from-s3-to-vast-s3-34#include-list), the source and destination paths can only contain the bucket name, without any object prefixes: `bucket-name`. 5. In the **Sync mode** step: - Under **Sync Mode**, specify if you want to run the migration once or [periodically](/documentation/docs/migrate-from-s3-to-vast-s3-34#periodic-resync): - To run the migration once, select **Manual** in the **SyncMode** field. Migration will be started after you submit the migration job manually. - To run the migration automatically at certain intervals, select **Automatic** in the **Sync Mode** field and enter an interval (in hours) in the **Sync Interval** field. The default interval is 24 hours. - Under **Include & exclude lists**, optionally add an include list for the migration: - Use the **Inclusion list** field to upload a file listing specific objects to be migrated. To learn more about this feature, see [Include list](/documentation/docs/migrate-from-s3-to-vast-s3-34#include-list). 6. In the **What you're copying** step: - Under **What you're copying**: - Review the settings you've made. If necessary, click **Back** and change as needed. - Under **S3 object metadata**: - If you do not want to copy S3 object tags during the migration, unselect the **Copy S3 object tags** option. By default, SyncEngine copies object tags together with the objects being migrated. - Optionally select **Stamp source last-modified via request header** to [preserve](/documentation/docs/migrate-from-s3-to-vast-s3-34#preserve-object-last-modified-date-and-time) the objects' original last-modified data and time during the migration. > **Note:** This option requires VAST Cluster 5.5.0 or later. - Under **Delete propagation**: - Decide how SyncEngine [handles the objects that have been deleted](/documentation/docs/migrate-from-s3-to-vast-s3-34#mirror-deletions) since the previous migration run. By default, SyncEngine does not delete such objects from the destination S3 bucket. If you want to have them deleted, select **Mirror source deletions on the destination**. - Under **Existing destination objects**: - Determine the behavior in case the destination VAST S3 bucket already contains an object with the same object key as the object being migrated. By default, SyncEngine overwrites existing objects using the new object data from the source. If you want to keep the existing objects in the destination bucket, select **Skip object keys that already exist at destination**. 7. In the **Change events (SQS)** step, optionally supply the SQS queue endpoint and the respective ARN to enable [event-driven resynchronization](/documentation/docs/migrate-from-s3-to-vast-s3-34#eventdriven-resync). > **Note:** This feature is available for AWS S3 source buckets only. - In the **SQS Queue URL**, enter the SQS queue URL in format: https://sqs..amazonaws.com// - In the **SQS Queue ARN** field, enter the ARN for accessing the SQS queue, in format: ``` arn:aws:sqs::: ``` 8. In the **Batch sizing** step, optionally adjust the size of batches that SyncEngine uses when copying the data. By default (if no values are entered in the fields provided), SyncEngine uses VAST cluster's defaults which are suitable for most workloads. - In the **Batch size** field, specify the maximum allowed batch size in GB. If not specified, a default of 1GB is used. - In the **Batch file count** field, specify the maximum allowed number of objects per batch. If not specified, a default of 1000 objects is used. 9. In the **Per-task timeouts** step, optionally configure suitable timeouts for the migration jobs. 10. In the **Review** step, review the settings you've made. If you want to make changes, click **Back**. Otherwise, click **Create Migration**. The migration is created and displayed in the **Migration** page. ### SyncEngine CLI The following commands creates a migration from an S3 bucket to a VAST S3 bucket named `syncengine.eng`. The migration is to be accomplished by workers labeled with `MYFIRSTMIGRATION` using connectors with connector IDs `15` and `37`. ``` syncengine-cli migration create \ --source-connector-id=15 \ --destination-connector-id=37 \ --bucket-name=syncengine.eng \ --label=MYFIRSTMIGRATION ``` Optional flags: ``` --sync-mode {manual | automatic --sync-interval } --enable-deletes={true|false} --ignore-existing={true|false} --copy-tags={true|false} --event-queue-url --event-queue-arn --batch-size --batch-file-count --inclusion-file --src-last-modified-header=={true|false} ``` The CLI creates a migration and displays its properties. > **Tip:** Note the migration ID in the output. ## Start Migration Manual migration start is only relevant for migrations that do not have periodic resynchronization configured. For migrations with periodic resynchronization, the migration starts automatically upon creation. ### SyncEngine UI Navigate to the ​**Migrations**​​ page and click the start button for the migration you want. ### SyncEngine CLI Run the command: ​ syncengine-cli migration start --migration-id=
​
The output should be similar to the following:
​
Starting sync job for migration: MYFIRSTMIGRATION
✓ Sync job created successfully!
Job ID: 2
Status: pending ## Monitor Migration As the migration progresses, its status changes as follows: `Not Started` (or `init`) -> `Waiting for Worker` (or `pending`) -> `Syncing` -> `Synced`. When complete, the destination S3 bucket will have objects migrated from the source bucket. ### SyncEngine UI Check the migration status in the **Migrations** page. ### SyncEngine CLI Run the command: ``` syncengine-cli migration status --migration-id= ``` The output should be similar to the following: ``` === Migration Status: MYFIRSTMIGRATION === ID: 33 Status: synced Source: my-source-bucket Destination: s3-data-dest Sync Mode: manual Last Sync: 2026-07-09 15:10:54 Copy Tags: always Settings: Error Retry: 0 Task Retry: 0 Batch Size: 0 GB Batch File Count: 0 Sync Jobs: 1 Recent Sync Jobs: +----+-----------+------------------+-------------+-------+------+ | ID | STATUS | START TIME | DURATION | FILES | SIZE | +----+-----------+------------------+-------------+-------+------+ | 95 | completed | 2026-07-09 15:10 | PT4.623401S | 1 | 29 B | +----+-----------+------------------+-------------+-------+------+ ``` ## Troubleshoot | Symptom | Likely Cause | Solution | | --- | --- | --- | | Sync job stays *pending* | Migration label does not match worker label | Update the migration to use the same label as the deployed worker. | | Sync fails writing to the VAST cluster | Invalid S3 credentials, or unreachable S3 virtual IP | Confirm access keys and S3 virtual IP reachability from workers. | | `Destination bucket <...> does not exist` | SyncEngine could not create the destination bucket. | Create the bucket manually, then recreate and start migration. The bucket name must start with the `syncengine.` prefix. | | Source S3 store returns HTTP error code `403` or S3 error `AccessDenied` | Insufficient permissions or wrong region. | Ensure that the worker has necessary permissions for the source bucket. For AWS S3, confirm that a correct bucket region is specified in connector settings. For other stores, check that the region field is blank. | | Connection timeout | Wrong endpoint, firewall, or virtual IP | Confirm URLs and ports 80/443 from workers. | | `S3 directory bucket` error | The source bucket is a S3 Express One Zone directory bucket. | Use a general-purpose bucket. | | Objects missing after synchronization | Wrong bucket or prefix | Verify the source and destination paths specified for the migration. | | Event-driven resync lists the entire bucket | SQS queue is not configured. | Ensure that the migration has an SQS queue specified, and the queue is configured to receive event notifications from the source bucket. | | Include file rejected | The source and/or destination path has an object key prefix specified, or event-driven incremental migration is configured. | Remove object key prefixes from the source and/or destination paths and disable event-driven resync. | | Objects not updated after event-driven resync | The SQS queue is missing events or is cont configured to receive event notifications from the source bucket | Verify that the SQS queue is subscribed for source bucket events; run a full resynchronization to reconcile. | | Migration very slow with tags enabled | Copying object tags may impact performance for large buckets. | Disable copying of object tags from the source bucket. |