Migrate from OneDrive to VAST S3

Overview

The OneDrive connector extracts files from Microsoft OneDrive into a VAST S3 bucket.

SyncEngine authenticates with credentials of a client application registered with Microsoft Entra ID, reads OneDrive content through through Microsoft Graph, streams each file to an S3 object on the VAST cluster, writes a sidecar metadata JSON object, and converts OneDrive permissions into S3 ACLs.

Files are copied in their original format as stored in OneDrive. The original file name extension is preserved on the destination object. No conversion to Markdown is performed.

The migration can include OneDrive data of all users within a certain Microsoft 365 tenant, or it can be limited to data of certain tenant users.

The following capabilities are supported for OneDrive to VAST S3 migrations:

• To manage the destination:
  • Auto-Create Destination Bucket
  • Add Key Prefix to Migrated Objects
• To determine when data synchronization occurs:
  • Periodic Resync
• To manage which items are being synchronized:
  • Tenant-Wide Migration
  • Mirror Deletions
• To manage what is synchronized per item:
  • Copy Extra Metadata
• To configure SyncEngine processes:
  • Handle Graph API Throttling
  • Graph API Rate Limits

Complete these steps to set up migration from OneDrive to a VAST S3 bucket:

1. Ensure that the prerequisites are met.

2. Create a connector for OneDrive.

3. Create a connector for VAST S3.

4. Create a migration from the source OneDrive to the destination VAST S3 bucket.

5. Start the migration.

   Note: This step is only required when the migration does not have periodic resynchronization enabled.

6. Monitor migration progress and completion.

7. If necessary, troubleshoot.

The following limitations apply:

• OneDrive content can only be migrated to a VAST S3 bucket. No other destination types are supported.
• Folders, OneNote packages, remote shortcuts, items larger than 250 GB, and items without a streamable MIME type are skipped.
• Graph permission roles other than read or owner are dropped (no destination ACEs are created for them).

Features

Periodic Resync

In addition to starting a migration once it is created, SyncEngine can be configured to automatically run a resynchronization at user-supplied intervals.

• In SyncEngine UI, set the SyncMode to Automatic and enter an interval (in hours) in the Sync Interval field.

• In SyncEngine CLI, include the --sync-mode automatic=true and --sync-interval <interval> options when submitting the migration create command.

For migrations set to Automatic synchronization mode, the first synchronization starts immediately after the migration is created, and the subsequent synchronizations are performed periodically with the specified interval. A subsequent synchronization cannot be triggered manually.

To be able to manually start a synchronization on demand, set SyncMode to Manual.

Auto-Create Destination Bucket

SyncEngine can automatically create a destination bucket on the VAST cluster based on the settings you've made when configuring the migration.

For migrations from OneDrive to VAST S3, SyncEngine attempts to automatically create a destination bucket when you submit a migration job. If it does not succeed, create the bucket manually so that the bucket name starts with the word 'syncengine' followed by a period (.): syncengine.<your-string>.

Add Key Prefix to Migrated Objects

You can configure SyncEngine to add a certain object key prefix to all objects created in the destination bucket as a result of a migration.
Adding migration-specific prefixes allows for migrating from multiple source stores to the same destination S3 bucket. otherwise, you would need to provision a separate destination bucket for each of the source stores.
To configure the object key prefix to be added:

• In SyncEngine UI, append the prefix to the bucket name in the Destination S3 Bucket field when creating a migration: /<bucket name>/<object key prefix>/. (Note that you need a slash before and after the prefix string.)
• In SyncEngine CLI, include the prefix on the --bucket-name option on the migration create command: --bucket-name=syncengine.<bucket name>/<object key prefix>/

Periodic Resync

In addition to starting a migration once it is created, SyncEngine can be configured to automatically run a resynchronization at user-supplied intervals.

• In SyncEngine UI, set the SyncMode to Automatic and enter an interval (in hours) in the Sync Interval field.

• In SyncEngine CLI, include the --sync-mode automatic=true and --sync-interval <interval> options when submitting the migration create command.

For migrations set to Automatic synchronization mode, the first synchronization starts immediately after the migration is created, and the subsequent synchronizations are performed periodically with the specified interval. A subsequent synchronization cannot be triggered manually.

To be able to manually start a synchronization on demand, set SyncMode to Manual.

Tenant-Wide Migration

In addition to migrating OneDrive data of a particular user, SyncEngine can migrate content of all users belonging to the Microsoft 365 tenant.

When running a tenant-wide migration, you can specify users to be excluded from the migration.

To configure tenant-wide migration:

• In SyncEngine UI:
  • (Optional) When creating the OneDrive connector, use the User filter (OData) and Excluded user UPNs fields to specify users to be excluded from migration.
  • When creating the migration, enter an asterisk (*) in the OneDrive user UPN field.
• In SyncEngine CLI:
  • (Optional) Include the --user-filter <OData filter> or --excluded-user-upns <upn1,upn2,...> options on the connector create command to specify users to be excluded from migration.
  • On the migration create command, include --user-upn *.

Mirror Deletions

You can determine how SyncEngine handles the items in the source store that have been deleted since the previous migration run.

By default, SyncEngine does not delete such items from the destination. If you want to have them deleted, configure the migration to mirror source deletions at the destination:

• In SyncEngine UI, select the Mirror source deletions at destination option when creating a migration.

• In SyncEngine CLI, include the --enable-deletes=true option when submitting the migration create command.

Note: Deletion of items at the destination cannot be undone by disabling this setting.

Copy Extra Metadata

You can confgure SyncEngine to copy additional Microsoft Graph metadata for each file being migrated. The metadata can include OneDrive-specific fields like description, and also file versioning information (latest version, last-modified date and time).

When enabled, SyncEngine writes the additional metadata to the file's sidecard metadata JSON object.

To enable copying of OneDrive metadata:

• In SyncEngine UI, set the Copy drive-item metadata and/or Copy version metadata flags.
• In SyncEngine CLI, include the --drive-item-metadata and `--version-metadata options on the migration create command.

By default, extra metadata is not copied.

Graph API Rate Limits

You can set request rate limits to prevent SyncEngine from causing excessive load on OneDrive:

• Cap the overall number of requests (per second) that SyncEngine sends to the Microsoft Graph API.
• Restrict throughput when downloading the files from OneDrive.

Each of the limits applies to the aggregate amount of requests or throughput by all workers used by the connector. Each limit is enforced independently of the other limit. On reaching a limit, workers wait and retry so that synchronization continues after the pause.

By default, no request rate limits are set.

Set rate limits if during a representative migration, vendor 429/503/SlowDown errors appear in the worker logs, and the Grafana Connector Rate Limits panel (or the Prometheus metrics connector_rate_limit_waits_total{connector_id, axis}) shows the wait counter growing. If no throttling and no vendor errors occur, you do not need to set rate limits.

For recommended request rates for OneDrive, see Graph API documentation. Typical rate limits you may need when migrating from OneDrive to VAST S3 are 20–50 API calls per second and 50–200 MB/s.

To set rate limits:

• In SyncEngine UI, use the API calls per second and Bytes per second fields in OneDrive connector settings.

Note: Connector rate limits can only be set in SyncEngine UI.

Avoid configuring multiple connectors for the same Microsoft 365 tenant or Entra ID registered application with rate limits that, when combined, exceed the maximum allowed request rate for the Microsoft Graph API. Use a single connector instead.

Handle Graph API Throttling

SyncEngine uses Microsoft Graph API to communicate with OneDrive. When the API queue is overloaded, the API responds with error code 429 Too Many Requests.

You can configure SyncEngine to keep retrying at a lower pace in case it gets this error frequently. To do so:

1. Set the flag in migration settings:

• In SyncEngine UI, in the Per-task timeouts step of the migration creation wizard:
  • Select the Treat Graph throttling (429 / Too Many Requests) as retriable option.
  • Set Task retry to 3.
• In SyncEngine CLI:

  • Include --task-retry 3 option on the migration create command.

  • After the migration is created but prior to migration start, submit the command for your migration ID:

      syncengine-cli migration excluded-exceptions add \
                    --migration-id=<migration ID> \
                    --pattern='429|Too Many Requests'
    

    The output should be similar to the following:

      Excluded exceptions add for migration 25:
        429|Too Many Requests
    

2. Set rate limits for the connector (optional).

If you need to undo the configuration via SyncEngine CLI, use commands:

syncengine-cli migration excluded-exceptions remove --migration-id=<ID> --pattern='429|Too Many Requests'
syncengine-cli migration excluded-exceptions reset --migration-id=<ID>       # removes all configured exceptions

Before You Begin

Ensure that the prerequisites are met:

OneDrive

• Microsoft Entra ID application registration in the Microsoft 365 tenant that owns the OneDrive data.
  You'll have to specify:
  • The Microsoft 365 tenant ID (displayed when you navigate to Entra ID in Microsoft Entra Admin Center).
  • The application (client) ID (Entra ID -> App registrations -> select your application and look for the application ID in the Overview page)
  • A client secret (must be generated under Entra ID -> App registrations -> Certificates & secrets -> Client secrets).
• Admin-consented Microsoft Graph application permissions that allow the application to read the OneDrive data subject to migration (Files.Read.All). For tenant-wide migrations, the User.Read.All permissions are also required (admin-consented).
• To migrate OneDrive data of a particular user: The user's UPN or email address

VAST Cluster

Networking

• VAST cluster's management IP.
• A DNS-resolvable name of the tenant's virtual IP pool to access the VAST S3 endpoint, or a virtual IPv4 address from that pool.
• Network connectivity from workers to the destination VAST S3 endpoint.

VAST Tenant and Admin Credentials

• If your VAST cluster has multiple VAST tenant configured:
  • To migrate to one of the tenants:
    • The name of the VAST tenant where the destination VAST S3 bucket will reside
    • Username and password for the VAST tenant admin user
  • To migrate to the default tenant:
    • Username and password for the VAST cluster admin user
• If your VAST cluster is not configured for multi-tenancy:
  • Username and password for the VAST cluster admin user

Create User

Create a user with S3 access keys allowing to create a bucket and put objects to it.

1. Create a local user on the VAST provider attached to the tenant where the destination VAST S3 bucket will reside.
   Ensure that the user has the Allow Bucket Creation option enabled.
2. Generate S3 access key for the user.
3. Create an identity policy to grant the user full access to the destination bucket ("Action": "s3:*", "Effect": "Allow").
4. Assign the identity policy to the user.
   Note the username and UID of the created user.

Active Directory

• The VAST cluster must be joined to Active Directory (required to map OneDrive user permissions to S3 ACLs).
• (Recommended) Ensure that Active Directory Domain Auto-Discovery is enabled for the Active Directory provider that is attached to the tenant where the destination bucket resides.

Active Directory

• The Active Directory must have mapping entries for OneDrive users and groups. For example, for a OneDrive user with email address alice@company.com, Active Directory must have sAMAccountName with the value alice.
• If you are going to use groups, each group must have email or mail attribute configured.

SyncEngine

• The SyncEngine control plane and at least one worker are up and running.
• At least one worker label of a deployed worker is known. (The worker label is set when you deploy the worker.)

Create OneDrive Connector

A OneDrive connector is used to connect to OneDrive where data to be migrated resides.

SyncEngine UI

1. In the Connectors page, click New Connector.
   A wizard dialog appears to walk you through the connector creation steps.

2. In the Type & name step:

   • Select the OneDrive connector type.
   • In the Connector name field, enter a unique name for the connector.
     The name can contain up to 63 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-) and underscores (_).

3. In the Credentials step, enter details of your client application registered in Microsoft Entra ID:

   Field	Description
   Client ID	Application (client) ID assigned when you registered your client application with Entra.
   Client secret	Client secret for your client application.
   Tenant ID	Your Microsoft 365 tenant ID.

4. In the Rate limits step, set optional request rate limits for the connector:

   Field	Description
   API calls per second	Maximum allowed number of Graph API calls per second. Leave blank for unlimited (default).
   Bytes per second	Maximum allowed download throughput. Leave blank for unlimited (default).

5. In the Review step, review the settings you've made.
   If you want to make changes, click Back. Otherwise, click Create Connector.
   The connector is created and displayed in the Connectors page.

SyncEngine CLI

The following command creates a OneDrive connector named src-onedrive:

syncengine-cli connector create \
  --type=onedrive \
  --name=src-onedrive \
  --client-id=<Entra application client ID> \
  --client-secret=<Entra client secret> \
  --tenant-id=<Entra tenant ID>

Note: Connector rate limits can only be set in SyncEngine UI.

The output should be similar to the following:

Successfully created connector 'src-onedrive' with ID 15

Tip: Note the connector ID in the output.

Create VAST S3 Connector

A VAST S3 connector is used to connect to the VAST S3 server on the VAST cluster to which you want to migrate the data.

SyncEngine UI

1. In the Connectors page, click New Connector.
   A wizard dialog appears to walk you through the connector creation steps.

2. In the Type & name step:

   • Select the Vast S3 connector type.
   • In the Connector name field, enter a unique name for the connector.
     The name can contain up to 63 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-) and underscores (_).

3. In the VMS cluster step, enter VAST cluster's VMS IP and admin credentials:

   Field	Description
   Management IP	VAST cluster's management IPv4 address or DNS-resolvable hostname (up to 253 characters).
   Username and Password	Username and password to log in to the VMS. If the destination bucket is on one of VAST cluster tenants, enter credentials of the admin user for that tenant. If the destination bucket is on the default tenant, enter credentials of the VAST cluster admin user.
   Tenant name	The name of the VAST cluster tenant where the destination bucket resides. If not specified, the default tenant is used.

4. In the S3 endpoint step, configure an S3 endpoint on the VAST cluster.

   Field	Description
   VIP	A DNS-resolvable name of the tenant's virtual IP pool to access the VAST S3 endpoint, or a virtual IPv4 address from that pool. A DNS-resolvable name is recommended.
   Access Key	Enter the S3 access key for the bucket owner user.
   Secret Key	Enter the private key for the bucket owner user.
   S3 Username	Enter the username for the bucket owner user.
   S3 UID	The UID of the bucket owner user.
   Endpoint URL	The URL to access the S3 endpoint. This field is optional. If not specified, the URL is derived from the VIP field.
   Use secure S3 connection (HTTPS)	Select to use HTTPS. If selected, you'll need to provide a TLS certificate in a next step. If not selected (default), HTTP is used.

5. In the Bucket & advanced step, optionally configure advanced settings for the destination S3 bucket:

   Field	Description
   Case-sensitive bucket names	When enabled, SyncEngine allows for bucket names that contain both lowercase and uppercase characters.
   Checksum Algorithm	Checksum algorithm for S3 PUT operations: CRC32C (default), CRC64NVME or SHA256. Note that CRC64NVME and SHA256 require VAST Cluster 5.5.0 or later. SyncEngine will verify if your VAST cluster supports checksum verification for the selected algorithm.
   Read Timeout	Socket read timeout, in seconds. Valid values: 60–3600. Default is 300.
   Connect Timeout	S3 endpoint connection timeout, in seconds. Valid values: 5–300. Default is 60.
   Max Retries	The maximum allowed number of retries by the VAST S3 client. Valid values: 1–100. Default is 3.
   Retry Mode	Select one of VAST S3 client retry modes: Adaptive (default) for rate-limiting of requests when throttled, Standard for exponential backoff with jitter, Legacy for simple exponential backoff.

6. In the Rate limits step, leave the defaults unchanged ('0', no limits set). These fields are reserved for future use.

7. In the SSL step, use the SSL Certificate Content field to enter the TLS certificate (in PEM format) to secure the S3 connection. If left blank, default CA certificates are used.

8. In the Review step, review the settings you've made.
   If you want to make changes, click Back. Otherwise, click Create Connector.
   The connector is created and displayed in the Connectors page.

SyncEngine CLI

The following command creates a destination VAST S3 connector named dest-vast-s3 to connect to the S3 endpoint at <VAST S3 VIP> on a VAST cluster with a management IP of <VMS IP>:

syncengine-cli connector create \
  --type=vast_s3 \
  --name=dest-vast-s3 \
  --mgmt-ip=<VMS IP> \
  --username=admin \
  --password=<password> \
  --vip=<VAST S3 VIP> \
  --access-key=<S3 access key> \
  --secret-key=<S3 secret key>

Note: To auto-generate S3 access keys, omit the --access-key and --secret-key options from the command.

Optional flags when creating a connector of the vast_s3 type:

--s3-username <name>                # Default: SYNCENGINE_USER
--s3-uid <UID>                      # Default: 270270270
--tenant-name <VAST tenant name>
--secure-s3={true|false}
--checksum-algorithm {CRC32C|CRC64NVME|SHA256}
--case-sensitive-bucket-names={true|false}
--read-timeout <seconds>
--client-connect-timeout <seconds>
--client-retries-max-attempts <count>
--client-retries-mode <adaptive|standard|legacy>
--ssl-cert-file <path to file>

The output should be similar to the following:

Successfully created connector 'dest-vast-s3' with ID 37

Tip: Note the connector ID in the output.

Create Migration

Create a migration to copy the data from the OneDrive folder to an S3 bucket on the VAST cluster.

SyncEngine UI

1. In the Migrations page, click New Migration.
   A wizard dialog appears to walk you through the migration creation steps.

2. In the Source step, select the OneDrive connector that will be used to connect to the folder to be migrated.

3. In the Destination step, select the VAST S3 connector that will be used to connect to the destination VAST cluster.

4. In the Paths step:

   • Under Migration Label, specify a SyncEngine worker label for the migration.
     The label is used to route migration jobs to the appropriate SyncEngine worker. One worker can be associated with one or more migration labels. Workers deployed with a certain label handle jobs for all migrations that have that label.
     You can create a new label or use an existing one.
     • To create a new label, select Create new label and enter a text string in the New Label field.
       The label can contain up to 128 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-) and underscores (_).
     • To use an existing label, select Use existing label and enter the label string in the Existing Label field.
   • Under Paths, specify the migration source and destination.
     • In the OneDrive user UPN field:
       • To migrate OneDrive data of a particular user, enter the user's User Principal Name (UPN) or email address.
       • To migrate OneDrive data of all users within the Microsoft 365 tenant, enter an asterisk (*).
         Note that in case of tenant-wide migration, the user filters set in the OneDrive connector still apply (if set).
     • In the Destination S3 Bucket, enter a name for the S3 bucket on the VAST cluster.
       SyncEngine will automatically create the bucket if it does not exist.
       If you want SyncEngine to add a certain object key prefix to the migrated objects, specify the prefix followed by a trailing slash: <bucket name>/<object key prefix>/.
       The string that you enter will automatically be prepended with the word 'syncengine' followed by a period (.): syncengine.<your string>. Special characters in the string will be replaced with hyphens.

5. In the Sync mode step, specify if you want to run the migration once or periodically:

   • To run the migration once, select Manual in the SyncMode field.
     Migration will be started after you submit the migration job manually.
   • To run the migration automatically at certain intervals, select Automatic in the Sync Mode field and enter an interval (in hours) in the Sync Interval field. The default interval is 24 hours.

6. In the What you're copying step:

   • Review the settings you've made. If necessary, click Back and change as needed.
   • If you want to copy additional OneDrive metadata together with the data, set the flags:
     • Copy drive-item metadata copies additional OneDrive item fields such as description.
     • Copy version metadata copies information about the latest file version and last-modified date and time.
       By default, the extra metadata is not copied.
   • Decide how SyncEngine handles the files that have been deleted since the previous migration run.
     By default, SyncEngine does not delete such files from the destination S3 bucket. If you want to have them deleted, select Mirror source deletions on the destination.

7. In the Per-task timeouts step, optionally configure suitable timeouts and retry options for the migration jobs.

   OneDrive First Task	First-task/setup timeout. Default: 600 seconds
   OneDrive Copy File	Per-file content copy. Default: 1800 seconds
   OneDrive Delete File	Per-file delete when mirroring deletions. Default: 600 seconds
   OneDrive Sync Changes	Incremental /delta sync. Default: 1800 seconds
   OneDrive Copy File Acl	Per-file ACL copy. Default: 900 seconds
   Task retry	Count of times to re-queue a failed per-file task. Default is 0 (no retry). For large migrations, set to 3.
   Error retry	Times the overall sync job is retried after a failed run.
   Treat Graph throttling (429 / Too Many Requests) as retriable	Enable to continue retrying when Graph API responds with 429 Too Many Requests. To learn more, see Handle Graph API throttling. By default, this flag is not set.

8. In the Review step, review the settings you've made.
   If you want to make changes, click Back. Otherwise, click Create Migration.
   The migration is created and displayed in the Migration page.

SyncEngine CLI

The following command creates a migration for OneDrive user alice@company.com to a VAST S3 bucket named syncengine.onedrive. The migration is to be accomplished by workers labeled with MYFIRSTMIGRATION using connectors with connector IDs 12 and 37.

syncengine-cli migration create \
  --source-connector-id=12 \
  --destination-connector-id=37 \
  --user-upn=alice@company.com \             #For tenant-wide, set to *
  --bucket-name=syncengine.onedrive \
  --label=MYFIRSTMIGRATION \

Optional flags:

--sync-mode {manual|automatic --sync-interval <hours>}
--enable-deletes={true|false}
--drive-item-metadata={true|false}
--version-metadata={true|false}
--task-retry <count>
--error-retry <count>

The CLI creates a migration and displays its properties.

The output should be similar to the following:

Creating migration: MYFIRSTMIGRATION
  Source type: onedrive
  Destination type: vast_s3
  OneDrive user UPN: alice@company.com
  Destination bucket: syncengine.onedrive

 Migration created successfully!
  ID: 25
  Label: MYFIRSTMIGRATION
  Sync Mode: manual
  File Deletions: Disabled

Tip: Note the migration ID in the output.

Start Migration

Manual migration start is only relevant for migrations that do not have periodic resynchronization configured. For migrations with periodic resynchronization, the migration starts automatically upon creation.

SyncEngine UI

Navigate to the ​Migrations​​ page and click the start button for the migration you want.

SyncEngine CLI

Run the command:
​
syncengine-cli migration start --migration-id=
​
The output should be similar to the following:
​
Starting sync job for migration: MYFIRSTMIGRATION
✓ Sync job created successfully!
Job ID: 2
Status: pending

Monitor Migration

As the migration progresses, its status changes as follows: Not Started (or init) -> Waiting for Worker (or pending) -> Syncing -> Synced.

When complete, the destination S3 bucket will have objects with object keys in format:

• <file ID>/onedrive.<file extension> - the migrated content, with the original file extensions preserved as part of the object key
• <file ID>/onedrive.metadata.json - metadata, such as source URL, source title, original last-modified timestamp, ACL-related fields, additional OneDrive metadata (if the option was enabled for the migration).

SyncEngine UI

Check the migration status in the Migrations page.

SyncEngine CLI

Run the command:

syncengine-cli migration status --migration-id=<migration ID>

The output should be similar to the following:

=== Migration Status: MYFIRSTMIGRATION ===
ID: 25
Status: synced
Source: alice@company.com
Destination: syncengine.onedrive
Sync Mode: manual
Last Sync: 2026-07-23 15:10:54

Settings:
  Error Retry: 0
  Task Retry: 3

Sync Jobs: 1

Recent Sync Jobs:
+----+-----------+------------------+-------------+-------+------+
| ID |  STATUS   |    START TIME    |  DURATION   | FILES | SIZE |
+----+-----------+------------------+-------------+-------+------+
| 42 | completed | 2026-07-23 15:10 | PT67S       |    58 | 12 MB|
+----+-----------+------------------+-------------+-------+------+

Troubleshoot