Migrate from POSIX to POSIX

Overview

The POSIX connector copies files between local POSIX paths that are already mounted and visible on SyncEngine workers. SyncEngine walks the source directory tree locally and copies with rsync. Use POSIX connectors when source and destination are ordinary filesystem paths on the worker (local disk, bind mounts, or NFS client mounts that the OS already mounted).

The following capabilities are supported for POSIX-to-POSIX migrations:

• To determine when data synchronization occurs:
  • Periodic Resync
• To manage which items are being synchronized:
  • Exclude List
  • Mirror Deletions

Complete these steps to set up migration between POSIX paths on a worker host:

1. Ensure that the prerequisites are met.

2. Create two POSIX connectors.

3. Create a migration.

4. Start the migration.

   Note: This step is only required when the migration does not have periodic resynchronization enabled.

5. Monitor migration progress and completion.

6. If necessary, troubleshoot.

Features

Periodic Resync

In addition to starting a migration once it is created, SyncEngine can be configured to automatically run a resynchronization at user-supplied intervals.

• In SyncEngine UI, set the SyncMode to Automatic and enter an interval (in hours) in the Sync Interval field.

• In SyncEngine CLI, include the --sync-mode automatic=true and --sync-interval <interval> options when submitting the migration create command.

For migrations set to Automatic synchronization mode, the first synchronization starts immediately after the migration is created, and the subsequent synchronizations are performed periodically with the specified interval. A subsequent synchronization cannot be triggered manually.

To be able to manually start a synchronization on demand, set SyncMode to Manual.

Mirror Deletions

You can determine how SyncEngine handles the items in the source store that have been deleted since the previous migration run.

By default, SyncEngine does not delete such items from the destination. If you want to have them deleted, configure the migration to mirror source deletions at the destination:

• In SyncEngine UI, select the Mirror source deletions at destination option when creating a migration.

• In SyncEngine CLI, include the --enable-deletes=true option when submitting the migration create command.

Note: Deletion of items at the destination cannot be undone by disabling this setting.

Exclude List

You can supply a list of files or directories to be skipped during migration.

The list can contain file paths or wildcard patterns, one path or pattern per line. Supported wildcards are: * (zero or more characters), ? (one character), and [seq] (one character from the sequence or range). The wildcards can appear anywhere in the path.

For example:

/mnt/source/projects/logs/*.log
/mnt/source/projects/data/temp/*
/mnt/source/projects/reports/image[1-2000].jpg

Do not add trailing slashes to directory paths, e.g. specify /path/dir (but not /path/dir/).

Common patterns include:

To supply an exclude list:

• In SyncEngine UI, upload the exclude list file in the Exclusion list field when creating a migration.
• In SyncEngine CLI:

  • Include the --exclusion-file <file path> option on the migration create command, or

  • Run this command after the migration is created:

      migration exclusion-file upload <file path> --migration-id=<migration ID>
    

The exclude list file must use the UTF-8 encoding.

The following limitations apply:

• An exclude list can contain up to 10,000 lines.
• Comment lines (#) are not supported.

Before You Begin

Ensure that the prerequisites are met:

SyncEngine

• The SyncEngine control plane and at least one worker are up and running.
• At least one worker label of a deployed worker is known. (The worker label is set when you deploy the worker.)
• The source and destination paths exist on all workers labeled for a particular POSIX migration. A worker must have read access to the source path and read/write access to the destination path.

Create POSIX Connector

A POSIX connector is used for POSIX filesystem migrations.

SyncEngine UI

1. In the Connectors page, click New Connector.
   A wizard dialog appears to walk you through the connector creation steps.

2. In the Type & name step:

   • Select the POSIX connector type.
   • In the Connector name field, enter a unique name for the connector.
     The name can contain up to 63 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-) and underscores (_).

3. In the Tunables step:

   • Under Content Types Supported, determine whether the connector will handle data and metadata, only data, or only metadata. By default, the connector handles both data and metadata.
   • Under Batch & Scan Tunables:
     • Optionally adjust the size of batches that SyncEngine uses when copying the data. By default (if no values are entered in the fields provided), SyncEngine uses VAST cluster's defaults which are suitable for most workloads.
       • In the Batch Size field, specify the maximum allowed batch size in GB. If not specified, a default of 1 GB is used.
       • In the Max Files per Batch field, specify the maximum allowed count of files per batch. If not specified, a default of 1000 files is used.
       • Optionally configure the Block Size and Task Flush Threshold values as needed.

4. In the Rsync step, optionally configure rsync flags for file transfers.

5. In the Review step, review the settings you've made.
   If you want to make changes, click Back. Otherwise, click Create Connector.
   The connector is created and displayed in the Connectors page.

SyncEngine CLI

The following command creates a POSIX connector named src-posix:

syncengine-cli connector create \
  --type=filesystem \
  --name=src-posix \

Note: The connector type can be specified either as filesystem or posix.

Optional flags:

--batch-size-gb <GB>
--batch-file-count <count>
--block-size <bytes>
--task-flush-threshold <count>
--rsync-archive
--rsync-compress
--rsync-checksum
--rsync-hard-links
--rsync-acls
--rsync-xattrs
--rsync-no-perms
--rsync-no-owner
--rsync-no-group
--rsync-bwlimit <KB/s>
--rsync-timeout <seconds>

The output should be similar to the following:

Successfully created connector 'src-posix' with ID 15

Tip: Note the connector ID in the output.

Create Migration

Create a migration to copy files from a source path to a destination path on a SyncEngine worker host.

SyncEngine UI

1. In the Migrations page, click New Migration.
   A wizard dialog appears to walk you through the migration creation steps.

2. In the Source step, select the source POSIX connector.

3. In the Destination step, select the destination POSIX connector.

4. In the Paths step:

   • Under Migration Label, specify a SyncEngine worker label for the migration.
     The label is used to route migration jobs to the appropriate SyncEngine worker. One worker can be associated with one or more migration labels. Workers deployed with a certain label handle jobs for all migrations that have that label.
     You can create a new label or use an existing one.
     • To create a new label, select Create new label and enter a text string in the New Label field.
       The label can contain up to 128 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-) and underscores (_).
     • To use an existing label, select Use existing label and enter the label string in the Existing Label field.
   • Under Paths, specify the paths to the migration source and destination.
   • Under Source Path, enter the absolute path where the data to be migrated are located.
   • In the Destination Path, enter the absolute path to which to copy the data.

5. In the Sync mode step:

   • Under Sync Mode, specify if you want to run the migration once or periodically:
     • To run the migration once, select Manual in the SyncMode field.
       Migration will be started after you submit the migration job manually.
     • To run the migration automatically at certain intervals, select Automatic in the Sync Mode field and enter an interval (in hours) in the Sync Interval field. The default interval is 24 hours.
   • Under Include & exclude lists: optionally add an exclude list for the migration:
     • Use the Exclusion list field to upload a file listing specific file paths to be skipped during the migration. To learn more about this feature, see Exclude list.

6. In the What you're copying step:

   • Under What you're copying:
     • Review the settings you've made. If necessary, click Back and change as needed.
   • Under Delete propagation:
     • Decide how SyncEngine handles the files that have been deleted since the previous migration run.
       By default, SyncEngine does not delete such files from the destination path. If you want to have them deleted, select Mirror source deletions on the destination.

7. In the Batch sizing step, optionally adjust the size of batches that SyncEngine uses when copying the data.
   By default (if no values are entered in the fields provided), SyncEngine uses VAST cluster's defaults which are suitable for most workloads.

   • In the Batch size field, specify the maximum allowed batch size in GB. If not specified, a default of 1024GB is used.
   • In the Batch file count field, specify the maximum allowed count of files per batch. If not specified, a default of 1000 files is used.

8. In the Per-task timeouts step, optionally configure suitable timeouts for the migration jobs.

9. In the Review step, review the settings you've made.
   If you want to make changes, click Back. Otherwise, click Create Migration.
   The migration is created and displayed in the Migration page.

SyncEngine CLI

The following command creates a migration from /mnt/posix-source to /mnt/posix-dest. The migration is to be accomplished by workers labeled with MYFIRSTMIGRATION using connectors with connector IDs 4 and 5.

syncengine-cli migration create \
  --source-connector-id=4 \
  --destination-connector-id=5 \
  --source-path=/mnt/posix-source \
  --destination-path=/mnt/posix-dest \
  --label=MYFIRSTMIGRATION

Optional flags:

--sync-mode {manual | automatic --sync-interval <hours>}
--enable-deletes={true|false}
--exclusion-file <path to file>
--batch-size <GB>
--batch-file-count <count>

The CLI creates a migration and displays its properties.

Tip: Note the migration ID in the output.

Start Migration

Manual migration start is only relevant for migrations that do not have periodic resynchronization configured. For migrations with periodic resynchronization, the migration starts automatically upon creation.

SyncEngine UI

Navigate to the ​Migrations​​ page and click the start button for the migration you want.

SyncEngine CLI

Run the command:
​
syncengine-cli migration start --migration-id=
​
The output should be similar to the following:
​
Starting sync job for migration: MYFIRSTMIGRATION
✓ Sync job created successfully!
Job ID: 2
Status: pending

Monitor Migration

As the migration progresses, its status changes as follows: Not Started (or init) -> Waiting for Worker (or pending) -> Syncing -> Synced.

When complete, the destination path will have files and directories copied from the source path. To verify, compare file counts or spot-check files on the destination path from a worker host.

SyncEngine UI

Check the migration status in the Migrations page.

SyncEngine CLI

Run the command:

syncengine-cli migration status --migration-id=<migration ID>

The output should be similar to the following:

=== Migration Status: MYFIRSTMIGRATION ===
ID: 125
Status: synced
Source: /mnt/posix-source
Destination: /mnt/posix-dest
Sync Mode: manual
Last Sync: 2026-07-15 14:06:08
Copy Tags: always

Settings:
  Error Retry: 0s
  Task Retry: 0
  Batch Size: 0 GB
  Batch File Count: 0

Sync Jobs: 1

Recent Sync Jobs:
+----+-----------+------------------+-------------+-------+------+
| ID |  STATUS   |    START TIME    |  DURATION   | FILES | SIZE |
+----+-----------+------------------+-------------+-------+------+
| 96 | completed | 2026-07-15 14:06 | PT4.623401S |     1 | 29 B |
+----+-----------+------------------+-------------+-------+------+

Troubleshoot