SyncEngine S3 Migration Guide

This guide walks you through creating an S3-to-S3 migration using syncengine-cli -- from any S3-compatible source to a VAST S3 destination.

Prerequisites

• syncengine-cli installed (RPM provides both CLI and web UI)
• Source S3 access key, secret key, and endpoint URL
• VAST S3 destination configured per the VAST Cluster Setup Guide (S3 user, identity policy, view/bucket)
• VAST S3 access key and secret key (saved from the VAST cluster setup)
• Network connectivity from worker hosts to both S3 endpoints (ports 80/443)

Step 1: Create the Source S3 Connector

syncengine-cli connector create \
  --type=s3 \
  --name="source-s3" \
  --access-key="<source-access-key>" \
  --secret-key="<source-secret-key>" \
  --endpoint="http://<source-s3-endpoint>:80" \
  --region="us-east-1"

Note the connector ID returned (e.g., Connector created with ID: 1).

Tip: The --region flag defaults to us-east-1. Adjust for your source environment if needed.

Step 2: Create the Destination S3 Connector (VAST)

syncengine-cli connector create \
  --type=s3 \
  --name="vast-s3" \
  --access-key="<vast-access-key>" \
  --secret-key="<vast-secret-key>" \
  --endpoint="http://<vast-dns-name>:80" \
  --region="us-east-1"

Note this connector ID as well (e.g., Connector created with ID: 2).

Step 3: Verify Connectors

syncengine-cli connector list --type=s3

You should see both connectors listed with their IDs and endpoint details.

Step 4: Create the Migration

S3 migrations use --source-bucket and --destination-bucket instead of --source-path and --destination-path.

syncengine-cli migration create \
  --source-connector-id=1 \
  --destination-connector-id=2 \
  --source-bucket=my-source-bucket \
  --destination-bucket=s3-data-dest \
  --label=s3-migration

Optional flags

Automatic mode example

syncengine-cli migration create \
  --source-connector-id=1 \
  --destination-connector-id=2 \
  --source-bucket=my-source-bucket \
  --destination-bucket=s3-data-dest \
  --label=s3-migration \
  --sync-mode=automatic \
  --sync-interval=12

Step 5: Start the Migration

If you used manual sync mode:

syncengine-cli migration start --migration-id=1

Step 6: Deploy a Worker

Workers connect to the control plane, authenticate, and start processing jobs.
Run these commands on the worker host, not on the control plane host.

1. Copy the ms script and worker bundle to the worker host (use scp, USB drive, or your preferred transfer method)

2. Make ms executable
   chmod +x ms

3. Install worker images
   ./ms install worker /path/to/Vastdata_SyncEngine_v3.1.0.tar.gz /opt/syncengine

4. Deploy the worker, pointing it at your control plane
   cd /opt/syncengine WORKER_LABEL=worker1 META_CONTROL_IP=<control_plane_ip> ./ms deploy worker

Replace <control_plane_ip> with the actual IP address of the host running the control plane (e.g., 172.31.31.55).

Verify the worker is running

Check the container

podman ps | grep worker

Watch the logs

podman logs -f syncengine-worker

You should see the worker register with the control plane and start polling for jobs.

Step 7: Monitor the Migration

View current status

syncengine-cli migration status --migration-id=1

View migration statistics

syncengine-cli migration stats --migration-id=1

Monitor in real-time

syncengine-cli migration monitor --migration-id=1

List all migrations

syncengine-cli migration list

Common Management Commands

S3 Limitations

Be aware of the following limitations in the current version:

• S3 object tags are not preserved during migration
• S3 Versions are not carried over
• S3 ACLs (access control lists) are not preserved -- use identity policies for access control on the VAST destination

Next Steps

• POSIX migration: See POSIX Migration Guide
• Metadata indexing: See Indexing Guide
• Worker tuning: See Worker Tuning in the Install Guide