SyncEngine Indexing Guide

New
  • Updated on Apr 15, 2026
  • Published on Apr 15, 2026
  • 2 minute(s) read
Prev Next

This guide walks you through creating metadata indexing jobs using syncengine-cli. Indexing scrapes file system metadata from a POSIX source and stores it in VastDB for fast querying and analysis.

Prerequisites

  • syncengine-cli installed (RPM provides both CLI and web UI)
  • Source path mounted and accessible on worker hosts (e.g., /mnt/data)
  • VAST cluster with VastDB available
  • VastDB management IP, VIP, and admin credentials

Step 1: Create the Source Connector

Create a filesystem connector pointing to the POSIX source you want to index:

syncengine-cli connector create --type=filesystem --name="source-filesystem"

Note the connector ID returned (e.g., Connector created with ID: 1).

Step 2: Create the VastDB Destination Connector

syncengine-cli connector create \
  --type=vast_db \
  --name="vast-indexing" \
  --mgmt-ip="<vast-mgmt-ip>" \
  --username="admin" \
  --password="<password>" \
  --vip="<vast-vip>"

Note this connector ID as well (e.g., Connector created with ID: 2).

VASTDB connector flags

Flag Required Description
--mgmt-ip Yes VAST management IP address (DNS names accepted)
--username Yes VAST admin username
--password Yes VAST admin password
--vip Yes VAST VIP address for data access
--bucket-name No Target bucket name in VastDB

Step 3: Verify Connectors

syncengine-cli connector list --type=filesystem
syncengine-cli connector list --type=vast_db

Step 4: Create the Indexing Job

syncengine-cli indexing create \
  --source-connector-id=1 \
  --destination-connector-id=2 \
  --source-path=/mnt/data \
  --label=data-index

All four flags are required:

Flag Description
--source-connector-id ID of the source filesystem connector
--destination-connector-id ID of the VastDB connector
--source-path Absolute path to index (must start with /)
--label Label for the indexing job (must be unique)

Note: If a label already exists, the system automatically appends a timestamp to make it unique.

Step 5: Deploy a Worker

Workers connect to the control plane, authenticate, and start processing jobs.
Run these commands on the worker host, not on the control plane host.

  1. Copy the ms script and worker bundle to the worker host (use scp, USB drive, or your preferred transfer method)

  2. Make ms executable
    chmod +x ms

  3. Install worker images
    ./ms install worker /path/to/Vastdata_SyncEngine_v3.1.0.tar.gz /opt/syncengine

  4. Deploy the worker, pointing it at your control plane
    cd /opt/syncengine WORKER_LABEL=worker1 META_CONTROL_IP=<control_plane_ip> ./ms deploy worker

Replace <control_plane_ip> with the actual IP address of the host running the control plane (e.g., 172.31.31.55).

Verify the worker is running

Check the container

podman ps | grep worker

Watch the logs

podman logs -f syncengine-worker

You should see the worker register with the control plane and start polling for jobs.

Step 6: Manage the Indexing Job

Start indexing

syncengine-cli indexing start --indexing-id=1

Check status

syncengine-cli indexing status --indexing-id=1

Monitor in real-time

syncengine-cli indexing monitor --indexing-id=1

Stop indexing

syncengine-cli indexing stop --indexing-id=1

List all indexing jobs

syncengine-cli indexing list

Filter by status:

syncengine-cli indexing list --status=syncing

Command Reference

Action Command
Create indexing job syncengine-cli indexing create --source-connector-id=1 --destination-connector-id=2 --source-path=/mnt/data --label=my-index
List all jobs syncengine-cli indexing list
List by status syncengine-cli indexing list --status=syncing
Check status syncengine-cli indexing status --indexing-id=1
Start job syncengine-cli indexing start --indexing-id=1
Stop job syncengine-cli indexing stop --indexing-id=1
Monitor job syncengine-cli indexing monitor --indexing-id=1

Next Steps

Related articles