Index POSIX Metadata in VAST Database

Overview

SyncEngine lets you extract metadata from a local POSIX store into VAST DataBase where they can be easily queried for further analysis.

An indexing job walks a local path on the worker and writes file and directory metadata into a VAST DataBase table. Indexing does not copy file contents; it stores metadata only.

On the first run, SyncEngine creates the following VAST DataBase resources (if they do not already exist) on the destination bucket view provisioned on the VAST cluster:

• A VAST database schema. The name of the schema is derived from the worker/indexing job label and the source POSIX path specified.
• A table under the schema. Each run of an indexing job creates one table in the metadata index database.

Complete these steps to set up POSIX metadata indexing:

1. Ensure that the prerequisites are met.

2. Create a POSIX connector for the local path to be indexed.

3. Create a VAST DB connector for the destination database.

4. Create an indexing job.

5. Start the indexing.

   Note: This step is only required when the migration does not have periodic resynchronization enabled.

6. Monitor indexing progress and completion.

7. If necessary, troubleshoot.

After indexing is complete, you can run queries against the VAST DataBase table containing the index using VAST DataBase querying methods described in the VAST Cluster Administrator's Guide.

The following limitation applies:

• One source path per indexing job. To index multiple paths, you need to create multiple indexing jobs.

Before You Begin

Ensure that the prerequisites are met:

VAST Cluster

Networking

• VAST cluster's management IP.
• A DNS-resolvable name of the tenant's virtual IP pool to access the destination database, or a virtual IPv4 address from that pool.
• Network connectivity from workers to the destination database.

VAST tenant and admin credentials

• If your VAST cluster has multiple VAST tenant configured:
  • To migrate to one of the tenants:
    • The name of the VAST tenant where the destination VAST S3 bucket will reside
    • Username and password for the VAST tenant admin user
  • To migrate to the default tenant:
    • Username and password for the VAST cluster admin user
• If your VAST cluster is not configured for multi-tenancy:
  • Username and password for the VAST cluster admin user

Create User

Create a user with S3 access keys allowing to create a bucket and put objects to it.

1. Create a local user on the VAST provider attached to the tenant where the destination database will reside.
   Ensure that the user has the Allow Bucket Creation option enabled.
2. Generate S3 access key for the user.
3. Create an identity policy to grant the user full access to the bucket ("Action": "s3:*", "Effect": "Allow").
4. Assign the identity policy to the user.
   Note the username and UID of the created user.

Create View Policy

Create a view policy that will control access to the view that will hold the destination database. In the view policy:

• Select the S3 Native security flavor.

Create View

Create an Element Store view exposing the destination database. In the view settings:

• Select the Database and S3 Bucket protocols.

  Note: In VAST Web UI, the S3 Bucket protocol is added automatically when you select the Database option in the Protocols field.

• Select the view policy you created.

• Select the user you created as the bucket owner.

SyncEngine

• The SyncEngine control plane and at least one worker are up and running.
• At least one worker label of a deployed worker is known. (The worker label is set when you deploy the worker.)
• The source path is mounted on all workers labeled for a particular POSIX indexing job. A worker must have read access to the source path.

Create POSIX Connector

A POSIX connector is used for POSIX filesystem migrations.

SyncEngine UI

1. In the Connectors page, click New Connector.
   A wizard dialog appears to walk you through the connector creation steps.

2. In the Type & name step:

   • Select the POSIX connector type.
   • In the Connector name field, enter a unique name for the connector.
     The name can contain up to 63 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-) and underscores (_).

3. In the Tunables step:

   • Under Content Types Supported, determine whether the connector will handle data and metadata, only data, or only metadata. By default, the connector handles both data and metadata.
   • Under Batch & Scan Tunables:
     • Optionally adjust the size of batches that SyncEngine uses when copying the data. By default (if no values are entered in the fields provided), SyncEngine uses VAST cluster's defaults which are suitable for most workloads.
       • In the Batch Size field, specify the maximum allowed batch size in GB. If not specified, a default of 1 GB is used.
       • In the Max Files per Batch field, specify the maximum allowed count of files per batch. If not specified, a default of 1000 files is used.
       • Optionally configure the Block Size and Task Flush Threshold values as needed.

4. In the Rsync step, optionally configure rsync flags for file transfers.

5. In the Review step, review the settings you've made.
   If you want to make changes, click Back. Otherwise, click Create Connector.
   The connector is created and displayed in the Connectors page.

SyncEngine CLI

The following command creates a POSIX connector named src-posix:

syncengine-cli connector create \
  --type=filesystem \
  --name=src-posix \

Note: The connector type can be specified either as filesystem or posix.

Optional flags:

--batch-size-gb <GB>
--batch-file-count <count>
--block-size <bytes>
--task-flush-threshold <count>
--rsync-archive
--rsync-compress
--rsync-checksum
--rsync-hard-links
--rsync-acls
--rsync-xattrs
--rsync-no-perms
--rsync-no-owner
--rsync-no-group
--rsync-bwlimit <KB/s>
--rsync-timeout <seconds>

The output should be similar to the following:

Successfully created connector 'src-posix' with ID 15

Tip: Note the connector ID in the output.

Create VAST DB Connector

A VAST DB connector is used to connect to a VAST database that will store POSIX metadata.

SyncEngine UI

1. In the Connectors page, click New Connector.
   A wizard dialog appears to walk you through the connector creation steps.

2. In the Type & name step:

   • Select the Vast DB connector type.
   • In the Connector name field, enter a unique name for the connector.
     The name can contain up to 63 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-) and underscores (_).

3. In the Configure step:

   • Under Vast parameters, provide the details for accessing the destination database:

     Field	Description
     Management Address	VAST cluster's management IPv4 address or DNS-resolvable hostname (up to 253 characters).
     VIP Address	A DNS-resolvable name of the tenant's virtual IP pool to access the destination database, or a virtual IPv4 address from that pool.
     Username and Password	Username and password to log in to the VMS. If the destination bucket is on one of VAST cluster tenants, enter credentials of the admin user for that tenant. If the destination bucket is on the default tenant, enter credentials of the VAST cluster admin user.
     Tenant name	The name of the VAST cluster tenant where the destination database resides. If not specified, the default tenant is used.

   • Under Catalog Bucket, use the Bucket Name field to enter a name for the S3 bucket that will hold the destination database.
     The bucket name can contain 3-63 characters, including lowercase English letters, digits, periods (.), and hyphens (-).
     By default, syncengine-catalog is used as the bucket name.

   • Under Content Types Supported, select Metadata.

     Note: Leave the Data checkbox unselected.

4. In the Rate limits step, leave the defaults unchanged ('0', no limits set). These fields are reserved for future use.

5. In the Review step, review the settings you've made.
   If you want to make changes, click Back. Otherwise, click Create Connector.
   The connector is created and displayed in the Connectors page.

SyncEngine CLI

The following command creates a destination database in a VAST S3 bucket named se-index :

syncengine-cli connector create \
  --type=vast_db \
  --name=vast-indexing \
  --mgmt-ip=<VMS IP> \
  --username=admin \
  --password=<password> \
  --vip=<VAST VIP>
  --bucket-name=se-index

Optional flags:

--tenant-name <VAST tenant name>

Note: Connector rate limits can only be set in SyncEngine UI.

The output should be similar to the following:

Successfully created connector 'vast-indexing' with ID 146

Tip: Note the connector ID in the output.

Create Indexing Job

Create an indexing job to retrieve metadata from a worker's local POSIX path and store it in a database on a VAST cluster.

SyncEngine UI

1. In the Indexing page, click Create Indexing Job.
   A dialog appears for you to enter indexing parameters.

2. In the Create Indexing Job dialog, complete the fields:

   Field	Description
   Label	Specify a SyncEngine worker label for the indexing job. The label is used to route indexing jobs to the appropriate SyncEngine worker. One worker can be associated with one or more labels. Workers deployed with a certain label handle all jobs that have that label.
   Source Connector (POSIX/S3)	Select the POSIX connector that will be used to connect to the source POSIX path.
   Destination Connector (VastDB)	Select the VAST DB connector that will be used to connect to the destination database on the VAST cluster.
   Source Path	Enter the source POSIX path (for example: /mnt/data).

3. Optionally, expand the Batch Settings pane to adjust the size of batches that SyncEngine uses when copying the data.
   By default (if no values are entered in the fields provided), SyncEngine uses VAST cluster's defaults which are suitable for most workloads.

   • In the Batch size field, specify the maximum allowed batch size in GB. If not specified, a default of 1GB is used.
   • In the Batch file count field, specify the maximum allowed number of objects per batch. If not specified, a default of 1000 objects is used.

4. In the Review step, review the settings you've made.
   If you want to make changes, click Back. Otherwise, click Create Indexing Job.
   The indexing job is created and displayed in the Indexing page.

SyncEngine CLI

The following commands creates an indexing job to extract metadata from /mnt/data . The migration is to be accomplished by workers labeled with data-index using connectors with connector IDs 1 and 2.

syncengine-cli indexing create \
  --source-connector-id=1 \
  --destination-connector-id=2 \
  --source-path=/mnt/data \
  --label=data-index

Note: Batch settings can only be configured in SyncEngine UI.

The CLI creates an indexing job and displays its properties.

The output should be similar to the following:

Creating indexing job: data-index
  Source type: posix
✓ Indexing job created successfully
  ID: 3
  Label: data-index
  Source Path: /mnt/data
  Status: init

Tip: Note the migration ID in the output.

Start Indexing Iob

Indexing jobs are started manually.

SyncEngine UI

Navigate to the Indexing page and click the start button for the indexing job you want.

SyncEngine CLI

Run the command:

syncengine-cli indexing start --indexing-id=<indexing job ID>

The output should be similar to the following:

✓ Indexing job started successfully
  Sync Job ID: 3
  Status: pending

Monitor Indexing Job

As the indexing progresses, its status changes as follows: Not Started (or init) -> Waiting for Worker (or pending) -> Syncing -> Synced.

When complete, the destination S3 bucket will have objects with object keys in format:

• <file ID>/onedrive.<file extension> - the migrated content, with the original file extensions preserved as part of the object key

• <file ID>/onedrive.metadata.json - metadata, such as source URL, source title, original last-modified timestamp, ACL-related fields, additional OneDrive metadata (if the option was enabled for the migration).

SyncEngine UI

Check the migration status in the Indexing page.

SyncEngine CLI

Run the command:

syncengine-cli indexing status --indexing-id=<indexing job ID>

The output should be similar to the following:

Indexing Job #3
================
Label: data-index
Status: synced
Source Path: /mnt/data
Created: N/A

Progress:
  Files Indexed: 42
  Total Size: 1.2 MB
  Current Rate: 0 files/sec

Troubleshoot

POSIX Metadata Index Reference

In the metadata index table, each row is one filesystem element (file, directory, symlink, and so on), and the columns are as follows: