--- title: "Index POSIX Metadata in VAST Database" slug: "index-posix-metadata-in-vast-database-34" updated: 2026-10-24T13:46:14Z published: 2026-10-24T12:58:19Z canonical: "kb.vastdata.com/index-posix-metadata-in-vast-database-34" --- > ## Documentation Index > Fetch the complete documentation index at: https://kb.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # Index POSIX Metadata in VAST Database ## Overview SyncEngine lets you extract metadata from a local POSIX store into VAST DataBase where they can be easily queried for further analysis. An indexing job walks a local path on the worker and writes file and directory metadata into a VAST DataBase table. Indexing does not copy file contents; it stores metadata only. On the first run, SyncEngine creates the following VAST DataBase resources (if they do not already exist) on the destination bucket view provisioned on the VAST cluster: - A VAST database schema. The name of the schema is derived from the worker/indexing job label and the source POSIX path specified. - A table under the schema. Each run of an indexing job creates one table in the metadata index database. Complete these steps to set up POSIX metadata indexing: 1. Ensure that the [prerequisites](/documentation/docs/index-posix-metadata-in-vast-database-34#before-you-begin) are met. 2. [Create](/documentation/docs/index-posix-metadata-in-vast-database-34#create-posix-connector) a POSIX connector for the local path to be indexed. 3. [Create](/documentation/docs/index-posix-metadata-in-vast-database-34#create-vast-db-connector) a VAST DB connector for the destination database. 4. [Create](/documentation/docs/index-posix-metadata-in-vast-database-34#create-indexing-job) an indexing job. 5. [Start](/documentation/docs/index-posix-metadata-in-vast-database-34#start-indexing) the indexing. > **Note:** This step is only required when the migration does not have periodic resynchronization enabled. 6. [Monitor](/documentation/docs/index-posix-metadata-in-vast-database-34#monitor-indexing-job) indexing progress and completion. 7. If necessary, [troubleshoot](/documentation/docs/index-posix-metadata-in-vast-database-34#troubleshoot). After indexing is complete, you can run queries against the VAST DataBase table containing the index using VAST DataBase querying methods described in the *VAST Cluster Administrator's Guide*. The following limitation applies: - One source path per indexing job. To index multiple paths, you need to create multiple indexing jobs. ## Before You Begin Ensure that the prerequisites are met: ### VAST Cluster #### Networking - VAST cluster's management IP. - A DNS-resolvable name of the tenant's virtual IP pool to access the destination database, or a virtual IPv4 address from that pool. - Network connectivity from workers to the destination database. #### VAST tenant and admin credentials - If your VAST cluster has multiple VAST tenant configured: - To migrate to one of the tenants: - The name of the VAST tenant where the destination VAST S3 bucket will reside - Username and password for the VAST tenant admin user - To migrate to the default tenant: - Username and password for the VAST cluster admin user - If your VAST cluster is not configured for multi-tenancy: - Username and password for the VAST cluster admin user #### Create User Create a user with S3 access keys allowing to create a bucket and put objects to it. 1. Create a local user on the VAST provider attached to the tenant where the destination database will reside. Ensure that the user has the *Allow Bucket Creation* option enabled. 2. Generate S3 access key for the user. 3. Create an identity policy to grant the user full access to the bucket (`"Action": "s3:*", "Effect": "Allow"`). 4. Assign the identity policy to the user. Note the username and UID of the created user. #### Create View Policy Create a view policy that will control access to the view that will hold the destination database. In the view policy: - Select the **S3 Native** security flavor. #### Create View Create an Element Store view exposing the destination database. In the view settings: - Select the **Database** and **S3 Bucket** protocols. > **Note:** In VAST Web UI, the **S3 Bucket** protocol is added automatically when you select the **Database** option in the **Protocols** field. - Select the view policy you created. - Select the user you created as the bucket owner. ### SyncEngine - The SyncEngine control plane and at least one worker are up and running. - At least one worker label of a deployed worker is known. (The worker label is set when you deploy the worker.) - The source path is mounted on all workers labeled for a particular POSIX indexing job. A worker must have read access to the source path. ## Create POSIX Connector A POSIX connector is used for POSIX filesystem migrations. ### SyncEngine UI 1. In the **Connectors** page, click **New Connector**. A wizard dialog appears to walk you through the connector creation steps. 2. In the **Type & name** step: - Select the **POSIX** connector type. - In the **Connector name** field, enter a unique name for the connector. The name can contain up to 63 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-) and underscores (_). 3. In the **Tunables** step: - Under **Content Types Supported**, determine whether the connector will handle data and metadata, only data, or only metadata. By default, the connector handles both data and metadata. - Under **Batch & Scan Tunables**: - Optionally adjust the size of batches that SyncEngine uses when copying the data. By default (if no values are entered in the fields provided), SyncEngine uses VAST cluster's defaults which are suitable for most workloads. - In the **Batch Size** field, specify the maximum allowed batch size in GB. If not specified, a default of 1 GB is used. - In the **Max Files per Batch** field, specify the maximum allowed count of files per batch. If not specified, a default of 1000 files is used. - Optionally configure the **Block Size** and **Task Flush Threshold** values as needed. 4. In the **Rsync** step, optionally configure `rsync` flags for file transfers. 5. In the **Review** step, review the settings you've made. If you want to make changes, click **Back**. Otherwise, click **Create Connector**. The connector is created and displayed in the **Connectors** page. ### SyncEngine CLI The following command creates a POSIX connector named `src-posix`: ``` syncengine-cli connector create \ --type=filesystem \ --name=src-posix \ ``` > **Note:** The connector type can be specified either as `filesystem` or `posix`. Optional flags: ``` --batch-size-gb --batch-file-count --block-size --task-flush-threshold --rsync-archive --rsync-compress --rsync-checksum --rsync-hard-links --rsync-acls --rsync-xattrs --rsync-no-perms --rsync-no-owner --rsync-no-group --rsync-bwlimit --rsync-timeout ``` The output should be similar to the following: ``` Successfully created connector 'src-posix' with ID 15 ``` > **Tip:** Note the connector ID in the output. ## Create VAST DB Connector A VAST DB connector is used to connect to a VAST database that will store POSIX metadata. ### SyncEngine UI 1. In the **Connectors** page, click **New Connector**. A wizard dialog appears to walk you through the connector creation steps. 2. In the **Type & name** step: - Select the **Vast DB** connector type. - In the **Connector name** field, enter a unique name for the connector. The name can contain up to 63 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-) and underscores (_). 3. In the **Configure** step: - Under **Vast parameters**, provide the details for accessing the destination database: | Field | Description | | --- | --- | | **Management Address** | VAST cluster's management IPv4 address or DNS-resolvable hostname (up to 253 characters). | | **VIP Address** | A DNS-resolvable name of the tenant's virtual IP pool to access the destination database, or a virtual IPv4 address from that pool. | | **Username** and **Password** | Username and password to log in to the VMS. If the destination bucket is on one of VAST cluster tenants, enter credentials of the admin user for that tenant. If the destination bucket is on the default tenant, enter credentials of the VAST cluster admin user. | | **Tenant name** | The name of the VAST cluster tenant where the destination database resides. If not specified, the default tenant is used. | - Under **Catalog Bucket**, use the **Bucket Name** field to enter a name for the S3 bucket that will hold the destination database. The bucket name can contain 3-63 characters, including lowercase English letters, digits, periods (.), and hyphens (-). By default, `syncengine-catalog` is used as the bucket name. - Under **Content Types Supported**, select **Metadata**. > **Note:** Leave the **Data** checkbox unselected. 4. In the **Rate limits** step, leave the defaults unchanged ('0', no limits set). These fields are reserved for future use. 5. In the **Review** step, review the settings you've made. If you want to make changes, click **Back**. Otherwise, click **Create Connector**. The connector is created and displayed in the **Connectors** page. ### SyncEngine CLI The following command creates a destination database in a VAST S3 bucket named `se-index` : ``` syncengine-cli connector create \ --type=vast_db \ --name=vast-indexing \ --mgmt-ip= \ --username=admin \ --password= \ --vip= --bucket-name=se-index ``` Optional flags: ``` --tenant-name ``` > **Note:** Connector rate limits can only be set in SyncEngine UI. The output should be similar to the following: ``` Successfully created connector 'vast-indexing' with ID 146 ``` > **Tip:** Note the connector ID in the output. ## Create Indexing Job Create an indexing job to retrieve metadata from a worker's local POSIX path and store it in a database on a VAST cluster. ### SyncEngine UI 1. In the **Indexing** page, click **Create Indexing Job**. A dialog appears for you to enter indexing parameters. 2. In the **Create Indexing Job** dialog, complete the fields: | Field | Description | | --- | --- | | **Label** | Specify a SyncEngine worker label for the indexing job. The label is used to route indexing jobs to the appropriate SyncEngine worker. One worker can be associated with one or more labels. Workers deployed with a certain label handle all jobs that have that label. | | **Source Connector (POSIX/S3)** | Select the POSIX connector that will be used to connect to the source POSIX path. | | **Destination Connector (VastDB)** | Select the VAST DB connector that will be used to connect to the destination database on the VAST cluster. | | **Source Path** | Enter the source POSIX path (for example: `/mnt/data`). | 3. Optionally, expand the **Batch Settings** pane to adjust the size of batches that SyncEngine uses when copying the data. By default (if no values are entered in the fields provided), SyncEngine uses VAST cluster's defaults which are suitable for most workloads. - In the **Batch size** field, specify the maximum allowed batch size in GB. If not specified, a default of 1GB is used. - In the **Batch file count** field, specify the maximum allowed number of objects per batch. If not specified, a default of 1000 objects is used. 4. In the **Review** step, review the settings you've made. If you want to make changes, click **Back**. Otherwise, click **Create Indexing Job**. The indexing job is created and displayed in the **Indexing** page. ### SyncEngine CLI The following commands creates an indexing job to extract metadata from `/mnt/data` . The migration is to be accomplished by workers labeled with `data-index` using connectors with connector IDs `1` and `2`. ``` syncengine-cli indexing create \ --source-connector-id=1 \ --destination-connector-id=2 \ --source-path=/mnt/data \ --label=data-index ``` > **Note:** Batch settings can only be configured in SyncEngine UI. The CLI creates an indexing job and displays its properties. The output should be similar to the following: ``` Creating indexing job: data-index Source type: posix ✓ Indexing job created successfully ID: 3 Label: data-index Source Path: /mnt/data Status: init ``` > **Tip:** Note the migration ID in the output. ## Start Indexing Iob Indexing jobs are started manually. ### SyncEngine UI Navigate to the **Indexing** page and click the start button for the indexing job you want. ### SyncEngine CLI Run the command: ``` syncengine-cli indexing start --indexing-id= ``` The output should be similar to the following: ``` ✓ Indexing job started successfully Sync Job ID: 3 Status: pending ``` ## Monitor Indexing Job As the indexing progresses, its status changes as follows: `Not Started` (or `init`) -> `Waiting for Worker` (or `pending`) -> `Syncing` -> `Synced`. When complete, the destination S3 bucket will have objects with object keys in format: - `<file ID>/onedrive.<file extension>` - the migrated content, with the original file extensions preserved as part of the object key - `<file ID>/onedrive.metadata.json` - metadata, such as source URL, source title, original last-modified timestamp, ACL-related fields, additional OneDrive metadata (if the option was enabled for the migration). ### SyncEngine UI Check the migration status in the **Indexing** page. ### SyncEngine CLI Run the command: ``` syncengine-cli indexing status --indexing-id= ``` The output should be similar to the following: ``` Indexing Job #3 ================ Label: data-index Status: synced Source Path: /mnt/data Created: N/A Progress: Files Indexed: 42 Total Size: 1.2 MB Current Rate: 0 files/sec ``` ## Troubleshoot | Symptom | Likely Cause | Solution | | --- | --- | --- | | Indexing job stays *pending* | Indexing label does not match worker label. | Update the indexing job to use the same label as the deployed worker. | | Indexing fails writing to the VAST cluster; VMS authentication or virtual IP reachability errors during indexing | Invalid credentials, or wrong VMS IP or virtual IP | Confirm VAST Cluster's management IP, the virtual IP, and user credentials. | | Source path not found or permission denied | Path missing or not readable on the worker | Mount the path on every worker for the indexing label used; verify permissions. | | Empty schema / no tables in Catalog Query | Job never reached the *Synced* status, or wrong VAST DB connector | Confirm job status; select the appropriate VAST DB connector. | ## POSIX Metadata Index Reference In the metadata index table, each row is one filesystem element (file, directory, symlink, and so on), and the columns are as follows: | Column | Type | Description | | --- | --- | --- | | `path` | string | Absolute path of the element | | `parent_path` | string | Absolute path of the parent directory | | `depth` | int32 | Depth relative to the indexing source path ('0' at the root) | | `name` | string | Base name of the element | | `element_type` | string | Element type: FILE, DIR, SYMLINK, HARD_LINK, or special types such as FIFO, SOCKET, CHAR_DEVICE, BLOCK_DEVICE, OTHER | | `extension` | string | File extension without the leading period; empty for directories | | `handle_id` | decimal | Inode number | | `size` | int64 | Size in bytes | | `total_used` | int64 | Allocated bytes | | `uid` | int32 | Owner user ID | | `gid` | int32 | Owner group ID | | `nfs_mode_bits` | int32 | Mode bits | | `ctime` | timestamp (seconds) | Change time | | `mtime` | timestamp (seconds) | Modification time | | `atime` | timestamp (seconds) | Access time | | `nlinks` | int32 | Hard-link count | | `scrape_duration` | int32 | Seconds spent scraping a directory; null for non-directories | | `user_tags_count` | int16 | Count of user tags (when present) | | `user_metadata` | map | User metadata map | | `user_tags` | map | User tags map | | `file_count` | int64 | For directories: number of child files counted during the walk; null for files | | `total_size` | int64 | For directories: aggregate size of child files; null for files |