--- title: "Migrate from Confluence to VAST S3" slug: "migrate-from-confluence-to-vast-s3-34" updated: 2026-10-24T15:07:10Z published: 2026-10-24T15:07:10Z canonical: "kb.vastdata.com/migrate-from-confluence-to-vast-s3-34" --- > ## Documentation Index > Fetch the complete documentation index at: https://kb.vastdata.com/llms.txt > Use this file to discover all available pages before exploring further. # Migrate from Confluence to VAST S3 ## Overview The Confluence connector extracts pages, blog posts, comments, attachments, and other Confluence space content from Confluence Cloud into a VAST S3 bucket. SyncEngine uses the Confluence REST API with an Atlassian API token to access the content, converts text to Markdown, and transforms Confluence permissions into S3 ACLs. The following capabilities are supported for Confluence to VAST S3 migrations: - To manage the destination: - [Auto-Create Destination Bucket](/documentation/docs/migrate-from-confluence-to-vast-s3-34#autocreate-destination-bucket) - [Add Key Prefix to Migrated Objects](/documentation/docs/migrate-from-confluence-to-vast-s3-34#add-key-prefix-to-migrated-objects) - To determine when data synchronization occurs: - [Periodic Resync](/documentation/docs/migrate-from-confluence-to-vast-s3-34#periodic-resync) - To manage which items are being synchronized: - [Skip File Types](/documentation/docs/migrate-from-confluence-to-vast-s3-34#skip-file-types) - [Mirror Deletions](/documentation/docs/migrate-from-confluence-to-vast-s3-34#mirror-deletions) - To manage conversions of data format and permissions: - [Convert to Markdown](/documentation/docs/migrate-from-confluence-to-vast-s3-34#convert-to-markdown) - [Convert Permissions to S3 ACLs](/documentation/docs/migrate-from-confluence-to-vast-s3-34#convert-permissions-to-s3-acls) - To configure SyncEngine processes: - [Confluence Rate Limits](/documentation/docs/migrate-from-confluence-to-vast-s3-34#confluence-rate-limits) To set up migration from Confluence to a VAST S3 bucket: 1. Ensure that the [prerequisites](/documentation/docs/migrate-from-confluence-to-vast-s3-34#before-you-begin) are met. 2. [Create](/documentation/docs/migrate-from-confluence-to-vast-s3-34#create-confluence-connector) a connector for Confluence. 3. [Create](/documentation/docs/migrate-from-confluence-to-vast-s3-34#create-vast-d3-connector) a connector for VAST S3. 4. [Create](/documentation/docs/migrate-from-confluence-to-vast-s3-34#create-migration) a migration from the source Confluence space to the destination VAST S3 bucket. 5. [Start](/documentation/docs/migrate-from-confluence-to-vast-s3-34#start-migration) the migration. > **Note:** This step is only required when the migration does not have periodic resynchronization enabled. 6. [Monitor](/documentation/docs/migrate-from-confluence-to-vast-s3-34#monitor-migration) migration progress and completion. 7. If necessary, [troubleshoot](/documentation/docs/migrate-from-confluence-to-vast-s3-34#troubleshoot). The following limitations apply: - Confluence content can only be migrated to a VAST S3 bucket. No other destination types are supported. - One Confluence space per migration. If you need to migrate multiple spaces, multiple migrations are to be set up. ## Features ### Auto-Create Destination Bucket SyncEngine can automatically create a destination bucket on the VAST cluster based on the settings you've made when configuring the migration. For migrations from Confluence to VAST S3, SyncEngine attempts to automatically create a destination bucket when you submit a migration job. If it does not succeed, create the bucket manually so that the bucket name starts with the word 'syncengine' followed by a period (.): `syncengine.<your string>`. ### Add Key Prefix to Migrated Objects You can configure SyncEngine to add a certain object key prefix to all objects created in the destination bucket as a result of a migration. Adding migration-specific prefixes allows for migrating from multiple source stores to the same destination S3 bucket. otherwise, you would need to provision a separate destination bucket for each of the source stores. To configure the object key prefix to be added: - In SyncEngine UI, append the prefix to the bucket name in the **Destination S3 Bucket** field when creating a migration: `/<bucket name>/<object key prefix>/`. (Note that you need a slash before and after the prefix string.) - In SyncEngine CLI, include the prefix on the `--bucket-name` option on the `migration create` command: `--bucket-name=syncengine.<bucket name>/<object key prefix>/` ### Periodic Resync In addition to starting a migration once it is created, SyncEngine can be configured to automatically run a resynchronization at user-supplied intervals. - In SyncEngine UI, set the **SyncMode** to **Automatic** and enter an interval (in hours) in the **Sync Interval** field. - In SyncEngine CLI, include the `--sync-mode automatic=true` and `--sync-interval <interval>` options when submitting the `migration create` command. For migrations set to **Automatic** synchronization mode, the first synchronization starts immediately after the migration is created, and the subsequent synchronizations are performed periodically with the specified interval. A subsequent synchronization cannot be triggered manually. To be able to manually start a synchronization on demand, set **SyncMode** to **Manual**. ### Skip File Types When migrating from Confluence, attachments with `.tmp` extensions are by default not migrated. You can alter the default behavior by supplying a regular expression to filter out attachments which you do not want to migrate. - In SyncEngine UI, use the Attachment Filter Regex field in connector settings. - In SyncEngine CLI, include the `--attachment-filter <regex>` option on the `connector create` command. The default attachment filter `(.*\.tmp$)` skips all files with the `.tmp` extension. ### Mirror Deletions You can determine how SyncEngine handles the items in the source store that have been deleted since the previous migration run. By default, SyncEngine does not delete such items from the destination. If you want to have them deleted, configure the migration to mirror source deletions at the destination: - In SyncEngine UI, select the **Mirror source deletions at destination** option when creating a migration. - In SyncEngine CLI, include the `--enable-deletes=true` option when submitting the `migration create` command. > **Note:** Deletion of items at the destination cannot be undone by disabling this setting. ### Convert to Markdown By default, files with `.pdf` and `.md` extensions are stored as-is. Files of other types are converted to Markdown where possible. You can alter the default behavior by supplying a comma-separated list of file extensions for file types that do no need to be converted to Markdown: - In SyncEngine UI, use the **AIQ Supported Extensions** field in connector settings. - In SyncEngine CLI, include the `--aiq-extensions` option on the `connector create` command. ### Convert Permissions to S3 ACLs When migrating from a non-S3 source to VAST S3, SyncEngine converts source permissions to S3 ACLs. You can configure SyncEngine to filter out certain users, groups and/or domains during the conversion by supplying a comma-separated list of domains or email addresses of users/groups to be excluded. - In SyncEngine UI, use the **Ignored Users**, **Ignored Groups**, **Ignored Domains** options in connector settings. - In SyncEngine CLI, include the `--ignored-users <user1,user2...>`, `--ignored-groups <group1,group2,...>`, `--ignored-domains <domain1,domain2,...>` options on the `connector create` command. ### Confluence Rate Limits You can set request rate limits to prevent SyncEngine from causing excessive load on Confluence: - Cap the overall number of requests (per second) that SyncEngine sends to the Confluence API. - Restrict throughput when downloading the files from Confluence. Each of the limits applies to the aggregate amount of requests or throughput by all workers used by the connector. Each limit is enforced independently of the other limit. On reaching a limit, workers wait and retry so that synchronization continues after the pause. By default, no request rate limits are set. Set rate limits if during a representative migration, vendor 429/503/SlowDown errors appear in the worker logs, and the Grafana **Connector Rate Limits** panel (or the Prometheus metrics `connector_rate_limit_waits_total{connector_id, axis}`) shows the wait counter growing. If no throttling and no vendor errors occur, you do not need to set rate limits. For recommended request rates for Confluence, see [Confluence documentation](https://developers.google.com/workspace/drive/api/guides/limits). Typical rate limits you may need when migrating from Confluence to VAST S3 are 5–10 API calls per second and 50–100 MB/s. To set rate limits: - In SyncEngine UI, use the **API calls per second** and **Bytes per second** fields in Confluence connector settings. > **Note:** Connector rate limits can only be set in SyncEngine UI. Avoid configuring multiple connectors for the same Confluence account with rate limits that, when combined, exceed the maximum allowed request rate for the Confluence API. Use a single connector instead. ## Before You Begin Ensure that the prerequisites are met: ### Confluence - Confluence Cloud site URL (for example `https://your-company.atlassian.net`). - Email address of the user with read access to the Confluence space to be migrated. - Atlassian API token with Confluence scope (create one from [API Tokens](https://id.atlassian.com/manage-profile/security/api-tokens) page of your Atlassian service account). - The space key of the Confluence space to be migrated (`https://your-company.atlassian.net/wiki/spaces/<space key>/<...>`). ### VAST Cluster #### Networking - VAST cluster's management IP. - A DNS-resolvable name of the tenant's virtual IP pool to access the VAST S3 endpoint, or a virtual IPv4 address from that pool. - Network connectivity from workers to the destination VAST S3 endpoint. #### VAST Tenant and Admin Credentials - If your VAST cluster has multiple VAST tenant configured: - To migrate to one of the tenants: - The name of the VAST tenant where the destination VAST S3 bucket will reside - Username and password for the VAST tenant admin user - To migrate to the default tenant: - Username and password for the VAST cluster admin user - If your VAST cluster is not configured for multi-tenancy: - Username and password for the VAST cluster admin user #### Create User Create a user with S3 access keys allowing to create a bucket and put objects to it. 1. Create a local user on the VAST provider attached to the tenant where the destination VAST S3 bucket will reside. Ensure that the user has the *Allow Bucket Creation* option enabled. 2. Generate S3 access key for the user. 3. Create an identity policy to grant the user full access to the destination bucket (`"Action": "s3:*", "Effect": "Allow"`). 4. Assign the identity policy to the user. Note the username and UID of the created user. #### Active Directory - The VAST cluster must be joined to Active Directory (required to map Google Drive user permissions to S3 ACLs). - (Recommended) Ensure that Active Directory Domain Auto-Discovery is enabled for the Active Directory provider that is attached to the tenant where the destination bucket resides. ### Active Directory - The Active Directory must have mapping entries for Confluence users and groups. For example, for a Confluence user with email address `alice@company.com`, Active Directory must have `sAMAccountName` with the value `alice`. ### SyncEngine - The SyncEngine control plane and at least one worker are up and running. - At least one worker label of a deployed worker is known. (The worker label is set when you deploy the worker.) ## Create Confluence Connector A Confluence connector is used to connect to the Confluence Cloud where your source Confluence space is located. ### SyncEngine UI 1. In the **Connectors** page, click **New Connector**. A wizard dialog appears to walk you through the connector creation steps. 2. In the **Type & name** step: - Select the **Confluence** connector type. - In the **Connector name** field, enter a unique name for the connector. The name can contain up to 63 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-) and underscores (_). 3. In the **Authentication** step, complete the fields: | Field | Description | | --- | --- | | **Confluence Base URL** | Your Atlassian Cloud base URL, for example: `https://you-company.atlassian.net`. | | **Username / Email** | The email address that you use to log in to the Confluence space. The email address must be up to 255 characters. | | **API Token** | Your Atlassian API token with Confluence scope. | 4. In the **Filters** step, configure the attachment filter, the user filter, and the Markdown conversion filter: | Field | Description | | --- | --- | | **Attachment Filter Regex** | Enter a regular expression to filter out attachments which you do not want to migrate. The default attachment filter `(.*\.tmp$)` skips all files with the `.tmp` extension. | | **Ignored Users** | Enter a comma-separated list of email addresses of users to be excluded when converting Confluence permissions to S3 ACLs. | | **Ignored Groups** | Enter a comma-separated list of email addresses of user groups to be excluded when converting Confluence permissions to S3 ACLs. | | **Ignored Domains** | Enter a comma-separated list of email address domains to be excluded when converting Confluence permissions to S3 ACLs. | | **AIQ Supported Extensions** | Enter a comma-separated list of file extensions to be extracted from Confluence as-is (without conversion to Markdown). By default, PDF and MD files are extracted. | 5. In the **Rate limits** step, set optional [request rate limits](/documentation/docs/confluence-rate-limits) for the connector: | Field | Description | | --- | --- | | **API calls per second** | Maximum allowed number of API calls per second. Leave blank for unlimited (default). | | **Bytes per second** | Maximum allowed download throughput. Leave blank for unlimited (default). | 6. In the **Review** step, review the settings you've made. If you want to make changes, click **Back**. Otherwise, click **Create Connector**. The connector is created and displayed in the **Connectors** page. ### SyncEngine CLI The following command creates a Confluence connector named `src-conf` for the Confluence user with email address `kirk@my-company.com` at the Confluence base URL `https://my-company.atlassian.net`: ``` syncengine-cli connector create \ --type=confluence \ --name=src-conf \ --base-url=htttps://my-company.atlassian.net \ --username=kirk@my-company.com \ --api-token= ``` Optional flags: ``` --attachment-filter --ignored-users --ignored-groups --ignored-domains --aiq-extensions ``` The output should be similar to the following: ``` Successfully created connector 'src-conf' with ID 15 ``` > **Tip:** Note the connector ID in the output. ## Create VAST S3 Connector A VAST S3 connector is used to connect to the VAST S3 server on the VAST cluster to which you want to migrate the data. ### SyncEngine UI 1. In the **Connectors** page, click **New Connector**. A wizard dialog appears to walk you through the connector creation steps. 2. In the **Type & name** step: - Select the **Vast S3** connector type. - In the **Connector name** field, enter a unique name for the connector. The name can contain up to 63 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-) and underscores (_). 3. In the **VMS cluster** step, enter VAST cluster's VMS IP and admin credentials: | Field | Description | | --- | --- | | **Management IP** | VAST cluster's management IPv4 address or DNS-resolvable hostname (up to 253 characters). | | **Username** and **Password** | Username and password to log in to the VMS. If the destination bucket is on one of VAST cluster tenants, enter credentials of the admin user for that tenant. If the destination bucket is on the default tenant, enter credentials of the VAST cluster admin user. | | **Tenant name** | The name of the VAST cluster tenant where the destination bucket resides. If not specified, the default tenant is used. | 4. In the **S3 endpoint** step, configure an S3 endpoint on the VAST cluster. | Field | Description | | --- | --- | | **VIP** | A DNS-resolvable name of the tenant's virtual IP pool to access the VAST S3 endpoint, or a virtual IPv4 address from that pool. A DNS-resolvable name is recommended. | | **Access Key** | Enter the S3 access key for the bucket owner user. | | **Secret Key** | Enter the private key for the bucket owner user. | | **S3 Username** | Enter the username for the bucket owner user. | | **S3 UID** | The UID of the bucket owner user. | | **Endpoint URL** | The URL to access the S3 endpoint. This field is optional. If not specified, the URL is derived from the **VIP** field. | | **Use secure S3 connection (HTTPS)** | Select to use HTTPS. If selected, you'll need to provide a TLS certificate in a next step. If not selected (default), HTTP is used. | 5. In the **Bucket & advanced** step, optionally configure advanced settings for the destination S3 bucket: | Field | Description | | --- | --- | | **Case-sensitive bucket names** | When enabled, SyncEngine allows for bucket names that contain both lowercase and uppercase characters. | | **Checksum Algorithm** | Checksum algorithm for S3 PUT operations: `CRC32C` (default), `CRC64NVME` or `SHA256`. Note that `CRC64NVME` and `SHA256` require VAST Cluster 5.5.0 or later. SyncEngine will verify if your VAST cluster supports checksum verification for the selected algorithm. | | **Read Timeout** | Socket read timeout, in seconds. Valid values: 60–3600. Default is 300. | | **Connect Timeout** | S3 endpoint connection timeout, in seconds. Valid values: 5–300. Default is 60. | | **Max Retries** | The maximum allowed number of retries by the VAST S3 client. Valid values: 1–100. Default is 3. | | **Retry Mode** | Select one of VAST S3 client retry modes: **Adaptive** (default) for rate-limiting of requests when throttled, **Standard** for exponential backoff with jitter, **Legacy** for simple exponential backoff. | 6. In the **Rate limits** step, leave the defaults unchanged ('0', no limits set). These fields are reserved for future use. 7. In the **SSL** step, use the **SSL Certificate Content** field to enter the TLS certificate (in PEM format) to secure the S3 connection. If left blank, default CA certificates are used. 8. In the **Review** step, review the settings you've made. If you want to make changes, click **Back**. Otherwise, click **Create Connector**. The connector is created and displayed in the **Connectors** page. ### SyncEngine CLI The following command creates a destination VAST S3 connector named `dest-vast-s3` to connect to the S3 endpoint at `<VAST S3 VIP>` on a VAST cluster with a management IP of `<VMS IP>`: ``` syncengine-cli connector create \ --type=vast_s3 \ --name=dest-vast-s3 \ --mgmt-ip= \ --username=admin \ --password= \ --vip= \ --access-key= \ --secret-key= ``` > **Note:** To auto-generate S3 access keys, omit the `--access-key` and `--secret-key` options from the command. Optional flags when creating a connector of the `vast_s3` type: ``` --s3-username # Default: SYNCENGINE_USER --s3-uid # Default: 270270270 --tenant-name --secure-s3={true|false} --checksum-algorithm {CRC32C|CRC64NVME|SHA256} --case-sensitive-bucket-names={true|false} --read-timeout --client-connect-timeout --client-retries-max-attempts --client-retries-mode --ssl-cert-file ``` The output should be similar to the following: ``` Successfully created connector 'dest-vast-s3' with ID 37 ``` > **Tip:** Note the connector ID in the output. ## Create Migration Create a migration to copy the data from your Confluence space to an S3 bucket on the VAST cluster. ### SyncEngine UI 1. In the **Migrations** page, click **New Migration**. A wizard dialog appears to walk you through the migration creation steps. 2. In the **Source** step, select the Confluence connector that will be used to connect to your Confluence space. 3. In the **Destination** step, select the VAST S3 connector that will be used to connect to the destination VAST cluster. 4. In the **Paths** step: - Under **Migration Label**, specify a SyncEngine worker label for the migration. The label is used to route migration jobs to the appropriate SyncEngine worker. One worker can be associated with one or more migration labels. Workers deployed with a certain label handle jobs for all migrations that have that label. You can create a new label or use an existing one. - To create a new label, select **Create new label** and enter a text string in the **New Label** field. The label can contain up to 128 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-) and underscores (_). - To use an existing label, select **Use existing label** and enter the label string in the **Existing Label** field. - Under **Paths**, specify the paths to the migration source and destination. - Under **Confluence Space**, select a Confluence space from the list of recent spaces, or enter the space key manually in the field provided. - In the **Destination S3 Bucket** field, enter a name for the S3 bucket on the VAST cluster. SyncEngine will automatically create the bucket if it does not exist. If you want SyncEngine to [add](/documentation/docs/migrate-from-confluence-to-vast-s3-34#add-key-prefix-to-migrated-objects) a certain object key prefix to the migrated objects, specify the prefix followed by a trailing slash: `<bucket name>/<object key prefix>/`. The string that you enter will automatically be prepended with the word 'syncengine' followed by a period (.): `syncengine.<your string>`. Special characters in the string will be replaced with hyphens. 5. In the **Sync mode** step, specify if you want to run the migration once or [periodically](/documentation/docs/migrate-from-confluence-to-vast-s3-34#periodic-resync): - To run the migration once, select **Manual** in the **SyncMode** field. Migration will be started after you submit the migration job manually. - To run the migration automatically at certain intervals, select **Automatic** in the **Sync Mode** field and enter an interval (in hours) in the **Sync Interval** field. The default interval is 24 hours. 6. In the **What you're copying** step: - Review the settings you've made. If necessary, click **Back** and change as needed. - Decide how SyncEngine [handles the files that have been deleted](/documentation/docs/migrate-from-confluence-to-vast-s3-34#mirror-deletions) since the previous migration run. By default, SyncEngine does not delete such files from the destination S3 bucket. If you want to have them deleted, select **Mirror source deletions on the destination**. 7. In the **Per-task timeouts** step, optionally configure suitable timeouts for the migration jobs. 8. In the **Review** step, review the settings you've made. If you want to make changes, click **Back**. Otherwise, click **Create Migration**. The migration is created and displayed in the **Migration** page. ### SyncEngine CLI The following command creates a migration from a Confluence space with the space key `ENG` to a VAST S3 bucket named `syncengine.eng`. The migration is to be accomplished by workers labeled with `MYFIRSTMIGRATION` using connectors with connector IDs `14` and `37`. ``` syncengine-cli migration create \ --source-connector-id=14 \ --destination-connector-id=37 \ --space-key=ENG \ --bucket-name=syncengine.eng \ --label=MYFIRSTMIGRATION ``` Optional flags: ``` --sync-mode {manual|automatic --sync-interval --enable-deletes={true|false} ``` The CLI creates a migration and displays its properties. > **Tip:** Note the migration ID in the output. ## Start Migration Manual migration start is only relevant for migrations that do not have periodic resynchronization configured. For migrations with periodic resynchronization, the migration starts automatically upon creation. ### SyncEngine UI Navigate to the ​**Migrations**​​ page and click the start button for the migration you want. ### SyncEngine CLI Run the command: ​ syncengine-cli migration start --migration-id=
​
The output should be similar to the following:
​
Starting sync job for migration: MYFIRSTMIGRATION
✓ Sync job created successfully!
Job ID: 2
Status: pending ## Monitor Migration As the migration progresses, its status changes as follows: `Not Started` (or `init`) -> `Waiting for Worker` (or `pending`) -> `Syncing` -> `Synced`. When complete, the destination S3 bucket will have objects prefixed with `pages/`, `blogposts/`, `comments/`, and `attachments/`. ### SyncEngine UI Check the migration status in the **Migrations** page. ### SyncEngine CLI Run the command: ``` syncengine-cli migration status --migration-id= ``` ## Troubleshoot | Symptom | Likely Cause | Solution | | --- | --- | --- | | Sync job stays *pending* | Migration label does not match worker label | Update the migration to use the same label as the deployed worker. | | Sync fails writing to the VAST cluster | Invalid S3 credentials, or unreachable S3 virtual IP | Confirm access keys and S3 virtual IP reachability from workers | | `Destination bucket ... does not exist` | SyncEngine could not create the destination bucket. | Create the bucket manually, then recreate and start migration. The bucket name must start with the `syncengine.` prefix. | | Confluence returns `401 Unauthorized` | Wrong email address or Atlassian API token | Recreate the token. Confirm that the email address matches the token owner. | | Migration complete but the destination bucket is empty | Wrong space key or insufficient access | Verify space key and user read permissions. | | Missing ACL grantees | Active Directory principal not mirrored on the VAST cluster | Create matching Active Directory user/group. Confirm that the VAST cluster has joined the Active Directory domain. |