SyncEngine Confluence Migration Guide

This guide walks you through migrating Confluence pages to a VAST S3 bucket.

Overview

The Confluence connector extracts pages, blog posts, comments, attachments, and other Confluence space content from Confluence Cloud into a VAST S3 bucket. SyncEngine uses the Confluence REST API with an Atlassian API token to access the content, converts text to Markdown, and transforms Confluence permissions into S3 ACLs.

To set up migration from Confluence to a VAST S3 bucket:

1. Ensure that the prerequisites are met.

2. Create a connector for Confluence.

3. Create a connector for VAST S3.

4. Create and start a migration from the source Confluence space to the destination VAST S3 bucket.

5. Monitor migration progress and completion.

6. If necessary, troubleshoot.

The following limitations apply:

• Confluence content can only be migrated to a VAST S3 bucket. No other destination types are supported.

• One Confluence space per migration. If you need to migrate multiple spaces, multiple migrations need to be set up.

Handling Attachments to Confluence Pages

By default, attached files with .pdf and .md extensions are stored as is. Attachments of other types are converted to Markdown where possible. Attachments with .tmp extensions are not migrated.

You can alter the default behavior when creating the Confluence connector.

Automatic Creation of Destination S3 Bucket

When you submit a migration job, SyncEngine attempts to automatically create a destination S3 bucket on the VAST cluster.

If it does not succeed, you can create the bucket manually with aws s3 mb. The name of the destination VAST S3 bucket must start with the word 'syncengine' followed by a period (.): syncengine..

Before You Begin

Ensure that the following prerequisites are met:

SyncEngine

• The SyncEngine control plane and at least one worker are up and running.

• At least one worker label of a deployed worker is known. (The worker label is set when you deploy the worker.)

VAST Cluster

• VAST cluster's management IP.

• A DNS-resolvable name of the default tenant's virtual IP pool to access the VAST S3 endpoint, or a virtual IPv4 address from that pool.

• User (username and UID) with S3 access keys allowing to create a bucket and put objects to it. To create this user, follow the guidelines in SyncEngine VAST Cluster Setup Guide.

• The VAST cluster must be joined to Active Directory (required to map Confluence user permissions to S3 ACLs).

• The Active Directory must have mapping entries for Confluence users and groups. For example, for a Confluence user with email address alice@company.com, Active Directory must have sAMAccountName with the value alice.

Confluence

• Confluence Cloud site URL (for example https://your-company.atlassian.net)

• Email address of the user with read access to the Confluence space to be migrated.

• Atlassian API token with Confluence scope (create one from API Tokens page of your Atlassian service account)

• The space key of the Confluence space to be migrated (https://your-company.atlassian.net/wiki/spaces/<space key>/<...>).

Create a Confluence Connector

Create a connector to connect to the Confluence Cloud where your source Confluence space is located.

If a suitable Confluence connector already exists, skip this step.

SyncEngine UI

1. In the Connectors page, click New Connector.

   A wizard dialog appears to walk you through the connector creation steps.

2. In the Type & name step:

   • Select the Confluence connector type.

   • In the Connector name field, enter a unique name for the connector.
     The name can contain up to 63 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-), and underscores (_).

3. In the Authentication step, complete the fields:

   Confluence Base URL	Your Atlassian Cloud base URL, for example: https://you-company.atlassian.net.
   Username / Email	The email address that you use to log in to the Confluence space. The email address must be up to 255 characters.
   API Token	Your Atlassian API token with Confluence scope. The token must be created beforehand.

4. In the Filters step, configure the attachment filter, the user filter, and the Markdown conversion filter:

   Attachment Filter Regex	Enter a regular expression to filter out attachments that you do not want to migrate. The default attachment filter (.*\.tmp$) skips all files with the .tmp extension.
   Ignored Users	Enter a comma-separated list of email addresses of users to be excluded when converting Confluence permissions to S3 ACLs.
   Ignored Groups	Enter a comma-separated list of email addresses of user groups to be excluded when converting Confluence permissions to S3 ACLs.
   Ignored Domains	Enter a comma-separated list of email address domains to be excluded when converting Confluence permissions to S3 ACLs.
   AIQ Supported Extensions	Enter a comma-separated list of file extensions to be extracted from Confluence as-is (without conversion to Markdown). By default, PDF and MD files are extracted.

5. In the Review step, review the settings you've made.

   • If you want to make changes, click Back. Otherwise, click Create Connector.

   • The connector is created and displayed on the Connectors page.

SyncEngine CLI

The following command creates a Confluence connector named src-conf for the Confluence user with email address kirk@my-company.com at the Confluence base URL https://my-company.atlassian.net:

syncengine-cli connector create \
  --type=confluence \
  --name=src-conf \
  --base-url=htttps://my-company.atlassian.net \
  --username=kirk@my-company.com \
  --api-token=<Atlassian API token>

Optional flags:

--attachment-filter <regex> 
--ignored-users <user1,user2...>
--ignored-groups <group1,group2,...>
--ignored-domains <domain1,domain2,...>
--aiq-extensions <ext1, ext2...>

TIP: Note the connector ID in the output.

Create a VAST S3 Connector

Create a connector to connect to the VAST S3 server on the VAST cluster to which you want to migrate the data.

If a suitable VAST S3 connector already exists, skip this step.

SyncEngine UI

1. In the Connectors page, click New Connector.

   A wizard dialog appears to walk you through the connector creation steps.

2. In the Type & name step:

   • Select the Vast S3 connector type.

   • In the Connector name field, enter a unique name for the connector.

     The name can contain up to 63 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-), and underscores (_).

3. In the VMS cluster step, enter the VAST cluster's VMS IP and admin credentials:

   Management IP	VAST cluster's management IPv4 address or DNS-resolvable hostname (up to 253 characters).
   Username	Username to log in to the VMS.
   Password	Password to log in to the VMS.

4. In the S3 endpoint step, configure an S3 endpoint on the VAST cluster.

   SyncEngine will use the S3 endpoint to automatically create the destination S3 bucket for the migration.

   VIP	A DNS-resolvable name of the default tenant's virtual IP pool to access the VAST S3 endpoint, or a virtual IPv4 address from that pool.
   Access Key	Enter the S3 access key for the bucket creator user.
   Secret Key	Enter the S3 access key for the bucket creator user.
   S3 Username	Enter the username for the bucket creator user.
   S3 UID	The UID of the bucket creator user.
   Endpoint URL	The URL to access the S3 endpoint. This field is optional. If not specified, the URL is derived from the VIP field.
   Use secure S3 connection  (HTTPS)	Select to use HTTPS. If selected, you'll need to provide a TLS certificate in a next step. If not selected (default), HTTP is used.

5. In the Bucket & advanced step, optionally configure advanced settings for the destination S3 bucket:

   Case-sensitive bucket names	When enabled, SyncEngine allows for bucket names that contain both lowercase and uppercase characters.
   Checksum Algorithm	Checksum algorithm for S3 PUT operations: CRC32C (default), CRC64NVME or SHA256. NOTE: CRC64NVME or SHA256 require VAST Cluster 5.5 or later. SyncEngine will verify if your VAST cluster supports checksum verification for the selected algorithm.
   Read Timeout	Socket read timeout, in seconds. Valid values: 60–3600. Default is 300.
   Connect Timeout	S3 endpoint connection timeout, in seconds. Valid values: 5–300. Default is 60.
   Max Retries	The maximum allowed number of retries by the VAST S3 client. Valid values: 1–100. Default is 3.
   Retry Mode	Select one of VAST S3 client retry modes: Adaptive (default) for rate-limiting of requests when throttled Standard for exponential backoff with jitter Legacy for simple exponential backoff

6. In the SSL step, use the SSL Certificate Content field to enter the TLS certificate (in PEM format) to secure the S3 connection. If left blank, default CA certificates are used.

7. In the Review step, review the settings you've made.

   If you want to make changes, click Back. Otherwise, click Create Connector.

   The connector is created and displayed in the Connectors page.

SyncEngine CLI

The following command creates a destination VAST S3 connector named dest-vast-s3 to connect to the S3 endpoint at <VAST S3 VIP> on a VAST cluster with a management IP of <VMS IP>:

syncengine-cli connector create \
  --type=vast_s3 \
  --name=dest-vast-s3 \
  --mgmt-ip=<VMS IP> \
  --username=admin \
  --password=<password> \
  --vip=<VAST S3 VIP> \
  --access-key=<S3 access key> \
  --secret-key=<S3 secret key>

NOTE: To auto-generate S3 access keys, omit the --access-key and --secret-key options from the command.

Optional flags when creating a connector of the vast_s3 type:

--s3-username <name>                # Default: SYNCENGINE_USER
--s3-uid <UID>                      # Default: 270270270                          
--secure-s3={true|false}
--checksum-algorithm {CRC32C|CRC64NVME|SHA256}
--case-sensitive-bucket-names={true|false}
--read-timeout <seconds>
--client-connect-timeout <seconds>
--client-retries-max-attempts <count>
--client-retries-mode <adaptive|standard|legacy>
--ssl-cert-file <path to file>

TIP: Note the connector ID in the output.

Create and Start a Migration

Create a migration to move the data from your Confluence space to an S3 bucket on the VAST cluster.

SyncEngine UI

1. In the Migrations page, click New Migration.

   A wizard dialog appears to walk you through the migration creation steps.

2. In the Source step, select the Confluence connector that will be used to connect to your Confluence space.

3. In the Destination step, select the VAST S3 connector that will be used to connect to the destination VAST cluster.

4. In the Paths step:

   • Under the Migration Label, specify a SyncEngine worker label for the migration.

     A label is used to route migration jobs to the appropriate SyncEngine worker. One worker can be associated with one or more migration labels. Workers deployed with a certain label handle jobs for all migrations that have that label.

     You can create a new label or use an existing one.

     • To create a new label, select Create new label and enter a name for the label in the New Label field.

       The name can contain up to 128 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-), and underscores (_).

     • To use an existing label, select Use existing label and enter the label name in the Existing Label field.

   • Under Paths, specify the paths to the migration source and destination.

     • Under Confluence Space, select a Confluence space from the list of recent spaces, or enter the space key manually in the field provided.

     • In the Destination S3 Bucket, enter a name for the S3 bucket on the VAST cluster.

       SyncEngine will automatically create the bucket if it does not exist.

       The string that you enter will automatically be prefixed with the word 'syncengine' followed by a period (.): syncengine.. Special characters in the bucket names will be replaced with hyphens.

5. In the Sync mode step, specify if you want to run the migration once or recurringly:

   • To run the migration once, select Manual in the SyncMode field.

     Migration will start after you manually submit the migration job.

   • To run the migration automatically at certain intervals, select Automatic in the Sync Mode field and enter an interval in the Sync Interval field.

6. In the What you're copying step:

   • Review the settings you've made. If necessary, click Back and change as needed.

   • Decide how SyncEngine handles the files that have been deleted since the previous migration run.

     By default, SyncEngine does not delete such files from the destination S3 bucket. If you want to have them deleted, select Mirror source deletions on the destination.

7. In the Per-task timeouts step, optionally configure suitable timeouts for the migration jobs.

8. In the Review step, review the settings you've made.

   If you want to make changes, click Back. Otherwise, click Create Migration.

   The migration is started and displayed in the Migration page.

SyncEngine CLI

1. Create a migration with migration create.

   The following commands creates a migration from a Confluence space with the space key ENG to a VAST S3 bucket named syncengine-eng. The migration is to be accomplished by workers labeled with MYFIRSTMIGRATION using connectors with connector IDs 14 and 37.

   syncengine-cli migration create \
     --source-connector-id=14 \
     --destination-connector-id=37 \
     --space-key=ENG \
     --bucket-name=syncengine-eng \
     --label=MYFIRSTMIGRATION

   Optional flags:

   --sync-mode {manual|automatic --sync-interval <hours>}
   --enable-deletes={true|false}

   The CLI creates a migration and displays its properties.

   TIP: Note the migration ID in the output.

2. Start the migration with migration start.

   The following command starts a migration with a migration ID of 24:

   syncengine-cli migration start --migration-id=24

Monitor Migration Status

As a migration progresses, its status changes as follows: Not Started (or init) -> Waiting for Worker (or pending) -> Syncing -> Synced.

When complete, the destination S3 bucket will have objects prefixed with pages/, blogposts/, comments/, and attachments/.

SyncEngine UI

Check the migration status in the Migrations page.

SyncEngine CLI

Run the command:

syncengine-cli migration status --migration-id=<migration ID>

Troubleshoot Migration