SyncEngine Google Drive Migration Guide

This guide walks you through migrating files stored in Google Drive to a VAST S3 bucket.

Overview

The Google Drive connector extracts files from a Google Drive folder into a VAST S3 bucket. The folder to be migrated can be your own folder, or it can be a different user's folder shared with you.

SyncEngine authenticates with a Google Cloud service account, reads content shared with that account, and copies the files to a S3 bucket on the VAST cluster, with Google Drive permissions converted into S3 ACLs.

To set up migration from Google Drive to a VAST S3 bucket:

1. Ensure that the prerequisites are met.

2. Create a connector for Google Drive.

3. Create a connector for VAST S3.

4. Create and start a migration from the source folder to the destination VAST S3 bucket.

5. Monitor migration progress and completion.

6. If necessary, troubleshoot.

The following limitations apply:

• Google Drive content can only be migrated to a VAST S3 bucket. No other destination types are supported.

• One folder per migration. If you need to migrate multiple drives or shared folders, multiple migrations are to be set up.

Handling Particular File Types

By default, files with .pdf and .md extensions are stored as-is. Files of other types are converted to Markdown where possible.

You can alter the default when creating the Google Drive connector.

Automatic Creation of Destination S3 Bucket

When you submit a migration job, SyncEngine attempts to automatically create a destination S3 bucket on the VAST cluster.

If it does not succeed, you can create the bucket manually with aws s3 mb. The name of the destination VAST S3 bucket must start with the word 'syncengine' followed by a period (.): syncengine..

Before You Begin

Ensure that the following prerequisites are met:

SyncEngine

• The SyncEngine control plane and at least one worker are up and running.

• At least one worker label of a deployed worker is known. (The worker label is set when you deploy the worker.)

VAST Cluster

• VAST cluster's management IP.

• A DNS-resolvable name of the default tenant's virtual IP pool to access the VAST S3 endpoint, or a virtual IPv4 address from that pool.

• User (username and UID) with S3 access keys allowing to create a bucket and put objects to it. To create this user, follow the guidelines in SyncEngine VAST Cluster Setup Guide.

• The VAST cluster must be joined to Active Directory (required to map Google Drive user permissions to S3 ACLs).

• The Active Directory must have mapping entries for Google Drive users and groups. For example, for a Google Drive user with email address alice@company.com, Active Directory must have sAMAccountName with the value alice.

Google Drive

• A Google Cloud service account with the Google Drive API enabled in its project. The account must have at least the Viewer role for the folder to be migrated.

• A JSON key for the Google Cloud service account.

• To create a JSON key, open the service account in the Google Cloud console, go to Keys, select Add key -> Create new key, choose JSON, and click Create to download the key file.

• The folder ID of the folder to be migrated.

• The folder ID can be found in the folder's URL: https://drive.google.com/drive/folders/<folder ID>

Create a Google Drive Connector

Create a connector to connect to the Google Drive folder to be migrated.

If a suitable Google Drive connector already exists, skip this step.

SyncEngine UI

1. In the Connectors page, click New Connector.

   A wizard dialog appears to walk you through the connector creation steps.

2. In the Type & name step:

   • Select the Google Drive connector type.

   • In the Connector name field, enter a unique name for the connector.

     The name can contain up to 63 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-), and underscores (_).

3. In the Service Account step, paste the JSON key for your Google Cloud service account.

4. In the Filters step, configure the user filter and Markdown conversion filter:

   Ignored Domains	Enter a comma-separated list of email address domains to be excluded when converting Google Drive permissions to S3 ACLs.
   Ignored Users	Enter a comma-separated list of email addresses of users to be excluded when converting Google Drive permissions to S3 ACLs.
   Ignored Groups	Enter a comma-separated list of email addresses of user groups to be excluded when converting Google Drive permissions to S3 ACLs.
   AIQ Supported Extensions	Enter a comma-separated list of file extensions to be extracted from Google Drive as-is (without conversion to Markdown). By default, PDF and MD files are extracted.

5. In the Review step, review the settings you've made.

   If you want to make changes, click Back. Otherwise, click Create Connector.

   The connector is created and displayed in the Connectors page.

SyncEngine CLI

The following command creates a Google Drive connector named src-gd:

syncengine-cli connector create \
  --type=google_drive \
  --name=src-gd \
  --service-account-file=<path to JSON key file>

Optional flags:

--ignored-domains <domain1,domain2,...>
--ignored-users <user1,user2...>
--ignored-groups <group1,group2,...>
--aiq-extensions <ext1, ext2...>

TIP: Note the connector ID in the output.

Create a VAST S3 Connector

Create a connector to connect to the VAST S3 server on the VAST cluster to which you want to migrate the data.

If a suitable VAST S3 connector already exists, skip this step.

SyncEngine UI

1. In the Connectors page, click New Connector.

   A wizard dialog appears to walk you through the connector creation steps.

2. In the Type & name step:

   • Select the Vast S3 connector type.

   • In the Connector name field, enter a unique name for the connector.

     The name can contain up to 63 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-), and underscores (_).

3. In the VMS cluster step, enter the VAST cluster's VMS IP and admin credentials:

   Management IP	VAST cluster's management IPv4 address or DNS-resolvable hostname (up to 253 characters).
   Username	Username to log in to the VMS.
   Password	Password to log in to the VMS.

4. In the S3 endpoint step, configure an S3 endpoint on the VAST cluster.

   SyncEngine will use the S3 endpoint to automatically create the destination S3 bucket for the migration.

   VIP	A DNS-resolvable name of the default tenant's virtual IP pool to access the VAST S3 endpoint, or a virtual IPv4 address from that pool.
   Access Key	Enter the S3 access key for the bucket creator user.
   Secret Key	Enter the S3 access key for the bucket creator user.
   S3 Username	Enter the username for the bucket creator user.
   S3 UID	The UID of the bucket creator user.
   Endpoint URL	The URL to access the S3 endpoint. This field is optional. If not specified, the URL is derived from the VIP field.
   Use secure S3 connection (HTTPS)	Select to use HTTPS. If selected, you'll need to provide a TLS certificate in a next step. If not selected (default), HTTP is used.

5. In the Bucket & advanced step, optionally configure advanced settings for the destination S3 bucket:

   Case-sensitive bucket names	When enabled, SyncEngine allows for bucket names that contain both lowercase and uppercase characters.
   Checksum Algorithm	Checksum algorithm for S3 PUT operations: CRC32C (default), CRC64NVME or SHA256. NOTE: CRC64NVME or SHA256 require VAST Cluster 5.5 or later. SyncEngine will verify if your VAST cluster supports checksum verification for the selected algorithm.
   Read Timeout	Socket read timeout, in seconds. Valid values: 60–3600. Default is 300.
   Connect Timeout	S3 endpoint connection timeout, in seconds. Valid values: 5–300. Default is 60.
   Max Retries	The maximum allowed number of retries by the VAST S3 client. Valid values: 1–100. Default is 3.
   Retry Mode	Select one of VAST S3 client retry modes: Adaptive (default) for rate-limiting of requests when throttled Standard for exponential backoff with jitter Legacy for simple exponential backoff

6. In the SSL step, use the SSL Certificate Content field to enter the TLS certificate (in PEM format) to secure the S3 connection. If left blank, default CA certificates are used.

7. In the Review step, review the settings you've made.

   If you want to make changes, click Back. Otherwise, click Create Connector.

   The connector is created and displayed in the Connectors page.

SyncEngine CLI

The following command creates a destination VAST S3 connector named dest-vast-s3 to connect to the S3 endpoint at <VAST S3 VIP> on a VAST cluster with a management IP of <VMS IP>:

syncengine-cli connector create \
  --type=vast_s3 \
  --name=dest-vast-s3 \
  --mgmt-ip=<VMS IP> \
  --username=admin \
  --password=<password> \
  --vip=<VAST S3 VIP> \
  --access-key=<S3 access key> \
  --secret-key=<S3 secret key>

NOTE: To auto-generate S3 access keys, omit the --access-key and --secret-key options from the command.

Optional flags when creating a connector of the vast_s3 type:

--s3-username <name>                # Default: SYNCENGINE_USER
--s3-uid <UID>                      # Default: 270270270                          
--secure-s3={true|false}
--checksum-algorithm {CRC32C|CRC64NVME|SHA256}
--case-sensitive-bucket-names={true|false}
--read-timeout <seconds>
--client-connect-timeout <seconds>
--client-retries-max-attempts <count>
--client-retries-mode <adaptive|standard|legacy>
--ssl-cert-file <path to file>

TIP: Note the connector ID in the output.

Create and Start a Migration

Create a migration to move the data from the Google Drive folder to an S3 bucket on the VAST cluster.

SyncEngine UI

1. In the Migrations page, click New Migration.

   A wizard dialog appears to walk you through the migration creation steps.

2. In the Source step, select the Google Drive connector that will be used to connect to the folder to be migrated.

3. In the Destination step, select the VAST S3 connector that will be used to connect to the destination VAST cluster.

4. In the Paths step:

   • Under Migration Label, specify a SyncEngine worker label for the migration.

     A label is used to route migration jobs to the appropriate SyncEngine worker. One worker can be associated with one or more migration labels. Workers deployed with a certain label handle jobs for all migrations that have that label.

     You can create a new label or use an existing one.

     • To create a new label, select Create new label and enter a name for the label in the New Label field.

       The name can contain up to 128 characters, including uppercase and lowercase English letters, digits, periods (.), hyphens (-) and underscores (_).

     • To use an existing label, select Use existing label and enter the label name in the Existing Label field.

   • Under Paths, specify the paths to the migration source and destination.

     • In the Google Drive Folder ID field, enter the folder ID of the folder you want to migrate.

     • In the Destination S3 Bucket, enter a name for the S3 bucket on the VAST cluster.

       SyncEngine will automatically create the bucket if it does not exist.

       The string that you enter will automatically be prefixed with the word 'syncengine' followed by a period (.): syncengine.. Special characters in the bucket names will be replaced with hyphens.

5. In the Sync mode step, specify if you want to run the migration once or recurringly:

   • To run the migration once, select Manual in the SyncMode field.

     Migration will start after you manually submit the migration job.

   • To run the migration automatically at certain intervals, select Automatic in the Sync Mode field and enter an interval in the Sync Interval field.

6. In the What you're copying step:

   • Review the settings you've made. If necessary, click Back and change as needed.

   • Decide how SyncEngine handles the files that have been deleted since the previous migration run.

     By default, SyncEngine does not delete such files from the destination S3 bucket. If you want to have them deleted, select Mirror source deletions on the destination.

7. In the Per-task timeouts step, optionally configure suitable timeouts for the migration jobs.

8. In the Review step, review the settings you've made.

   If you want to make changes, click Back. Otherwise, click Create Migration.

   The migration is started and displayed in the Migration page.

SyncEngine CLI

1. Create a migration with migration create.

   The following commands create a migration from a Google Drive folder with the specified folder ID to a VAST S3 bucket named syncengine-eng. The migration is to be accomplished by workers labeled with MYFIRSTMIGRATION using connectors with connector IDs 12 and 37.

   syncengine-cli migration create \
     --source-connector-id=12 \
     --destination-connector-id=37 \
     --folder-id=<folder ID> \
     --bucket-name=syncengine-eng \
     --label=MYFIRSTMIGRATION

   Optional flags:

   --sync-mode {manual|automatic --sync-interval <hours>}
   --enable-deletes={true|false}

   The CLI creates a migration and displays its properties.

   TIP: Note the migration ID in the output.

2. Start the migration with migration start.

   The following command starts a migration with migration ID of 25:

   syncengine-cli migration start --migration-id=25

Monitor Migration Status

As a migration progresses, its status changes as follows: Not Started (or init) -> Waiting for Worker (or pending) -> Syncing -> Synced.

When complete, the destination S3 bucket will have objects with prefixes similar to <file ID>/gdrive.pdf.

SyncEngine UI

Check the migration status in the Migrations page.

SyncEngine CLI

Run the command:

syncengine-cli migration status --migration-id=<migration ID>

Troubleshoot Migration