Trino Managed CNode Applications

The configuration.yaml file for Trino should include this section with details for connecting to a VAST Database:

  vast: connector.name: vast
    endpoint: ENDPOINT
    access_key_id: VAST_ACCESS_KEY
    secret_access_key: VAST_SECRET_KEY
    data_endpoints: DATA_ENDPOINTS
    vast.http-client.request-timeout: 100h
    vast.http-client.idle-timeout: 290s
    num_of_splits: 512
    num_of_subsplits: 4  

where VAST_ACCESS_KEY and VAST_SECRET_KEY are the access and secret keys for a VMS  user with permissions to create and access VAST Databases (see Pre-requisites), and ENDPOINT and DATA_ENDPOINTS are the Coordinator VIP.

The Trino App includes all Trino connectors, including connectors for Apache Hive.

To configure the Hive connector, add a section like this to the configuration.yaml file in the catalogs section:

catalogs:
  hive: connector.name: hive
    fs.native-local.enabled=true
    fs.native-s3.enabled=true
    s3.region=us-east-1
    s3.path-style-access: true
    s3.socket-read-timeout: 10m
    hive.non-managed-table-writes-enabled:true
    hive.storage-format: TEXTFILE
    hive.recursive-directories: true
    hive.metastore: FILE
    hive.metastore.catalog.dir: /var/trino/hive-metastore
    s3.endpoint=ENDPOINT
    s3.aws-access-key=ACCESS_KEY
    s3.aws-secret-key=SECRET_KEY

where ACCESS_KEY and SECRET_KEY are the S3 keys, and ENDPOINT is the VIP for the Trino Coordinator. The field s3.region must be included, and indicate a valid AWS region, however if the database is located in the VAST Cluster, the value is ignored.

You can upload P12, PEM, or JKS certificates to the Trino App, to use when authenticating with clients.  This is done in the Create Application workflow (see Creating Applications on CNodes).

You can also remove or replace a certificate.

To use certificates with the Trino App, add the following to the configuration.yaml file. This includes configuring LDAP.

additionalConfigFiles:
  password-authenticator:
    body: password-authenticator.name=ldap

      ldap.url=ldap://<DOMAIN>
      ldap.user-bind-pattern=${USER}@<DC>
      ldap.allow-insecure=true
    extension: properties
additionalConfigProperties:
- http-server.https.keystore.key=KEY
- http-server.process-forwarded=true
- internal-communication.https.required=true
- internal-communication.shared-secret=SECRET

...
server:
  config:
    authenticationType: PASSWORD
    coordinatorUrl: localhost
    http:
      port: 8080
    https:
      enabled: true
      keystore:
        path: /managed_app_mnt_internal/userfiles/trino.cert
      port: 8443
      ....

where DOMAIN is the LDAP domain, and KEY and SECRET are a key pair used for communication between the Trino Coordinator and Worker nodes (generate any key pair for this).

If you configure LDAP for the Trino App, a P12 type certificate must be uploaded to the App. In addition, the configuration.yaml file must include the additions described in Configuring Certificates for the Trino App

You can configure the Trino App to save temporary data in persistent storage, for example saving data between queries.

The persistent storage is configured to use the same bucket as the database that the app uses.

To configure this, add the following to the configuration.yaml file.

additionalConfigProperties:
- exchange.deduplication-buffer-size=32MB
- retry-policy=TASK
additionalExchangeManagerProperties:
- exchange-manager.name=filesystem
- exchange.base-directories=s3://tabular-youthful-hypercritical-quetzal
- exchange.s3.aws-access-key=ACCESS_KEY
- exchange.s3.aws-secret-key=SECRET_KEY
- exchange.s3.region=us-east-1
- exchange.s3.endpoint=DATA_ENDPOINTS
- exchange.s3.path-style-access=true

where ACCESS_KEY and SECRET_KEY are the S3 keys, and DATA_ENDPOIN is the VIP for the Trino Coordinator.  The field exchange.s3.region must be included, and indicate a valid AWS region, however if the database is located in the VAST Cluster, the value is ignored.

This is an example of the complete configuration.yaml file, including settings to connect to the VAST Database, HIve,

accessControl: {}
additionalConfigProperties: []
additionalExchangeManagerProperties: []
additionalLogProperties:
- com.vastdata=INFO
additionalNodeProperties: []
auth: {}
catalogs:
  hive: connector.name: hive
    fs.native-local.enabled=true
    fs.native-s3.enabled=true
    s3.region=us-east-1
    s3.path-style-access: true
    s3.socket-read-timeout: 10m
    hive.non-managed-table-writes-enabled:true
    hive.storage-format: TEXTFILE
    hive.recursive-directories: true
    hive.metastore: FILE
    hive.metastore.catalog.dir: /var/trino/hive-metastore
    s3.endpoint=ENDPOINT
    s3.aws-access-key=ACCESS_KEY
    s3.aws-secret-key=SECRET_KEY
  jmx: connector.name=jmx
    
  tpcds: connector.name=tpcds
    tpcds.splits-per-node=4
    
  tpch: connector.name=tpch
    tpch.splits-per-node=4
    
  vast: connector.name: vast
    endpoint: ENDPOINT
    access_key_id: ACCESS_KEY
    secret_access_key: SECRET_KEY
    data_endpoints: DATA_ENDPOINTS
    vast.http-client.request-timeout: 100h
    vast.http-client.idle-timeout: 290s
    num_of_splits: 512
    num_of_subsplits: 4
coordinator:
  config:
    memory:
      heapHeadroomPerNode: ""
    query:
      maxMemoryPerNode: ""
eventListenerProperties: []
resourceGroups: {}
server:
  config:
    authenticationType: 
    coordinatorUrl: localhost
    http:
      port: 8080
    https:
      enabled: true
      keystore:
        path: /managed_app_mnt_internal/userfiles/trino.cert
      port: 8443
    query:
      maxMemory: ""
  coordinatorExtraConfig: ""
  exchangeManager: {}
  log:
    trino:
      level: INFO
  workerExtraConfig: ""
worker:
  config:
    memory:
      heapHeadroomPerNode: ""
    query:
      maxMemoryPerNode: ""

Run this command on the client to connect to the Trino app:

./trino --server <VIP>:8080 --catalog vast --schema <schema> --user <username> 

where VIP is the Coordinator VIP, schema is your schema in the VAST database, and username is the VMS user created for Trino.