Trino on VAST CNodes

TL;DR: Skip to the Minimum Viable Configuration or the full-fat Config w/Certs, LDAP + Row & Column Access Control

To build from minimum up to full fat, with all the “why” and “how” along the way, journey onward. This is the 5.4 version and above - Trino Workers & Coordinator running as Docker containers on CNodes.

Minimum Viable Configuration

This configuration.yaml was run in our lab. It was built by downloading the template from the 5.4.1-sp3 release and modifying it as indicated by these bullets:

80: endpoint (singular) - used by Trino to initiate connections to VAST DB, vs
84: data_endpoints (plural) - used by worker threads / Connector to balance work across CNodes in parallel. Since Trino is written in Java, using a DNS name here instead of listing all the VIPs in the VIP Pool would compromise parallelism [skipping the rabbit hole about DNS in Java]
81: region - completely ignored by the Connector, but has to be set to some string for various libraries to pass checks. By convention, we frequently set it to us-east-1, but any string should work if you feel lucky
82 & 83: access_key_id & secret_access_key - set to the owner of the VAST database for “easy mode”; no other Access Policies required if you run this way
70 - 77: other catalogs uncommented (nice to have for Trino health testing, demo, etc)

server:
  log:
    trino:
      level: INFO
  config:
    coordinatorUrl: localhost
    http:
      port: 8080
    https:
      enabled: false
      port: 8443
      keystore:
        path: ""
    # -- Trino supports multiple [authentication
    # types](https://trino.io/docs/current/security/authentication-types.html):
    # PASSWORD, CERTIFICATE, OAUTH2, JWT, KERBEROS.
    authenticationType: ""
    query:
      maxMemory: ""

  workerExtraConfig: ""
  coordinatorExtraConfig: ""

coordinator:
  config:
    memory:
      heapHeadroomPerNode: ""
    query:
      maxMemoryPerNode: ""

worker:
  config:
    memory:
      heapHeadroomPerNode: ""
    query:
      maxMemoryPerNode: ""



additionalConfigProperties: []
# additionalConfigProperties -- [Additional config
# properties](https://trino.io/docs/current/admin/properties.html).
# @raw
# Example:
# ```yaml
#  - internal-communication.shared-secret=random-value-999
#  - http-server.process-forwarded=true
# ```

additionalLogProperties:
- com.vastdata=INFO
# additionalLogProperties -- [Additional log
# properties](https://trino.io/docs/current/installation/deployment.html#log-levels).
# @raw
# Example:
# ```yaml
#  - io.airlift=DEBUG
# ```

additionalNodeProperties: []
# additionalNodeProperties -- [Additional node
# properties](https://trino.io/docs/current/installation/deployment.html#log-levels).
# @raw
# Example, assuming the NODE_ID environment variable has been set:
# ```yaml
#  - node.id=${NODE_ID}
# ```

catalogs:
  tpch: |
    connector.name=tpch
    tpch.splits-per-node=4
  tpcds: |
    connector.name=tpcds
    tpcds.splits-per-node=4
  jmx: |
    connector.name=jmx
  vast: |
    connector.name=vast
    endpoint=http://172.200.205.1
    region=us-east-1
    access_key_id=RZV2NP1HT89CUN97QJ2B
    secret_access_key=tp+RYus0xq0wE2AS6lW2vciUt14/4bXF+Wojzq2i
    data_endpoints=http://172.200.205.1,http://172.200.205.2,http://172.200.205.3,http://172.200.205.4,http://172.200.205.5,http://172.200.205.6
    vast.http-client.request-timeout=100h
    vast.http-client.idle-timeout=290s
    num_of_splits=512
    num_of_subsplits=4

# catalogs -- Configure
# [catalogs](https://trino.io/docs/current/installation/deployment.html#catalog-properties).

# additionalConfigFiles:
#   password-authenticator:
#     extension: properties
#     body: |
#       password-authenticator.name=ldap
#       ldap.url=ldap://xx.aaa.lab
#       ldap.user-bind-pattern=${USER}@aaa.lab
#       ldap.allow-insecure=true

Initial Provisioning

Prerequisites

access_key_id & secret_access_key for a user who owns a database on the VAST cluster, or has access to the Catalog on the cluster.
That’s it! This one is Minimum Viable after all. 😀

Create Managed Application

The VMS screen is listed under Data Engine in the left navigation. There is a “Create” button in the upper right. The wizard is straightforward, but the IP Address requirements are very specific:

Notes on IP Addresses

Addressing is weird enough that I made a whole topic heading for it. If you’re used to running a Trino worker container that is also a coordinator, that’s not what Applications on CNodes will expect. Instead, provide at least 1 IP address per worker (obviously) plus an additional IP address for the Coordinator. When provisioning completes successfully, you will see just the Coordinator in your VIP Pools & Addresses.

Network IP address associations

Just the Coordinator - Where are the Worker IP Addresses?

However, when you ask Trino what IP addresses it is using for Workers, it will say something like this:

trino> SELECT * FROM system.runtime.nodes;
               node_id                |        http_uri         | node_version | coordinator | state
--------------------------------------+-------------------------+--------------+-------------+--------
 a05dbf1e-9e00-4f5c-af58-ae6145974780 | http://172.16.2.19:8080 | 475          | false       | active
 3cd92ae2-13ec-41b4-89ed-0a3e700f1d7f | http://172.16.2.26:8080 | 475          | true        | active
 d554f67a-1c6c-43f8-ab1f-4d4002326481 | http://172.16.2.18:8080 | 475          | false       | active
(3 rows)

Which happen to be CNode addresses as seen from physical inventory:

Infrastructure - CNodes UI

Which begs the question: Does traffic from Trino to the VAST DB VIP Pool use Northbound or Southbound ports/switches? Some references say each Trino worker container talks only to VASD DB VIP addresses on the same physical CNode. However, when I look inside a trino_worker, I see all of the endpoints from configuration.yaml.

poc-var-205 vastdata@poc-arrow-cb1-cn-3 ~:$ docker exec -it trino_worker bash
root@trino:/usr/local/bin# cat /managed_app_mnt_internal/userfiles/configs/trino_worker/catalog/vast.properties
connector.name=vast
endpoint=http://172.200.205.1
region=us-east-1
access_key_id=RZV2NP1HT89CUN97QJ2B
secret_access_key=tp+RYus0xq0wE2AS6lW2vciUt14/4bXF+Wojzq2i
data_endpoints=http://172.200.205.1,http://172.200.205.2,http://172.200.205.3,http://172.200.205.4,http://172.200.205.5,http://172.200.205.6
vast.http-client.request-timeout=100h
vast.http-client.idle-timeout=290s
num_of_splits=512
num_of_subsplits=4

From each CNode, it is apparent that one is running a trino_coordinator container:

poc-var-205 vastdata@poc-arrow-cb4-cn-1 ~:$ docker ps
CONTAINER ID   IMAGE                                                                     COMMAND                  CREATED       STATUS      PORTS     NAMES
42e88a3edf74   59084463d6a2                                                              "/usr/local/trino/bi…"   5 days ago    Up 5 days             trino_coordinator
353b8c493998   vastdata.registry.local:5000/dev/orion:release-5.4.1-sp3-2209349-x86_64   "dumb-init /vast/bin…"   5 days ago    Up 5 days             vast_platform
e16ad209e0f8   local_registry:latest                                                     "/entrypoint.sh /etc…"   7 weeks ago   Up 5 days             registry

While the other(s) are running a worker Trino container:

poc-var-205 vastdata@poc-arrow-cb1-cn-3 ~:$ docker ps
CONTAINER ID   IMAGE                                                                     COMMAND                  CREATED       STATUS      PORTS     NAMES
7babb2a382c1   59084463d6a2                                                              "/usr/local/trino/bi…"   5 days ago    Up 5 days             trino_worker
8b3eaaacf91e   vastdata.registry.local:5000/dev/orion:release-5.4.1-sp3-2209349-x86_64   "dumb-init /vast/bin…"   5 days ago    Up 5 days             vast_platform
f8a11d81f55a   vastdata.registry.local:5000/dev/orion/registry:latest                    "/entrypoint.sh /etc…"   4 weeks ago   Up 5 days             registry

Trivia

A CNode reboots to change the grub command line when adding or removing a Managed Application:

isolcpus=4,5,6-34,68,69,70-98 nvme.allow_dev_partitions_pci_vendor=0x1b4b nvme.allow_dev_partitions_pci_device=0x2241 iommu=pt

isolcpus=4,5,6-40,68,69,70-104 nvme.allow_dev_partitions_pci_vendor=0x1b4b nvme.allow_dev_partitions_pci_device=0x2241 iommu=pt

When Provisioning Fails

Don’t panic! All your settings are probably correct, and you just need to give the Cluster a nudge to carry on with the Provisioning. If the Create Application fails (on the VMS Activities UI), help it along by:

Activate any inactive CNode(s) from VMS Infrastructure → CNodes UI
Start (not Restart!) the Create Application process from VMS Data Engine → Applications
Repeat as necessary until the Coordinator process starts / Update Managed Application Activity Succeeds

If you don’t succeed, Activate CNode(s) and try Start(ing) the Application (again)

If at 1st you don’t succeed, Activate CNode(s) and try Start(ing) the Application (again)

Note 1: Restart is how you swap in a new configuration.yaml after provisioning succeeds. Start is what you want to keep doing until provisioning succeeds.

Note 2: When deleting the Managed Application (Trino on CNodes) you may have to Activate CNodes and Delete again - very similar to above. Takes 30+ minutes to create (or delete), so plan accordingly. Activate CNode(s) is serial, but 60-90 seconds per CNode, so doesn’t cost you much time if you keep an eye on the Provisioning Activity

Updating configuration.yaml

To swap to a new configuration.yaml into your running Trino on CNodes Application:

Edit Configuration & Security
Delete the current configuration.yaml
upload the new configuration.yaml
Restart the Trino on the CNodes Application

Updating configuration.yaml

Configuring TLS Certs

Modify the server section of your configuration.yaml to enable HTTPS and look for a cert at /managed_app_mnt_internal/userfiles/trino.cert, which is where your uploaded cert file will be mounted within the Trino Coordinator container. Edit the Trino Managed Application, and then Restart it.

Upload the new configuration.yaml & cert

If the Managed Application doesn’t restart, revert to the last known good configuration.yaml, and troubleshoot

Troubleshooting

It may be useful to SSH to the cnode and get a shell inside the Trino coordinator.

docker exec -it trino_coordinator bash

From within the trino_coordinator container, you can check the validity of the cert.

openssl rsa -in /managed_app_mnt_internal/userfiles/trino.cert -check -noout

If you have access to OpenSSL elsewhere, you may want to check the cert is valid before restarting the Managed Application.

Connecting with HTTPS

Our command-line client invocation is getting longer as we include --add-hosts, but the /etc/hosts file inside Docker containers is created dynamically, so this is how we launch:

[vastdata@poc-var-n1 ~]$ docker run --add-host "bwtrino.selab.vastdata.com:172.200.205.20" -it --rm trinodb/trino:475 trino --server https://bwtrino.selab.vastdata.com:8443 --user Alice
trino> SELECT current_user;
 _col0
-------
 Alice
(1 row)

Be sure to run a command that exercises the VAST Connector to confirm that the above actually talked to the database, rather than doing something local to the client without crossing the network at all (see above). Congratulations, you have working TLS on your Trino coordinator.

Configure DBeaver

Your laptop running DBeaver may use a different DNS than the lab. In that case, create an entry in /etc/hosts or C:\Windows\System32\drivers\etc\hosts as appropriate for your platform. DBeaver will insist that the hostname of the Trino coordinator matches the cert. Set SLL to true in Driver Properties, and change the port number on the Main page to 8443 if it doesn’t update automatically.

DBeaver Trino Connection Driver Properties

Again, run a query that requires crossing the network and exercising the VAST connector to verify the path end-to-end.

Configure Secure Internal Communications

When using any form of Authentication, Trino requires secure communications between workers and the coordinator. See, at a minimum, add the following two items to the configuration.yaml:

internal-communications.shared-secret - Note: this is an example only, and not sufficiently complex to provide security. See Trino documentation for practical advice regarding shared secret values.
internal-communications.https-required - while the above Trino documentation seems to imply that a shared secret is sufficient, queries will fail with communication errors unless this setting is also ‘true’.

additionalConfigProperties:
- internal-communication.shared-secret='Vast!2026'
- internal-communication.https.required=true

As the “known good” syntax for these configuration elements is complex, please see the full configuration.yaml examples in the LDAP or Password sections of this guide.

Authentication & Access Control

Prerequisites

Trino requires a secure network connection for password authentication. Please configure the Trino Coordinator for TLS before proceeding.
Trino requires secure communication between workers and the coordinator when any form of Authentication is used. Please configure Secure Internal Communications before proceeding.

Key Concepts

The key thing to understand here: Trino tells VAST the username. VAST enforces allowing or denying visibility of database objects and what operations are permitted on objects (except for masking, which is a shared responsibility between VAST and Trino in the current implementation.

Trino is built to overlay many Catalogs, each with different capabilities. When working with VAST Database Tables, the Authentication and Access Control responsibilities are distributed per the following contract:

First, Trino requires every user to have a username. When using LDAP or a Password File, Trino directly verifies the user’s name & password on an LDAP server (or in a password file). Trino passes this username through the VAST Connector.
Second, VAST evaluates Identity Policies to determine the user’s Access Controls on schemas, tables, rows, and columns within each Database - this can be as granular as (for example) Row & Column masking and/or Allow / Deny specific objects and operations.

Note: each Managing Identity Policies VAST Access Control may be set as an Identity (user) or Bucket (database, or objects within a database) policy.

User Identity

A username is always required to connect to a Trino instance, even if no authentication is configured. Trino passes the user identity to VAST, either the default user “trino” or another user determined by the authentication method. Demonstrating Row / Column features, e.g., column deny or masking, is trivial when no authentication method is configured (as in the Minimum Viable configuration.yaml) since Trino will accept any username in this configuration. In normal use, the Trino application is trusted to convey the user’s identity appropriately after verifying the user in LDAP or a Password file.

Obviously, using a password file or LDAP provides a “real” user identity. However, this involves sending a password from the Client to the Coordinator over the network (Trino doesn’t implement Challenge/Response in its Driver), so TLS also becomes a requirement to avoid sending passwords in the clear (especially for LDAP, where this password is probably the user’s AD login too).

For example, at the command line client:

$ docker run -it --rm trinodb/trino:475 trino --user Alice --server http://172.200.205.20:8080
trino> SELECT current_user AS whoami;
 whoami
--------
 Alice
(1 row)

Or via GUI clients such as DBeaver:

Setting User Name in DBeaver Connection Properties

Configuring Trino LDAP Authentication

LDAP allows Trino to verify a user’s identity and then pass the username to VAST for policy enforcement on objects in VAST Database(s). See Authentication & Access Control for a detailed explanation

Prerequisites

Trino requires a secure network connection for password authentication. Please configure the Trino Coordinator for TLS before proceeding.

Trino requires secure communication between workers and the coordinator when any form of Authentication is used. Please configure Secure Internal Communications before proceeding.

Minimum LDAP Configuration

authenticationType: Password
additionalConfigFilessection - This will become another file inside the trino_coordinator CNode, so the syntax needs to be precise to avoid breaking the yaml. Please start with this working example.

server:
  log:
    trino:
      level: INFO
  config:
    coordinatorUrl: localhost
    http:
      port: 8080
    https:
      enabled: true
      port: 8443
      keystore:
        path: "/managed_app_mnt_internal/userfiles/trino.cert"
    # -- Trino supports multiple [authentication
    # types](https://trino.io/docs/current/security/authentication-types.html):
    # PASSWORD, CERTIFICATE, OAUTH2, JWT, KERBEROS.
    authenticationType: "PASSWORD"
    query:
      maxMemory: ""

  workerExtraConfig: ""
  coordinatorExtraConfig: ""

coordinator:
  config:
    memory:
      heapHeadroomPerNode: ""
    query:
      maxMemoryPerNode: ""

worker:
  config:
    memory:
      heapHeadroomPerNode: ""
    query:
      maxMemoryPerNode: ""

additionalConfigProperties:
- internal-communication.shared-secret='Vast!2026'
- internal-communication.https.required=true

additionalLogProperties:
- com.vastdata=INFO
# additionalLogProperties -- [Additional log
# properties](https://trino.io/docs/current/installation/deployment.html#log-levels).
# @raw
# Example:
# ```yaml
#  - io.airlift=DEBUG
# ```

additionalNodeProperties: []
# additionalNodeProperties -- [Additional node
# properties](https://trino.io/docs/current/installation/deployment.html#log-levels).
# @raw
# Example, assuming the NODE_ID environment variable has been set:
# ```yaml
#  - node.id=${NODE_ID}
# ```

catalogs:
  tpch: |
    connector.name=tpch
    tpch.splits-per-node=4
  tpcds: |
    connector.name=tpcds
    tpcds.splits-per-node=4
  jmx: |
    connector.name=jmx
  vast: |
    connector.name=vast
    endpoint=http://172.200.205.1
    region=us-east-1
    access_key_id=RZV2NP1HT89CUN97QJ2B
    secret_access_key=tp+RYus0xq0wE2AS6lW2vciUt14/4bXF+Wojzq2i
    data_endpoints=http://172.200.205.1,http://172.200.205.2,http://172.200.205.3,http://172.200.205.4,http://172.200.205.5,http://172.200.205.6
    vast.http-client.request-timeout=100h
    vast.http-client.idle-timeout=290s
    num_of_splits=512
    num_of_subsplits=4

# catalogs -- Configure
# [catalogs](https://trino.io/docs/current/installation/deployment.html#catalog-properties).

additionalConfigFiles:
  password-authenticator:
    extension: properties
    body: |
      password-authenticator.name=ldap
      ldap.url=ldap://selab-var-dc2.selab.vastdata.com
      ldap.user-bind-pattern=${USER}@selab.vastdata.com
      ldap.allow-insecure=true

Finding LDAP Servers

If you are using Active Directory, you will need to find the IP Address(es) or hostname(s) of the LDAP servers (Domain Controllers). To find the SRV records in DNS, you can query from any host on the network. From Linux on the network:

$ host -t srv _ldap._tcp.selab.vastdata.com
_ldap._tcp.selab.vastdata.com has SRV record 0 100 389 selab-var-dc2.selab.vastdata.com.
_ldap._tcp.selab.vastdata.com has SRV record 0 100 389 selab-var-dc1.selab.vastdata.com.

It is always safer to use FQDNs in the ldap:// URL, and it is practically mandatory for ldaps:// since an organization will typically put host names, not IP addresses, in a Certificate.

If the above host application is not installed, find the package name that contains it and the appropriate package manager for your Linux distribution. For example, on Rocky:

sudo dnf install bind-utils -y

Troubleshooting LDAP

Obviously, basic connectivity checks (ping, etc.) from the CNode(s) need to pass for LDAP to work, and beyond that, the protocol itself is quite complex. If you can’t query LDAP from your command line, what chance does Trino have of getting the configuration right the first time :wink:

Connectivity from the CNode

Firewalls, incorrect routing configuration, flaky port bonding - any of these could get in your (and the trino_coordinator container’s) way. Don’t be afraid to SSH to CNodes and:

ping the hostname and/or IP address of the LDAP server. Firewalls may block ICMP (which is fine), but can the CNode resolve the name?
curl the LDAP server on specific ports - it is not uncommon for only the TLS ports to be open (insecure ports may immediately return “connection reset by peer”, vs timeout is probably a firewall).
Of the following ports, you only need 1 of the 4 open to get LDAP working.
Note: Even without TLS, LDAP typically uses SASL challenge/response, not sending the user’s password, so unencrypted LDAP is not unsafe.

	Insecure Port	TLS (Secure) Port
Typical OpenLDAP	389	636
Typical Active Directory	3268	3269

LDAP Protocol

This command mimics the way Trino will query an Active Directory Domain Controller via LDAP when confirming a user’s identity:

ldapsearch -x -H ldap://selab-var-dc2.selab.vastdata.com -D "bryan.whitmore@selab.vastdata.com" -W -b "DC=selab,DC=vastdata,DC=com" "(sAMAccountName=bryan.whitmore)"

The ladapsearch command is already available on CNodes if you SSH there. To install on your Linux, e.g. Rocky

sudo dnf install openldap-clients -y

When run with the -W option, the command will prompt for the user’s password. Replace the Domain and User info as appropriate for your site. If this command returns without error, you have the correct info to format the ldap.user-bind-pattern for your site.

ldap.allow-insecure=true in the configuration will allow Trino to query a secure LDAP (ldaps://) even if it cannot verify the LDAP server’s certificate. It will also allow Trino to use clear (ldap://) connections. Again, even without TLS, LDAP typically uses SASL challenge/response, not sending the user’s password, so unencrypted LDAP is not unsafe.

Configuring Trino Password File Authentication

Prerequisites

Trino requires a secure network connection for password authentication. Please configure the Trino Coordinator for TLS before proceeding.
Trino requires secure communication between workers and the coordinator when any form of Authentication is used. Please configure Secure Internal Communications before proceeding.

Installing & Using bcrypt

To install & use Bcrypt on Rocky Linux without installing specialized developer tools, use the htpasswd utility from the Apache web server:

sudo dnf install httpd-tools

and then (for the example password Vast!2026)

$ htpasswd -bnBC 10 "" 'Vast!2026' | tr -d ':\n'
$2y$10$qWKtf6lxbeYaSn175lwoSeqWp/I0lTygxEn1IP5RLDQlXQPpHTPJi

Note 1: Bcrypt generates a different hash every time for the same input because it automatically generates and includes a unique, random salt for each hashing operation. So you can use the example from this guide, but also don’t be alarmed if you generate your own and it differs from this example.

Note 2: Bcrypt libraries exist for Python + other common languages if you prefer.

Documentation Index

Trino on VAST CNodes

Minimum Viable Configuration

Initial Provisioning

Prerequisites

Create Managed Application

Notes on IP Addresses

Trivia

When Provisioning Fails

Updating configuration.yaml

Configuring TLS Certs

Troubleshooting

Connecting with HTTPS

Configure DBeaver

Configure Secure Internal Communications

Authentication & Access Control

Prerequisites

Key Concepts

User Identity

Configuring Trino LDAP Authentication

Prerequisites

Minimum LDAP Configuration

Finding LDAP Servers

Troubleshooting LDAP

Connectivity from the CNode

LDAP Protocol

Configuring Trino Password File Authentication

Prerequisites

Installing & Using bcrypt