Overview
When running Trino on VAST CNodes, users may want to ensure that long-running or resource-intensive queries can recover from node failures. This article explains how to configure query resiliency using Trino's Fault-Tolerant Execution (FTE) backed by an S3 exchange target.
Classic "Spill to Disk" vs. Exchange Spooling
Historically, users have relied on classic local SSD "spill to disk" to handle resource constraints. However, VAST engineering strongly advises against using classic "spill to disk". Trino generally runs faster without it, and local-SSD spill is less well-supported in Trino than the S3 path.
Instead, the recommended approach for query resiliency is Trino's Fault-Tolerant Execution (FTE) using exchange spooling to S3. FTE mitigates query failures by spooling intermediate exchange data to storage. In the event of a worker outage or fault during execution, that data can be reused by another worker, allowing the query or individual tasks to be retried automatically.
The Performance Trade-Off
While FTE significantly improves query resiliency, it can reduce raw query performance. Because intermediate data is written to and read from S3, customers who prioritize maximum raw performance over resiliency may choose not to enable this feature. Furthermore, a TASK retry policy is best suited for large batch queries and can result in higher latency for short-running queries.
Storage Options: Who Provides the S3 Backend? FTE requires an S3 storage backend to act as the exchange target.
Default Recommendation (VAST S3): The default assumption is that the customer will use their own VAST cluster's S3 capabilities. This is usually already enabled and offers ample capacity.
Alternative (External S3): If a customer does not want S3 enabled on their VAST cluster, Trino's exchange manager can be configured to point to an external S3-compatible storage system, such as AWS S3 or MinIO.
Step-by-Step Configuration Guide
To enable FTE on VAST CNodes, it must be configured via the Trino configuration exposed by VAST.
Step 1: Create a VAST S3 Bucket and Credentials:
First, provision an S3 bucket on the VAST cluster to serve as the spooling destination. Next, generate S3 credentials (access_key_id and secret_access_key) for a user with permission to access this bucket.
Step 2: Configure Trino's Exchange Manager
Update the VAST Trino configuration to point the exchange manager to your newly created S3 bucket. You will need to define the retry policy (e.g., TASK) and supply the S3 credentials and endpoint.
Reference Configuration
Below is a sample configuration adapted for VAST CNodes using an S3 exchange target. Customers can adapt the bucket name, endpoint, and secrets to match their environment:
additionalConfigFiles:
exchange-manager:
extension: properties
body: |
exchange-manager.name=filesystem
exchange.base-directories=s3://<your-exchange-spooling-bucket>
exchange.s3.aws-access-key=<REDACTED>
exchange.s3.aws-secret-key=<REDACTED>
exchange.s3.region=us-east-1
exchange.s3.endpoint=http://<vast-s3-vip>:80
exchange.s3.path-style-access=true
additionalConfigProperties:
- exchange.deduplication-buffer-size=32MB
- retry-policy=TASK
additionalExchangeManagerProperties:
- exchange-manager.name=filesystem
- exchange.base-directories=s3://<your-exchange-spooling-bucket>
- exchange.s3.aws-access-key=<REDACTED>
- exchange.s3.aws-secret-key=<REDACTED>
- exchange.s3.region=us-east-1
- exchange.s3.endpoint=http://<vast-s3-vip>:80
- exchange.s3.path-style-access=true
additionalLogProperties:
- com.vastdata=INFO
Note: Ensure you replace <your-exchange-spooling-bucket>, <REDACTED> credentials, and the <vast-s3-vip> endpoint with your actual VAST S3 environment details.