This guide explains how to configure AWS-compatible applications to authenticate with temporary credentials from the Security Token Service (STS) instead of long-lived access keys.
STS provides short-lived, automatically expiring credentials that improve security by eliminating the need to distribute or store permanent credentials in applications. Temporary credentials can be issued on demand, scoped to specific permissions, and integrated with existing identity providers and authentication workflows.
Many applications and tools that support the Amazon S3 API - including Boto3, AWS CLI, Apache Hadoop, and other S3-compatible clients - can use STS-issued credentials with little or no application changes. In most cases, the integration consists of obtaining temporary credentials from an STS endpoint and configuring the application to use the resulting access key, secret key, and session token.
This document provides step-by-step instructions for configuring common applications to authenticate with STS. Each application is described, including the required configuration, how to supply temporary credentials, and any application-specific considerations or limitations.
If you want to set up STS on your VAST cluster: STS AssumeRole with a Permanent Access Key Pair (No OIDC)
Applications Usage
For this section, we assume you have the application set up and ready for use.
If you're installing for the first time, refer to the appendix
boto3
boto3.py
#!/usr/bin/env python3
"""VAST STS AssumeRole smoke test (boto3)."""
import os
import boto3
from botocore.config import Config
ROLE_ARN = os.environ.get(
"VAST_ROLE_ARN", "arn:vast::default:role/<YOUR_ROLE>"
)
STS_ENDPOINT = os.environ.get("VAST_STS_ENDPOINT", "https://vip")
S3_ENDPOINT = os.environ.get("VAST_S3_ENDPOINT", "http://vip")
CA_BUNDLE = os.environ.get("AWS_CA_BUNDLE", "/path/to/you/ca/budnle")
BUCKET = os.environ.get("VAST_TEST_BUCKET", "<YOUR_BUCKET>")
def main():
session = boto3.session.Session(region_name="us-east-1") #Needed
sts = session.client(
"sts",
endpoint_url=STS_ENDPOINT,
verify=CA_BUNDLE,
config=Config(signature_version="v4"),
)
print(f"Calling AssumeRole on {STS_ENDPOINT} ...")
out = sts.assume_role(
RoleArn=ROLE_ARN,
RoleSessionName="boto3-vast-tutorial",
DurationSeconds=3600, #15 minutes minimum to 36 hours maximum
)
creds = out["Credentials"]
print("OK assume_role")
print(" AccessKeyId :", creds["AccessKeyId"])
print(" Expiration :", creds["Expiration"])
print(" SessionToken:", creds["SessionToken"][:24] + "...")
# Optional: prove temp creds work on S3
s3 = boto3.client(
"s3",
endpoint_url=S3_ENDPOINT,
aws_access_key_id=creds["AccessKeyId"],
aws_secret_access_key=creds["SecretAccessKey"],
aws_session_token=creds["SessionToken"],
region_name="us-east-1",
config=Config(
s3={"addressing_style": "path"},
signature_version="s3v4",
),
)
print(f"Listing s3://{BUCKET}/tables/ via {S3_ENDPOINT} ...")
resp = s3.list_objects_v2(
Bucket=BUCKET, Prefix="...", Delimiter="/", MaxKeys=20
)
prefixes = [p["Prefix"] for p in resp.get("CommonPrefixes", [])]
print(" Prefixes:", prefixes)
if not prefixes:
print(" WARNING: empty listing — check role IAM policy for this bucket")
else:
print("OK S3 list with temporary credentials")
if __name__ == "__main__":
main()Output example
Details
Calling AssumeRole on https://vip ...
OK assume_role
AccessKeyId : TEMP................
Expiration : 2026-07-21 ...
SessionToken: AQAAAFRFTVA................
Listing s3://<>/tables/ via http://vip ...
Prefixes: ['tables/revenue_sts/', 'tables/sample_sts/', ...]
OK S3 list with temporary credentialsaws cli
Details
export AWS_ACCESS_KEY_ID='<STATIC_ACCESS_KEY>'
export AWS_SECRET_ACCESS_KEY='<STATIC_SECRET_KEY>'
export AWS_DEFAULT_REGION=us-east-1
export AWS_CA_BUNDLE=/path/to/your/ca/bundle
aws sts assume-role \
--role-arn arn:vast::default:role/<YOUR_ROLE> \
--role-session-name cli-check \
--duration-seconds 3600 \ #15 minutes minimum to 36 hours maximum
--endpoint-url https://vipOutput example
Details
OK assume_role
AccessKeyId: TEMP...
Expiration: 2026-07-21 12:35:23+00:00
S3 prefixes: tables/revenue_by_region/, revenue_sts/, sample/, sample_sts/Spark (Hadoop)
Prerequisites
Before you begin, ensure that your VAST cluster is configured with a Java KeyStore (JKS) certificate to enable secure HTTPS communication.
In case your Spark context is longer then the sts expiration time, Spark (Hadoop) will ask for new keys (by default, 5 minutes before the key is expiried)
spark_sts.py
#!/usr/bin/env python3
"""
Spark job using Hadoop AssumedRoleCredentialProvider with VAST STS.
With proper SSL/TLS configuration for HTTPS.
"""
from botocore.exceptions import ClientError
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DoubleType
from datetime import datetime
import urllib3
urllib3.disable_warnings()
# Configuration
VAST_ENDPOINT = "<VIP>"
BUCKET_NAME = "<BUCKET_NAME>"
BASE_ACCESS_KEY = ""
BASE_SECRET_KEY = ""
CA_CERT = "/opt/spark/conf/ssl/vast_cert/RootCA.crt"
def create_spark_session():
"""
Create Spark session with AssumedRoleCredentialProvider.
Uses SSL truststore for HTTPS STS endpoint.
"""
print("=" * 60)
print("STEP 2: Creating Spark with AssumedRoleCredentialProvider")
print("=" * 60)
print("Configuration:")
print(" credentials.provider = AssumedRoleCredentialProvider")
print(" assumed.role.sts.endpoint = " + VAST_ENDPOINT)
print(" SSL: Using RootCA.crt truststore")
print("")
spark = SparkSession.builder \
.appName("VAST-STS-HTTPS-Demo") \
.config("spark.hadoop.fs.s3a.endpoint", VAST_ENDPOINT) \
.config("spark.hadoop.fs.s3a.path.style.access", "true") \
.config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \
.config("spark.hadoop.fs.s3a.connection.ssl.enabled", "true") \
.config("spark.hadoop.fs.s3a.aws.credentials.provider",
"org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider") \
.config("spark.hadoop.fs.s3a.assumed.role.arn",
"arn:vast::default:role/<YOUR_ROLE>") \
.config("spark.hadoop.fs.s3a.assumed.role.sts.endpoint", VAST_ENDPOINT) \
.config("spark.hadoop.fs.s3a.assumed.role.sts.endpoint.region", "us-east-1") \
.config("spark.hadoop.fs.s3a.assumed.role.session.duration", "1h") \
.config("spark.hadoop.fs.s3a.assumed.role.session.name", "spark-vast-only") \
.config("spark.hadoop.fs.s3a.assumed.role.credentials.provider",
"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider") \
.config("spark.hadoop.fs.s3a.access.key", BASE_ACCESS_KEY) \
.config("spark.hadoop.fs.s3a.secret.key", BASE_SECRET_KEY) \
.getOrCreate()
spark.sparkContext.setLogLevel("WARN")
print("✅ Spark session created with SSL")
return spark
def write_parquet(spark):
"""Write sample Parquet data via S3A."""
print("=" * 60)
print("STEP 3: Writing Parquet to VAST via S3A (HTTPS)")
print("=" * 60)
data = [
("Alice", 34, "Engineering", 85000.0),
("Bob", 45, "Marketing", 72000.0),
("Charlie", 29, "Engineering", 78000.0),
("Diana", 38, "Sales", 92000.0),
("Eve", 41, "Engineering", 95000.0),
]
schema = StructType([
StructField("name", StringType(), False),
StructField("age", IntegerType(), False),
StructField("department", StringType(), False),
StructField("salary", DoubleType(), False),
])
df = spark.createDataFrame(data, schema)
df.show()
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
output_path = f"s3a://{BUCKET_NAME}/employees_{timestamp}"
print(f"Writing to: {output_path}")
df.write.mode("overwrite").parquet(output_path)
print("✅ Parquet written successfully")
return output_path
def read_parquet(spark, path):
"""Read back and verify."""
print("=" * 60)
print("STEP 4: Reading Parquet back")
print("=" * 60)
df = spark.read.parquet(path)
print(f"✅ Read {df.count()} rows")
df.createOrReplaceTempView("employees")
spark.sql("""
SELECT department, COUNT(*) as count, ROUND(AVG(salary), 2) as avg_salary
FROM employees GROUP BY department ORDER BY avg_salary DESC
""").show()
def main():
print("=" * 60)
print("SPARK + VAST STS with HTTPS")
print("Using Hadoop AssumedRoleCredentialProvider")
print("=" * 60)
print("")
spark = create_spark_session()
try:
output_path = write_parquet(spark)
read_parquet(spark, output_path)
print("")
print("=" * 60)
print("✅ SUCCESS - VAST STS works with HTTPS!")
print(f" Output: {output_path}")
print("=" * 60)
finally:
spark.stop()
if __name__ == "__main__":
main()
docker-compose.yml
services:
spark-job:
build: .
container_name: spark-vast-ssl
environment:
AWS_REGION: "us-east-1"
AWS_DEFAULT_REGION: "us-east-1"
command: >
/opt/spark/bin/spark-submit
--master local[*]
--conf "spark.driver.extraJavaOptions=-Dcom.amazonaws.services.s3.enableV4=true -Djavax.net.ssl.trustStore=/opt/spark/conf/ssl/truststore.jks -Djavax.net.ssl.trustStorePassword=password"
/app/spark_vast.py
volumes:
- ./spark_vast_only.py:/app/spark_vast_only.py:ro
- ./https:/opt/spark/conf/ssl:roOutput example
Details
docker compose run --rm spark-job
WARN[0000] Found orphan containers ([minio]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
============================================================
SPARK + VAST STS with HTTPS
Using Hadoop AssumedRoleCredentialProvider
============================================================
============================================================
STEP 1: Checking bucket exists (HTTPS with CA cert)
============================================================
============================================================
STEP 2: Creating Spark with AssumedRoleCredentialProvider
============================================================
Configuration:
credentials.provider = AssumedRoleCredentialProvider
assumed.role.sts.endpoint = https://172.27.117.1
SSL: Using RootCA.crt truststore
26/01/05 09:20:37 INFO SparkContext: Running Spark version 3.5.3
26/01/05 09:20:37 INFO SparkContext: OS info Linux, 4.18.0-372.26.1.el8.lb.x86_64, amd64
26/01/05 09:20:37 INFO SparkContext: Java version 11.0.24
26/01/05 09:20:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
26/01/05 09:20:37 INFO ResourceUtils: ==============================================================
26/01/05 09:20:37 INFO ResourceUtils: No custom resources configured for spark.driver.
26/01/05 09:20:37 INFO ResourceUtils: ==============================================================
26/01/05 09:20:37 INFO SparkContext: Submitted application: VAST-STS-HTTPS-Demo
26/01/05 09:20:37 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
26/01/05 09:20:37 INFO ResourceProfile: Limiting resource is cpu
26/01/05 09:20:37 INFO ResourceProfileManager: Added ResourceProfile id: 0
26/01/05 09:20:37 INFO SecurityManager: Changing view acls to: spark
26/01/05 09:20:37 INFO SecurityManager: Changing modify acls to: spark
26/01/05 09:20:37 INFO SecurityManager: Changing view acls groups to:
26/01/05 09:20:37 INFO SecurityManager: Changing modify acls groups to:
26/01/05 09:20:37 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: spark; groups with view permissions: EMPTY; users with modify permissions: spark; groups with modify permissions: EMPTY
26/01/05 09:20:37 INFO Utils: Successfully started service 'sparkDriver' on port 33435.
26/01/05 09:20:37 INFO SparkEnv: Registering MapOutputTracker
26/01/05 09:20:37 INFO SparkEnv: Registering BlockManagerMaster
26/01/05 09:20:37 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
26/01/05 09:20:37 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
26/01/05 09:20:37 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
26/01/05 09:20:37 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-32072331-a150-460c-b1d6-4551e2b525b1
26/01/05 09:20:37 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
26/01/05 09:20:37 INFO SparkEnv: Registering OutputCommitCoordinator
26/01/05 09:20:37 INFO JettyUtils: Start Jetty 0.0.0.0:4040 for SparkUI
26/01/05 09:20:37 INFO Utils: Successfully started service 'SparkUI' on port 4040.
26/01/05 09:20:38 INFO Executor: Starting executor ID driver on host 8a85850b5ba8
26/01/05 09:20:38 INFO Executor: OS info Linux, 4.18.0-372.26.1.el8.lb.x86_64, amd64
26/01/05 09:20:38 INFO Executor: Java version 11.0.24
26/01/05 09:20:38 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): ''
26/01/05 09:20:38 INFO Executor: Created or updated repl class loader org.apache.spark.util.MutableURLClassLoader@746b2070 for default.
26/01/05 09:20:38 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38647.
26/01/05 09:20:38 INFO NettyBlockTransferService: Server created on 8a85850b5ba8:38647
26/01/05 09:20:38 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
26/01/05 09:20:38 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 8a85850b5ba8, 38647, None)
26/01/05 09:20:38 INFO BlockManagerMasterEndpoint: Registering block manager 8a85850b5ba8:38647 with 434.4 MiB RAM, BlockManagerId(driver, 8a85850b5ba8, 38647, None)
26/01/05 09:20:38 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 8a85850b5ba8, 38647, None)
26/01/05 09:20:38 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 8a85850b5ba8, 38647, None)
✅ Spark session created with SSL
============================================================
STEP 3: Writing Parquet to VAST via S3A (HTTPS)
============================================================
+-------+---+-----------+-------+
| name|age| department| salary|
+-------+---+-----------+-------+
| Alice| 34|Engineering|85000.0|
| Bob| 45| Marketing|72000.0|
|Charlie| 29|Engineering|78000.0|
| Diana| 38| Sales|92000.0|
| Eve| 41|Engineering|95000.0|
+-------+---+-----------+-------+
Writing to: s3a://test-view2/employees_20260105_092041
26/01/05 09:20:41 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
✅ Parquet written successfully
============================================================
STEP 4: Reading Parquet back
============================================================
✅ Read 5 rows
+-----------+-----+----------+
| department|count|avg_salary|
+-----------+-----+----------+
| Sales| 1| 92000.0|
|Engineering| 3| 86000.0|
| Marketing| 1| 72000.0|
+-----------+-----+----------+
============================================================
✅ SUCCESS - VAST STS works with HTTPS!
Output: s3a://test-view2/employees_20260105_092041
============================================================Appendix
Boto3 Installation
pip install boto3AWS CLI installation
Installing or updating to the latest version of the AWS CLI