STS for S3 Troubleshooting Guide

This guide helps diagnose and resolve issues with OIDC-based STS federation on VAST.

Quick Diagnostic Checklist

Before diving into specific errors, verify these common configuration points:

Error Reference

STS Errors (AssumeRoleWithWebIdentity)

S3 Errors (after successful STS)

Symptom-Based Troubleshooting

"AssumeRoleWithWebIdentity fails with InvalidIdentityToken"

Possible causes:

1. Using access_token instead of id_token

   The access_token from Azure is meant for Microsoft Graph API, not VAST. Always use the id_token.

   How to verify:

   # Decode your token and check the audience
   cat /tmp/jwt_token.txt | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '.aud'

   • If aud is <https://graph.microsoft.com> or similar → wrong token

   • If aud is your Azure App ID → correct token

2. Audience (aud) mismatch

   The aud claim must match the condition in your trust policy.

   How to verify:

   # Check JWT audience
   cat /tmp/jwt_token.txt | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '.aud'

   This value must exactly match the StringEquals condition in your trust policy:

   "Condition": {
     "StringEquals": {
       "login.microsoftonline.com/<TENANT>/v2.0:aud": "<THIS_MUST_MATCH>"
     }
   }

3. JWT signature cannot be verified

   VAST fetches public keys from the OIDC provider to verify JWT signatures. If this fails:

   vcli: admin> oidc list
   # Check State column
   
   vcli: admin> oidc refresh_keys --id <OIDC_ID>
   # Force a key refresh

"AssumeRoleWithWebIdentity fails with ExpiredToken"

The JWT has passed its expiration time.

How to verify:

# Check expiration timestamp
cat /tmp/jwt_token.txt | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '.exp'

# Compare to current time
date +%s

If current time > exp value, the token is expired.

Resolution: Get a new JWT from your OIDC provider (re-authenticate).

Note: Azure id_token typically expires in ~1 hour. Plan your credential renewal accordingly.

"AssumeRoleWithWebIdentity fails with AccessDenied"

Possible causes:

1. Role doesn't exist

   vcli: admin> iamrole list --tenant-id <VAST_TENANT_ID>

   Verify the role name matches what you're requesting.

2. Wrong tenant name in role ARN

   The role ARN format is: arn:vast::<TENANT_NAME>:role/<ROLE_NAME>

   Verify TENANT_NAME matches exactly (case-sensitive).

3. Trust policy denies based on claims

   If your trust policy has conditions on claims like sub, verify the JWT contains matching values:

   cat /tmp/jwt_token.txt | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '.sub'

"S3 operations fail with InvalidAccessKeyId"

Possible causes:

1. STS credentials have expired

   STS credentials expire after the DurationSeconds specified (default: 900 seconds = 15 minutes).

   Resolution: Call AssumeRoleWithWebIdentity again to get fresh credentials.

2. Using wrong VIP

   STS credentials are tenant-scoped. You must send S3 requests to a VIP assigned to that tenant.

   vcli: admin> vippool list --tenant-id <VAST_TENANT_ID>

   Use a VIP from this list.

3. Access keys were revoked

   An administrator may have revoked all access keys for the role:

   vcli: admin> iamrole revoke_access_keys --id <ROLE_ID>

   Resolution: Call AssumeRoleWithWebIdentity again to get new credentials.

4. Credentials not exported correctly

   Verify environment variables are set:

   echo $AWS_ACCESS_KEY_ID
   echo $AWS_SECRET_ACCESS_KEY
   echo $AWS_SESSION_TOKEN

   All three must be non-empty. The AWS_SESSION_TOKEN is required for STS-based credentials.

"S3 operations fail with AccessDenied"

The STS call succeeded, but the identity policy doesn't permit this operation.

How to diagnose:

1. Check what policy is attached to the role:

   vcli: admin> iamrole show --id <ROLE_ID> --identity-policies

2. Review the policy actions:

   vcli: admin> identitypolicy show --id <POLICY_ID>

3. Common issues:

   • Policy allows s3:GetObject but not s3:PutObject

   • Policy restricts to specific bucket but you're accessing a different one

   • Policy restricts to specific object prefix

Example: Policy allows only GET operations:

{
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:ListBucket"],
    "Resource": "*"
  }]
}

This policy will deny PUT, DELETE, etc.

"OIDC provider shows PROVIDER_ACCESS_FAILURE"

VAST cannot reach the OIDC discovery URL or fetch public keys.

How to diagnose:

1. Check the discovery URL is correct:

   vcli: admin> oidc list
   # Note the discovery URL

2. Test connectivity from VAST cluster (or a machine with similar network access):

   curl -s "https://login.microsoftonline.com/<TENANT_ID>/v2.0/.well-known/openid-configuration" | jq .

3. Common causes:

   • Firewall blocking outbound HTTPS to login.microsoftonline.com

   • DNS resolution failure

   • Typo in tenant ID

   • Proxy required but not configured

4. Force a key refresh after fixing connectivity:

   vcli: admin> oidc refresh_keys --id <OIDC_ID>

"OIDC provider creation fails"

Possible causes:

1. Invalid discovery URL

   The URL must be reachable and return valid OIDC metadata.

   Test it:

   curl -s "<DISCOVERY_URL>" | jq .

   Should return JSON with issuer, jwks_uri, authorization_endpoint, etc.

2. Discovery URL returns an unexpected format

   Some identity providers have non-standard configurations. Ensure the response includes required fields.

OIDC Provider Scenarios

Azure AD v1.0 vs v2.0 Endpoints

Azure AD has two endpoint versions with different behaviors:

Important: Your trust policy Principal.Federated must match the iss claim format for your chosen endpoint version.

If you're getting InvalidIdentityToken errors, verify you're using consistent endpoint versions between:

• OIDC provider discovery URL on VAST

• Token request endpoint in your client

• Trust policy principal value

Credential Lifecycle Issues

Understanding Expiration

JWT Lifetime (~1 hour)
├── STS Creds #1 (15 min) ─► Expires ─► Get new STS creds with same JWT
├── STS Creds #2 (15 min) ─► Expires ─► Get new STS creds with same JWT
├── STS Creds #3 (15 min) ─► Expires ─► Get new STS creds with same JWT
└── STS Creds #4 (15 min) ─► JWT expires ─► Must re-authenticate

Renewal Strategy

For long-running applications:

1. Track both JWT expiration (exp claim) and STS credential expiration (Expiration field)

2. Renew STS credentials before they expire (e.g., at 80% of duration)

3. When JWT expires, trigger re-authentication flow

4. Consider using refresh tokens if your OIDC provider supports them

After Key Revocation

If an administrator revokes access keys for a role:

vcli: admin> iamrole revoke_access_keys --id <ROLE_ID>

• All existing STS credentials for that role become invalid immediately

• New AssumeRoleWithWebIdentity calls will succeed and return new, valid credentials

• This is useful for emergency credential rotation

Diagnostic Commands Reference

OIDC Provider

# List all OIDC providers
vcli: admin> oidc list

# Refresh public keys (useful after connectivity issues)
vcli: admin> oidc refresh_keys --id <OIDC_ID>

Tenant Configuration

# List tenants and their OIDC provider associations
vcli: admin> tenant list

# Check VIP pools for a tenant
vcli: admin> vippool list --tenant-id <TENANT_ID>

IAM Roles and Policies

# List IAM roles in a tenant
vcli: admin> iamrole list --tenant-id <TENANT_ID>

# Show role details including trust policy
vcli: admin> iamrole show --id <ROLE_ID>

# Show role with attached identity policies
vcli: admin> iamrole show --id <ROLE_ID> --identity-policies

# List identity policies
vcli: admin> identitypolicy list --tenant-id <TENANT_ID>

# Show policy details
vcli: admin> identitypolicy show --id <POLICY_ID>

# Revoke all access keys for a role (emergency)
vcli: admin> iamrole revoke_access_keys --id <ROLE_ID>

JWT Inspection (Client-Side)

# Decode JWT header
cat /tmp/jwt_token.txt | cut -d'.' -f1 | base64 -d 2>/dev/null | jq .

# Decode JWT payload (claims)
cat /tmp/jwt_token.txt | cut -d'.' -f2 | base64 -d 2>/dev/null | jq .

# Check specific claims
cat /tmp/jwt_token.txt | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '{iss, aud, sub, exp, email}'

# Check if token is expired
EXP=$(cat /tmp/jwt_token.txt | cut -d'.' -f2 | base64 -d 2>/dev/null | jq -r '.exp')
NOW=$(date +%s)
if [ "$NOW" -ge "$EXP" ]; then echo "Token EXPIRED"; else echo "Token valid for $((EXP - NOW)) seconds"; fi

AWS CLI Debugging

# Enable debug output for AWS CLI
aws s3 ls s3://mybucket/ \
  --endpoint-url "https://<VAST_VIP>" \
  --no-verify-ssl \
  --debug

boto3 Debugging (Python)

import boto3
import logging

# Enable debug logging
boto3.set_stream_logger('', logging.DEBUG)

Common Configuration Mistakes