Scenario:
S3 access has been working for a given user, but then suddenly stops working with the following error:
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS access key Id you provided does not exist in our records.
Root Cause:
This can happen when the user's UID is changed or added in Active Directory. The next time VAST queries AD for the user and sees that the UID needs to be updated, it detects a conflict between the user's VIDs in the local UDB. It then overwrites the older VID that has the S3 keys, removing them. An error like this is found in the CNode leader trace:
2022-07-22 16:00:20.884879801 (P04985:E001:S000:F0ee455de) {PLAS:UMR--:HI-:WRN} [notifier.cpp:48:notify]: NOTIFY: [severity:info counter: 0] Removing s3 data from vid=105 because it conflicts with vid=138 file=src/proto/common/udb/udb2.cpp:2252 function=void Auth::UDB2::fill_udb2_user_result_v2(Auth::UDB2UserEntry *, GetUdb2UserResultV2::RootBuilder *) silo=0