Cluster Token Mapper alerts

Overview

This KB covers the relevant information for the following alerts:

Cluster Token Mapper usage is above 52%

The alerts usually mean the system is getting full and approaching the Token mapper's limits.

Background

Token mapper is an internal resource that consumes space in the SCMs and is used to optimize specific background flows in the system.
For example, one use of the token mapper is to specify which elements the delete background flows should work on.
In some scenarios, a massive delete (for example, by the Trash folder) may increase the token mapper - when a massive delete is followed by delete background flows that not keeping up to handle the deletion fast enough, the mapping of the handles in the token mapper will increase and not cleared until the delete background flows will get to clear the delete backlog.

Capacity thresholds

ABUNDANT - Token mapper usage is below 60%
SCARCE - Token mapper usage is above 60%
HALT_WRITE -
- In versions prior to 5.0.0-sp24, if Token mapper usage reaches 65%, the cluster will be read-only.
- Version 5.0.0-sp24 and above: if Token mapper usage reaches 95%, the cluster will be read-only.

Monitor

For tracking the Token mapper usage of the cluster, please create a graph in Analytics, called Token mapper usage, which includes the following metric NVRamHashMapMetrics,name=new_token_mapper,block_used_space_percent__avg - in this example, the Token mapper usage percent is ~69.93%:

In such scenarios of Token mapper growth, it is worth also looking at the Cluster Delete graph. In this case, the token mapper usage increase correlates with the time of the Delete Element Backlog increase - this means that probably the delete flows are not keeping up with the rate of the deletion that comes from the clients:
Delete the element backlog

Troubleshooting

If the token mapper is increasing and it correlates with the Delete Backlog increase, the Delete flows may not be keeping up with the delete rate from clients. We recommend contacting Customer Success support for further investigation and considering adjusting the system settings relevant to deleting flows.
If the token mapper is increasing and does not correlate with the Delete Backlog increase, we recommend contacting Customer Success support for further investigation.

Capacity terms can be found at VAST Cluster Administrator's Guide (requires a valid user to access the Vast customer portal) under the Capacity section.