NFS 16 Group Limit

TL;DR - Set the Group Membership Source in the View Policy to include Providers.

Background

Like the quote attributed to Bill Gates of “640K ought to be enough for anybody.” NFS has a built-in limit to the number of groups included when using sec=sys. Namely, by definition, the NFS RPC is limited to 16 groups at most. However, if a user is part of more than 16 groups, it’s wholly up to the client to determine which 16 groups are sent in the NFS RPC call, and that generally isn’t controllable by the end user.

This is the “16 Group Problem” and in most cases includes a reference to:

https://xkyle.com/solving-the-nfs-16-group-limit-problem/

Configuration

What is the VAST equivalent to --manage-gids, and how does VAST handle cases where a user is part of more than 16 groups?

For VAST, the group membership that is used to perform access checks for NFS is controlled by the NFS Group Membership Source setting in the View Policy. This can be set in one of three different ways:

NFS RPC - Use only the information in the NFS RPC call to determine group membership when doing access checks. Only the UID and the 16 GIDs that are in the NFS RPC will be used. These IDs can be locally defined on the client (i.e.,/etc/passwd or /etc/group). No external providers need to be consulted because everything is contained in the NFS RPC call. In this case, the VAST cluster doesn’t need to be bound to AD / LDAP / NIS. If there are providers defined, they will not be considered. This is going to be subject to the 16 group limit because that’s all that’s in the NFS RPC call.

Providers - Use the UID in the NFS RPC and perform lookups in the configured providers (Active Directory / LDAP / NIS / Local) to determine group membership. Ignore any groups contained in the NFS RPC call itself. The group membership for the user will be combined from all providers, even if the group in question has no POSIX Attributes (i.e., gidNumber). VAST currently supports up to 1024 groups per user. (See here.) The caveat here is that if a UID cannot be found in any Provider, the user is denied access. This also rules out users and groups defined locally on the client (not to be confused with users and groups defined in the Local provider on the VAST Cluster itself).

NFS RPC and Providers - Use the group membership contained in the NFS RPC and combine it with group membership as determined from the configured providers, up to 1024 groups total. Unlike Providers alone, this will allow for both client local users and client local group membership (again users in a client’s /etc/passwd or group membership in a client’s /etc/group).

Possible Limitations

If attempting to set a group owner via NFS on a View that has a View Policy of Providers alone, the group must be resolvable via the Providers. For NFSv3 or when setting a group owner using a numeric GID with NFSv4, this requires that the Group have a gidNumber defined in a Provider. If one cannot be located, the operation will fail.

With NFS RPC and Providers as the Group Membership source, depending on how the Linux client determined its UID and GID numbers, things can appear to be still subject to the 16 group limit, but for an entirely different reason.

For clients using Centrify, UID and GID numbers are not stored as attributes in Active Directory that can be directly queried. Centrify does offer an LDAP Proxy that fills this role and allows lookup of UID and GID numbers.

Likewise, for clients that use sssd on Linux, the option ldap_id_mapping controls where UID and GID numbers are sourced from. If set ldap_id_mapping = False, sssd will obtain these from specific attributes in the user objects in AD, typically uidNumber and gidNumber, which the VAST cluster can query as well. However, if ldap_id_mapping = True, then sssd will generate the UID and GID numbers based on the Windows Domain and the user’s Windows SID. VAST clusters have no way of looking up groups or users via these numbers.

Since NFS RPC and Providers combine the groups provided in the NFS RPC call and what is able to be determined from the Providers, users that cannot be located in the Providers will still be granted access, however it will be based ONLY on the information in the NFS RPC call itself and thus limited to 16 groups.

Additionally, if a group cannot be located in a Provider based on its GID number, that group can still be set as a group owner, though access based on the GID number will only be granted to users whose NFS RPC call contains that GID number in the included 16 groups. This can be especially confusing to end users as the Linux client will happily translate the GID number into a group name.

Troubleshooting

When troubleshooting group access issues for users who are part of more than 16 groups, validate the following.

The Group Membership Source is set to either Providers or NFS RPC and Providers.
Determine where the client is getting its UID and GID numbers from. Look for ldap_id_mapping in /etc/sssd/sssd.conf or determine the Centrify configuration.
Validate that the necessary Providers are configured on the VAST Cluster
Perform a lookup on the VAST cluster using the UID number of the user. Use either the user query from vcli or User Management → Users → Search from the VMS GUI.
Perform a lookup of the group that should be granting access using the GID number. Use either the group query from vcli or User Management → Groups → Search from the VMS GUI

Possible Workarounds

Populate the UID and GID Numbers in the providers, or fix the attribute mapping so they are resolvable.
Change from using sec=sys to sec=krb5. This will require NFSv4. The advantage of doing this is that Linux clients will no longer use numeric UIDs and GIDs in the NFS RPC calls, but will instead use user principals (i.e., username@domain), which are easily resolvable.