TACACS+ Integration on NVIDIA Cumulus Linux with Cisco ISE

1. What is TACACS+ on Cumulus?

Cumulus Linux implements TACACS+ as a client for centralized AAA (Authentication, Authorization, and Accounting) of administrative access to the switch. It uses PAM and NSS to integrate TACACS+ transparently into the Linux login system no local accounts need to be created for TACACS+ users.

What TACACS+ controls:

• SSH logins
• Console access
• sudo command execution
• NVUE (nv set, nv show, nv apply) commands

What TACACS+ does NOT control:

• Data plane traffic forwarding through the switch
• Routing protocol operation
• OpenTelemetry (OTEL) telemetry export see Section 9

Note: The TACACS+ client is only supported through the management interface (eth0) or management VRF. It is not supported through bond, SVI, or swp interfaces.

2. Version Support Matrix

Recommendation: Use 5.11 (ESR) for maximum stability or 5.16 (latest) for the most features.

Warning: Upgrading from 5.14 to 5.15+ requires updating TACACS+ NVUE command syntax. exclude-users changed to exclude-user (singular), and server config syntax changed. Update any automation scripts before upgrading.

5.15 Syntax Change Reference

3. How Authentication Works

User SSH login → Cumulus PAM → TACACS+ server (port 49) → ISE/AAA server
                                                                │
                                                     Identity lookup (AD/LDAP)
                                                                │
                                                  priv_lvl attribute returned
                                                                │
                                              Privilege level enforced on switch

Key behaviors:

• Authentication uses PAM and includes login, SSH, sudo, and su
• NSS maps TACACS+ users to local privilege-level accounts (tacacs0 through tacacs15)
• No need to create local accounts for TACACS+ users
• Accounting records are sent to all configured TACACS+ servers by default
• Up to 8 TACACS+ servers supported (5.16+); 7 in earlier versions

4. Privilege Levels

The TACACS+ server returns a priv_lvl attribute (0–15) after authentication. Cumulus enforces access based on this value.

Note: The privilege level is configured on the TACACS+ server (e.g., ISE Shell Profile), not on the Cumulus switch. The switch receives and enforces the level returned by the server.

5. Cumulus Linux Configuration

Required Settings

# TACACS+ server syntax for Cumulus 5.15+
nv set system aaa tacacs server <TACACS-SERVER-IP> priority 5
nv set system aaa tacacs server <TACACS-SERVER-IP> secret '<shared-secret>'

# VRF use mgmt for out-of-band management interface
nv set system aaa tacacs vrf mgmt

# Authentication order TACACS+ first, local fallback
nv set system aaa authentication order tacacs local

# Exclude local accounts from TACACS+ (always authenticate locally)
nv set system aaa tacacs exclude-user cumulus

# Enable accounting
nv set system aaa tacacs accounting state enabled

nv config apply

Note on priority: The priority value controls which server is tried first when multiple servers are configured. It is not the TACACS+ privilege level. Lower number = higher priority.

Multiple Servers

nv set system aaa tacacs server <PRIMARY-IP>   priority 5
nv set system aaa tacacs server <SECONDARY-IP> priority 10
nv config apply

Verify Configuration

nv show system aaa tacacs
nv show system aaa tacacs server

Optional Parameters

# Change timeout (default: 5 seconds)
nv set system aaa tacacs timeout 10

# Change port (default: 49)
nv set system aaa tacacs server <IP> port 49

# Set source IP for TACACS+ packets (useful for ACLs on ISE)
nv set system aaa tacacs source-ip <LOOPBACK-IP>

# Change authentication mode (default: pap)
nv set system aaa tacacs authentication mode chap

# Send accounting to first responding server only (default: all)
nv set system aaa tacacs accounting send-records first-response

Test Connectivity

# Verify reachability
ping -I eth0 <TACACS-SERVER-IP>

# Verify port 49 is open
nc -zv <TACACS-SERVER-IP> 49
# Expected: Connection to <IP> 49 port [tcp/tacacs] succeeded!

Warning: Connection refused on port 49 means the Device Admin Service is not enabled on ISE. This is the most common initial issue.

6. Cisco ISE Configuration

Prerequisites on ISE

1. Enable Device Admin Service on the PSN node:
   Administration → System → Deployment → [node] → Enable Device Admin Service
   (Without this, ISE will not listen on port 49)

2. Join Active Directory (if using AD authentication):
   Administration → Identity Management → External Identity Sources → Active Directory

3. Import AD Groups to use as authorization conditions:
   AD join point → Groups tab → Add → Select Groups from Directory

Add the Cumulus Switch as a Network Device

Work Centers → Device Administration → Network Resources → Network Devices → Add

Create a TACACS Shell Profile

Work Centers → Device Administration → Policy Elements → Results → TACACS Profiles → Add

Create a Device Admin Policy Set

Work Centers → Device Administration → Device Admin Policy Sets

1. Add a new policy set with a condition matching your network devices (e.g. by Device Type)
2. Set Allowed Protocols to Default Device Admin
3. Open the policy set and configure:

Authentication Policy:

• Default rule → Identity Source: your AD join point

Authorization Policy (add above the default deny rule):

Note: ISE 3.x does not require an Essentials/Advantage/Premier tier license for TACACS+ Device Administration. Only the Device Admin license (L-ISE-TACACS-ND=) one per PSN is required.

7. Local User Behavior with TACACS+ Enabled

When TACACS+ is enabled, local users and TACACS+ users coexist on the switch.

Authentication Order

nv set system aaa authentication order tacacs local

Local vs TACACS+ Users Compared

Excluding Users from TACACS+

Always exclude accounts that should never go to TACACS+:

# Exclude the default admin account
nv set system aaa tacacs exclude-user cumulus

# Exclude a monitoring service account
nv set system aaa tacacs exclude-user <monitoring-user>

nv config apply

Critical: Always keep at least one local break-glass account excluded from TACACS+. If ISE becomes unreachable and all accounts go through TACACS+, no one can log in to the switches.

Known Limitation Simultaneous Logins

If two TACACS+ users log in simultaneously with the same privilege level, they share the same mapped local UID (e.g., both map to tacacs15). Files and processes are attributed to both users. This is an inherent Linux TACACS+ limitation. TACACS+ accounting on the server side tracks the real username regardless.

8. Monitoring User Best Practice

Automated monitoring scripts that SSH into Cumulus switches should use a dedicated local account with minimum required privileges not a TACACS+ user and not the cumulus admin account.

Why a Local Account

• Monitoring must continue even if ISE is unreachable
• Avoids generating large volumes of TACACS+ accounting records from automated polling
• Easier to audit and scope to exactly the commands needed

Minimum Required Access for Common Monitoring Tasks

Create the Monitoring User

# Create local user with read-only NVUE role
nv set system aaa user <monitoring-user> role nvue-monitor
nv set system aaa user <monitoring-user> password

# Exclude from TACACS+ always authenticate locally
nv set system aaa tacacs exclude-user <monitoring-user>

nv config apply

The built-in nvue-monitor role grants:

• ✅ All nv show commands
• ✅ Read-only system commands including decode-syseeprom
• ❌ No nv set, nv unset, or nv apply
• ❌ No sudo access

User Account Summary

9. OTEL Telemetry is Not Affected by TACACS+

TACACS+ and OpenTelemetry (OTEL) operate on completely separate planes of the switch. They are independent, use different protocols, and enabling or configuring TACACS+ has zero effect on OTEL operation.

Comparison

What OTEL Collects (Always Independent of TACACS+)

• Interface statistics counters, errors, drops, utilization
• Buffer statistics ingress/egress occupancy and watermarks
• Histogram data queue depth and latency
• Platform statistics CPU, memory, temperature, fans, PSUs
• Routing metrics BGP peer state, route table counts
• Control plane statistics control plane packet counters
• ACL statistics packet/byte counts per rule (5.15+)
• AI Ethernet statistics adaptive routing, SRv6, packet trimming (5.14+)
• Systemd service health statistics

OTEL Hardware Support

Basic OTEL Setup (Unchanged by TACACS+)

# Enable OTEL completely independent of TACACS+ state
nv set system telemetry enable on
nv set system telemetry export otlp state enabled
nv set system telemetry export otlp grpc destination <COLLECTOR-IP> port 4317
nv set system telemetry export otlp grpc insecure enabled
nv set system telemetry export vrf mgmt
# Enable specific metric categories
nv set system telemetry interface-stats egress-buffer traffic-class 3
nv set system telemetry interface-stats export state enabled
nv set system telemetry interface-stats ingress-buffer priority-group 3
nv set system telemetry interface-stats sample-interval 30
nv set system telemetry interface-stats switch-priority 3
nv set system telemetry label cluster description <Cluster>
nv set system telemetry label switch_hostname description <TAG>
nv set system telemetry label switch_ip description <IP>


nv config apply

Warning: Do not enable OTEL and gNMI streaming simultaneously, they conflict with each other. This restriction is unrelated to TACACS+.

10. Accounting

Accounting is enabled by default when TACACS+ is configured. All commands run by TACACS+ users generate records sent to the configured server(s).

What Gets Logged

• All Linux commands
• All NVUE commands
• Login and logout events
• Sub-processes spawned by commands
• Sudo commands (logged against the original TACACS+ login name)

What Does NOT Get Logged

• Inline editor actions (vi, nano file edits)
• vtysh internal actions
• Commands longer than 240 bytes are truncated (TACACS+ protocol limit)

Accounting Commands

# Enable accounting
nv set system aaa tacacs accounting state enabled

# Disable accounting
nv set system aaa tacacs accounting state disabled

# Send to first responding server only (default: all servers)
nv set system aaa tacacs accounting send-records first-response

# Send to all servers (default)
nv set system aaa tacacs accounting send-records all

nv config apply

11. Per-Command Authorization

Per-command authorization allows granular control over which commands specific privilege levels can run. It operates at two layers:

Layer 1 Cumulus Switch (Coarse)

On Cumulus, authorization is enforced at the initial word of the command only:

# Allow privilege level 0 to run ip and nv commands
nv set system aaa tacacs authorization 0 command ip
nv set system aaa tacacs authorization 0 command nv
nv config apply

nv permits all NVUE commands beginning with nv. For more granular control of specific sub-commands, use ISE.

Layer 2 ISE (Granular)

Configure TACACS Command Sets in ISE for fine-grained per-command control:

Work Centers → Device Administration → Policy Elements → Results → TACACS Command Sets

Example command sets:

• Permit show only: permit nv show, deny everything else
• Permit interface config: permit nv set interface, deny nv set system

Attach command sets to authorization rules in your Device Admin Policy Set.

Note: As of Cumulus 5.16, per-command authorization supports NVUE tab completion and ? option listing.

12. Troubleshooting

Common Issues

Debug on Cumulus

# Watch TACACS+ events in real time
tail -f /var/log/syslog | grep -i <username>

# Check audit records
cat /var/log/audit/audit.log

# Debug per-command authorization (run as TACACS+ user)
export TACACSAUTHDEBUG=1

# Restart audit daemon if accounting records are missing
sudo systemctl restart auditd.service

# Notify audisp plugin after config changes
sudo killall -HUP audisp-tacplus

# Add debug logging to accounting config
# Edit /etc/audisp/audisp-tac_plus.conf → add: debug=1

Debug on ISE

Operations → TACACS → Live Logs shows every authentication and authorization attempt in real time with the exact pass/fail reason, matched policy, and applied shell profile.

13. Reference

Key Files on Cumulus

NVUE Quick Reference

# Show TACACS+ config
nv show system aaa tacacs

# Show auth order
nv show system aaa

# Show server status
nv show system aaa tacacs server

# Show monitoring user
nv show system aaa user <username>

Documentation Links

• Cumulus Linux TACACS+ (5.16): https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-516/System-Configuration/Authentication-Authorization-and-Accounting/TACACS/
• Cisco ISE Device Administration: https://www.cisco.com/c/en/us/td/docs/security/ise/3-4/admin_guide/b_ise_admin_3_4/b_ISE_admin_Device_Administration.html
• Cisco ISE Licensing Guide: https://www.cisco.com/c/en/us/products/collateral/security/identity-services-engine/ise-licensing-guide-og.html
• Cumulus Linux Support Policy: https://docs.nvidia.com/networking-ethernet-software/knowledge-base/Support/Support-Offerings/Cumulus-Linux-Release-Versioning-and-Support-Policy/

Author: Dima Kaputkin