Documentation Index

Fetch the complete documentation index at: https://kb.vastdata.com/llms.txt

Use this file to discover all available pages before exploring further.

Converged Cisco NX-OS - iBGP EVPN VXLAN

New
  • Published on Jul 8, 2026
  • 10 minute(s) read
Prev Next

NX-OS reference configurations for the converged VAST EVPN/VXLAN fabric.

Replace placeholders (<asn>, <tenant-vrf>, <vip-prefix>, <vtep>, <spine-lo0>, Ethernet<x>, etc.) per site.

Platform features

Enable on all switches before overlay config:

feature nxapi
nv overlay evpn
feature ospf
feature bgp
feature pim
feature interface-vlan
feature vn-segment-vlan-based
feature bfd
feature nv overlay

MTU 9216 on fabric and host-facing ports. OSPF process UNDERLAY with BFD on spine-leaf /30 links.

VLAN / VNI mapping (leaves)

vlan 10
  name VAST-Storage_VLAN_10
  vn-segment 30010
vlan 69
  name VAST-Storage-Network
  vn-segment 30069
vlan 100
  name vip-pool
  vn-segment 100
vlan 2000
  vn-segment 50000

VLAN

VNI

Purpose

10

30010

Native on storage trunks (discovery/infra)

69

30069

RoCEv2 inter-EBox backend

100

100

VIP pool + Anycast SVI

2000

50000

L3 VRF (associate-vrf on NVE) 

evpn
  vni 100 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 30010 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 30069 l2
    rd auto
    route-target import auto
    route-target export auto

Spines carry VLAN 2000 / VNI 50000 only (L3 VNI). L2 VNIs terminate on leaves.

OSPF underlay

OSPF carries loopback and /30 reachability only - not VIP subnets or client prefixes.

router ospf UNDERLAY
  bfd
  router-id <loopback0>

interface loopback0
  ip address <router-id>/32
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode

interface loopback1
  description VTEP
  ip address <vtep>/32
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode

interface Ethernet<x>          ! spine uplink
  mtu 9216
  ip address <local>/30
  ip ospf network point-to-point
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode
  priority-flow-control mode on
  service-policy type qos input QOS_CLASSIFICATION
  service-policy type queuing output QOS_EGRESS_PORT

Out of OSPF: tenant VRF, VIP-pool SVI, client ports, mgmt0.

NVE / VXLAN (leaf)

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 100
    mcast-group <bum-group>
  member vni 30010
    mcast-group <bum-group>
  member vni 30069
    mcast-group <bum-group>
  member vni 50000 associate-vrf

host-reachability protocol bgp - BGP EVPN tells NVE which VTEP owns each MAC/IP.

Spine (route reflector)

Spines are EVPN route reflectors. No host ports. L3 VNI only on NVE.

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 50000 associate-vrf

router bgp <asn>
  neighbor <leaf-lo0>
    remote-as <asn>
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
      route-reflector-client
    ! repeat per leaf

Inter-spine iBGP EVPN between spine loopback0s. Spines do not run OSPF to each other - they learn leaf loopbacks via leaf adjacencies.

Tenant VRF + Anycast gateway (leaves)

fabric forwarding anycast-gateway-mac 2020.0000.00aa

vrf context <tenant-vrf>
  vni 50000
  rd auto
  address-family ipv4 unicast
    route-target both auto evpn

interface Vlan100
  vrf member <tenant-vrf>
  mtu 9216
  ip address <virtual-gw>/<mask> tag <redist-tag>
  fabric forwarding mode anycast-gateway
  arp gratuitous accept

route-map <tagged-svi-rm> permit 10
  match tag <redist-tag>

EBox ARPs on VLAN 100. Leaf learns MAC+IP and originates EVPN Type-2. Same Anycast MAC on all leaves.

iBGP EVPN (leaf)

router bgp <asn>
  router-id <loopback0>
  neighbor <spine-lo0>
    remote-as <asn>
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
    ! repeat per spine

  vrf <tenant-vrf>
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map <tagged-svi-rm>
      aggregate-address <vip-prefix> summary-only
      maximum-paths <n>

eBGP spine to client

interface Ethernet<x>.150
  mtu 9216
  encapsulation dot1q 150
  vrf member <tenant-vrf>
  ip address <gw>/<mask>

ip prefix-list PL-VIP-EXPORT seq 10 permit <vip-prefix>
route-map <export-aggregate-only> permit 10
  match ip address prefix-list PL-VIP-EXPORT

router bgp <asn>
  vrf <tenant-vrf>
    address-family ipv4 unicast
      aggregate-address <vip-prefix> summary-only
      advertise l2vpn evpn
      maximum-paths <n>
    neighbor <client-if>
      remote-as external
      address-family ipv4 unicast
        route-map <export-aggregate-only> out

Clients receive the aggregate VIP prefix. Individual /32 VIPs stay in EVPN inside the fabric.

PIM anycast-RP (when NVE uses mcast-group)

Required when L2 VNIs use mcast-group for BUM. Skip if using ingress replication.

interface loopback254
  ip address <rp>/32
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode

ip pim rp-address <rp> group-list <bum-range>
ip pim anycast-rp <rp> <spine-lo0>
! repeat per spine

PIM sparse-mode on underlay /30s and loopbacks.

RoCE / QoS / PFC

Apply to storage trunks and spine uplinks.

Classification:

class-map type qos match-any ROCEv2
  match dscp 26
class-map type qos match-any CNP
  match dscp 48

policy-map type qos QOS_CLASSIFICATION
  class ROCEv2
    set qos-group 3
  class CNP
    set qos-group 7
  class class-default
    set qos-group 0

Egress queuing:

policy-map type queuing QOS_EGRESS_PORT
  class type queuing c-out-8q-q3
    bandwidth remaining percent 50
    random-detect minimum-threshold 950 kbytes maximum-threshold 3000 kbytes \
      drop-probability 7 weight 0 ecn
  class type queuing c-out-8q-q7
    priority level 1
  class type queuing c-out-8q-q-default
    bandwidth remaining percent 50

PFC (system-wide):

policy-map type network-qos qos_network
  class type network-qos c-8q-nq3
    mtu 4200
    pause pfc-cos 3

system qos
  service-policy type network-qos qos_network
  service-policy type queuing output QOS_EGRESS_PORT

Traffic

DSCP

QoS group

Egress

RoCEv2

26

3

q3, ECN WRED

CNP

48

7

strict priority

Default

—

0

best effort

Host-facing ports (leaf)

VIP-POOL (EBox client VIPs):

interface Ethernet<x>
  description VIP-POOL
  switchport mode trunk
  switchport trunk native vlan 100
  switchport trunk allowed vlan 100
  mtu 9216

Internal-RoCEv2 (storage backend):

interface Ethernet<x>
  description Internal-RoCEv2
  switchport mode trunk
  switchport trunk native vlan 10
  switchport trunk allowed vlan 10,69
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on
  service-policy type qos input QOS_CLASSIFICATION
  service-policy type queuing output QOS_EGRESS_PORT
  mtu 9216

No IP on host ports. Bridging only into VXLAN.

Verification

Check

Command

Expected result

OSPF adjacencies 

show ip ospf neighbors

All fabric neighbors FULL; one adjacency per configured uplink

Remote VTEP ECMP 

show ip route <remote-vtep>

/32 to remote VTEP with multiple best paths (ECMP) via underlay next-hops

BFD 

show bfd neighbors

All BFD sessions Up on OSPF-enabled fabric links

EVPN BGP 

show bgp l2vpn evpn summary

All L2VPN EVPN neighbors Established; AF active

VIP VNI 

show bgp l2vpn evpn vni-id 100

VNI 100 routes present; Type-2/3 from local and remote VTEPs

RoCE VNI 

show bgp l2vpn evpn vni-id 30069

VNI 30069 routes present; Type-2 MAC/IP for RoCE hosts

NVE status 

show nve vni

Site VNIs Up on leaves (100, 30010, 30069, 50000); source Loopback1

Client eBGP 

show bgp vrf <tenant-vrf> ipv4 unicast summary

External peer(s) Established; <vip-prefix> in RIB per design

PFC 

show interface <port> priority-flow-control detail

PFC on; pause enabled for configured priority (e.g. CoS 3)

QoS 

show policy-map interface <port> input

Input policy QOS_CLASSIFICATION attached; RoCE/CNP class counters increment under load

Templates

Spine

hostname <hostname>

feature nxapi
nv overlay evpn
feature ospf
feature bgp
feature pim
feature interface-vlan
feature vn-segment-vlan-based
feature bfd
feature nv overlay

! ----- QoS / PFC (fabric uplinks) -----
class-map type qos match-any ROCEv2
  match dscp 26
class-map type qos match-any CNP
  match dscp 48
policy-map type qos QOS_CLASSIFICATION
  class ROCEv2
    set qos-group 3
  class CNP
    set qos-group 7
  class class-default
    set qos-group 0
policy-map type queuing QOS_EGRESS_PORT
  class type queuing c-out-8q-q3
    bandwidth remaining percent 50
    random-detect minimum-threshold 950 kbytes maximum-threshold 3000 kbytes \
      drop-probability 7 weight 0 ecn
  class type queuing c-out-8q-q7
    priority level 1
  class type queuing c-out-8q-q-default
    bandwidth remaining percent 50
policy-map type network-qos qos_network
  class type network-qos c-8q-nq3
    mtu 4200
    pause pfc-cos 3
system qos
  service-policy type network-qos qos_network
  service-policy type queuing output QOS_EGRESS_PORT

! ----- VLAN / VNI (L3 only) -----
vlan 2000
  vn-segment 50000

! ----- VRF -----
vrf context <tenant-vrf>
  vni 50000
  rd auto
  address-family ipv4 unicast
    route-target both auto evpn

! ----- Loopbacks -----
interface loopback0
  description Routing loopback
  ip address <loopback0>/32
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode

interface loopback1
  description VTEP
  ip address <vtep>/32
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode

interface loopback254
  description PIM anycast-RP
  ip address <rp>/32
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode

! ----- OSPF underlay -----
router ospf UNDERLAY
  bfd
  router-id <loopback0>

! ----- PIM anycast-RP -----
ip pim rp-address <rp> group-list <bum-range>
ip pim anycast-rp <rp> <spine-lo0>
! repeat per spine

! ----- Leaf uplinks (repeat per leaf link) -----
interface Ethernet<leaf-uplink>
  description leaf-uplink
  mtu 9216
  ip address <local>/30
  ip ospf network point-to-point
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode
  priority-flow-control mode on
  service-policy type qos input QOS_CLASSIFICATION
  service-policy type queuing output QOS_EGRESS_PORT
  no shutdown

! ----- NVE (L3 VNI only) -----
interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 50000 associate-vrf

! ----- L3 VNI SVI -----
interface Vlan2000
  mtu 9216
  vrf member <tenant-vrf>
  no ip redirects
  ip forward
  no shutdown

! ----- Client eBGP (repeat per client-facing port) -----
interface Ethernet<client-port>.150
  mtu 9216
  encapsulation dot1q 150
  vrf member <tenant-vrf>
  ip address <gw>/<mask>
  no shutdown

ip prefix-list PL-VIP-EXPORT seq 10 permit <vip-prefix>
route-map <export-aggregate-only> permit 10
  match ip address prefix-list PL-VIP-EXPORT

! ----- BGP -----
router bgp <asn>
  router-id <loopback0>
  cluster-id <asn>
  neighbor <leaf-lo0>
    bfd
    remote-as <asn>
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
      route-reflector-client
    ! repeat per leaf
  neighbor <spine-lo0>
    remote-as <asn>
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
    ! inter-spine iBGP EVPN
  vrf <tenant-vrf>
    address-family ipv4 unicast
      advertise l2vpn evpn
      aggregate-address <vip-prefix> summary-only
      maximum-paths <n>
      maximum-paths ibgp <n>
    neighbor <client-if>
      remote-as external
      address-family ipv4 unicast
        route-map <export-aggregate-only> out
      ! repeat per client interface

Leaf

hostname <hostname>

feature nxapi
nv overlay evpn
feature ospf
feature bgp
feature pim
feature interface-vlan
feature vn-segment-vlan-based
feature bfd
feature nv overlay

fabric forwarding anycast-gateway-mac 2020.0000.00aa

! ----- QoS / PFC -----
class-map type qos match-any ROCEv2
  match dscp 26
class-map type qos match-any CNP
  match dscp 48
policy-map type qos QOS_CLASSIFICATION
  class ROCEv2
    set qos-group 3
  class CNP
    set qos-group 7
  class class-default
    set qos-group 0
policy-map type queuing QOS_EGRESS_PORT
  class type queuing c-out-8q-q3
    bandwidth remaining percent 50
    random-detect minimum-threshold 950 kbytes maximum-threshold 3000 kbytes \
      drop-probability 7 weight 0 ecn
  class type queuing c-out-8q-q7
    priority level 1
  class type queuing c-out-8q-q-default
    bandwidth remaining percent 50
policy-map type network-qos qos_network
  class type network-qos c-8q-nq3
    mtu 4200
    pause pfc-cos 3
system qos
  service-policy type network-qos qos_network
  service-policy type queuing output QOS_EGRESS_PORT

! ----- VLAN / VNI -----
vlan 10
  name VAST-Storage_VLAN_10
  vn-segment 30010
vlan 69
  name VAST-Storage-Network
  vn-segment 30069
vlan 100
  name vip-pool
  vn-segment 100
vlan 2000
  vn-segment 50000

evpn
  vni 100 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 30010 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 30069 l2
    rd auto
    route-target import auto
    route-target export auto

! ----- VRF -----
vrf context <tenant-vrf>
  vni 50000
  rd auto
  address-family ipv4 unicast
    route-target both auto evpn

! ----- Loopbacks -----
interface loopback0
  description Routing loopback
  ip address <loopback0>/32
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode

interface loopback1
  description VTEP
  ip address <vtep>/32
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode

! ----- OSPF underlay -----
router ospf UNDERLAY
  bfd
  router-id <loopback0>

! ----- Spine uplinks (repeat per link) -----
interface Ethernet<spine-uplink>
  description spine-uplink
  mtu 9216
  ip address <local>/30
  ip ospf network point-to-point
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode
  priority-flow-control mode on
  service-policy type qos input QOS_CLASSIFICATION
  service-policy type queuing output QOS_EGRESS_PORT
  no shutdown

! ----- NVE -----
interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 100
    mcast-group <bum-group>
  member vni 30010
    mcast-group <bum-group>
  member vni 30069
    mcast-group <bum-group>
  member vni 50000 associate-vrf

! ----- L3 VNI SVI -----
interface Vlan2000
  mtu 9216
  vrf member <tenant-vrf>
  no ip redirects
  ip forward
  no shutdown

! ----- VIP anycast gateway -----
interface Vlan100
  mtu 9216
  vrf member <tenant-vrf>
  ip address <virtual-gw>/<mask> tag <redist-tag>
  fabric forwarding mode anycast-gateway
  arp gratuitous accept
  no shutdown

route-map <tagged-svi-rm> permit 10
  match tag <redist-tag>

! ----- Host: VIP-POOL (repeat per EBox port) -----
interface Ethernet<vip-pool-port>
  description VIP-POOL
  switchport mode trunk
  switchport trunk native vlan 100
  switchport trunk allowed vlan 100
  spanning-tree port type edge trunk
  mtu 9216
  no shutdown

! ----- Host: Internal-RoCEv2 (repeat per EBox port) -----
interface Ethernet<roce-port>
  description Internal-RoCEv2
  switchport mode trunk
  switchport trunk native vlan 10
  switchport trunk allowed vlan 10,69
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on
  spanning-tree port type edge trunk
  mtu 9216
  service-policy type qos input QOS_CLASSIFICATION
  service-policy type queuing output QOS_EGRESS_PORT
  no shutdown

! ----- BGP -----
router bgp <asn>
  router-id <loopback0>
  neighbor <spine-lo0>
    bfd
    remote-as <asn>
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
    ! repeat per spine
  vrf <tenant-vrf>
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map <tagged-svi-rm>
      aggregate-address <vip-prefix> summary-only
      maximum-paths <n>
      maximum-paths ibgp <n>

Related articles