Converged EVPN/VXLAN - Cisco NX-OS Design

Converged design

In the converged data network, front-end (FE) and back-end (BE) attach to the same VAST spine-leaf switches. That covers EBox (CNode + DNode on one server) and disaggregated CNode/DNode servers. One network to cable and operate.

Cisco NX-OS notes for the converged VAST fabric.

VIP /32s and storage MACs stay out of OSPF. Clients don't run EVPN. EBox nodes don't run BGP.

ECMP

Each leaf has /30 uplinks to spines per the site design. OSPF + BFD install equal-cost routes to every remote VTEP; VXLAN encap hashes across the same paths. Clients peer eBGP with each spine and load-balance the VIP aggregate. Client routes do not change on VIP mobility - only the fabric VTEP next-hop updates via EVPN Type-2.

Dual-leaf failure in a single EBox pair is not an ECMP problem: both VTEPs remain unavailable until those leaves recover.

Front-end (FE)

FE is client-facing storage traffic. VAST publishes protocol VIPs from a shared VIP pool (e.g. <vip-prefix>/16). Each active CNode owns a set of /32 VIPs on VLAN 100.

L2: VLAN 100 is stretched to every leaf on VNI 100. EBox connects to a VIP-POOL trunk (native VLAN 100; allowed only 100). VIP frames are untagged on the wire.

L3: every leaf has the same anycast gateway on VLAN 100 SVI inside the tenant VRF. Same virtual IP, same Anycast MAC (2020.0000.00aa) on all leaves. Symmetric IRB lets the leaf route between the L2 VNI and the VRF.

interface Vlan100
  vrf member <tenant-vrf>
  mtu 9214
  ip address <virtual-gw>/<mask>
  fabric forwarding mode anycast-gateway
  arp gratuitous accept

When an EBox claims a VIP, it ARPs/GARPs on VLAN 100. The local leaf learns MAC+IP and originates EVPN Type-2. Other leaves and spines learn which VTEP owns each /32. Client routing stays stable (aggregate /16 toward clients); only the overlay next-hop changes on failover.

Optional: VLAN 110 for VMS mgmt-data VIP on the data network (HTTPS, Prometheus). Same Anycast SVI pattern, policy-restrict who can reach it.

Back-End (BE)

BE is intra-cluster traffic: CNode-to-DNode RDMA over RoCEv2. DNodes don't run BGP. No routing on the EBox for backend - it's L2 bridged into the overlay.

L2: VLAN 69 stretched to all leaves on VNI 30069. VLAN 10 is native on the Internal-RoCEv2 trunk (allowed 10, 69). VLAN 10 carries discovery/infra; VLAN 69 carries the actual RoCE data plane.

EBox-to-EBox on the same leaf pair stays local L2. Cross-leaf (CNode on one leaf, DNode on another) goes VXLAN on VNI 30069. That path needs PFC end-to-end - see RoCEv2 section.

BE does not use the client VRF or spine eBGP. It's purely an overlay L2 with lossless QoS.

Client connections

Clients (compute, Kubernetes nodes, backup servers) connect to spines only, not to VAST leaves.

Physical: routed port or VLAN 150 subinterface on spine, in tenant VRF:

interface Ethernet<x>.150
  mtu 9216
  encapsulation dot1q 150
  vrf member <tenant-vrf>
  ip address <gw>/<mask>

BGP: eBGP from spine VRF to client. EBox nodes do not run BGP.

router bgp <asn>
  vrf <tenant-vrf>
    address-family ipv4 unicast
      aggregate-address <vip-prefix> summary-only
      advertise l2vpn evpn
    neighbor <client-if>
      remote-as <client-asn>
      address-family ipv4 unicast
        route-map <export-aggregate-only> out

What the client learns: one aggregate VIP prefix. Not individual VIP /32s. Inside the fabric, EVPN carries hundreds of /32 Type-2 routes; the export route-map strips them.

ECMP: client peers with each spine, same aggregate from each.

Traffic path client -> VIP:

1. Client forwards to VIP /32 (longest match inside learned /16).

2. The packet arrives at the spine in the tenant VRF. Spine has the /32 from EVPN or forwards into overlay.

3. Spine NVE encap to remote VTEP (or local if VIP is on attached leaf path via L3VNI).

4. Leaf delivers on VLAN 100 trunk to EBox.

Clients never see EVPN, VNI, or VTEP addresses. They only need reachability to the VIP pool prefix and L3 adjacency to all spines.

EBox connectivity

Each EBox needs two logical attachments to its leaf pair (can be two physical ports or combined per hardware layout):

! VIP-POOL
interface Ethernet<x>
  description VIP-POOL
  switchport mode trunk
  switchport trunk native vlan 100
  switchport trunk allowed vlan 100
  mtu 9216

! Internal-RoCEv2
interface Ethernet<x>
  description Internal-RoCEv2
  switchport mode trunk
  switchport trunk native vlan 10
  switchport trunk allowed vlan 10,69
  priority-flow-control mode on
  service-policy type qos input QOS_CLASSIFICATION
  service-policy type queuing output QOS_EGRESS_PORT
  mtu 9216

No IP on leaf host ports. Bridging only into VXLAN.

Leaf pairing

EBox pairs share a leaf failure domain. Each pair connects to two leaves for path redundancy and VIP spread. One VTEP (loopback1) per leaf.

Each EBox typically has VIP-POOL and RoCE ports cabled to both leaves in its pair. VIP /32 Type-2s originate from the leaf where the EBox is active. Sibling EBoxes in the pair may host VIPs on the other leaf in normal operation.

Failure behavior

EBox reload: one VTEP withdraws its Type-2s. VIPs move to surviving leaves in the fabric. OSPF unchanged. Client routes unchanged (/16 still valid).

Dual-leaf reload: both VTEPs in a pair are gone. All Type-2s with those next-hops withdraw. No alternate VTEP in the pair - cluster impact until leaves recover.

Single-leaf reload: The surviving leaf in the pair continues to host VIPs for both EBoxes in that pair. Cross-pair VIPs unaffected.

What EBox does not do

• No BGP to spine or leaf

• No OSPF

• No EVPN

• No routing for VIPs - L2 on VLAN 100, kernel owns VIP addresses

The network learns EBox presence from ARP/GARP and EVPN Type-2, not from routing protocols on the host.

OSPF underlay

OSPF is only there so every switch can route to every VTEP (loopback1). Not vlan 100 subnets, not client prefixes.

In OSPF:

• loopback0 - router-id, iBGP EVPN source

• loopback1 - VTEP, NVE source-interface

• spine-leaf /30 links (ECMP paths per site design)

Out of OSPF: tenant VRF, VIP-pool SVI, client ports, mgmt0.

router ospf UNDERLAY
  bfd
  router-id <loopback0>

interface loopback1
  ip address <vtep>/32
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode

interface Ethernet<x>          ! spine uplink
  ip address <local>/30
  ip ospf network point-to-point
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode

BFD on the process. When a /30 drops, adjacency goes in ~300 ms instead of waiting for the dead interval.

Area 0 is enough for a small leaf/spine pod. Multiple ECMP paths leaf to remote VTEP is normal.

Check: show ip ospf neighbors (FULL on all expected uplinks), show ip route <remote-vtep> (multiple next-hops).

OSPF can't reach a VTEP = VXLAN to that leaf is dead, even if BGP EVPN still has routes in the RIB.

VXLAN / NVE

NVE is the tunnel interface. Frames on VLAN 100 / 69 / 10 get VXLAN-encapped with a VNI and sent to the remote VTEP from BGP EVPN.

feature nv overlay evpn
feature vn-segment-vlan-based

vlan 100
  vn-segment 100

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 100      mcast-group 239.1.1.0
  member vni 30010    mcast-group 239.1.1.0
  member vni 30069    mcast-group 239.1.1.0
  member vni 50000 associate-vrf

host-reachability protocol bgp - no flood-and-learn. BGP tells NVE which VTEP has the MAC/IP.

member vni binds VNI to the tunnel. L2 VNIs (100, 30010, 30069) stretch vlans. associate-vrf on the L3 VNI hooks the tenant VRF into the overlay.

Leaf: all L2 + L3 VNIs terminate here.

Spine: member vni <l3vni> associate-vrf only. No L2 VNIs on spine NVE. VIP-Pool VNI on spine = wrong.

Outer DIP = remote VTEP (OSPF route to loopback1). Inner = Ethernet + VNI.

EVPN routes

BGP AF l2vpn evpn. Route types that matter:

EBox ARPs for VIP; leaf learns MAC+IP and originates Type-2. Everyone else sees "VIP /32 behind VTEP x.x.x.x, VNI 100."

Node reboot: Type-2 withdraws. Other VTEPs hosting those VIPs publish new Type-2s. VIP moves without client route changes.

evpn
  vni 100 l2
    rd auto
    route-target import auto
    route-target export auto

rd auto / route-target auto - Cisco derives per switch. Fine for single fabric.

Symmetric IRB: VLAN 100 SVI in tenant VRF, anycast gateway. L2 VNI traffic can be routed via an SVI into a VRF and back. Same Anycast MAC on all leaves.

fabric forwarding anycast-gateway-mac 2020.0000.00aa

interface Vlan100
  vrf member <tenant-vrf>
  ip address <virtual-gw>/<mask>
  fabric forwarding mode anycast-gateway

iBGP EVPN (leaf/spine)

Same ASN everywhere. Leaves don't peer at each other. Spines are route reflectors.

Leaf peers to each spine loopback0:

router bgp <asn>
  router-id <loopback0>
  neighbor <spine-lo0>
    remote-as <asn>
    update-source loopback0
    address-family l2vpn evpn
      send-community extended
    ! repeat per spine

Spine: route-reflector-client toward every leaf, iBGP EVPN between spines:

router bgp <asn>
  neighbor <leaf-lo0>
    update-source loopback0
    address-family l2vpn evpn
      route-reflector-client

Spine reflects Type-2 from one leaf to another. No leaf-to-leaf BGP needed.

Tenant VRF BGP (separate from EVPN AF):

  vrf <tenant-vrf>
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map <tagged-svi-rm>
      aggregate-address <vip-prefix> summary-only

advertise l2vpn evpn pulls EVPN routes into the VRF. aggregate-address summary-only sends one prefix to eBGP clients.

iBGP EVPN needs OSPF reachability to loopback0. Broken underlay = BGP flap or unreachable update-source.

Check: show bgp l2vpn evpn summary, show bgp l2vpn evpn evi 100, show bgp l2vpn evpn mac-ip-addr.

PIM (BUM when NVE has mcast-group)

Skip PIM if NVE has no mcast-group. With mcast-group, broadcast/unknown-unicast/multicast for that VNI goes to the group (e.g., 239.1.1.0). PIM builds the replication tree on the underlay.

BUM still happens with EVPN: ARP before Type-2 is learned; GARP after failover; some control traffic even when all data is known unicast.

PIM sparse-mode on underlay /30s and loopbacks. RP on spines as anycast, same RP IP:

interface loopback254
  ip address <rp>/32
  ip pim sparse-mode

ip pim rp-address <rp> group-list 239.1.1.0/25
ip pim anycast-rp <rp> <spine-lo0>
! repeat per spine

Anycast-RP because overlay BUM on L2 VNIs uses multicast. No mcast-group on NVE = use ingress/head-end replication instead.

RPF checks use the OSPF table toward the sourcing VTEP.

eBGP spine to client

See Client connections above for topology. Spine VRF config detail:

ip prefix-list PL-VIP-EXPORT seq 10 permit <vip-prefix>
route-map <export-aggregate-only> permit 10
  match ip address prefix-list PL-VIP-EXPORT

router bgp <asn>
  vrf <tenant-vrf>
    address-family ipv4 unicast
      aggregate-address <vip-prefix> summary-only
      advertise l2vpn evpn
    neighbor <client-if>
      remote-as external
      address-family ipv4 unicast
        route-map <export-aggregate-only> out

Inside fabric: hundreds of /32 Type-2. Outside: one /16.

Traffic flow

Client to VIP (known unicast):

1. Client sends to VIP /32. eBGP aggregate on client; spine has /32 from EVPN or encapsulates.

2. Spine VRF routes into EVPN; next-hop = remote VTEP.

3. OSPF path to that VTEP loopback.

4. NVE encap VNI 100, deliver on VLAN 100 trunk to EBox.

ARP for unknown VIP:

1. Flood on VLAN 100, VXLAN BUM via mcast-group 239.1.1.0 (needs PIM).

2. EBox replies, leaf learns MAC, and originates Type-2.

3. Next packet is unicast VXLAN.

EBox-to-EBox storage (vlan 69): L2 VNI 30069. Same VXLAN, different VNI. PFC on every hop for RoCE (section below).

VLANs / VNIs

VIP trunk:

interface Ethernet<x>
  switchport trunk native vlan 100
  switchport trunk allowed vlan 100

RoCE trunk:

interface Ethernet<x>
  switchport trunk native vlan 10
  switchport trunk allowed vlan 10,69
  priority-flow-control mode on
  service-policy type qos input QOS_CLASSIFICATION

RoCEv2 (vlan 69)

VAST uses RoCEv2 for inter-EBox backend traffic (CNode to DNode RDMA). VLAN 69 is stretched L2 across all leaves on VNI 30069. VLAN 10 is the native VLAN on storage trunks (discovery / infra).

RoCE is not "best effort TCP." RDMA assumes lossless delivery on the priority that carries data. One dropped RoCE frame can stall or kill a connection. That means PFC on every hop: EBox port, leaf, spine uplink, remote leaf, remote EBox port.

Cross-leaf traffic is still RoCE inside VXLAN (VNI 30069). PFC has to work on the underlay uplinks too, not just on the host-facing trunk.

Marking

CNP must get through fast. If CNP is delayed or dropped, senders don't back off and you get persistent loss on the RoCE queue.

Classification (input on RoCE ports)

class-map type qos match-any CNP
  match dscp 48
class-map type qos match-any ROCEv2
  match dscp 26

policy-map type qos QOS_CLASSIFICATION
  class ROCEv2
    set qos-group 3
  class CNP
    set qos-group 7
  class class-default
    set qos-group 0

Egress queuing

policy-map type queuing QOS_EGRESS_PORT
  class type queuing c-out-8q-q3
    bandwidth remaining percent 50
    random-detect minimum-threshold 950 kbytes maximum-threshold 3000 kbytes \
      drop-probability 7 weight 0 ecn
  class type queuing c-out-8q-q7
    priority level 1
  class type queuing c-out-8q-q-default
    bandwidth remaining percent 50

ECN on Q3 lets the network signal congestion before buffers fill. CNP from the host does the actual rate reduction on the RNIC.

PFC (system-wide)

policy-map type network-qos qos_network
  class type network-qos c-8q-nq3
    mtu 4200
    pause pfc-cos 3

system qos
  service-policy type network-qos qos_network
  service-policy type queuing output QOS_EGRESS_PORT

COS 3 is paused under congestion, not dropped. mtu 4200 on the network-qos class is for the RoCE queue MTU handling on NX-OS.

Host RoCE trunk

interface Ethernet<x>
  description Internal-RoCEv2
  switchport mode trunk
  switchport trunk native vlan 10
  switchport trunk allowed vlan 10,69
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on
  mtu 9216
  service-policy type qos input QOS_CLASSIFICATION
  service-policy type queuing output QOS_EGRESS_PORT

Spine uplinks

Same PFC + QoS policies on leaf-to-spine /30 interfaces. MTU 9216 on fabric links. Without PFC on uplinks, same-leaf EBox pairs work, and cross-leaf RoCE fails under load.

Overlay

VLAN 69 maps to VNI 30069. EVPN Type-2 learns remote EBox MACs. Unicast RoCE between leaves goes VXLAN encap over ECMP underlay. BUM on VLAN 69 uses the same mcast-group as other L2 VNIs if configured.

Check

show interface <roce-port> priority-flow-control detail
show queuing interface <spine-uplink>
show policy-map interface <roce-port> input
show bgp l2vpn evpn evi 30069
show nve vni 30069

Look for PFC pause counters incrementing on congested links. If pauses only show on host ports but not on uplinks, cross-leaf RoCE will break.

References

• Cisco Nexus 9000 VXLAN BGP EVPN Design and Implementation Guide — NX-OS spine-leaf EVPN/VXLAN fundamentals