VastOS Network Configuration Guide

The configure_network.py script is the primary tool for initializing and managing network interfaces on VAST nodes (CBox, DBox, and EBox). It handles Layer 2 and Layer 3 topologies, hardware detection, firmware configuration, and VMS (management service) setup. The script manages configuration for the following network roles:

• Management - Cluster management using VMS, SSH, API, etc. Also provides the outbound interface for DNS services, NTP, Active Directory, LDAP, etc. By default, holds the VMS VIP and the DNS VIP.

• External/Northband/Frontend - Application clients connect to the VAST cluster through this network using common protocols (NFS, S3, SMB, block, etc).

• Internal/Backend - Handles communication between CNodes and DNodes, within the same cluster and across clusters when using replication / global namespace.

• BMC/IPMI - Manage server hardware settings. Used to configure BIOS, etc.

Relevant paths on a VAST cluster:

• Script path: /usr/bin/configure_network.py

• Log location: /vast/log/configure_network/configure_network.log

• Config file location: /etc/vast-configure_network.py-params.ini (Stores the arguments used in the last successful run. Use the *--load-params-from-file* option to re-apply).

For a full list of supported hardware platforms, see the Appendix.

Configuration Modes

The script supports two primary network topologies.

Layer 2 (Standard/Legacy)

The default mode. It uses VLANs for internal traffic and assumes a flat Layer 2 network for the external interface.

• External Network: Configured on a specific physical interface or bond. May be shared with management functions.

• Internal Network: Uses bond0 (or physical interfaces) with a specific VLAN (default 69).

Layer 3 (BGP)

Enables BGP routing for leaf-spine topologies (--l3-network-mode). Uses FRR to manage BGP capabilities.

• External Network: IPs are assigned to physical interfaces. BGP advertises these routes.

• Internal Network: IPs are assigned to loopback or physical interfaces without VLAN tagging. BGP advertises these routes.

• Requirement: Must specify ASN (--l3-asn) and Router ID (--l3-router-id).

• Community Tags: Supports default or community based BGP tagging for DNode/VIPpool segregation.

Usage

The script requires root privileges. It can be run in several modes:

Full install

Perform a full configuration of network settings based upon the specified options.

sudo configure_network.py <NODE_ID> [OPTIONS]

**<NODE_ID>**: Integer representing the node ID (e.g., 1, 2). Used to calculate default internal IPs unless overridden. The ID must be unique across the cluster.

Dry run

Generates network configuration files without applying changes to the system. Configuration files are stored in the folder /tmp/configure_network.

sudo configure_network.py <NODE_ID> [OPTIONS] --dry-run

The --dry-run flag supports one of the following optional parameters:

• --dry-run=clean - Remove the /tmp/configure_network folder before starting

• --dry-run=keep - Keep the /tmp/configure_network folder, overwriting existing files

Apply previous settings

Apply network settings using the configuration stored in /etc/vast-configure_network.py-params.ini.

sudo configure_network.py --load-params-from-file

Identify Hardware

Print supported hardware identified by the script on the local node. No changes are made to the system.

sudo configure_network.py --identify-hw

Special flags

These flags change the default topology. Use the flags below when planning L3, edge, or platform-specific deployments.

--combo-ip

Use when: A single IP address must serve management, internal, and external roles on a single interface. Common on EBox, single-NIC, and L3 edge deployments where separate address pools are not available.

How it works: One address replaces the usual split between --ext-ip and the internal addresses from --template. In L3 mode, it also becomes the internal mgmt_ip. Requires --combo-ip-netmask and --combo-ip-gateway.

Typical scenarios:

• Supermicro Milan EBox (required)

• Single-port edge nodes with --l3-network-mode

• Lab or customer networks with one routable IP per node

Not intended for: Standard dual-NIC CBox/DBox deployments where management (--ext-ip) and data (--template) are on separate networks.

--mgmt-data-vip

Use when: VMS must be reachable via the data network in addition to (or instead of) the management network. Usually for L3 resiliency or sites where operators manage the cluster through the data-plane fabric.

How it works: A second floating VMS VIP on --mgmt-data-interface. Unlike --mgmt-vip, routing is not inherited from --ext-gateway; you must supply --mgmt-data-netmask, --mgmt-data-vip-gateway, and optionally --mgmt-data-vip-vlan.

Typical scenarios:

• L3 inband deployments (e.g., Supermicro Genoa single-NIC L3)

• The customer wants a backup VMS path if the OOB management network is down

Not intended for: Standard outband deployments. --mgmt-vip on the management network is sufficient for most clusters.

--no-bond

Use when: The node should not create bond0. Required for L3/BGP mode, which configures internal addresses directly on physical interfaces.

How it works: Skips bond0 creation and its virtual-function slaves. Internal IPs land on physical interface aliases (:a, :b, :m) instead of bond0.69:*. L3 mode sets this automatically if not already specified.

Typical scenarios:

• Any --l3-network-mode deployment

• BGP leaf-spine where routes are learned, not bridged

Not intended for: Standard L2 clusters. bond0 with VLAN 69 is the default internal topology.

--no-vlan

Use when: Internal traffic should not be VLAN-tagged (no bond0.69 or enp*s0f0.69 style interfaces).

How it works: Internal IPs are assigned directly to physical interfaces or to untagged bonds. Setting --data-vlan 0 automatically enables --no-vlan if not already set.

Typical scenarios:

• L3 deployments where VLAN 69 is not used on the backend network

• Customer fabric carries internal traffic without a dedicated data VLAN tag

Not intended for: Standard L2 clusters. Internal VLAN 69 is the default.

What’s New

The following capabilities were added in the VastOS 5.5 release:

• Dry-run mode --dry-run (clean or keep) generates configuration files without applying changes; cleans up prior dry-run output by default.

• Standalone hardware identification --identify-hw prints the detected platform and exits (cannot be combined with other flags).

• VRF support for L3 mode --l3-vrf-mode configures a VRF for the L3 data path (requires a non-zero --data-vlan).

• Granular BGP community tagging per-segment. Replaces the coarse default/community choice from 5.0-5.4. Configured via the following parameters:

  • --l3-community-tagging-internal

  • --l3-community-tagging-external

  • --l3-community-tagging-ext

  • --l3-community-tagging-interfaces

  • --l3-community-tagging-ext-mgmt-vip

  • --l3-community-tagging-ext-vip-ranges

• Custom routing tables

  • --l3-route-table-id assigns learned routes to a non-main routing table.

  • --l3-default-route-table-id puts the default route to management into its own table.

• Internal-interface QoS suite: the full QoS surface that previously existed only for external interfaces is now available for internal interfaces:

  • --qos-internal-pfc-enable

  • --qos-internal-pfc-traffic-class

  • --qos-internal-rdma-dscp2prio

  • --qos-internal-cnp-dscp2prio

• CNP DSCP mapping - map Congestion Notification Packets to a DSCP/traffic-class pair (previously RDMA-only).

  • --qos-external-cnp-dscp2prio

  • --qos-internal-cnp-dscp2prio

• VLAN-less internal network --no-vlan removes the internal VLAN tag from internal interfaces.

• VXLAN switch support --vxlan now adjusts ARP and vast_pfc settings for Cumulus/VXLAN switch fabrics.

• BF3 DPU configuration --bf3-dpu configures networking for BlueField-3 DPU-based deployments.

• FRR IPv6 link-local prefix override --frr-ipv6-prefix overrides the Vast prefix used by FRR for the global next hop.

• L2 port label in L2/L3 hybrid --l2-internal-port-in-l2l3 (a or b) labels, where the internal port stays L2 in a hybrid L2-on-L3 topology.

• Force-reduce internal IPs --force-reduce-internal-ips reduces the internal IP count to 2 regardless of cluster type (whereas the existing --reduce-internal-ips only applies to single-interface SuperMicro DPU clusters).

The following capabilities have been deprecated in the 5.5 release:

• --enable-pfc and --traffic-class remain in 5.5 but are now explicitly DEPRECATED. Use the --qos-external-pfc-* flags instead.

Argument Reference

General Network Settings

Management Interface

Management VIP (VMS and related services)

L3 / BGP Settings

IPMI & Back-to-Back (B2B)

QoS & Flow Control (New in v5.5)

Notes:

• Using the options below will erase all previous QoS settings.

• --enable-pfc and --traffic-class are DEPRECATED. Use the options below instead.

• --vxlan must be set when using QoS.

• QoS can be set on a live cluster. Example:

  • clush -g cnodes "sudo configure_network.py --qos-only --qos-external-pfc-enable --qos-external-rdma-dscp2prio 40,3 --qos-external-cnp-dscp2prio 48,6 --vxlan"

Auto-Ports Feature

The --auto-ports flag automatically detects the hardware platform type and maps interfaces accordingly. This is the recommended method for supported hardware.

Modes:

• eth: Both internal and external networks use Ethernet

• ib: Both internal and external networks use InfiniBand (IB)

• int_eth_ext_ib: Configure the internal network to use Ethernet and the external to use IB

• int_ib_ext_eth: Configure the internal network to use IB and the external to use Ethernet

Helper Flags:

• --auto-ports-ext-iface: Defines where management traffic flows.

  • outband: Dedicated management port (default).

  • inband: Shared with internal data traffic.

  • northband: Uses one of the external data interfaces.

Examples

Scenario 1: Standard CNode Setup (L2)

Setting up node 1 with a dedicated management port eno1 and IPMI configuration.

sudo configure_network.py 1 \
  --auto-ports eth \
  --ext-ip 10.0.0.101 \
  --ext-netmask 255.255.255.0 \
  --ext-gateway 10.0.0.1 \
  --mgmt-vip 10.0.0.200 \
  --ipmi-ip 192.168.10.101 \
  --ipmi-netmask 255.255.255.0 \
  --ipmi-gateway 192.168.10.1

Scenario 2: L3 / BGP Configuration

Setting up node 5 in a BGP leaf-spine architecture.

sudo configure_network.py 5 \
  --l3-network-mode \
  --l3-asn 65001 \
  --l3-router-id 192.168.100.5 \
  --ext-interface eno1 \
  --ext-ip 10.0.5.10 \
  --ext-netmask 255.255.255.0 \
  --mgmt-vip 10.0.100.1

Scenario 3: QoS Update (No Reboot/Network Reset)

Updating DSCP mappings on a running system.

sudo configure_network.py --qos-only \
  --qos-external-pfc-enable \
  --qos-external-rdma-dscp2prio 46,5

Scenario 4: "Combo" IP (Single Port Edge)

Using a single IP address for Management, Internal, and External traffic (common in Edge/EBox deployments).

sudo configure_network.py 1 \
  --l3-network-mode \
  --l3-asn 65000 \
  --l3-router-id 10.10.10.1 \
  --combo-ip 10.10.10.15 \
  --combo-ip-netmask 255.255.255.0 \
  --combo-ip-gateway 10.10.10.254

Troubleshooting

1. Reboot Required: If the script changes firmware settings (SR-IOV, Link Type), it will create a file /run/configure_network_requires_reboot. You must reboot the node and run the script again.

2. Log Analysis: Always check /vast/log/configure_network/configure_network.log for failures in specific steps (e.g., ipmitool failures or frr stabilization issues).

3. Dry Run: Use --only-generate-conf-files to verify the generated /etc/sysconfig/network-scripts/ifcfg-* files without applying changes.

Appendix: Supported Hardware

The configure_network.py script will automatically recognize the following hardware platforms and devices, applying hardware-specific changes as appropriate.

Legacy CBox / DBox

Ceres DBox

HPE

Dell

Supermicro

Cisco

Third-party EBox / GBox (OEM)