EVPN and VXLAN networking

Overview

F5 BIG-IP Next for Kubernetes (BNK) uses Virtual Extensible LAN (VXLAN) and Ethernet VPN (EVPN) to provide scalable, multi-tenant networking across a cloud fabric. VXLAN handles the data plane by encapsulating tenant traffic in tunnels between VXLAN Tunnel Endpoints (VTEPs). EVPN handles the control plane by using BGP to dynamically discover remote VTEPs and distribute reachability information — eliminating the need for static tunnel configuration or network flooding.

This design aligns with NVIDIA’s Cloud Partner reference architecture, where VXLAN represents customer and tenant traffic and BGP EVPN acts as the control plane for distributing segmentation and reachability information across the fabric.

This document explains:

  • What VXLAN and EVPN do and how they work together

  • How tenants are isolated using VRFs and VNIs

  • Which EVPN route types the system uses

  • How traffic flows through the system for north–south and east–west scenarios

What VXLAN and EVPN do

VXLAN (data plane)

VXLAN creates overlay tunnels that carry tenant traffic across the physical network fabric. Each tunnel uses a Virtual Network Identifier (VNI) to tag traffic for a specific tenant or network segment. Traffic Management Microkernel (TMM) performs VXLAN encapsulation when sending traffic and decapsulation when receiving it.

VXLAN solves a key problem in multi-tenant environments: it lets you create thousands of isolated network segments without relying on traditional VLANs, which are limited to 4,094 IDs.

EVPN (control plane)

EVPN uses BGP to distribute MAC addresses, IP addresses, and IP prefixes across VTEPs. When a new endpoint (such as a pod) appears on a VTEP, EVPN advertises that endpoint’s reachability to all other VTEPs through BGP. Remote VTEPs learn how to reach that endpoint without flooding the network.

EVPN provides:

  • Dynamic VTEP discovery: Remote VTEPs are learned automatically through BGP route exchange. You don’t need to configure static tunnel endpoints.

  • Optimized forwarding: Traffic is sent directly to the correct VTEP based on learned MAC/IP information, instead of being flooded across the fabric.

  • ARP/ND suppression: The fabric can answer Address Resolution Protocol (ARP) and Neighbor Discovery (ND) requests locally using information learned through EVPN, which reduces broadcast traffic.

How they work together

VXLAN and EVPN serve complementary roles:

Layer

Component

Role

Control plane

EVPN (over BGP)

Discovers remote VTEPs, distributes MAC/IP and IP prefix reachability, manages VNI-to-VRF mappings

Data plane

VXLAN

Encapsulates tenant traffic in tunnels between VTEPs, using VNIs for segmentation

EVPN tells each VTEP where to send traffic. VXLAN carries that traffic through the fabric.

Tenant and VNI model

Each tenant is isolated using a combination of a VRF and one or more VNIs.

VRF-to-VNI mapping

  • Each tenant VRF maps 1:1 with an L3 VNI. This L3 VNI handles inter-subnet routing for the tenant.

  • Each L3 VNI maps to exactly one VRF.

  • VNI-to-VRF mappings must be identical across all VTEPs. If VTEP A maps VNI 1000 to VRF “tenant-1,” then every other VTEP in the fabric must use the same mapping. BGP EVPN acts as the source of truth for maintaining this consistency.

L3 VNI

Type

Purpose

When to use

L3 VNI

Provides Layer 3 routing between subnets within a tenant VRF, or between a tenant and external networks.

Use when endpoints need to communicate across different subnets, or when you need north–south connectivity to external networks.

Tenant isolation

Tenant isolation is enforced at multiple levels:

  • VRF isolation: Each tenant’s routing table is separate. Routes from one tenant cannot leak into another tenant’s VRF.

  • VNI isolation: Each VNI carries traffic for a single tenant or segment. TMM forwards traffic only to VTEPs that share the same VNI.

  • Route target policies: EVPN uses route targets to control which VRFs import and export routes. A VRF only receives routes that match its configured route targets.

EVPN route type

The system uses EVPN route type to distribute reachability information across the fabric.

Route Type 5 (RT-5): IP prefix advertisement

RT-5 routes advertise subnet-level reachability. They carry IP prefixes (such as pod CIDRs or service CIDRs) associated with an L3 VNI.

What RT-5 carries:

  • IP prefix (for example, a virtual server address or snatpool IP)

  • Associated L3 VNI

  • IP address of the originating VTEP

When RT-5 is used:

  • Inter-subnet routing within a tenant VRF

  • North–south traffic between tenant workloads and external networks

  • Advertising default or summarized routes into the fabric

Example:

A tenant pod CIDR (such as 192.168.10.0/24) is advertised as an RT-5 route in the tenant’s L3 VNI. The fabric uses this route to direct traffic destined for that subnet to the correct VTEP, which then routes it to the correct pod through the tenant VRF.

When the route type applies

Scenario

Route type

Why

Remote VXLAN tunnel is not within the same subnet as the local VTEP

RT-5 (L3 VNI)

Layer 3 routing based on IP prefix reachability

Subnet or CIDR advertised into the fabric

RT-5

Prefix-level routing for north–south or inter-subnet traffic

How traffic flows

North–south traffic (fabric-facing)

North–south traffic flows between tenant workloads and external networks through the fabric.

Outbound (pod to external network):

  1. A pod on Node A sends traffic to an external destination.

  2. TMM on Node A performs a VRF lookup to identify the tenant routing domain.

  3. TMM encapsulates the traffic in a VXLAN header with the tenant’s L3 VNI.

  4. The fabric forwards the encapsulated packet to the border gateway or leaf switch based on EVPN RT-5 routes.

  5. The border gateway decapsulates the VXLAN header and routes the traffic to the external network.

Inbound (external network to pod):

  1. Traffic from an external network reaches the border gateway.

  2. The border gateway looks up the destination in its EVPN routing table and identifies the correct VTEP and VNI.

  3. The border gateway encapsulates the traffic in a VXLAN header and sends it to the correct VTEP.

  4. TMM on the destination node decapsulates the VXLAN header, performs a VRF and forwarding lookup, and delivers the traffic to the correct pod.

East–west traffic (inter-VTEP)

East–west traffic flows between workloads on different nodes through VXLAN tunnels.

Different subnets (L3 VNI):

  1. A pod on Node A sends traffic to a pod on Node B in a different subnet but within the same tenant VRF.

  2. TMM on Node A performs an IP routing lookup in the tenant VRF. The route was learned through an EVPN RT-5 route.

  3. TMM encapsulates the traffic in a VXLAN header with the L3 VNI and sends it to VTEP B.

  4. TMM on Node B decapsulates the VXLAN header, performs a local routing lookup in the tenant VRF, and delivers the traffic to the destination pod.

EVPN route lifecycle

The following sequence shows how EVPN routes are advertised, learned, and withdrawn:

  1. Advertisement: When a pod is created or TMM learns a new MAC/IP locally, OcNOS advertises an RT-5 route through BGP.

  2. Learning: Remote VTEPs receive the EVPN route through BGP. OcNOS on each remote VTEP sends the learned information (FDB entries, ARP entries, routes) to TMM through tmrouted. TMM updates its forwarding tables with the remote endpoint data. This includes the VTEP entry on the VXLAN tunnel, the ARL entry for the remote MAC address, and ARP and route entries for the remote destination.

  3. Withdrawal: When a pod is removed or a route is no longer valid, OcNOS withdraws the EVPN route. Remote VTEPs remove the corresponding FDB, ARP, and route entries from TMM.

When a VNI interface is received in a specific route domain, OcNOS updates the router MAC on the VRF to the MAC address of the VNI. This is required for traffic to flow correctly between the router and the server leaf connected to TMM.

How EVPN integrates with CRDs

You configure EVPN and VXLAN networking through Kubernetes CRDs. The CRDs let you define your intent, and the CNE controller translates that intent into the configuration that OcNOS and TMM need.

CRD

EVPN/VXLAN role

GlobalRoutingConfig

Defines global BGP neighbors (numbered or unnumbered) and IPv4/IPv6 unicast address families

RoutingConfig

Defines tenant-specific BGP neighbors and associates the tenant with a VXLAN/VRF through vxlanRef or inline fields.

RoutingTemplate

Contains the Go template that renders the EVPN and BGP configuration for OcNOS. You can include EVPN address family blocks, route targets, and VNI mappings in the template.

Vrf

Creates the tenant VRF that isolates routing for the tenant. Maps 1:1 with an L3 VNI.

Vxlan

Configures the local VXLAN tunnel endpoint, including the VNI, VTEP IP, MTU, port, and VRF association.

The workflow is:

  1. You create a Vrf and Vxlan for the tenant.

  2. You create a RoutingConfig that references the VXLAN CR and defines BGP neighbors.

  3. The CNE controller renders the RoutingTemplate, which produces the OcNOS configuration including EVPN, VRF, and VXLAN settings.

  4. The controller pushes the configuration to the routing container through gRPC.

  5. OcNOS establishes BGP EVPN sessions with the fabric and begins advertising and learning routes.

  6. TMM receives learned routes from OcNOS through tmrouted and updates its forwarding tables.

For step-by-step instructions, see the related how-to guides.