BGP routing with Kubernetes CRDs

Overview

F5 BIG-IP Next for Kubernetes (BNK) lets you configure Border Gateway Protocol (BGP) routing using Kubernetes custom resource definitions (CRDs). Instead of managing network configuration manually, you can define routing intent as standard Kubernetes resources. The controller reads these resources, renders the configuration using a Go template, and delivers it to the Traffic Management Microkernel (TMM) routing container automatically.

This approach supports:

  • North–south BGP routing in the default or tenant-specific Virtual Routing and Forwarding (VRF) instances.

  • East–west traffic using Virtual Extensible LAN (VXLAN) tunnels with Ethernet VPN (EVPN) overlays.

  • BGP Unnumbered for simplified fabric peering without assigned IP addresses.

Use cases

You can use BGP routing with Kubernetes CRDs to address the following scenarios:

  • North–south BGP in the default VRF: Route traffic between your Kubernetes workloads and external networks using a single, shared routing domain.

  • North–south BGP with tenant VRF isolation: Provide each tenant a dedicated routing domain so traffic stays isolated at the network level.

  • East–west VXLAN/EVPN overlays: Connect workloads across nodes using VXLAN tunnels, with EVPN providing dynamic discovery of remote endpoints and route distribution.

  • EVPN-only overlays: Use EVPN as the control plane for VXLAN without additional north–south BGP neighbors. A global BGP configuration is still required.

  • BGP Unnumbered for fabric neighborship: Simplify peering in large data center fabrics by using IPv6 link-local addresses instead of assigned IP addresses.

Key concepts

Declarative networking

You declare your desired network state using Kubernetes CRDs. The CNE controller reads these resources, renders a configuration using a Go template, and delivers it to the routing container as a configuration payload. You don’t need to configure the routing software directly.

The workflow is event-driven. Every time you create or delete a routing CRD, the controller renders the full configuration and pushes it to the routing container. The routing container merges and applies the configuration automatically.

Important

You cannot update routing CRDs in place. To make changes, delete the existing resource and create a new one.

CRD reference

The following table describes each CRD and its purpose.

CRD

Scope

Purpose

GlobalRoutingConfig

Cluster (singleton)

Defines the global base BGP configuration, including the Autonomous System Number (ASN) range, neighbors, and template reference. Create this resource first and only once.

RoutingConfig

Tenant

Defines tenant-specific overlay intent, including neighbors and VXLAN/VRF attachment. The controller processes this resource only after GlobalRoutingConfig reaches a successful status.

RoutingTemplate

Cluster

Contains a Go template that the controller uses to render the final routing configuration. Both GlobalRoutingConfig and RoutingConfig reference a template.

Vrf

Tenant

Creates a VRF for tenant-specific routing domain isolation. Maps a route domain to the VRF name.

Vxlan

Tenant

Configures VXLAN tunnel encapsulation for tenant traffic, including the Virtual Network Identifier (VNI), VTEP IPs, MTU, and VRF association.

Secret

Namespace

Stores BGP neighbor authentication passwords. Referenced by the neighborSecrets field in routing CRDs.

How the controller processes CRDs

  1. You create a GlobalRoutingConfig resource with your base BGP settings and a reference to a RoutingTemplate.

  2. The CNE controller validates basic fields, including ASN range, neighbor definitions, IP formats, and the existence of referenced resources.

  3. The controller renders the Go template for each TMM pod, substituting values like the per-device ASN and VTEP IP.

  4. The controller pushes the rendered configuration to the TMM routing container through gRPC.

  5. The routing container merges and applies the configuration, then returns an acknowledgment.

  6. The controller updates the CRD status to show whether the configuration was delivered or rejected.

For tenant-specific routing, you follow the same pattern with a RoutingConfig resource. The controller renders and pushes tenant configurations only after the global configuration succeeds.

ASN assignment

The GlobalRoutingConfig resource defines how ASNs are assigned to TMM pods:

  • startASN (required): The base ASN. If endASN is not set, all TMM pods use this single ASN.

  • endASN (optional): When set, the controller assigns one ASN per TMM slot from the range defined by startASN through endASN.

VRF isolation

A VRF provides tenant-specific routing domain isolation. When you create a Vrf resource, the system generates a deterministic route domain ID. Other CRDs — such as F5SPKVlan, F5SPKStaticRoute, and Vxlan — reference the VRF by name through the vrf field.

This lets you:

  • Isolate tenant traffic at the routing level

  • Associate VLANs, static routes, and VXLAN tunnels with specific tenants

  • Prevent cross-tenant data leakage

VXLAN and EVPN

VXLAN provides tunnel encapsulation for tenant traffic. EVPN acts as the control plane, dynamically discovering remote VXLAN Tunnel Endpoints (VTEPs) and distributing reachability information across the fabric.

The system supports:

  • EVPN Route Type 5 (RT-5): Advertises IP prefix routes for subnet-level reachability. These routes enable Layer 3 connectivity when the remote VXLAN tunnel is not within the same subnet as the local VTEP.

  • L3 VNI: Layer 3 routing between tenants or between a tenant and external networks. Each tenant VRF maps 1:1 with an L3 VNI, and VNI-to-VRF mappings must be consistent across all VTEPs.

Tenant VXLAN/VRF attachment options

When you create a RoutingConfig for a tenant, you can attach VXLAN and VRF data in two ways:

Option

Method

Description

Option 1 (preferred)

vxlanRef

Reference an existing Vxlan CR that contains the VNI, VTEP IPs, and VRF configuration.

Option 2

Inline fields + vrfRef

Define VNI, VTEP subnet, and prefix inline in the RoutingConfig, and reference a separate Vrf CR.

BGP Unnumbered

BGP Unnumbered simplifies peering by using IPv6 link-local addresses instead of traditional IP addresses.

With BGP Unnumbered:

  1. The controller identifies VLANs without self IPs as BGP Unnumbered interfaces.

  2. Traffic Management Microkernel (TMM) sends periodic Router Advertisements (RAs) on these interfaces to enable neighbor discovery.

  3. Remote peers respond with their own RAs, and TMM forwards the link-local address information to tmrouted (the communication bridge between TMM and OcNOS) and then to OcNOS.

  4. OcNOS establishes the BGP session using IPv6 link-local addresses.

In the GlobalRoutingConfig, you use the interface field instead of ip to identify BGP Unnumbered neighbors. The fields are mutually exclusive.

Traffic flows

North–south (external): Traffic between tenant pods and external networks flows through TMM, which performs VRF lookup, VXLAN encapsulation or decapsulation, and forwards packets to or from the fabric.

East–west (inter-VTEP): Traffic between pods on different nodes flows through VXLAN tunnels. TMM on the source node encapsulates the packet with the correct VNI, the fabric forwards it based on EVPN routes, and TMM on the destination node decapsulates and delivers it.