How BGP routing works

Overview

F5 BIG-IP Next for Kubernetes (BNK) uses a split architecture for BGP routing. OcNOS owns the control plane and manages all routing protocols, BGP sessions, and EVPN route exchange. TMM owns the data plane and handles packet forwarding, VXLAN encapsulation and decapsulation, and VRF isolation. A Kubernetes-native controller sits between you and these components, translating your CRD-based intent into configuration that the routing container can apply.

This document explains how these components work together to deliver BGP routing configuration from a Kubernetes CRD to a running network.

Architecture components

The BGP routing architecture has four main components:

Component	Role
CNE controller	Watches Kubernetes CRDs, validates input, renders Go templates, and pushes configuration to the routing container through gRPC
TMM	Owns the data plane. Handles packet forwarding, VXLAN encapsulation and decapsulation, VRF interface creation, and kernel networking
OcNOS	Owns the control plane. Runs BGP, EVPN, and BFD. Establishes BGP sessions, exchanges routes, and signals reachability to TMM through tmrouted
tmrouted	Acts as the communication bridge between TMM and OcNOS. Translates protobuf messages from TMM into Netlink messages that OcNOS understands, and forwards OcNOS route updates back to TMM

Below diagram shows the CRD hierarchy and configuration delivery workflow for BGP routing.

Control plane and data plane separation

OcNOS (control plane)

OcNOS (the routing operating system running inside the f5-tmm-routing container) handles all control plane functions. It is responsible for:

• Routing protocols: BGP session establishment, route advertisement, and route learning

• EVPN: Distributes IP prefix routes (Route Type 5) across the fabric

• BGP Unnumbered: Establishes BGP sessions using IPv6 link-local addresses, with neighbor discovery support provided by TMM and tmrouted

• BFD: The oamd daemon configures and manages Bidirectional Forwarding Detection (BFD) sessions, while the TMM BFD Proxy handles BFD packets

OcNOS uses the Linux network stack for BGP peer connections. Interfaces, IP addresses, and routes must be present in the network namespace used by the BGP daemon. Otherwise, OcNOS may select an incorrect local IP address, which causes failures when establishing peer connections.

Traffic Management Microkernel (TMM)

TMM handles all the data plane forwarding logic. It is responsible for:

• VRF interface creation: TMM creates VRF interfaces in the Linux kernel and binds tap interfaces to the correct VRF

• VXLAN encapsulation and decapsulation: TMM wraps tenant traffic in VXLAN headers for north–south and east–west flows, and unwraps incoming VXLAN traffic

• L2/L3 forwarding: TMM performs MAC and IP lookups, updates forwarding tables with local and remote endpoint information, and routes traffic to the correct destination

• Static route management: TMM adds and removes static routes in the correct kernel routing table based on VRF configuration

Configuration delivery workflow

The following sequence describes how your CRD intent becomes a running BGP configuration.

Step 1: You create CRDs

You declare your routing intent by creating Kubernetes custom resources:

• GlobalRoutingConfig: Defines the global base BGP configuration, including the Autonomous System Number (ASN) range, neighbors, and a reference to a RoutingTemplate.

• RoutingConfig: Defines tenant-specific overlay intent, including neighbors and VXLAN/VRF attachment.

• RoutingTemplate: Contains a Go template that the controller uses to render the final routing configuration.

Step 2: The CNE controller validates and renders

The CNE controller watches for changes to GlobalRoutingConfig, RoutingConfig, RoutingTemplate, and all referenced dependency CRDs (Vxlan, Vrf) and Secrets.

When it detects a create or delete event, the controller:

1. Validates basic fields:

   • startASN and endASN consistency with the number of TMM pods

   • Neighbor definitions (each neighbor must have either ip or interface, not both)

   • IP, subnet, and prefix formats

   • Existence of all referenced dependency CRDs and Secrets

2. Renders the Go template. The controller substitutes values from the CRD into the referenced RoutingTemplate. It loops over each TMM pod in the inventory, generating a per-device configuration with the correct ASN and VTEP IP for that pod.

3. Produces a ConfigBlob. The rendered output is a CLI configuration payload ready for the routing container.

Note

The controller performs only basic validation. It does not validate the rendered CLI against OcNOS syntax. The routing container performs deep validation when it applies the configuration.

Step 3: The controller pushes configuration through gRPC

The controller sends each ConfigBlob to the TMM routing container through a gRPC channel secured with mutual TLS (mTLS).

The gRPC communication works as follows:

• TLS certificates are mounted into the f5-tmm-routing container.

• The CNE controller (f5ingress) targets the f5-tmm pod by its pod IP.

• Traffic reaches TMM’s eth0 interface first. A forwarding virtual server then routes it to the grpcconf_watcher process inside the routing container.

• The gRPC payload is not decrypted until it reaches the f5-tmm-routing container.

Step 4: The routing container applies the configuration

The grpcconf_watcher process inside the routing container receives the gRPC message containing the full configuration payload. It performs post-processing and applies the configuration through the OcNOS CLI (cmlsh).

The routing container:

1. Persists per-config files as needed.

2. Merges and applies the configuration.

3. Returns an acknowledgment (success or failure) to the controller.

Step 5: The controller updates CRD status

The controller writes the delivery outcome to the GlobalRoutingConfig or RoutingConfig status. The status reflects whether the payload was delivered to or rejected by the routing container.

Important

A successful delivery status means the routing container received and processed the payload. It does not guarantee that the routing container applied every CLI command without error. The rendered CLI is not stored in the CRD status or spec.

Template rendering

The RoutingTemplate CRD contains a Go template that the controller evaluates for each TMM pod. The template has access to the following variable contexts:

Variable	Description
.global.spec	The spec from the GlobalRoutingConfig resource (ASN range, neighbors, IP subnet, secrets)
.tenant.spec	The spec from the RoutingConfig resource (tenant neighbors, VXLAN/VRF attachment)
.device.localAsn	The ASN assigned to the specific TMM pod, derived from the startASN/endASN range
.vxlan	VXLAN configuration from the referenced Vxlan CR (Option 1) or inline fields (Option 2)
.vrf	VRF configuration from the referenced Vrf CR

The controller renders the global slice first and appends the tenant overlay. Each reconcile produces a full ConfigBlob — the controller never computes diffs or sends partial updates. This means every create or delete event results in a complete template render and push.

If a referenced Secret or RoutingTemplate changes, the controller triggers a full re-render and push for all affected CRDs.

Example: Basic RoutingTemplate

apiVersion: k8s.f5net.com/v1alpha1
kind: RoutingTemplate
metadata:
  name: ocnos-routing-template
spec:
  body: |
    router bgp {{ .global.spec.bgp.startASN }}
    bgp log-neighbor-changes
    {{- if .global.spec.bgp.neighbors }}
    {{- range $nbr := .global.spec.bgp.neighbors }}
    neighbor {{ or $nbr.ip $nbr.interface }} remote-as {{ $nbr.remoteAs }}
    {{- end }}
    {{- end }}
    exit

The template is flexible by design. You can add static CLI lines (such as BFD configuration) directly into the template body without changing the CRD schema.

ASN assignment

The GlobalRoutingConfig defines how ASNs are assigned across TMM pods:

• Single ASN: Set startASN only. All TMM pods use the same ASN.

• ASN range: Set both startASN and endASN. The controller assigns one ASN per TMM slot from the range.

The controller validates that the ASN range is large enough for the number of TMM pods in the cluster.

Communication between TMM and OcNOS

TMM and OcNOS communicate through tmrouted (the communication bridge between TMM and OcNOS), which acts as a message translator between the two systems.

TMM to OcNOS (configuration and state)

TMM sends the following information to OcNOS through tmrouted:

Data	Transport	Purpose
VRF (route domain) configuration	Protobuf → tmrouted → Netlink (TMOS_NEWRD / TMOS_DELRD)	Creates or deletes routing domains in OcNOS
VLAN and interface binding	Protobuf → tmrouted → Netlink (RTM_NEWLINK with IFLA_MASTER)	Associates interfaces with VRFs in OcNOS
Virtual addresses and self IPs	Protobuf → tmrouted → Netlink	Provides addresses for BGP advertisement
VXLAN configuration	Protobuf → tmrouted (handled as VLAN objects)	Configures VXLAN tunnel endpoints
Router Advertisement data (BGP Unnumbered)	Protobuf → tmrouted → NDD server socket	Forwards neighbor discovery information to OcNOS for unnumbered BGP sessions

OcNOS to TMM (learned routes and reachability)

OcNOS sends the following information back to TMM through tmrouted:

Data	Transport	Purpose
FDB entries (EVPN RT-5)	Netlink (TMOS_ADD_FDB / TMOS_DEL_FDB) → tmrouted → Protobuf	Provides remote MAC and VTEP IP for VXLAN forwarding
ARP entries	Netlink (RTM_NEWNEIGH / RTM_DELNEIGH) → tmrouted → Protobuf	Provides remote IP-to-MAC mappings
Dynamic routes	Netlink (RTM_NEWROUTE) → tmrouted → Protobuf	Provides L3 VNI routes with destination prefix and VNI

When TMM receives these updates, it:

• Updates the ARL (Address Resolution Logic) table with remote MAC addresses

• Creates VTEP entries on the correct VXLAN tunnel for the remote VTEP IP

• Creates dynamic ARP entries for remote endpoints

• Adds interface routes for L3 VNI destinations

If the connection between TMM and tmrouted is reestablished (for example, after a restart), TMM clears all ARL and ARP entries received from tmrouted and deletes the corresponding VTEPs. OcNOS then resends current state.

VRF processing in TMM

When you create a Vrf resource, the following sequence occurs:

1. The controller sends a route domain message to TMM containing a name and an ID.

2. TMM creates a route domain internally and notifies the Linux kernel to create a VRF interface.

3. TMM sends the VRF name, interface index, and route table ID to tmrouted.

4. Tmrouted forwards this information to OcNOS through Netlink.

5. OcNOS creates a corresponding VRF in its routing table.

When you associate a VLAN with a VRF (by setting the vrf field on an F5SPKVlan resource):

1. TMM receives the VLAN configuration with the VRF ID.

2. TMM binds the tap interface to the VRF interface in the kernel.

3. TMM sends the VLAN-to-VRF binding to tmrouted, which forwards it to OcNOS.

Self IPs, static routes, and address lists associated with a VRF must include the VRF ID so they are placed in the correct routing domain.

Note

The default route domain (route domain 0) is always present on TMM. Unless you specify a VRF, all configuration belongs to route domain 0. No VRF interface is created for route domain 0.

VXLAN and EVPN processing

Local VTEP setup

When you create a Vxlan resource:

1. The controller configures TMM with the local VXLAN tunnel data (VNI, VTEP IP, MTU, port) and maps it to the specified VRF.

2. TMM sends the VXLAN configuration to tmrouted.

3. TMM sends virtual addresses (including the underlay IP used by the local VTEP) to OcNOS for BGP advertisement.

4. OcNOS advertises Route Type 5 (IP prefix) routes for the local VTEP.

Remote VTEP discovery

OcNOS learns about remote VTEPs through EVPN route exchange:

1. OcNOS receives RT-5 routes from BGP peers.

2. OcNOS sends FDB entries (containing the VNI, remote MAC, and remote VTEP IP) to tmrouted through Netlink.

3. Tmrouted forwards these entries to TMM through protobuf.

4. TMM updates its forwarding tables:

   • Creates a VTEP entry on the VXLAN tunnel for the remote VTEP IP

   • Updates ARL data with the remote MAC address

   • Creates ARP and route entries for the remote destination

EVPN route lifecycle

When OcNOS receives a new EVPN route from a BGP peer, it adds forwarding entries. When a route is withdrawn, OcNOS sends corresponding delete messages to remove the FDB, ARP, and route entries from TMM.

When a VNI interface is received in a specific route domain, OcNOS updates the router MAC on the VRF to the MAC address of the VNI. This is required for traffic to flow correctly between the router and the server leaf connected to TMM.

BGP Unnumbered neighbor discovery

Because OcNOS relies on hardware support for BGP Unnumbered neighbor discovery (which is not available in a Kubernetes environment), TMM and tmrouted provide this function instead.

The neighbor discovery flow works as follows:

1. The controller identifies unnumbered interfaces: VLANs without self IPs are treated as BGP Unnumbered interfaces when the ENABLE_UNNUMBERED_VLAN environment variable is set.

2. TMM sends Router Advertisements: TMM sends periodic RAs on the unnumbered interface (minimum interval: 10 seconds, maximum interval: 60 seconds). The RA contains the TMM interface’s link-local address.

3. TMM receives remote RAs: When TMM receives an RA from a remote peer on an RA-enabled interface, it extracts the peer’s link-local address.

4. TMM forwards RA data to tmrouted: TMM pushes the remote RA data to tmrouted through a protobuf message.

5. Tmrouted forwards to OcNOS: Tmrouted runs an NDD server socket (/tmos/.ndmgrserv) that accepts client connections from OcNOS. When OcNOS connects and registers for IPv6 neighbor updates, tmrouted pushes all stored RA data to the OcNOS client.

6. OcNOS establishes the BGP session: With the remote peer’s link-local address known, OcNOS establishes the BGP session using IPv6 link-local addresses.

RA data lifecycle

• When a neighbor state changes to NEXTHOP_REACHABLE, TMM sends an ADD message to tmrouted, which stores it.

• When a neighbor state changes to NEXTHOP_DELETED, TMM sends a DELETE message to tmrouted to clean up the entry.

Security model

gRPC channel security

The gRPC channel between the CNE controller and the f5-tmm-routing container is secured with mutual TLS (mTLS):

• TLS certificates are mounted into the routing container.

• The controller authenticates to the routing container, and the routing container authenticates back to the controller.

• Traffic is encrypted in transit. The gRPC payload is not decrypted until it reaches the grpcconf_watcher process inside the routing container.

BGP session security

BGP neighbor authentication passwords are stored in Kubernetes Secrets and referenced by the neighborSecrets field in GlobalRoutingConfig and RoutingConfig resources. The controller injects these passwords into the rendered configuration during template processing.

Trust boundaries

The architecture has four trust boundaries:

Boundary	Description
Tenant-to-fabric	External vendor peers join the fabric through controlled interfaces. Per-tenant VRFs with explicit import/export route-target policies provide isolation.
Fabric control plane	Internal BGP sessions between spines and leaves. Route policy filtering restricts allowed route types, route targets, and prefixes per vendor.
Overlay data plane	VXLAN tunnels between VTEPs carry tenant traffic. Unique VNI assignment per tenant VRF prevents cross-tenant data leakage.
Management plane	The CNE controller pushes configuration through the gRPC channel. Access is restricted to cluster-internal endpoints through headless service DNS resolution.

For BGP Unnumbered sessions, the architecture relies on the NVIDIA cloud-managed leaf-spine fabric for link-layer trust. Link-local neighbor discovery cannot be cryptographically validated, and BGP authentication (TCP-AO/MD5) requires mutual provider support. Risk is mitigated by restricting BGP adjacencies to interface-bound neighbors, explicitly configuring router IDs, enforcing strict inbound and outbound routing policy, and monitoring for session instability.

Error handling

Scenario	Behavior
Validation failure	The controller writes the error to the CRD status and does not push a ConfigBlob.
gRPC delivery failure	The controller writes the error to the CRD status. The routing container does not receive the configuration.
Routing container rejection	The routing container returns a failure acknowledgment. The controller writes the error to the CRD status.
Delete event	The controller renders a template-driven delete variant (or empty payload) and pushes it to the routing container.
TMM-to-tmrouted reconnection	TMM clears all ARL and ARP entries received from tmrouted and deletes corresponding VTEPs. OcNOS resends current state.

Related topics

• BGP routing with Kubernetes CRDs

• BGP Unnumbered

• EVPN and VXLAN networking

• Configure global BGP routing

• Configure BGP Unnumbered

• Configure VXLAN for EVPN

• Configure VRF isolation

• Configure tenant BGP routing

• CRD reference

• Sample YAML files by use case

• Troubleshooting BGP routing