Top of Rack BGP


The Service Proxy for Kubernetes (SPK) Top of Rack (ToR) BGP feature enables the Traffic Management Microkernel (TMM) Pods to establish BGP peer relationships based on the cluster node the TMM Pod is running on. Without the ToR BGP feature, all TMM Pods must share the same BGP neighbor configuration, requiring a full BGP mesh.


The table below describes the BIG-IP Controller’s BGP Helm parameters used in this document, refer to the BGP Overview for the full list of parameters.

Parameter Description
asn The AS number of the f5-tmm-routing container.
maxPathsEbgp Set the maxPathsEbgp to install multiple paths in External BGP
maxPathsIbgp Set the maxPathsIbgp to install multiple paths in Internal BGP
hostname The hostname of the f5-tmm-routing container.
neighbors.nodeLookup Reference the ConfigMap holding the IP addresses.
neighbors.asn The AS number of the BGP peer.
neighbors.ebgpMultihop Enables connectivity between external peers that do not have a direct connection (1-255).
neighbors.acceptsIPv4 Enables advertising IPv4 virtual server addresses to the peer (true / false). The default is false.
neighbors.softReconf Enables BGP4 policies to be activated without clearing the BGP session.

BGP peer groups

The SPKs ToR feature relies on BGP peer-groups to support per-node BGP peering relationships. Because BGP peer-groups support only a one AS (autonomous system) number, you must create a separate SPKs ConfigMap for each AS number when using the procedure below. You can use multiple ConfigMaps to map IP addresses within a single AS number, for example to map IPv4 and IPv6 peers within the AS


Use these steps to configure SPKs ToR BGP peering.

  1. Determine the cluster node names:

    kubectl get nodes
    NAME                          STATUS   ROLES                Ready    control-plane,master   Ready    <none>               Ready    <none>   
  2. Create a new ConfigMap using the node names and the IP addresses of the BGP peers connected to the node. As mentioned in the BGP peer groups section, you must create one ConfgMap per AS number:

    Note: Be certain to add the namespace of the BIG-IP Controller.

    apiVersion: v1
    kind: ConfigMap
      name: "spk-tor-bgp"
      namespace: "spk-ingress"

    You can also use multiple IP addresses per entry.

    apiVersion: v1
    kind: ConfigMap
      name: "spk-tor-bgp"
      namespace: "spk-ingress"
  3. Reference the ConfigMap name using the BIG-IP Controller’s neighbors.nodeLookup parameter:

      asn: 64443
      maxPathsEbgp: 4
      maxPathsIbgp: "null"
       - nodeLookup: spk-tor-bgp
         asn: 3535
         ebgpMultihop: 100
         acceptsIPv4: true
         softReconf: true            
  4. After installing the BIG-IP Controller, to verify the BGP configuration is correct, first determine where the TMMs have scheduled:

    kubectl get pods -n spk-ingress -o wide | grep tmm
    f5-tmm-f98ff99cc-l76hc  4/4  Running
    f5-tmm-f98ff99cc-nfkgw  4/4  Running  172.18.0:6
  5. View the ZebOS configuration once the BIG-IP Controller has installed:

    In this example, the f5-tmm-routing container is in the spk-ingress Project:

    oc exec -it deploy/f5-tmm -c f5-tmm-routing -n spk-ingress \
    -- imish -e 'show running-config'
  6. Verify the correct IP address is configured:

    In this example, the mapped IP address is used in the configuration.

    router bgp 64443
    bgp router-id
    no bgp default ipv4-unicast
    bgp log-neighbor-changes
    bgp graceful-restart restart-time 120
    max-paths ebgp 4
    redistribute kernel
    neighbor spk-peers peer-group
    neighbor spk-peers remote-as 3535
    neighbor spk-peers ebgp-multihop 100
    neighbor spk-peers activate
    neighbor spk-peers soft-reconfiguration inbound
    neighbor peer-group spk-peers
    neighbor activate
  7. The TMM Pods should now peer with the BGP neighbors based on the node they are running.

Configuration updates

The configuration updates section describes how to update the SPKs BGP peering settings when either adding a new cluster node, or changing BGP peer IP addresses.

Adding new nodes

Use the steps below to add a new node to the cluster.

Note: These steps can also be followed for replacing nodes without interruption of service.

  1. Integrate the new cluster node.
  2. Configure the peer BGP Router for the new node.
  3. Update the SPKs BGP ConfigMap with the node and node IP address information.
  4. New TMM Pods can now be scheduled to the new node.

Changing BGP peer IPs

Use these step when changing BGP peer IP addresses.

Note: These steps can also be followed when BGP peers need to be replaced.

  1. Configure the BGP peers with new IP addresses.

  2. Update the SPKs ConfigMap by mapping each new BGP peer IP address with the correct cluster node name.

  3. Delete the TMM Pods in the namespace:

    kubectl delete pods -l app=f5-tmm -n spk-ingress
  4. The new TMM Pods should deploy automatically, reading the new BGP peer IP addresses from the SPKs config map.


Provide feedback to improve this document by emailing