BGP Overview

Overview

To scale the Cloud-Native Network Functions (CNFs) Traffic Management Microkernel (TMM), learn and announce routing prefixes between networks, and to advertise subscriber NAT IP addresses to downstream routers, Border Gateway Protocol (BGP) sessions must be established on TMM’s upstream and downstream interfaces. The TMM Proxy Pod’s f5-tmm-routing container can be enabled and configured when installing the BIG-IP Controller. By default, the f5-tmm-routing container is disabled.

Review the sections below for BGP configuration assistance:

_images/spk_info.png Note: Review CNF BGP topology for a high-level overview.

ZebOS ConfigMaps

The CNFs f5-tmm-routing container can reference native ZebOS.conf files as ConfigMaps using the BIG-IP Controller Helm values. One of the benefits of referencing the ZebOS.conf file as a ConfigMap is the ability to modify BGP configurations while the CNFs F5ingress and TMM Proxy Pods are running. The CNFs Controller detects modifications made to the ConfigMap file, and applies the updates the running f5-tmm-routing container. Refer to the ZebOS ConfigMaps overview.

BGP Parameters

The tables below describe the BIG-IP Controller’s tmm.dynamicRouting.tmmRouting.config Helm parameters.

tmm.dynamicRouting

Parameter Description
enabled Enables the f5-tmm-routing container: true or false (default).
exportZebosLogs Enables sending f5-tmm-routing logs to Fluentd Logging: true (default) or false.

tmm.dynamicRouting.tmmRouting.config.bgp

Configure and establish BGP peering relationships.

Parameter Description
asn The AS number of the f5-tmm-routing container.
hostname The hostname of the f5-tmm-routing container.
logFile Specifies a file used to capture BGP logging events: /var/log/zebos.log.
debugs Sets the BGP logging level to debug for troublshooting purposes: ["bgp"]. It is not recommended to run in debug level for extended periods.
secret Set the name of the Kubernetes secret containing the BGP neighbor password. See the BGP Secret section below.
neighbors.ip The IPv4 or IPv6 address of the BGP peer.
neighbors.asn The AS number of the BGP peer.
neighbors.nodeLookup Reference the ConfigMap holding the IP addresses. For more infor, refer to Top of Rack BGP.
neighbors.password The BGP peer MD5 authentication password. Note: The password is stored in the f5-tmm-dynamic-routing configmap unencrypted.
neighbors.ebgpMultihop Enables connectivity between external peers that do not have a direct connection (1-255).
neighbors.acceptsIPv4 Enables advertising IPv4 virtual server addresses to the peer (true / false). The default is false.
neighbors.acceptsIPv6 Enables advertising IPv6 virtual server addresses to the peer (true / false). The default is false.
neighbors.softReconf Enables BGP4 policies to be activated without clearing the BGP session.
neighbors.maxPathsEbgp The number of parallel application (downstream) eBGP neighbours installed as Gateways or Next hops. The configurable range is 2 to 64.
neighbors.maxPathsIbgp The number of parallel subscriber (upstream) iBGP neighbours installed as Gateways or Next hops. The configurable range is 2 to 64.
neighbors.fallover Enables bidrectional forwarding detection (BFD) between neighbors (true / false). The default is false.
neighbors.routeMap References the routeMaps.name parameter, and applies the filter to the BGP neighbor.

tmm.dynamicRouting.tmmRouting.config.prefixList

Create prefix lists to filter specified IP address subnets.

Parameter Description
name The name of the prefixList entry.
seq The order of the prefixList entry.
deny Allow or deny the prefixList entry.
prefix The IP address subnet to filter.

tmm.dynamicRouting.tmmRouting.config.routeMaps

Create route maps that apply to BGP neighbors, referencing specified prefix lists.

Parameter Description
name The name of the routeMaps object applied to the BGP neighbor.
seq The order of the routeMaps entry.
deny Allow or deny routeMaps entry.
match The name of the referenced prefixList.

tmm.dynamicRouting.tmmRouting.config.bfd

Enable BFD and configure the control packet intervals.

Parameter Description
interface Selects the BFD peering interface if specified.
interval Sets the minimum transmission interval in milliseconds: 50 (default) - 999.
minrx Sets the minimum receive interval in milliseconds: 50 (default) - 999.
multiplier Sets the Hello multiplier value 3 - 50. The default is 10.
multihop_peer Enables multi-hop BFD to BGP neighbor: true or false (default).

BGP Secrets

BGP neighbor passwords can be stored as Kubernetes secrets using the bgpSecret parameter described in the BGP Parameters section above. When using Secrets, the value must be the neighbor.ip, and the data must be the base64 encoded password. When using IPv6, replace any colon : characters, with dash *- characters. For example:

apiVersion: v1
kind: Secret
metadata:
 name: bgp-secret
 namespace: cnf-gateway
data:
 10.1.2.3: c3dvcmRmaXNo
 2002--10-1-2-3: cGFzc3dvcmQK

Enabling ECMP for Load Balancing Traffic

Currently, tmrouted only selects the first route configured before sending it to TMM/ZebOS. With this setup, it will accept multiple routes (instead of just one) and enable ECMP across upstream or downstream paths. Since ECMP is a round-robin load balancing method, traffic will load balance across all available ECMP routes.

  1. When installing the BIG-IP Contoller, set the maxPathsEbgp parameter to the maxiumum number of ECMP paths you intend to use:

    Note: In this example, set the maxpathsEbgp or maxpathsIbgp parameter to number of paths for load balancing traffic.

    tmm:
      dynamicRouting:
        enabled: true
        tmmRouting:
          config:
            bgp:
              asn: 100
              maxPathsEbgp: 2
              maxPathsIbgp: 2
              hostname: cnf-bgp
              neighbors:
              - ip: 10.10.10.200
                asn: 200
                ebgpMultihop: 10
              - ip: 192.168.10.200
                asn: 400
                ebgpMultihop: 10
    

Advertising IP addresses

To ensure upstream routers use TMM as a gatway, and to ensure downstream routers correclty route back through TMM, BGP peering relationships should be established on TMM’s upstream and downstream interfaces. With BGP relationships, TMM can advertise routes learned from the downstream router, and any destination IP addresses defined in the Traffic Management CNF CRs to the upstream router. TMM will also advertise IP addresses defined in the F5BigNatPolicy CR, to ensure the downstream router sends connections back through TMM.

  1. When installing the BIG-IP Contoller, set the acceptsIPv4 and the acceptsIPv6 parameters to advertise IPv4 and IPv6 destination addresses:

    Note: In this example, the acceptsIPv4 and acceptsIPv6 parameters are set on the upstream BGP peer:

    tmm:
      dynamicRouting:
        enabled: true
        tmmRouting:
          config:
            bgp:
              asn: 100
              maxPathsEbgp: 4
              maxPathsIbgp: 'null'
              hostname: cnf-bgp
              neighbors:
              - ip: 10.10.10.200
                asn: 200
                ebgpMultihop: 10
                acceptsIPv4: true
                acceptsIPv6: true
              - ip: 192.168.10.200
                asn: 400
                ebgpMultihop: 10
    
  2. Once the BIG-IP Controller is installed, log in to the peer router and verify the advertised routing prefixes are being advertised with the TMM self-IPs as next hops:

    show ip bgp
    

    In this example, the TMM replicas with self IP addresses 10.10.10.250 and 10.10.10.251 are advertising the CR destination address 192.168.10.100:

    show ip bgp
    
    Network             Next Hop         Metric     Path
    192.168.10.100/32   10.10.10.250          0     200
                        10.10.10.251          0     200
    

Filtering IP addresses

By default, all F5BigNatPolicy IP addresses are advertised (redistributed) to BGP neighbors. To advertise specific NAT IP addresses, configure a prefixList defining the IP addresses to advertise, and apply a routeMap to the BGP neighbor configuration referencing the prefexList. In the example below, only the 10.244.10.0/24 and 10.244.20.0/24 IP address subnets will be advertised to the BGP neighbor:

dynamicRouting:
  enabled: true
  tmmRouting:
    config:
      prefixList:
        - name: 10pod
          seq: 10
          deny: false
          prefix: 10.244.10.0/24 le 32
        - name: 20pod
          seq: 10
          deny: false
          prefix: 10.244.20.0/24 le 32

      routeMaps:
        - name: snatpoolroutemap
          seq: 10
          deny: false
          match: 10pod
        - name: snatpoolroutemap
          seq: 11
          deny: false
          match: 20pod

      bgp:
        asn: 100
        hostname: cnf-bgp
        neighbors:
        - ip: 10.10.10.200
          asn: 200
          routeMap: snatpoolroutemap

Once the BIG-IP Controller is installed, verify the expected SNAT pool IP addresses are being advertised.

  1. Install the F5BigNatPolicy Custom Resource (CR).

  2. Log in to the f5-tmm-routing container:

    kubectl exec -it deploy/f5-tmm -c f5-tmm-routing -n <namespace> -- bash
    

    In this example, the f5-tmm-routing container is in the cnf-gateway Project:

    kubectl exec -it deploy/f5-tmm -c f5-tmm-routing -n cnf-gateway -- bash
    
  3. Log in IMI shell and turn on privileged mode:

    imish
    en
    
  4. Verify the SNAT pool IP addresses are being advertised:

    show bgp ipv4 neighbors <ip address> advertised-routes
    

    In this example, the NAT pool IP addresse lists are being advertised, and TMM’s external interface is the next hop:

    show bgp ipv4 neighbors 10.10.10.200 advertised-routes
    
         Network          Next Hop            Metric    LocPrf        Weight
    *>   10.244.10.1/32   10.20.2.207         0         100           32768
    *>   10.244.10.2/32   10.20.2.207         0         100           32768
    *>   10.244.20.1/32   10.20.2.207         0         100           32768
    *>   10.244.20.2/32   10.20.2.207         0         100           32768
    
    Total number of prefixes 4
    

Enabling BFD

Bidirectional Forwarding Detection (BFD) rapidly detects loss of connectivity between BGP neighbors by exchanging periodic BFD control packets on the network link. After a specified interval, if a control packet is not received, the connection is considered down, enabling fast network convergence. The BFD configuration requires the interface name of the external BGP peer. Use the following command to obtain the external interface name:

kubectl get ingressroutevlan <external vlan> -o "custom-columns=VLAN Name:.spec.name"
  1. When installing the BIG-IP Contoller, configure the bfd for the interface to monitor:

    Note: In this example, BFD is enabled on TMM’s external interface:

    tmm:
      dynamicRouting:
        enabled: true
        tmmRouting:
          config:
            bgp:
              asn: 100
              maxPathsEbgp: 4
              maxPathsIbgp: 'null'
              hostname: cnf-bgp
              neighbors:
              - ip: 10.10.10.200
                asn: 200
                ebgpMultihop: 10
                acceptsIPv4: true
                acceptsIPv6: true
              - ip: 192.168.10.200
                asn: 400
                ebgpMultihop: 10
            bfd:
              interface: external
              interval: 100
              minrx: 100
              multiplier: 3
    
  2. Once the BIG-IP Controller is installed, verify the BFD configuration is working.

    In this example, the f5-tmm-routing container is in the cnf-gateway Namespace:

    kubectl exec -it deploy/f5-tmm -c f5-tmm-routing -n cnf-gateway -- bash
    
  3. Log in IMI shell and turn on privileged mode:

    imish
    en
    
  4. View the bfd session status:

    _images/spk_info.png Note: You can append the detail argument for verbose session information.

    show bfd session 
    

    In this example, the Sess-State is Up:

    BFD process for VRF: (DEFAULT VRF)
    =====================================================================================
    Sess-Idx   Remote-Disc  Lower-Layer  Sess-Type   Sess-State  UP-Time   Remote-Addr
    2          1            IPv4         Single-Hop  Up          00:03:16  10.10.10.200/32
    Number of Sessions:    1
    
  5. BGP should now quickly detect link failures between neighbors.

Troubleshooting

When BGP neighbor relationships fail to establish, begin troubleshooting by reviewing BGP log events to gather useful diagnostic data. If you installed the Fluentd logging collector, review the Log file locations and Viewing logs sections of the FLuentd Logging guide before proceeding to the steps below. If the Fluentd logging collector is not installed, use the steps below to verify the current BGP state, and enable and review log events to resolve a simple connectivity issue.

_images/spk_info.png Note: BGP connectivity is established over TCP port 179.

  1. Run the following command to verify the BGP state:

    kubectl exec -it deploy/f5-tmm -c f5-tmm-routing -n cnf-gateway \
    -- imish -e 'show bgp neighbors' | grep state
    

    In this example, the BGP state is Active, indicating neighbor relationships are not currently established:

     BGP state = Active
     BGP state = Active
    
  2. To enable BGP logging, log in to the f5-tmm-routing container:

    kubectl exec -it deploy/f5-tmm -c f5-tmm-routing -n cnf-gateway \
    -- bash
    
  3. Run the following commands to enter configuration mode:

    imish
    en
    config t
    
  4. Enable BGP logging:

    log file /var/log/zebos.log
    
  5. Exit configuration mode, and return to the shell:

    exit
    exit
    exit
    
  6. View the BGP log file events as they occur:

    tail -f /var/log/zebos.log
    

    In this example, the log messages indicate the peers (neighbors), are not reachable:

    Jan 01 12:00:00 : BGP : ERROR [SOCK CB] Could not find peer for FD - 11 (error:107)
    Jan 01 12:00:01 : BGP : INFO 10.20.2.206-Outgoing [FSM] bpf_timer_conn_retry: Peer down,
    Jan 01 12:00:02 : BGP : ERROR [SOCK CB] Could not find peer for FD - 11 (error:107)
    Jan 01 12:00:01 : BGP : INFO 10.30.2.206-Outgoing [FSM] bpf_timer_conn_retry: Peer down,
    
  7. Fix: The tag ID on the F5BigNetVlan was set to the correct ID value:

    The messages indicate the neighbors are now Up. It can take up to two minutes for the relationships to establish:

    Jan 01 12:00:05 : BGP : ERROR [SOCK CB] Could not find peer for FD - 13 (error:107)
    Jan 01 12:00:06 : BGP : INFO %BGP-5-ADJCHANGE: neighbor 10.20.2.206 Up
    Jan 01 12:00:07 : BGP : ERROR [SOCK CB] Could not find peer for FD - 11 (error:107)
    Jan 01 12:00:08 : BGP : INFO %BGP-5-ADJCHANGE: neighbor 10.30.2.206 Up
    
  8. The BGP state should now be Established:

    imish -e 'show bgp neighbors' | grep state
    
      BGP state = Established, up for 00:00:36
      BGP state = Established, up for 00:00:19
    
  9. If the BGP state is still not established, and there are issues other than connectivity, set BGP logging to debug, and continue reviewing the lower-level log events:

    debug bgp all  
    
  10. Once the BGP troubleshooting is complete, remove the BGP log and debug configurations:

    no log file
    
    no debug bgp
    

Feedback

Provide feedback to improve this document by emailing cnfdocs@f5.com.

Supplemental

  • Top of Rack BGP enables TMM Pods to peer with neighbors based on the cluster node they run on.