Troubleshooting BGP routing

Overview

This guide helps you diagnose and resolve common issues with BGP routing, VXLAN/EVPN overlays, VRF isolation, and BGP Unnumbered in your environment. Each section describes the symptom, how to verify the current state, and how to resolve the issue.

Verify VRF configuration

Use the debug sidecar to confirm that your VRF and VLAN associations are configured correctly on TMM.

Check VRF devices and VLAN bindings

Run the following command in the TMM debug sidecar:

ip a

What to look for:

  • A VRF device should appear as a separate interface with NOARP,MASTER,UP,LOWER_UP flags.

  • Any VLAN associated with that VRF should show master <vrf-name> in its output, which confirms the VLAN is bound to the correct routing domain.

Example output (healthy state):

```
7: l3-int: <NOARP,MASTER,UP,LOWER_UP> mtu 65575 qdisc noqueue state UP group default qlen 1000
    link/ether 22:10:66:4c:8a:78 brd ff:ff:ff:ff:ff:ff
8: int_vlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master l3-int state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:3e:76:f7 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.167/24 brd 10.0.0.255 scope global int_vlan
      valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe3e:76f7/64 scope link
      valid_lft forever preferred_lft forever
```

In this example:

  • l3-int is the VRF device.

  • int_vlan shows master l3-int, which confirms it is bound to the l3-int VRF.

What to check if the VRF is missing

  • Confirm that you created the Vrf resource and that its metadata.name is 15 characters or fewer.

  • Confirm that the F5SPKVlan resource includes the vrf field set to the correct VRF name.

  • Check the TMM logs for error messages related to VRF or route domain creation.

Test connectivity across VRFs

To test connectivity to an IP address in a non-default VRF, you must specify the VLAN or VRF interface name with the -I argument.

Ping using the VLAN interface name

ping -I int_vlan 10.0.0.231

Expected output:

PING 10.0.0.231 (10.0.0.231) from 10.0.0.167 int_vlan: 56(84) bytes of data.
64 bytes from 10.0.0.231: icmp_seq=1 ttl=64 time=1.35 ms

Ping using the VRF device name

ping -I l3-int 10.0.0.231

Expected output:

```
PING 10.0.0.231 (10.0.0.231) from 10.0.0.167 l3-int: 56(84) bytes of data.
64 bytes from 10.0.0.231: icmp_seq=1 ttl=64 time=1.87 ms
```

Note

A standard ping without the -I flag uses the default route domain (route domain 0). Traffic in a non-default VRF is not reachable without specifying the correct interface.

What to check if ping fails

  • Verify that the VRF device and VLAN binding are correct using ip a.

  • Confirm that the self IP on the VLAN is in the correct subnet.

  • Confirm that the static route (if any) includes the correct vrf field in the F5SPKStaticRoute resource.

Verify VXLAN tunnel status

Use the OcNOS CLI to confirm that the VXLAN tunnel between local and remote VTEPs is established.

Check VXLAN tunnel state

sh nvo vxlan tunnel

Example output (healthy state):

VXLAN Network tunnel Entries
Source           Destination      Status        Up/Down       Update
========================================================================
100.100.100.100  100.100.100.200  Resolved      00:11:53      00:11:53        ---
Total number of entries are 1

What to look for:

Field

Healthy value

Problem indicator

Source

Your local VTEP IP

Missing or incorrect IP

Destination

The remote VTEP IP

Missing entry means the remote VTEP was not discovered

Status

Resolved

Any other status indicates the tunnel is not established

What to check if the tunnel is missing

  • Confirm that the Vxlan resource exists and has the correct vni, vrf, and vlan fields.

  • Verify that the remote VTEP is advertising EVPN routes (see the next section).

Verify EVPN route exchange

Use the OcNOS CLI to confirm that EVPN routes are being received and advertised correctly.

Check EVPN routes

sh bgp l2vpn evpn

Example output:

BGP table version is 2, local router ID is 100.100.100.100
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              l - labeled, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

[EVPN route type]:[ESI]:[VNID]:[relevant route information]
  1 - Ethernet Auto-discovery Route
  2 - MAC/IP Route
  3 - Inclusive Multicast Route
  4 - Ethernet Segment Route
  5 - Prefix Route

Network          Next Hop            Metric    LocPrf       Weight     Path  Peer          Encap

RD[100.100.100.200:2]
*>   [5]:[0]:[0]:[24]:[8.8.8.0]:[0.0.0.0]:[1000]
                100.100.100.200      0        100       0   65002 ?  100.100.100.200 VXLAN

Total number of prefixes 1

Understanding EVPN route types

Route type

Label

What it means

[5]

Prefix Route

Advertises IP prefix routes for subnet-level reachability. Used for L3 VNI traffic across different subnets.

What to look for

  • *> indicates the route is valid and selected as the best path.

  • Next Hop should show the remote VTEP IP address.

  • Encap should show VXLAN.

  • VNI (shown in the route entry as [1000] in the example above) should match your configured VNI.

What to check if no EVPN routes appear

  • Confirm that the GlobalRoutingConfig neighbor includes l2vpn-evpn in the families list.

  • Confirm that the remote VTEP is running and advertising routes.

  • Confirm that route targets match between the local and remote VTEPs.

Common issue: Configuration rejected by the routing container

Symptom: You create a GlobalRoutingConfig or RoutingConfig resource, but the status shows an error instead of success.

  1. Check CR status

kubectl get globalroutingconfig <name> -o yaml

or

kubectl get routingconfig <name> -o yaml

Look at the status section for delivery outcome and error messages.

  1. Identify the failure type

Failure type

Where it happens

What to check

Validation failure

CNE controller

The controller rejected the CRD before pushing configuration. Check the status message for the specific validation rule that failed.

gRPC delivery failure

Between controller and routing container

The controller could not deliver the configuration payload. Check network connectivity between f5ingress and the f5-tmm-routing container.

Routing container rejection

Routing container (OcNOS)

The routing container received the payload but could not apply it. The rendered CLI may contain syntax errors or unsupported commands.

  1. Review common validation errors

Error message

Cause

Resolution

"GlobalRoutingConfig updates are not allowed. Please delete and recreate."

You attempted to update an existing GlobalRoutingConfig

Delete the resource and create a new one with the updated values

"RoutingConfig updates are not allowed. Please delete and recreate."

You attempted to update an existing RoutingConfig

Delete the resource and create a new one

"RoutingTemplate updates are not allowed. Please delete and recreate."

You attempted to update an existing RoutingTemplate

Delete the resource and create a new one

"endASN must be >= startASN"

The endASN value is less than startASN

Set endASN to a value greater than or equal to startASN

"at least one of routerIdSource or ipSubnet+prefix must be specified"

Neither routerIdSource nor routerIdSubnet + routerIdPrefix is set in the BGP configuration

Add one of these fields to the bgp section

"One of vxlanRef or vrf is required"

A RoutingConfig was created without either vxlanRef or vrf

Add one of these fields to the spec

"vxlanRef and vrf are mutually exclusive - specify only one"

A RoutingConfig includes both vxlanRef and vrf

Remove one of the two fields

"remoteAs is required when ip is specified"

A neighbor uses the ip field but does not include remoteAs

Add the remoteAs field to the neighbor definition

Template contains os.Exit, syscall., exec.Command, os.Remove, os.RemoveAll, ioutil.WriteFile, or os.WriteFile

The RoutingTemplate body includes a forbidden operation

Remove the forbidden operation from the template

  1. Check processing order

If a RoutingConfig is rejected, confirm that the GlobalRoutingConfig exists and has reached a successful status. Tenant routing configurations are processed only after the global configuration succeeds.

Important

If you delete the GlobalRoutingConfig, all associated RoutingConfig resources are also deleted. You must recreate both the global and tenant configurations.