Troubleshoot Your OpenShift Deployment

How to get help

If the issue you’re experiencing isn’t covered here, try one of the following options:

General OpenShift troubleshooting

The following troubleshooting doc(s) may help with OpenShift-specific issues.

BIG-IP Controller troubleshooting


You can use oc commands to check the BIG-IP Controller configurations using the command line.

oc get pod -o yaml                \\ Returns the Pod's YAML settings
oc describe pod myBigIpCtlr       \\ Returns an information dump about the Pod you can use to troubleshoot specific issues

I just deployed the Controller; how do I verify that it’s running?

  1. Find the name of the k8s-bigip-ctlr Pod.

    oc get pod
    NAME                             READY     STATUS    RESTARTS   AGE
    k8s-bigip-ctlr-687734628-7fdds   1/1       Running   0          15d
  2. Check the status of the Pod.

    oc get pod k8s-bigip-ctlr-687734628-7fdds -o yaml
  3. View the Controller logs.

    View the logs
    oc logs k8s-bigip-ctlr-687734628-7fdds
    Follow the logs
    oc logs -f k8s-bigip-ctlr-687734628-7fdds
    View logs for a container that isn’t responding
    oc logs --previous k8s-bigip-ctlr-687734628-7fdds

How do I set the log level?

To change the log level for the BIG-IP Controller:

  1. Edit the Deployment yaml and add the following to the args section.

  2. Replace the BIG-IP Controller deployment

    oc replace -f f5-k8s-bigip-ctlr.yaml
  3. Verify the Deployment updated successfully.

    oc describe deployment k8s-bigip-ctlr -o wide

What happened to my BIG-IP configuration changes?

If you make changes to objects in the partition managed by the BIG-IP Controller – whether via configuration sync or manually – the Controller will overwrite them. By design, the BIG-IP Controller keeps the BIG-IP system in sync with what it knows to be the desired configuration. For this reason, F5 does not recommend making any manual changes to objects in the partition(s) managed by the BIG-IP Controller.

Why didn’t the BIG-IP Controller create any objects on my BIG-IP?

Here are a few basic things to check:

Did you provide valid JSON?

The settings provided in the section of your ConfigMap must be valid JSON. Run your desired configurations through a JSON linter before use to avoid potential object creation errors.

Have you used the correct version of the F5 schema?

Additions to the F5 schema made with each version release support the features in that specific version. For example, if you use v1.3.0 of the Controller with v0.1.2 of the schema, the Controller’s core functionality would be fine. You wouldn’t, however, be able to use the features from k8s-bigip-ctlr v1.3.0.

Are you looking in the correct partition on the BIG-IP system?

If you’re in the Common partition, switch to the partition managed by the BIG-IP Controller to find the objects it deployed.

  • In the BIG-IP configuration utility (aka, the GUI), check the partition drop-down menu.

  • In the BIG-IP Traffic Management shell (TMSH), check the name of the partition shown in the prompt.


Network troubleshooting

How do I verify connectivity between the BIG-IP VTEP and the OSE Node?

  1. Ping the Node’s VTEP IP address.

    Use the -s flag to set the MTU of the packets to allow for VxLAN encapsulation.

    ping -s 1600 <OSE_Node_IP>
  2. View the logs for the k8s-bigip-ctlr.


    Use the -f option to follow the logs.

  3. In a TMOS shell, output the REST requests from the BIG-IP logs.

    • Do a tcpdump of the underlay network.

      tcpdump -i <name_of_vlan_used_for_VTEP>

      Example showing two-way communication between the BIG-IP VTEP IP and the OSE node VTEP IPs.
      admin@(BIG-IP)(cfg-sync Standalone)(Active)(/Common)(tmos)$ tcpdump -i ocpvlan
      08:08:06.933951 IP > VXLAN, flags [I] (0x08), vni 0
      IP > Flags [.], ack 9, win 219, options [nop,nop,TS val 573988389 ecr 3961177660], length 0 in slot1/tmm1 lis=_wcard_tunnel_/Common/ose-tunnel
      08:08:06.934310 IP > VXLAN, flags [I] (0x08), vni 0
      IP > Flags [.], ack 923, win 251, options [nop,nop,TS val 3961177661 ecr 573988389], length 0 out slot1/tmm0 lis=_wcard_tunnel_/Common/ose-tunnel
    • Do a tcpdump of the overlay network.

      tcpdump -i <name_of_VXLAN_tunnel_on_BIG-IP>

      Example showing traffic on the overlay network; at minimum, you should see BIG-IP health monitors for the Pod IP addresses.
      admin@(BIG-IP)(cfg-sync Standalone)(Active)(/Common)(tmos)$ tcpdump -i ose-tunnel
      08:09:51.911667 IP > Flags [.], ack 1, win 229, options [nop,nop,TS val 3961282640 ecr 574093366], length 0 out slot1/tmm0 lis=
      08:09:51.911672 IP > Flags [P.], seq 1:8, ack 1, win 229, options [nop,nop,TS val 3961282640 ecr 574093366], length 7 out slot1/tmm0 lis=
      08:09:51.913161 IP > Flags [.], ack 8, win 219, options [nop,nop,TS val 574093369 ecr 3961282640], length 0 in slot1/tmm0 lis=
      08:09:51.913265 IP > Flags [P.], seq 1:922, ack 8, win 219, options [nop,nop,TS val 574093369 ecr 3961282640], length 921 in slot1/tmm0 lis=
  4. In a TMOS shell, view the VLAN statistics.

    • Underlay

      tmsh show /net vlan <name_of_vlan_used_for_VTEP>

      admin@(BIG-IP)(cfg-sync Standalone)(Active)(/Common)(tmos)$ show /net vlan ocpvlan
      Net::Vlan: ocpvlan
      Interface Name      ocpvlan
      Mac Address (True)  00:0c:29:fe:f9:4e
      MTU                 1500
      Tag                 4094
        | Net::Vlan-Member: 1.1
        | Tagged    no
        | Tag-Mode  none
           | Net::Interface
           | Name  Status   Bits   Bits   Pkts  Pkts  Drops  Errs  Media
           |                  In    Out     In   Out
           | 1.1       up  52.8G  17.0G  14.6M  7.4M      0     0   none
    • Overlay

      tmsh show /net vlan <name_of_VXLAN_tunnel_on_BIG-IP>

      admin@(BIG-IP)(cfg-sync Standalone)(Active)(/Common)(tmos)$ show /net tunnels tunnel ose-tunnel
      Net::Tunnel: ose-tunnel
      Incoming Discard Packets            0
      Incoming Error Packets              0
      Incoming Unknown Proto Packets      0
      Outgoing Discard Packets            0
      Outgoing Error Packets              0
      HC Incoming Octets               1.8G
      HC Incoming Unicast Packets     10.2M
      HC Incoming Multicast Packets       0
      HC Incoming Broadcast Packets       5
      HC Outgoing Octets               1.8G
      HC Outgoing Unicast Packets     10.2M
      HC Outgoing Multicast Packets   91.6K
      HC Outgoing Broadcast Packets   92.7K
  5. In a TMOS shell, view the MAC address entries for the OSE tunnel. This will show the mac address and IP addresses of all of the OpenShift endpoints.

    tmsh show /net fdb tunnel <name_of_VXLAN_tunnel on BIG-IP>

    admin@(BIG-IP)(cfg-sync Standalone)(Active)(/Common)(tmos)$ show /net fdb tunnel ose-tunnel
    Tunnel      Mac Address        Member                 Dynamic
    ose-tunnel  0a:58:0a:82:00:1b  endpoint:  yes
    ose-tunnel  0a:58:0a:82:00:21  endpoint:  yes
    ose-tunnel  0a:58:0a:82:00:25  endpoint:  yes
  6. In a TMOS shell, view the ARP entries.

    This will show all of the ARP entries; you should see the VTEP entries on the ocpvlan and the Pod IP addresses on ose-tunnel.

    admin@(BIG-IP)(cfg-sync Standalone)(Active)(/Common)(tmos)$ show /net arp
    Name          Address       HWaddress          Vlan                Expire-in-sec  Status
    ------------------------------------------------------------------------------------------   0a:58:0a:82:00:1b  /Common/ose-tunnel  224            resolved   0a:58:0a:82:00:21  /Common/ose-tunnel  220            resolved   0a:58:0a:82:00:25  /Common/ose-tunnel  222            resolved  00:0c:29:c8:4c:dc  /Common/ocpvlan     220            resolved  00:0c:29:8d:ac:42  /Common/ocpvlan     220            resolved  00:0c:29:cd:ba:44  /Common/ocpvlan     220            resolved