Troubleshoot Your OpenShift Deployment

How to get help

If the issue you’re experiencing isn’t covered here, try one of the following options:

General OpenShift troubleshooting

The following troubleshooting doc(s) may help with OpenShift-specific issues.

BIG-IP Controller troubleshooting

Tip

You can use oc commands to check the BIG-IP Controller configurations using the command line.

oc get pod -o yaml                \\ Returns the Pod's YAML settings
oc describe pod myBigIpCtlr       \\ Returns an information dump about the Pod you can use to troubleshoot specific issues

I just deployed the Controller; how do I verify that it’s running?

  1. Find the name of the k8s-bigip-ctlr Pod.

    oc get pod
    NAME                             READY     STATUS    RESTARTS   AGE
    k8s-bigip-ctlr-687734628-7fdds   1/1       Running   0          15d
    
  2. Check the status of the Pod.

    oc get pod k8s-bigip-ctlr-687734628-7fdds -o yaml
    
  3. View the Controller logs.

    View the logs
    oc logs k8s-bigip-ctlr-687734628-7fdds
    
    Follow the logs
    oc logs -f k8s-bigip-ctlr-687734628-7fdds
    
    View logs for a container that isn’t responding
    oc logs --previous k8s-bigip-ctlr-687734628-7fdds
    

How do I set the log level?

To change the log level for the BIG-IP Controller:

  1. Annotate the Deployment for the BIG-IP Controller.

    oc annotate k8s-bigip-ctlr.yaml "--log-level=DEBUG"
    
  2. Verify the Deployment updated successfully.

    oc describe deployment k8s-bigip-ctlr -o wide
    

What happened to my BIG-IP configuration changes?

If you make changes to objects in the partition managed by the BIG-IP Controller – whether via configuration sync or manually – the Controller will overwrite them. By design, the BIG-IP Controller keeps the BIG-IP system in sync with what it knows to be the desired configuration. For this reason, F5 does not recommend making any manual changes to objects in the partition(s) managed by the BIG-IP Controller.


Why didn’t the BIG-IP Controller create any objects on my BIG-IP?

Here are a few basic things to check:

Did you provide valid JSON?

The settings provided in the data.data section of your ConfigMap must be valid JSON. Run your desired configurations through a JSON linter before use to avoid potential object creation errors.

Have you used the correct version of the F5 schema?

Additions to the F5 schema made with each version release support the features in that specific version. For example, if you use v1.3.0 of the Controller with v0.1.2 of the schema, the Controller’s core functionality would be fine. You wouldn’t, however, be able to use the features from k8s-bigip-ctlr v1.3.0.

Are you looking in the correct partition on the BIG-IP system?

If you’re in the Common partition, switch to the partition managed by the BIG-IP Controller to find the objects it deployed.

  • In the BIG-IP configuration utility (aka, the GUI), check the partition drop-down menu.

    ../_images/bigip-partition_gui.png
  • In the BIG-IP Traffic Management shell (TMSH), check the name of the partition shown in the prompt.

    ../_images/bigip-partition_tmsh.png

Why did I see a traffic group error when I deployed my iApp?

When deploying an iApp with the BIG-IP Controller for OpenShift , the iApp may create a virtual IP in the wrong traffic group. If this occurs, you will see an error message like that below.

Configuration error: Unable to to create virtual address (/openshift/127.0.0.2) as part of application
(/os/default_os.http.app/default_os.http) because it matches the self ip (/Common/selfip.external)
which uses a conflicting traffic group (/Common/traffic-group-local-only)

If you’ve seen this error, you can override or change the default traffic-group as follows:

  • Set the specific traffic group you need in the iappOptions section of the virtual server F5 Resource definition.

  • Preferred Set the desired traffic group as the default for the partition you want the BIG-IP Controller to manage. This option doesn’t require OpenShift to know about BIG-IP traffic groups.

    "trafficGroup": "/Common/traffic-group-local-only"
    

Network troubleshooting

How do I verify connectivity between the BIG-IP VTEP and the OSE Node?

  1. Ping the Node’s VTEP IP address.

    Use the -s flag to set the MTU of the packets to allow for VxLAN encapsulation.

    ping -s 1600 <OSE_Node_IP>
    
  2. View the logs for the k8s-bigip-ctlr.

    Tip

    Use the -f option to follow the logs.

  3. In a TMOS shell, output the REST requests from the BIG-IP logs.

    • Do a tcpdump of the underlay network.

      tcpdump -i <name-of-BIG-IP-VXLAN-tunnel>
      

      Example showing two-way communication between the BIG-IP VTEP IP and the OSE node VTEP IPs.
      admin@(BIG-IP)(cfg-sync Standalone)(Active)(/Common)(tmos)$ tcpdump -i ocpvlan
      08:08:06.933951 IP 10.214.1.102.58472 > 10.214.1.23: VXLAN, flags [I] (0x08), vni 0
      IP 10.130.0.27.http > 10.128.2.10.37542: Flags [.], ack 9, win 219, options [nop,nop,TS val 573988389 ecr 3961177660], length 0 in slot1/tmm1 lis=_wcard_tunnel_/Common/ose-tunnel
      08:08:06.934310 IP 10.214.1.23.28277 > 10.214.1.102: VXLAN, flags [I] (0x08), vni 0
      IP 10.128.2.10.37542 > 10.130.0.27.http: Flags [.], ack 923, win 251, options [nop,nop,TS val 3961177661 ecr 573988389], length 0 out slot1/tmm0 lis=_wcard_tunnel_/Common/ose-tunnel
      
    • Do a tcpdump of the overlay network.

      tcpdump -i <name-of-BIG-IP-VXLAN-tunnel>
      

      Example showing traffic on the overlay network; at minimum, you should see BIG-IP health monitors for the Pod IP addresses.
      admin@(BIG-IP)(cfg-sync Standalone)(Active)(/Common)(tmos)$ tcpdump -i ose-tunnel
      08:09:51.911667 IP 10.128.2.10.38036 > 10.130.0.27.http: Flags [.], ack 1, win 229, options [nop,nop,TS val 3961282640 ecr 574093366], length 0 out slot1/tmm0 lis=
      08:09:51.911672 IP 10.128.2.10.38036 > 10.130.0.27.http: Flags [P.], seq 1:8, ack 1, win 229, options [nop,nop,TS val 3961282640 ecr 574093366], length 7 out slot1/tmm0 lis=
      08:09:51.913161 IP 10.130.0.27.http > 10.128.2.10.38036: Flags [.], ack 8, win 219, options [nop,nop,TS val 574093369 ecr 3961282640], length 0 in slot1/tmm0 lis=
      08:09:51.913265 IP 10.130.0.27.http > 10.128.2.10.38036: Flags [P.], seq 1:922, ack 8, win 219, options [nop,nop,TS val 574093369 ecr 3961282640], length 921 in slot1/tmm0 lis=
      
  4. In a TMOS shell, view the VLAN statistics.

    • Underlay

      tmsh show /net vlan <name_of_vlan_used_for_VTEP>
      

      Example
      admin@(BIG-IP)(cfg-sync Standalone)(Active)(/Common)(tmos)$ show /net vlan ocpvlan
      -------------------------------------
      Net::Vlan: ocpvlan
      -------------------------------------
      Interface Name      ocpvlan
      Mac Address (True)  00:0c:29:fe:f9:4e
      MTU                 1500
      Tag                 4094
      Customer-Tag
        -----------------------
        | Net::Vlan-Member: 1.1
        -----------------------
        | Tagged    no
        | Tag-Mode  none
           -------------------------------------------------------------
           | Net::Interface
           | Name  Status   Bits   Bits   Pkts  Pkts  Drops  Errs  Media
           |                  In    Out     In   Out
           -------------------------------------------------------------
           | 1.1       up  52.8G  17.0G  14.6M  7.4M      0     0   none
      
    • Overlay

      tmsh show /net vlan <name_of_VXLAN_tunnel_on_BIG-IP>
      

      Example
      admin@(BIG-IP)(cfg-sync Standalone)(Active)(/Common)(tmos)$ show /net tunnels tunnel ose-tunnel
      -------------------------------------
      Net::Tunnel: ose-tunnel
      -------------------------------------
      Incoming Discard Packets            0
      Incoming Error Packets              0
      Incoming Unknown Proto Packets      0
      Outgoing Discard Packets            0
      Outgoing Error Packets              0
      HC Incoming Octets               1.8G
      HC Incoming Unicast Packets     10.2M
      HC Incoming Multicast Packets       0
      HC Incoming Broadcast Packets       5
      HC Outgoing Octets               1.8G
      HC Outgoing Unicast Packets     10.2M
      HC Outgoing Multicast Packets   91.6K
      HC Outgoing Broadcast Packets   92.7K
      
  5. In a TMOS shell, view the MAC address entries for the OSE tunnel. This will show the mac address and IP addresses of all of the OpenShift endpoints.

    tmsh show /net fdb tunnel <name_of_VXLAN_tunnel on BIG-IP>
    

    Example
    admin@(BIG-IP)(cfg-sync Standalone)(Active)(/Common)(tmos)$ show /net fdb tunnel ose-tunnel
    -------------------------------------------------------------
    Net::FDB
    Tunnel      Mac Address        Member                 Dynamic
    -------------------------------------------------------------
    ose-tunnel  0a:58:0a:82:00:1b  endpoint:10.214.1.102  yes
    ose-tunnel  0a:58:0a:82:00:21  endpoint:10.214.1.102  yes
    ose-tunnel  0a:58:0a:82:00:25  endpoint:10.214.1.102  yes
    
  6. In a TMOS shell, view the ARP entries.

    This will show all of the ARP entries; you should see the VTEP entries on the ocpvlan and the Pod IP addresses on ose-tunnel.

    admin@(BIG-IP)(cfg-sync Standalone)(Active)(/Common)(tmos)$ show /net arp
    ------------------------------------------------------------------------------------------
    Net::Arp
    Name          Address       HWaddress          Vlan                Expire-in-sec  Status
    ------------------------------------------------------------------------------------------
    10.130.0.27   10.130.0.27   0a:58:0a:82:00:1b  /Common/ose-tunnel  224            resolved
    10.130.0.33   10.130.0.33   0a:58:0a:82:00:21  /Common/ose-tunnel  220            resolved
    10.130.0.37   10.130.0.37   0a:58:0a:82:00:25  /Common/ose-tunnel  222            resolved
    10.214.1.100  10.214.1.100  00:0c:29:c8:4c:dc  /Common/ocpvlan     220            resolved
    10.214.1.101  10.214.1.101  00:0c:29:8d:ac:42  /Common/ocpvlan     220            resolved
    10.214.1.102  10.214.1.102  00:0c:29:cd:ba:44  /Common/ocpvlan     220            resolved