Debug Sidecar

Overview

The Service Proxy Pod’s debug sidecar provides a set of command line tools for obtaining low-level, diagnostic data and statistics about the Service Proxy Traffic Management Microkernel (TMM). The debug sidecar deploys by default with the Ingress Controller.

Command line tools

The table below lists and describes the available command line tools:

Tool Description
tmctl Displays various TMM traffic processing statistics, such as pool and virtual server connections.
core-tmm Creates a diagnostic core file of the TMM process. Refer to the TMM core files guide.
bdt_cli Displays TMM networking information such as ARP and route entries.
qkview Creates a diagnostic data TAR file for F5 support. See the Qkview section below.
configviewer Displays a log of the configuration objects created and deleted using SPK Custom Resources (CRs).
tcpdump Displays packets sent and received on the specified network interface.
ping Send ICMP ECHO_REQUEST packets to remote hosts.
traceroute Displays the packet route in hops to a remote host.

Note: Type man f5-tools in the debug container to get a full list of TMM specific commands.

Connecting to the sidecar

To connect to the debug sidecar and begin gathering diagnostic information, use the commands below.

  1. Connect to the debug sidecar:

    oc exec -it deploy/f5-tmm -c debug -n <project> -- bash
    

    In this example, the debug sidecar is in the spk-ingress Project:

    oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
    
  2. Execute one of the available diagnostic commands:

    In this example, ping is used to test connectivity to a remote host with IP address 192.168.10.100:

    ping 192.168.10.100
    
    PING 192.168.10.100 (192.168.10.100): 56 data bytes
    64 bytes from 192.168.10.100: icmp_seq=0 ttl=64 time=0.067 ms
    64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.067 ms
    64 bytes from 192.168.10.100: icmp_seq=2 ttl=64 time=0.067 ms
    64 bytes from 192.168.10.100: icmp_seq=3 ttl=64 time=0.067 ms
    
  3. Type Exit to leave the debug sidecar.

Command examples

tmctl

Use the tmctl tool to query Service Proxy TMM for application traffic processing statistics.

Virtual server connections

To view virtual server connection statistics run the following command:

tmctl -f /var/tmstat/blade/tmm0 virtual_server_stat -s name,serverside.tot_conns

Pool member connections

To view pool member connection statistics run the following command:

tmctl -f /var/tmstat/blade/tmm0 pool_member_stat -s pool_name,serverside.tot_conns

bdt_cli

Use the bdt_cli tool to query the Service Proxy TMM for networking data.

  1. Connect to TMM referencing the gRPC channel SSL/TL certificates and key:

    bdt_cli -tls=true -use_fqdn=true -server_addr=tmm0:8850 \
            -ca_file=/etc/ssl/certs/ca_root.crt \
            -client_crt=/etc/ssl/certs/f5-ing-demo-f5ingress.crt \
            -client_key=/etc/ssl/private/f5-ing-demo-f5ingress.key
    
  2. Once connected, select an entry from the menu:

    Enter the request type(number or string):
    1. check
    2. arp
    3. connection
    4. route
    5. exit
    

    In this example, the 2. arp entry is selected:

    Enter ArpRequest(override fields as necessary, defaults are listed here):
    e.g. {}
    msg:  name:"10.244.2.1" ip_addr:"10.244.2.1" mac_addr:"0a:06:be:84:6d:ca" vlan:"eth0" expire:87 status:"resolved"
    msg:  name:"10.244.2.101" ip_addr:"10.244.2.101" mac_addr:"92:b5:9d:c5:92:69" vlan:"eth0" expire:91 status:"resolved"
    msg:  name:"169.254.0.1" ip_addr:"169.254.0.1" mac_addr:"00:01:23:45:67:00" vlan:"tmm" status:"permanent"
    

configviewer

Use the configviewer utility to show events related to configuring Custom Resources.

  1. You must set the CONFIG_VIEWER_ENABLE parameter to true when deploying the Ingress Controller. For example:

    tmm:
    
      customEnvVars:
        - name: CONFIG_VIEWER_ENABLE
          value: "true"
    
  2. After deploying a Custom Resource (CR), you can view the current configuration event with the following command:

    Note: The example respresents a portion of the TMM configuration.

    configviewer --ipport=tmm0:11211 --displayall
    
    GetAll Connect!
    GetAll Connect Complete!
    pattern: 006f40782e*
    binlookup config_viewer_bin
     Query: get/th /6552fc31.0/*
    
    --------------------------------------------------------------------------------------------------
    Config for pool_member_list updated at <some date / time>
    {
            "name": "apps-nginx-crd-pool-member-list",
            "id": "apps-nginx-crd-pool-member-list",
            "members": [
                    "apps-nginx-crd-pool-member-10.244.1.22",
                    "apps-nginx-crd-pool-member-10.244.1.23",
                    "apps-nginx-crd-pool-member-10.244.2.21"
            ]
    }
    

Persisting files

Some diagnostic tools such as qkview and tcpdump produce diagnostic files requiring further analysis by F5. Diagnostic files can be saved to a filesystem by configuring the debug container’s persistence parameter in the Ingress Controller Helm values file. Ensure a Kubernetes PersistentVolume is available for binding. Use the steps below to configure the debug container’s persistence parameter.

  1. Verify a PersistentVolume is available for the debug container:

    oc get storageclass
    
    NAME                  PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   
    managed-nfs-storage   storage.io/nfs   Delete          Immediate          
    
  2. Set the persistence.enabled parameter to true, and configure the storageClass name:

    Note: In this example, the storageClass name managed-nfs-storage was gathered in the previous step:

    debug:
    
      persistence:
        enabled: true
        storageClass: "managed-nfs-storage"
        accessMode: ReadWriteOnce
        size: 1Gi
    
  3. After you deploy the Ingress and Service Proxy Pods, find the bound persistent volume claim:

    oc get pv -n <project> | grep f5-debug-sidecar
    

    In this example, the pv is in the spk-ingress Project:

    oc get pv -n spk-ingress | grep f5-debug-sidecar
    
    pvc-c4d5a2bc-6b74-4124-a19c-d9213966253a   1Gi   spk-ingress/f5-debug-sidecar     manages-nfs-storage
    
  4. Diagnostic files placed in the debug sidecar’s /shared directory, will map to the Path from the PersistenceVolume claim object:

    The directory path exists on the worker node of the debug container.

    oc describe pv pvc-c4d5a2bc-6b74-4124-a19c-d9213966253a | grep Path:
    
    Path: /opt/local-path-provisioner/pvc-c4dabc-6b74-4124-a19c-d92139653a_ingress_f5-debug-sidecar
    

Qkview

The qkview utility collects diagnostic and logging information from the f5-tmm container, and stores the data in a Linux TAR file. Qkview files are typically generated and sent to F5 for further analysis. Use the steps below to run the qkview utility on the Service Proxy TMM Pod’s debug container, and copy the file to your local workstation. If you enabled the Fluentd Logging collector, also run the qkview utility on f5-fluentd container.

  1. Switch to the Service Proxy TMM Pod Project:

    oc project <project>
    

    In this example, the spk-ingress Project is selected:

    oc project spk-ingress
    
  2. Obtain the name of the Service Proxy TMM Pod:

    oc get pods --selector app=f5-tmm
    

    In this example, the Service Proxy TMM Pod name is f5-tmm-79df567d45-ssjv9:

    NAME                      READY   STATUS  
    f5-tmm-79df567d45-ssjv9   5/5     Running
    
  3. Set the Service Proxy TMM Pod name as an environment variable:

    In this example, the environment variable is named TMM_POD:

    TMM_POD=f5-tmm-79df567d45-ssjv9
    
  4. Open a remote shell to the TMM Pod’s debug container:

    oc rsh -c debug $TMM_POD bash   
    

    The shell will display the name of the Service Proxy TMM Pod:

    [root@f5-tmm-79df567d45-ssjv9 /]#
    
  5. Change into the /shared directory mapped to the persistent volume:

    cd /shared
    
  6. Run the qkview utility:

    qkview
    
  7. The qkview file appears similar to the following:

    qkview.20210219-223559.tar.gz
    
  8. Type Exit to exit the debug container.

  9. Copy the Qkview file to your local workstation:

    oc rsync -c debug $TMM_POD:/shared/<file> .
    

    In this example, the /shared/qkview.20210219-223559.tar.gz Qkview file is copied to the local workstation:

    oc rsync -c debug $TMM_POD:/shared/qkview.20210219-223559.tar.gz .
    
  10. Switch to the Fluentd logging Pod project:

    oc project <project>
    

    In this example, the spk-utilities Project is selected:

    oc project spk-utilities
    
  11. Obtain the name of the Fluentd logging Pod:

    oc get pods --selector run=f5-fluentd
    

    In this example, the Fluentd logging Pod is named f5-toda-fluentd-768b475dc-pk6bp:

    NAME                              READY   STATUS 
    f5-toda-fluentd-768b475dc-pk6bp   1/1     Running
    
  12. Set the Fluentd logging Pod name as an environment variable:

    FLUENTD_POD=f5-toda-fluentd-768b475dc-pk6bp
    
  13. Connect to the f5-fluentd container:

    oc rsh deploy/f5-toda-fluentd bash
    
  14. Change into the /var/log/f5 directory mapped to the persistent volume:

    cd /var/log/f5
    
  15. Run the qkview utility:

    qkview
    
  16. The Qkview file appears similar to the following, on the worker node’s mapped filesystem:

    qkview.20210219-273529.tar.gz
    
  17. Type Exit to exit the f5-fluent container.

  18. Copy the Qkview file to the local filesystem:

    In this example, the file /shared/qkview.20210730-231942.tar.gz is copied to the local workstation:

    oc rsync $FLUENTD_POD:/shared/qkview.20210730-231942.tar.gz .
    

Disabling the sidecar

The TMM debug sidecar installs by default with the Ingress Controller. You can disable the debug sidecar by setting the debug.enabled parameter to false in the Ingress Controller Helm values file:

debug:
  enabled: false

Feedback

Provide feedback to improve this document by emailing spkdocs@f5.com.

Supplemental