Debug Sidecar

Overview

The TMM Proxy Pod’s debug sidecar provides a set of command line tools for obtaining low-level, diagnostic data and statistics about the Service Proxy Traffic Management Microkernel (TMM). The debug sidecar deploys by default with the BIG-IP Controller.

Command line tools

The table below lists and describes the available command line tools:

Tool Description
tmctl Displays various TMM traffic processing statistics, such as pool and virtual server connections.
bdt_cli Displays TMM networking information such as ARP and route entries.
tmm_cli Sets the TMM logging level. For an example, see the tmm_cli section below.
mrfdb Performs read and write dSSM Database operations.
configviewer Displays a log of the configuration objects created and deleted using SPK Custom Resources (CRs).
qkview Creates a diagnostic data TAR file for F5 support. See the Qkview section below.

Connecting to the sidecar

To connect to the debug sidecar and begin gathering diagnostic information, use the commands below.

  1. Connect to the debug sidecar:

    In this example, the debug sidecar is in the cnf-gateway Project:

    kubectl exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash
    
  2. Execute one of the available diagnostic commands:

    In this example, ping is used to test connectivity to a remote host with IP address 192.168.10.100:

    ping 192.168.10.100
    
    PING 192.168.10.100 (192.168.10.100): 56 data bytes
    64 bytes from 192.168.10.100: icmp_seq=0 ttl=64 time=0.067 ms
    64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.067 ms
    64 bytes from 192.168.10.100: icmp_seq=2 ttl=64 time=0.067 ms
    64 bytes from 192.168.10.100: icmp_seq=3 ttl=64 time=0.067 ms
    
  3. Type Exit to leave the debug sidecar.

Command examples

tmctl

Use the tmctl tool to query Service Proxy TMM for application traffic processing statistics.

Virtual server connections

To view virtual server connection statistics run the following command:

Client side statstics

tmctl -d blade virtual_server_stat -s name,clientside.tot_conns

Server side statstics

tmctl -d blade virtual_server_stat -s name,serverside.tot_conns

bdt_cli

Use the bdt_cli tool to query the Service Proxy TMM for networking data.

  1. Connect to TMM referencing the gRPC channel SSL/TL certificates and key:

    bdt_cli -tls=true -use_fqdn=true -server_addr=tmm0:8850 \
            -ca_file=/etc/ssl/certs/ca_root.crt \
            -client_crt=/etc/ssl/certs/f5-ing-demo-f5ingress.crt \
            -client_key=/etc/ssl/private/f5-ing-demo-f5ingress.key
    
  2. Once connected, enter a number representing the network data of interest:

    Enter the request type(number or string):
    1. check
    2. arp
    3. connection
    4. route
    5. exit
    

    The output will resemble the following:

    "2" looks like a number.
    Enter ArpRequest(override fields as necessary, defaults are listed here):
    e.g. {}
    
  3. Select the Enter key again to view the networking data:

     name:169.254.0.254 ipAddr:169.254.0.254 macAddr:00:01:23:45:67:fe vlan:tmm expire:0 status:permanent
     name:169.254.0.253 ipAddr:169.254.0.253 macAddr:00:98:76:54:32:10 vlan:tmm expire:0 status:permanent
     name:169.254.0.1 ipAddr:169.254.0.1 macAddr:00:01:23:45:67:00 vlan:tmm expire:0 status:permanent
     name:10.244.1.98 ipAddr:10.244.1.98 macAddr:22:22:fe:6d:59:e1 vlan:eth0 expire:0 status:permanent
     name:10.20.200.210 ipAddr:10.20.200.210 macAddr:96:b3:23:d4:7c:69 vlan:net1 expire:0 status:permanent
    

tmm_cli

By default, the f5-tmm container logs events at the Notice level. You can use the tmm_cli command to modify the logging level.

The logging levels listed below generally log messages from the lower severity levels as well.

1-Debug, 2-Informational, 3-Notice, 4-Warning, 5-Error, 6-Critical, 7-Alert, 8-Emergency

  1. Connect to the debug sidecar:

    In this example, the debug sidecar is in the cnf-gateway Project:

    kubectl exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash
    
  2. To set the f5-tmm container’s logging level to Debug, run the following command:

    tmm_cli -logLevel 1
    
    ok
    

    The f5-tmm container will log an event message simlilar to the following:

    Set bigdb var 'log.tmm.level'='Debug'
    

configviewer

Use the configviewer utility to show events related to installing CNF CRs.

  1. You must set the CONFIG_VIEWER_ENABLE parameter to true when deploying the BIG-IP Controller. For example:

    tmm:
    
      customEnvVars:
        - name: CONFIG_VIEWER_ENABLE
          value: "true"
    
  2. After deploying a Custom Resource (CR), you can view the current configuration event with the following command:

    Note: The example respresents a portion of the TMM configuration.

    configviewer --ipport=tmm0:11211 --displayall
    
    GetAll Connect!
    GetAll Connect Complete!
    pattern: 006f40782e*
    binlookup config_viewer_bin
     Query: get/th /6552fc31.0/*
    
    --------------------------------------------------------------------------------------------------
    Config for pool_member_list updated at <some date / time>
    {
            "name": "apps-nginx-crd-pool-member-list",
            "id": "apps-nginx-crd-pool-member-list",
            "members": [
                    "apps-nginx-crd-pool-member-10.244.1.22",
                    "apps-nginx-crd-pool-member-10.244.1.23",
                    "apps-nginx-crd-pool-member-10.244.2.21"
            ]
    }
    

Qkview

The qkview utility collects diagnostic and logging information from the f5-tmm container, and stores the data in a Linux TAR file. If you enabled the Fluentd Logging collector, run the qkview utility on f5-fluentd container to gather log files from all of the SPK Pods. Qkview files are typically generated and sent to F5 for further analysis. Use the steps below to run the qkview utility, and copy the file to your local workstation.

  1. Obtain the name of the TMM Proxy Pod:

    In this example, the TMM Proxy Pod is in the cnf-gateway namespace.

    kubectl get pods --selector app=f5-tmm -n cnf-gateway
    

    In this example, the TMM Proxy Pod is named f5-tmm-77b95f699f-5zv8n.

    NAME                      READY   STATUS    
    f5-tmm-77b95f699f-5zv8n   5/5     Running 
    
  2. Connect to the debug sidecar:

    kubectl exec -it f5-tmm-77b95f699f-5zv8n -c debug -n cnf-gateway -- bash
    

    The shell prompt should contain the name of the TMM Proxy Pod.

    debuguser@f5-tmm-77b95f699f-5zv8n:~$
    
  3. Run the qkview utility:

    qkview
    

    The command output should indicate the file was created and saved.

    Diagnostic snapshot file saved: qkview.20220511-185024.tar.gz
    
  4. Copy the file to your workstation:

    kubectl cp <namespace>/<podname>:<file> ./<file> -c debug
    

    In this example, the qkview named qkview.20220511-185024.tar.gz is copied to the local workstation.

    kubectl cp cnf-gateway/f5-tmm-77b95f699f-k8bfh:qkview.20220511-185024.tar.gz \
    ./qkview.20220511-185024.tar.gz -c debug
    
  5. Obtain the name of the Fluent logging Pod:

    kubectl get pods --selector run=f5-fluentd -n cnf-gateway
    

    In this example the Fluentd logging Pod is named f5-toda-fluentd-84f96b6757-v5wj9.

    f5-toda-fluentd-84f96b6757-v5wj9       1/1     Running
    
  6. Connect to the Fluentd logging Pod:

    kubectl exec -it f5-toda-fluentd-84f96b6757-v5wj9 -n cnf-gateway -- bash
    
  7. Run the qkview utility:

    qkview
    

    The command output should indicate the file was created and saved.

    Diagnostic snapshot file is saved: qkview.20220511-195129.tar.gz
    
  8. Copy the file to your workstation:

    kubectl cp <namespace>/<podname>:<file> ./<file> 
    

    In this example, the qkview named qkview.20220511-195129.tar.gz is copied to the local workstation.

    kubectl cp cnf-gateway/ff5-toda-fluentd-84f96b6757-v5wj:qkview.20220511-195129.tar.gz \
    ./qkview.20220511-195129.tar.gz
    

Disabling the sidecar

The TMM debug sidecar installs by default with the SPK Controller. You can disable the debug sidecar by setting the debug.enabled parameter to false in the BIG-IP Controller Helm values file:

debug:
  enabled: false

Feedback

Provide feedback to improve this document by emailing cnfdocs@f5.com.