Debug Sidecar

Overview

The Service Proxy Pod’s debug sidecar provides a set of command line tools for obtaining low-level, diagnostic data and statistics about the Service Proxy Traffic Management Microkernel (TMM). The debug sidecar deploys by default with the SPK Controller.

Command line tools

The table below lists and describes the available command line tools:

Tool Description
tmctl Displays various TMM traffic processing statistics, such as pool and virtual server connections.
core-tmm Creates a diagnostic core file of the TMM process.
bdt_cli Displays TMM networking information such as ARP and route entries. See the bdt_cli section below.
tmm_cli Sets the TMM logging level. See the tmm_cli section below.
mrfdb Enables reading and writing dSSM database records. See the mrfdb section below.
configviewer Displays a log of the configuration objects created and deleted using SPK Custom Resources (CRs). See the configviewer section below.
tcpdump Displays packets sent and received on the specified network interface.
ping Send ICMP ECHO_REQUEST packets to remote hosts.
traceroute Displays the packet route in hops to a remote host.

Note: Type man f5-tools in the debug container to get a full list of TMM specific commands.

Connecting to the sidecar

To connect to the debug sidecar and begin gathering diagnostic information, use the commands below.

  1. Connect to the debug sidecar:

    In this example, the debug sidecar is in the spk-ingress Project:

    oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
    
  2. Execute one of the available diagnostic commands:

    In this example, ping is used to test connectivity to a remote host with IP address 192.168.10.100:

    ping 192.168.10.100
    
    PING 192.168.10.100 (192.168.10.100): 56 data bytes
    64 bytes from 192.168.10.100: icmp_seq=0 ttl=64 time=0.067 ms
    64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.067 ms
    64 bytes from 192.168.10.100: icmp_seq=2 ttl=64 time=0.067 ms
    64 bytes from 192.168.10.100: icmp_seq=3 ttl=64 time=0.067 ms
    
  3. Type Exit to leave the debug sidecar.

Command examples

tmctl

Use the tmctl tool to query Service Proxy TMM for application traffic processing statistics.

  1. Connect to the debug sidecar:

    oc exec -it deploy/f5-tmm -c debug -n <project> -- bash
    

    In this example, the debug sidecar is in the spk-ingress Project:

    oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
    
  2. To view virtual server connection statistics run the following command:

    tmctl -d blade virtual_server_stat -s name,clientside.tot_conns
    
  3. To view pool member connection statistics run the following command:

    tmctl -d blade pool_member_stat -s pool_name,serverside.tot_conns
    

bdt_cli

Use the bdt_cli tool to query the Service Proxy TMM for networking data.

Commands:

  • arp - Get ARP routes and their status.
  • check - Get TMM Check Magic.
  • completion - Generate the autocompletion script for the specialized shell.
  • connection - Get Connection List.
  • help - Help about any command.
  • l2forward - Get L2 Forwarding entries.
  • route - Get Route List.

Command example:

  1. Connect to the debug sidecar:

    oc exec -it deploy/f5-tmm -c debug -n <project> -- bash 
    

    In this example, the debug sidecar is in the spk-ingress Project:

    oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
    
  2. Connect to TMM:

    bdt_cli -u -s tmm0:8850 [command] 
    
  3. Example of showing routes:

    bdt_cli -u -s tmm0:8850 route
    
    routeType:1 isIpv6:false destNet:{ip:{addr:<none>, rd:0} pl:0} gw:{ip:{addr:10.59.147.121, rd:0}} gwType:1 interface:external
    routeType:1 isIpv6:false destNet:{ip:{addr:10.19.148.120, rd:0} pl:29} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:external
    routeType:1 isIpv6:false destNet:{ip:{addr:192.168.202.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:internal
    routeType:0 isIpv6:false destNet:{ip:{addr:169.254.1.1, rd:0} pl:32} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:eth0
    routeType:1 isIpv6:false destNet:{ip:{addr:169.254.0.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:tmm
    

tmm_cli

By default, the f5-tmm container logs events at the Notice level. You can use the tmm_cli command to modify the logging level. The logging levels are listed below in the order of message severity. More severe levels generally log messages from the lower severity levels as well.

1-Debug, 2-Informational, 3-Notice, 4-Warning, 5-Error, 6-Critical, 7-Alert, 8-Emergency

  1. Connect to the debug sidecar:

    oc exec -it deploy/f5-tmm -c debug -n <project> -- bash
    

    In this example, the debug sidecar is in the spk-ingress Project:

    oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
    
  2. To set the f5-tmm container’s logging level to Debug, run the following command:

    tmm_cli -logLevel 1
    
    ok
    

    The f5-tmm container will log an event message simlilar to the following:

    Set bigdb var 'log.tmm.level'='Debug'
    

mrfdb

The mrfdb utility enables reading and writing dSSM database records. The mrfdb tool queries the dSSM Database Sentinel Pod, sending commands to the dssmmaster DB, and relaying the response back to the debug sidecar. The mrfdb command uses these four subcomands:

  • The IP address of the dSSM Sentinel service to be queried.
  • The serverName designating the dSSM server-farm controlled by the dssmmaster DB.
  • The type designating the command category: dns46, cgnat, custom.
  • The command that is specific to the chosen type (category).

Command example:

  1. Obtain the IP address of the dSSM Sentinel:

    In this example, dSSM is installed in the spk-utilities Project.

    oc get svc -n spk-utilities
    

    In this example, the Sentinel IP address is 10.203.180.204.

    NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)  
    f5-dssm-db         ClusterIP   10.108.254.57    <none>        6379/TCP 
    f5-dssm-sentinel   ClusterIP   10.103.180.204   <none>        26379/TCP
    
  2. Login to the debug sidecar container:

    In this example, the debug sidecar is in the spk-ingress Project.

    oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
    
  3. Run the mrfdb utility:

    In this example, the mrfdb utility queries for all DB records.

    mrfdb -ipport=10.103.180.204:26379 -serverName=server -displayAllBins
    

Detailed examples:

For detailed mrfdb usage examples, refer to the following guides:

configviewer

Use the configviewer utility to show events related to installing SPK CRs.

  1. You must set the CONFIG_VIEWER_ENABLE parameter to true when deploying the SK Controller. For example:

    tmm:
    
      customEnvVars:
        - name: CONFIG_VIEWER_ENABLE
          value: "true"
    
  2. Connect to the debug sidecar:

    oc exec -it deploy/f5-tmm -c debug -n <project> -- bash
    

    In this example, the debug sidecar is in the spk-ingress Project:

    oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
    
  3. After deploying a Custom Resource (CR), you can view the current configuration event with the following command:

    Note: The example respresents a portion of the TMM configuration.

    configviewer --ipport=tmm0:11211 --displayall
    
    GetAll Connect!
    GetAll Connect Complete!
    pattern: 006f40782e*
    binlookup config_viewer_bin
     Query: get/th /6552fc31.0/*
    
    --------------------------------------------------------------------------------------------------
    Config for pool_member_list updated at <some date / time>
    {
            "name": "apps-nginx-crd-pool-member-list",
            "id": "apps-nginx-crd-pool-member-list",
            "members": [
                    "apps-nginx-crd-pool-member-10.244.1.22",
                    "apps-nginx-crd-pool-member-10.244.1.23",
                    "apps-nginx-crd-pool-member-10.244.2.21"
            ]
    }
    

Persisting files

Some diagnostic tools such as tcpdump produce files that require further analysis by F5. When you install the SPK Controller, you can configure the debug.persistence Helm parameter to ensure diagnostic files created in the debug sidecar container are saved to a filesystem. Use the steps below to verify a PersistentVolume is available, and to configure persistence.

  1. Verify a StoraceClass is available for the debug container:

    oc get storageclass
    
    NAME                  PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   
    managed-nfs-storage   storage.io/nfs   Delete          Immediate          
    
  2. Set the persistence.enabled parameter to true, and configure the storageClass name:

    Note: In this example, managed-nfs-storage value is obtained from the NAME field in step 1:

    debug:
    
      persistence:
        enabled: true
        storageClass: "managed-nfs-storage"
        accessMode: ReadWriteOnce
        size: 1Gi
    
  3. After you deploy the Controller and Service Proxy Pods, find the bound PersistentVolume:

    oc get pv | grep f5-debug-sidecar
    

    In this example, the pv is Bound in the spk-ingress Project as expected:

    pvc-42a5ef7-5c5f-4518-930f-851abf32c67   1Gi   Bound  spk-ingress/f5-debug-sidecar  managed-nfs-storage
    
  4. Use the PersistentVolume ID to find the Server name and the Path, or location on the cluster node where diagnostic files are storeed.

    _images/spk_warn.png Important: Files must be placed in the the debug sidecar’s /shared directory to be persisted.

    oc describe pv <pv_id> | grep -iE 'path|server'
    

    In this example, the PersistentVolume ID is pvc-42a5ef7-5c5f-4518-930f-851abf32c67:

    oc describe pv pvc-42a5ef7-5c5f-4518-930f-851abf32c67 | grep -iE 'path|server'
    

    The Server and Path information will resemble the following:

    Server:  provisioner.ocp.f5.com
    Path:    /opt/local-path-provisioner/pvc-42a5ef7-5c5f-4518-930f-851abf32c67_ingress_f5-debug-sidecar
    

Disabling the sidecar

The TMM debug sidecar installs by default with the Controller. You can disable the debug sidecar by setting the debug.enabled parameter to false in the Controller Helm values file:

debug:
  enabled: false

Feedback

Provide feedback to improve this document by emailing spkdocs@f5.com.

Supplemental