Debug Sidecar

Overview

The TMM Proxy Pod’s debug sidecar provides a set of command line tools for obtaining low-level, diagnostic data and statistics about the Service Proxy Traffic Management Microkernel (TMM). The debug sidecar deploys by default with the BIG-IP Controller.

Command line tools

The table below lists and describes the available command line tools:

Tool Description
tmctl Displays various TMM traffic processing statistics, such as pool and virtual server connections.
bdt_cli Displays TMM networking information such as ARP and route entries. See the bdt_cli section below.
mrfdb Enables reading and writing dSSM database records. See the mrfdb section below.
configview Displays Custom Resource (CR) configuration objects using their logged UUID.
qkview Creates a diagnostic data TAR file for F5 support. See the Qkview section below.

Connecting to the sidecar

To connect to the debug sidecar and begin gathering diagnostic information, use the commands below.

  1. Connect to the debug sidecar:

    In this example, the debug sidecar is in the cnf-gateway Project:

    oc exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash

  2. Execute one of the available diagnostic commands:

    In this example, ping is used to test connectivity to a remote host with IP address 192.168.10.100:

    ping 192.168.10.100

    PING 192.168.10.100 (192.168.10.100): 56 data bytes
64 bytes from 192.168.10.100: icmp_seq=0 ttl=64 time=0.067 ms
64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.067 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=64 time=0.067 ms
64 bytes from 192.168.10.100: icmp_seq=3 ttl=64 time=0.067 ms

  3. Type Exit to leave the debug sidecar.

Command examples

tmctl

Use the tmctl tool to query Service Proxy TMM for application traffic processing statistics.

Virtual server connections

To view virtual server connection statistics run the following command:

Client side statstics

tmctl -d blade virtual_server_stat -s name,clientside.tot_conns

Server side statstics

tmctl -d blade virtual_server_stat -s name,serverside.tot_conns

bdt_cli

Use the bdt_cli tool to query the Service Proxy TMM for networking data.

Commands:

  • arp - Get ARP routes and their status
  • check - Get TMM Check Magic
  • completion - Generates the autocompletion script for the specialized shell
  • connection - Get Connection List
  • help - Help about any command
  • l2forward - Get L2 Forwarding entries
  • logLevel - Set the TMM log level
  • route - Get Route List

Command example:

  1. Connect to the debug sidecar:

    oc exec -it deploy/f5-tmm -c debug -n <project> -- bash

    In this example, the debug sidecar is in the cnf-gateway Project:

    oc exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash

  2. Connect to TMM:

    bdt_cli -u -s tmm0:8850 [command]

  3. Example routes:

     bdt_cli -u -s tmm0:8850 route 

    routeType:1 isIpv6:false destNet:{ip:{addr:<none>, rd:0} pl:0} gw:{ip:{addr:10.59.147.121, rd:0}} gwType:1 interface:external
routeType:1 isIpv6:false destNet:{ip:{addr:10.19.148.120, rd:0} pl:29} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:external
routeType:1 isIpv6:false destNet:{ip:{addr:192.168.202.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:internal
routeType:0 isIpv6:false destNet:{ip:{addr:169.254.1.1, rd:0} pl:32} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:eth0
routeType:1 isIpv6:false destNet:{ip:{addr:169.254.0.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:tmm

  4. To set the logging level of f5-tmm container to Error, run the following command:

    bdt_cli logLevel -l 5

    Following are the logging levels listed in the order of message severity.
    1-Debug, 2-Informational, 3-Notice (Default), 4-Warning, 5-Error, 6-Critical, 7-Alert, 8-Emergency

    Note: The logging levels generally log messages from the lower severity levels.

mrfdb

The mrfdb utility enables reading and writing dSSM database records. The mrfdb tool queries the dSSM Database Sentinel Pod, sending commands to the dssmmaster DB, and relaying the response back to the debug sidecar. The mrfdb command uses these four subcomands:

  • The IP address of the dSSM Sentinel service to be queried.
  • The serverName designating the dSSM server-farm controlled by the dssmmaster DB.
  • The type designating the command category: dns46, cgnat, custom.
  • The command that is specific to the chosen type (category).

Command example:

  1. Login to the debug sidecar container:

    In this example, the debug sidecar is in the cnf-gateway namespace.

    oc exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash

  2. Run the mrfdb utility:

    In this example, the mrfdb utility queries for all DB records.

    mrfdb -ipport=f5-dssm-sentinel:26379 -serverName=server -displayAllBins

configview

Use the configview utility to show configuration objects created by the installed CNF CRs.

  1. View the TMM deployment logs, and grep for UUID events:

    In this example, TMM is in the cnf-gateway Project:

    oc logs deploy/f5-tmm -c f5-tmm -n cnf-gateway | grep UUID

    In this example, the first log UUID cnf-gateway-net-external-vlan will be used to query with configview.

    <134>Jan 1 1:10:11 f5-tmm-7d5b489c5b-fffgt tmm1[36]: 01010058:6: audit log: action: CREATE; UUID: cnf-gateway-net-external-vlan; event: declTmm.vlan; Error: No error

  2. Connect to the debug sidecar:

    In this example, the debug sidecar is in the cnf-gateway Project:

    oc exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash

  3. Execute the configview utility:

    configview uuid cnf-gateway-net-external-vlan

    The example output displays the CR parameters and values.

    request:[declTmm.vlan]:{name:"external" id:"cnf-gateway-net-external-vlan" tag:3350 mtu:9000 tagged_interfaces:"1.2"}

Qkview

The qkview utility collects diagnostic and logging information from the f5-tmm container, and stores the data in a Linux TAR file. If you enabled the Fluentd Logging collector, run the qkview utility on f5-fluentd container to gather log files from all of the CNFs Pods. Qkview files are typically generated and sent to F5 for further analysis. Use the steps below to run the qkview utility, and copy the file to your local workstation.

  1. Obtain the name of the TMM Proxy Pod:

    In this example, the TMM Proxy Pod is in the cnf-gateway namespace.

    oc get pods --selector app=f5-tmm -n cnf-gateway

    In this example, the TMM Proxy Pod is named f5-tmm-77b95f699f-5zv8n.

    NAME                      READY   STATUS    
f5-tmm-77b95f699f-5zv8n   5/5     Running

  2. Connect to the debug sidecar:

    oc exec -it f5-tmm-77b95f699f-5zv8n -c debug -n cnf-gateway -- bash

    The shell prompt should contain the name of the TMM Proxy Pod.

    debuguser@f5-tmm-77b95f699f-5zv8n:~$

  3. Run the qkview utility:

    qkview

    The command output should indicate the file was created and saved.

    Diagnostic snapshot file saved: qkview.20220511-185024.tar.gz

  4. Copy the file to your workstation:

    oc cp <namespace>/<podname>:<file> ./<file> -c debug

    In this example, the qkview named qkview.20220511-185024.tar.gz is copied to the local workstation.

    oc cp cnf-gateway/f5-tmm-77b95f699f-k8bfh:qkview.20220511-185024.tar.gz \
./qkview.20220511-185024.tar.gz -c debug

  5. Obtain the name of the Fluent logging Pod:

    oc get pods --selector run=f5-fluentd -n cnf-gateway

    In this example the Fluentd logging Pod is named f5-toda-fluentd-84f96b6757-v5wj9.

    f5-toda-fluentd-84f96b6757-v5wj9       1/1     Running

  6. Connect to the Fluentd logging Pod:

    oc exec -it f5-toda-fluentd-84f96b6757-v5wj9 -n cnf-gateway -- bash

  7. Run the qkview utility:

    qkview

    The command output should indicate the file was created and saved.

    Diagnostic snapshot file is saved: qkview.20220511-195129.tar.gz

  8. Copy the file to your workstation:

    oc cp <namespace>/<podname>:<file> ./<file>

    In this example, the qkview named qkview.20220511-195129.tar.gz is copied to the local workstation.

    oc cp cnf-gateway/ff5-toda-fluentd-84f96b6757-v5wj:qkview.20220511-195129.tar.gz \
./qkview.20220511-195129.tar.gz

Disabling the sidecar

The TMM debug sidecar installs by default with the CNFs Controller. You can disable the debug sidecar by setting the debug.enabled parameter to false in the BIG-IP Controller Helm values file:

debug:
  enabled: false

Feedback

Provide feedback to improve this document by emailing cnfdocs@f5.com.