Debug Sidecar¶
Overview¶
The TMM Proxy Pod’s debug sidecar provides a set of command line tools for obtaining low-level, diagnostic data and statistics about the Service Proxy Traffic Management Microkernel (TMM). The debug sidecar deploys by default with the BIG-IP Controller.
Command line tools¶
The table below lists and describes the available command line tools:
Tool | Description |
---|---|
tmctl | Displays various TMM traffic processing statistics, such as pool and virtual server connections. |
bdt_cli | Displays TMM networking information such as ARP and route entries. See the bdt_cli section below. |
mrfdb | Enables reading and writing dSSM database records. See the mrfdb section below. |
configview | Displays Custom Resource (CR) configuration objects using their logged UUID. |
qkview | Creates a diagnostic data TAR file for F5 support. See the Qkview section below. |
Connecting to the sidecar¶
To connect to the debug sidecar and begin gathering diagnostic information, use the commands below.
Connect to the debug sidecar:
In this example, the debug sidecar is in the cnf-gateway Project:
kubectl exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash
Execute one of the available diagnostic commands:
In this example, ping is used to test connectivity to a remote host with IP address 192.168.10.100:
ping 192.168.10.100
PING 192.168.10.100 (192.168.10.100): 56 data bytes 64 bytes from 192.168.10.100: icmp_seq=0 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=2 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=3 ttl=64 time=0.067 ms
Type Exit to leave the debug sidecar.
Command examples¶
tmctl¶
Use the tmctl tool to query Service Proxy TMM for application traffic processing statistics.
Virtual server connections
To view virtual server connection statistics run the following command:
Client side statstics
tmctl -d blade virtual_server_stat -s name,clientside.tot_conns
Server side statstics
tmctl -d blade virtual_server_stat -s name,serverside.tot_conns
bdt_cli¶
Use the bdt_cli tool to query the Service Proxy TMM for networking data.
Commands:
- arp - Get ARP routes and their status
- check - Get TMM Check Magic
- completion - Generate the autocompletion script for the specialized shell
- connection - Get Connection List
- help - Help about any command
- l2forward - Get L2 Forwarding entries
- logLevel - Set the TMM log level
- route - Get Route List
Command example:
Connect to the debug sidecar:
kubectl exec -it deploy/f5-tmm -c debug -n <project> -- bash
In this example, the debug sidecar is in the cnf-gateway Project:
kubectl exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash
Connect to TMM:
bdt_cli -u -s tmm0:8850 [command]
Example routes:
bdt_cli -u -s tmm0:8850 route
routeType:1 isIpv6:false destNet:{ip:{addr:<none>, rd:0} pl:0} gw:{ip:{addr:10.59.147.121, rd:0}} gwType:1 interface:external routeType:1 isIpv6:false destNet:{ip:{addr:10.19.148.120, rd:0} pl:29} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:external routeType:1 isIpv6:false destNet:{ip:{addr:192.168.202.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:internal routeType:0 isIpv6:false destNet:{ip:{addr:169.254.1.1, rd:0} pl:32} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:eth0 routeType:1 isIpv6:false destNet:{ip:{addr:169.254.0.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:tmm
To set the logging level of f5-tmm container to Error, run the following command:
bdt_cli logLevel -l 5
Following are the logging levels listed in the order of message severity.
1-Debug, 2-Informational, 3-Notice (Default), 4-Warning, 5-Error, 6-Critical, 7-Alert, 8-Emergency
Note: The logging levels generally log messages from the lower severity levels.
mrfdb¶
The mrfdb utility enables reading and writing dSSM database records. The mrfdb tool queries the dSSM Database Sentinel Pod, sending commands to the dssmmaster DB, and relaying the response back to the debug sidecar. The mrfdb command uses these four subcomands:
- The IP address of the dSSM Sentinel service to be queried.
- The serverName designating the dSSM server-farm controlled by the dssmmaster DB.
- The type designating the command category: dns46, cgnat, custom.
- The command that is specific to the chosen type (category).
Command example:
Login to the debug sidecar container:
In this example, the debug sidecar is in the cnf-gateway namespace.
kubectl exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash
Run the mrfdb utility:
In this example, the mrfdb utility queries for all DB records.
mrfdb -ipport=f5-dssm-sentinel:26379 -serverName=server -displayAllBins
configview¶
Use the configview utility to show configuration objects created by the installed CNF CRs.
View the TMM deployment logs, and grep for UUID events:
In this example, TMM is in the cnf-gateway Project:
kubectl logs deploy/f5-tmm -c f5-tmm -n cnf-gateway | grep UUID
In this example, the first log UUID cnf-gateway-net-external-vlan will be used to query with configview.
<134>Jan 1 1:10:11 f5-tmm-7d5b489c5b-fffgt tmm1[36]: 01010058:6: audit log: action: CREATE; UUID: cnf-gateway-net-external-vlan; event: declTmm.vlan; Error: No error
Connect to the debug sidecar:
In this example, the debug sidecar is in the cnf-gateway Project:
kubectl exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash
Execute the configview utility:
configview uuid cnf-gateway-net-external-vlan
The example output displays the CR parameters and values.
request:[declTmm.vlan]:{name:"external" id:"cnf-gateway-net-external-vlan" tag:3350 mtu:9000 tagged_interfaces:"1.2"}
Qkview¶
The qkview utility collects diagnostic and logging information from the f5-tmm container, and stores the data in a Linux TAR file. If you enabled the Fluentd Logging collector, run the qkview utility on f5-fluentd container to gather log files from all of the CNFs Pods. Qkview files are typically generated and sent to F5 for further analysis. Use the steps below to run the qkview utility, and copy the file to your local workstation.
Obtain the name of the TMM Proxy Pod:
In this example, the TMM Proxy Pod is in the cnf-gateway namespace.
kubectl get pods --selector app=f5-tmm -n cnf-gateway
In this example, the TMM Proxy Pod is named f5-tmm-77b95f699f-5zv8n.
NAME READY STATUS f5-tmm-77b95f699f-5zv8n 5/5 Running
Connect to the debug sidecar:
kubectl exec -it f5-tmm-77b95f699f-5zv8n -c debug -n cnf-gateway -- bash
The shell prompt should contain the name of the TMM Proxy Pod.
debuguser@f5-tmm-77b95f699f-5zv8n:~$
Run the qkview utility:
qkview
The command output should indicate the file was created and saved.
Diagnostic snapshot file saved: qkview.20220511-185024.tar.gz
Copy the file to your workstation:
kubectl cp <namespace>/<podname>:<file> ./<file> -c debug
In this example, the qkview named qkview.20220511-185024.tar.gz is copied to the local workstation.
kubectl cp cnf-gateway/f5-tmm-77b95f699f-k8bfh:qkview.20220511-185024.tar.gz \ ./qkview.20220511-185024.tar.gz -c debug
Obtain the name of the Fluent logging Pod:
kubectl get pods --selector run=f5-fluentd -n cnf-gateway
In this example the Fluentd logging Pod is named f5-toda-fluentd-84f96b6757-v5wj9.
f5-toda-fluentd-84f96b6757-v5wj9 1/1 Running
Connect to the Fluentd logging Pod:
kubectl exec -it f5-toda-fluentd-84f96b6757-v5wj9 -n cnf-gateway -- bash
Run the qkview utility:
qkview
The command output should indicate the file was created and saved.
Diagnostic snapshot file is saved: qkview.20220511-195129.tar.gz
Copy the file to your workstation:
kubectl cp <namespace>/<podname>:<file> ./<file>
In this example, the qkview named qkview.20220511-195129.tar.gz is copied to the local workstation.
kubectl cp cnf-gateway/ff5-toda-fluentd-84f96b6757-v5wj:qkview.20220511-195129.tar.gz \ ./qkview.20220511-195129.tar.gz
Disabling the sidecar¶
The TMM debug sidecar installs by default with the CNFs Controller. You can disable the debug sidecar by setting the debug.enabled
parameter to false
in the BIG-IP Controller Helm values file:
debug:
enabled: false
Feedback¶
Provide feedback to improve this document by emailing cnfdocs@f5.com.