Debug Sidecar¶
The Debug sidecar provides a set of command-line utilities for obtaining low-level, diagnostic data and statistics about the Service Proxy Traffic Management Microkernel (TMM).
Command-Line Utilities¶
The table below lists and describes the available command-line utilities.
| Utility | Description |
|---|---|
| tmctl | Displays various TMM traffic processing statistics, such as pool and virtual server connections. See the tmctl section below. |
| core-tmm | Creates a diagnostic core file of the TMM process. |
| bdt_cli | Displays TMM networking information such as ARP and route entries. See the bdt_cli section below. |
| mrfdb | Enables reading and writing dSSM database records. See the mrfdb section below. |
| configview | Displays Custom Resource (CR) configuration objects using their logged UUID. See the configview section below. |
| tcpdump | Displays packets sent and received on the specified network interface. |
| ping | Send ICMP ECHO_REQUEST packets to remote hosts. |
| traceroute | Displays the packet route in hops to a remote host. |
| netkvest | Performs connectivity checks to a remote host from the specified source SNAT pool using the ping and traceroute diagnostic utilities. See the netkvest section below. |
Note: Type man f5-tools in the debug container to get a full list of TMM specific commands.
Connecting to the debug sidecar¶
To connect to the debug sidecar and begin gathering diagnostic information, use the commands below.
Connect to the debug sidecar.
In this example, the debug sidecar is a container in TMM pod in default Project:
kubectl exec -it f5-tmm-6cdbc6bb65-j2r7d -c debug -n default -- bash
Execute one of the available diagnostic commands:
In this example, ping is used to test connectivity to a remote host with IP address 192.168.10.100:
ping 192.168.10.100
PING 192.168.10.100 (192.168.10.100): 56 data bytes 64 bytes from 192.168.10.100: icmp_seq=0 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=2 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=3 ttl=64 time=0.067 ms
Type exit to leave the debug sidecar.
Command examples¶
tmctl¶
Use the tmctl utility to query TMM for application traffic processing statistics.
Connect to the debug sidecar.
kubectl exec -it f5-tmm-6cdbc6bb65-j2r7d -c debug -n <project> -- bash
In this example, the debug sidecar is in the default Project:
kubectl exec -it f5-tmm-6cdbc6bb65-j2r7d -c debug -n default -- bash
To view virtual server connection statistics, run the following command:
tmctl -d blade virtual_server_stat -s name,clientside.tot_conns
To view pool member connection statistics, run the following command:
tmctl -d blade pool_member_stat -s pool_name,serverside.tot_conns
To confirm flow offloading on the DPU, run the following command:
tmctl -w 150 -d /var/tmstat/blade/ doca_flow_entries
To view fw rules connection statistics, run the following command:
tmctl -d blade fw_rule_stat -w 200
To view rst cause connection statistics, run the following command:
tmctl -d blade rst_cause_stat
To view irules connection statistics, run the following command:
tmctl -d blade -w 999 rule_stat
To view flow redirect connection statistics, run the following command:
watch -n 1 tmctl -d blade tmm/flow_redir_stats
To view profile UDP connection statistics, run the following command:
tmctl -w400 -dblade profile_udp_statexit
To view protocol inspection connection statistics, run the following command:
watch -n 1 tmctl -d blade protocol_inspection_stats -s insp_name,hit_count,last_hit_time
To view DNS Cache Resolver connection statistics, run the following command:
watch tmctl -d /var/tmstat/blade dns_cache_resolver_stat -s name,queries,msg.hits,msg.misses
To view DOS connection statistics, run the following command:
watch tmctl -d /var/tmstat/blade dos_stat -s vector_name,attack_count,stats,drops,stats_1,status
To view tmm connection statistics, run the following command:
watch -n 1 tmctl -d blade tmm_stat -s cpu_usage_5secs,memory_total,memory_used,client_side_traffic.cur_conns,client_side_traffic.tot_conns
To confirm forward flow on the DPU, run the following command:
watch -n 1 tmctl -d blade doca_flow_fwd
bdt_cli¶
Use the bdt_cli utility to query the TMM for networking data.
Commands:
arp - Get ARP routes and their status.
check - Check the grpc connection to TMM.
completion - Generate the autocompletion script for the specialized shell.
connection or connection list - Get the list of connections.
help - Help about any command.
l2forward - Get L2 Forwarding entries.
route - Get Route List.
logLevel - Set the TMM log level.
connection delete - Delete the connections based on filter operations.
tcpdumpSSLProvider - Enables a tcpdump option to decrypt pcap data in packet capture.
Supported flags to filter connections for both list and delete commands:
cs_client_addr - Clientside client IP address
cs_client_port - Clientside client port
cs_server_addr - Clientside server IP address
cs_server_port - Clientside server port
ss_server_addr - Serverside server IP address
ss_server_port - Serverside server port
ss_client_addr - Serverside client IP address
ss_client_port - Serverside client port
type - Connection Type
protocol - Protocol
idle_time - Idle Time
connection_id - Connection ID
vs_name - Virtual Server Name
cs_client_prefix - Clientside client prefix
cs_server_prefix - Clientside server prefix
vlan_name - Vlan Name
Command example:
Connect to the debug sidecar.
kubectl exec -it f5-tmm-6cdbc6bb65-j2r7d -c debug -n <project> -- bash
In this example, the debug sidecar is in the default Project:
kubectl exec -it f5-tmm-6cdbc6bb65-j2r7d -c debug -n default -- bash
Connect to TMM.
bdt_cli -u -s tmm0:8850 [command]
Example of showing routes.
bdt_cli -u -s tmm0:8850 route
routeType:1 isIpv6:false destNet:{ip:{addr:<none>, rd:0} pl:0} gw:{ip:{addr:10.59.147.121, rd:0}} gwType:1 interface:external routeType:1 isIpv6:false destNet:{ip:{addr:10.19.148.120, rd:0} pl:29} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:external routeType:1 isIpv6:false destNet:{ip:{addr:192.168.202.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:internal routeType:0 isIpv6:false destNet:{ip:{addr:169.254.1.1, rd:0} pl:32} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:eth0 routeType:1 isIpv6:false destNet:{ip:{addr:169.254.0.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:tmm
To set the f5-tmm container’s logging level to Error, run the following command.
The logging levels are listed below in the order of message severity. More severe levels generally log messages from the lower severity levels as well.
1-Debug, 2-Informational, 3-Notice (Default), 4-Warning, 5-Error, 6-Critical, 7-Alert, 8-Emergency
bdt_cli logLevel -l 5
List all connections.
bdt_cli -u -s tmm0:8850 connection
(or)
bdt_cli -u -s tmm0:8850 connection list
List Connection with a filter.
Note: The system supports both filter and wildcard operations for retrieving the list of connections.
bdt_cli -u -s tmm0:8850 connection list --flag
In this example, listing a connection with a filter like Clientside client port is
5506:bdt_cli -u -s tmm0:8850 connection delete --cs_client_port 5506
Delete Connection with a filter.
Note: Currently, the system only supports filter operations but not wildcard for deleting connections.
bdt_cli -u -s tmm0:8850 connection delete --flag
In this example, deleting a connection with a filter like Serverside server port is
8051:bdt_cli -u -s tmm0:8850 connection delete --cs_server_port 8051
mrfdb¶
The mrfdb utility enables reading and writing dSSM database records. The mrfdb tool queries the dSSM Database Sentinel Pod, sending commands to the dssmmaster DB, and relaying the response back to the debug sidecar. The mrfdb command uses these four subcomands.
The IP address of the dSSM Sentinel service to be queried.
The serverName designating the dSSM server-farm controlled by the dssmmaster DB.
The type designating the command category: cgnat, custom.
The command that is specific to the chosen type (category).
Command example:
Obtain the IP address of the dSSM Sentinel.
In this example, dSSM is installed in the f5-utils Project.
kubectl get svc -n f5-utils
In this example, the Sentinel IP address is 10.203.180.204.
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) f5-dssm-db ClusterIP 10.108.254.57 <none> 6379/TCP f5-dssm-sentinel ClusterIP 10.103.180.204 <none> 26379/TCP
Login to the debug sidecar container.
In this example, the debug sidecar is in the default Project.
kubectl exec -it f5-tmm-6cdbc6bb65-j2r7d -c debug -n default -- bash
Run the mrfdb utility.
In this example, the mrfdb utility queries for all DB records.
mrfdb -ipport=10.103.180.204:26379 -serverName=dssm-svc -displayAllBins
configview¶
Use the configview utility to show configuration object created by the installed BIG-IP Next for Kubernetes CRs.
View the TMM deployment logs, and grep for UUID events.
In this example, TMM is in the default Project:
kubectl logs f5-tmm-6cdbc6bb65-j2r7d -n default | grep UUID
In this example, the first log UUID default-net-external-vlan will be used to query with configview.
<134>Jan 1 1:10:11 f5-tmm-7d5b489c5b-fffgt tmm1[36]: 01010058:6: audit log: action: CREATE; UUID: default-net-external-vlan; event: declTmm.vlan; Error: No error
Connect to the debug sidecar.
In this example, the debug sidecar is in the default Project:
kubectl exec -it f5-tmm-6cdbc6bb65-j2r7d -c debug -n default -- bash
Execute the configview utility.
configview uuid default-net-external-vlan
The example output displays the CR parameters and values.
request:[declTmm.vlan]:{name:"external" id:"default-net-external-vlan" tag:3350 mtu:1500 tagged_interfaces:"1.2"}
netkvest¶
Note: The netkvest utility supports only the ping and traceroute diagnostic utilities.
Use the netkvest utility to check connectivity to a remote host from a specified source SNAT pool.
Connect to the debug sidecar.
kubectl exec -it deploy/f5-tmm -c debug -n <project> -- bash
In this example, the the debug sidecar is in the default Project.
kubectl exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
To check the connectivity to a remote host from a specified source SNAT pool using the ping diagnostic utility, run the following command.
kubectl exec -it deploy/f5-tmm -c debug -- netkvest -s <source_SNAT_pool_name> -d <remote_host> -u <diagnostic utility>
In this example, the netkvest utility checks for destination 22.22.22.100 from egress-snatpool source SNAT pool using ping diagnostic utility.
kubectl exec -it deploy/f5-tmm -c debug -- netkvest -s egress-snatpool -d 22.22.22.100 -u ping
Sample Output:
PING 22.22.22.100 (22.22.22.100) 64 data bytes 64 bytes from 22.22.22.100: icmp_seq=0 ttl=63 64 bytes from 22.22.22.100: icmp_seq=1 ttl=63 64 bytes from 22.22.22.100: icmp_seq=2 ttl=63 64 bytes from 22.22.22.100: icmp_seq=3 ttl=63 64 bytes from 22.22.22.100: icmp_seq=4 ttl=63 64 bytes from 22.22.22.100: icmp_seq=5 ttl=63 64 bytes from 22.22.22.100: icmp_seq=6 ttl=63 64 bytes from 22.22.22.100: icmp_seq=7 ttl=63 64 bytes from 22.22.22.100: icmp_seq=8 ttl=63 64 bytes from 22.22.22.100: icmp_seq=9 ttl=63 64 bytes from 22.22.22.100: icmp_seq=10 ttl=63 PING 22.22.22.100 (22.22.22.100) 64 data bytes 64 bytes from 22.22.22.100: icmp_seq=0 ttl=63 64 bytes from 22.22.22.100: icmp_seq=1 ttl=63 64 bytes from 22.22.22.100: icmp_seq=2 ttl=63 64 bytes from 22.22.22.100: icmp_seq=3 ttl=63 64 bytes from 22.22.22.100: icmp_seq=4 ttl=63 64 bytes from 22.22.22.100: icmp_seq=5 ttl=63 64 bytes from 22.22.22.100: icmp_seq=6 ttl=63 64 bytes from 22.22.22.100: icmp_seq=7 ttl=63 64 bytes from 22.22.22.100: icmp_seq=8 ttl=63 64 bytes from 22.22.22.100: icmp_seq=9 ttl=63 64 bytes from 22.22.22.100: icmp_seq=10 ttl=63 2025-06-18 14:21:12 [info]: main.main: Execution is successful
To check the connectivity to a remote host from a specified source SNAT pool using the traceroute diagnostic utility, run the following command.
kubectl exec -it deploy/f5-tmm -c debug -- netkvest -s <source_SNAT_pool_name> -d <remote_host> -u <diagnostic utility>
In this example, the netkvest utility checks for destination 22.22.22.100 from source SNAT pool using the traceroute diagnostic utility.
kubectl exec -it deploy/f5-tmm -c debug -- netkvest -s egress-snatpool -d 22.22.22.100 -u traceroute
Sample Output:
traceroute to 22.22.22.100 (22.22.22.100), 64 hops max, 64 byte packets 1 33.33.33.254 2 22.22.22.100 traceroute to 22.22.22.100 (22.22.22.100), 64 hops max, 64 byte packets 1 33.33.33.254 2 22.22.22.100 2025-06-18 14:21:12 [info]: main.main: Execution is successful
Limitations:
The netkvest utility has limitations based on the IP version, as shown below:
Note: When using the netkvest utility, make sure the source and destination IP addresses are of the same type—either both IPv4 or both IPv6. Mixing them will cause the command to fail.
If the user specifies a diagnostic command with an IPv4 source, but provides an IPv6 destination, the command will fail with an error.
Example 1 – IPv4 source with IPv6 destination.
kubectl exec -it deploy/tmm -c debug -- netkvest -s 11.11.11.11 -d 2002::22:22:22:100 -u ping
Sample Output:
2025-06-18 12:01:06 [error] main.main: Execution failed: Destination type is IPv6, but no IPv6 addresses found in source. Command terminated with exit code 2.
Similarly, if the user specifies a diagnostic command with an IPv6 source, but provides an IPv4 destination, the command will also fail:
Example 2 – IPv6 source with IPv4 destination.
oc exec -it deploy/tmm -c debug -- netkvest -s 2002::11:11:11:11 -d 22.22.22.100 -u ping
Sample Output:
2025-06-18 12:05:45 [error] main.main: Execution failed: Destination type is IPv4, but no IPv4 addresses found in source. Command terminated with exit code 2.
Supplemental
The Debug API can run diagnostic commands on a targeted TMM from your local workstation.