Debug Sidecar¶
The TMM Proxy Pod’s debug sidecar provides a set of command-line utilities for obtaining low-level, diagnostic data and statistics about the Service Proxy Traffic Management Microkernel (TMM). The debug sidecar deploys by default with the BIG-IP Controller.
Command-Line Utilities¶
The table below lists and describes the available command-line utilities.
| Utility | Description |
|---|---|
| tmctl | Displays various TMM traffic processing statistics, such as pool and virtual server connections. |
| bdt_cli | Displays TMM networking information such as ARP, route entries and DNS Cache records. See the bdt_cli section below. |
| mrfdb | Enables reading and writing dSSM database records. See the mrfdb section below. |
| configview | Displays Custom Resource (CR) configuration objects using their logged UUID. |
| ipint_dump | Inspects the IP Intelligence policy binary blob for diagnostics. |
| iprep_dump | Lists IP Addresses in IP Reputation database. |
| iprep_lookup | Validate IP Address entry in IP Reputation database. |
| netkvest | Performs connectivity checks to a remote host from the specified source SNAT pool using the ping and traceroute diagnostic utilities. See the netkvest section below. |
Connecting to the debug sidecar¶
To connect to the debug sidecar and begin gathering diagnostic information, use the commands below.
Connect to the debug sidecar.
In this example, the debug sidecar is in the cnf-gateway Project:
oc exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash
Execute one of the available diagnostic commands:
In this example, ping is used to test connectivity to a remote host with IP address 192.168.10.100:
ping 192.168.10.100
PING 192.168.10.100 (192.168.10.100): 56 data bytes 64 bytes from 192.168.10.100: icmp_seq=0 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=2 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=3 ttl=64 time=0.067 ms
Type Exit to leave the debug sidecar.
Command Examples¶
tmctl¶
Use the tmctl utility to query Service Proxy TMM for application traffic processing statistics.
Virtual server connections
To view virtual server connection statistics run the following command:
Client side statstics
tmctl -d blade virtual_server_stat -s name,clientside.tot_conns
Server side statstics
tmctl -d blade virtual_server_stat -s name,serverside.tot_conns
bdt_cli¶
Use the bdt_cli tool to query the Service Proxy TMM for networking data.
Commands:
arp - Get ARP routes and their status
check - Get TMM Check Magic
completion - Generates the autocompletion script for the specialized shell
connection - Get Connection List
help - Help about any command
l2forward - Get L2 Forwarding entries
logLevel - Set the TMM log level
route - Get Route List
dnsCacheRecords
list - Get the list of DNS cache records
Sample Output:
bdt_cli dnsCacheRecords list <flags>
count - Get the count of DNS cache records
Sample Output:
bdt_cli dnsCacheRecords count <flags>
delete - Delete the DNS cache records
Sample Output:
bdt_cli dnsCacheRecords delete <flags>
Flags Supported for dnsCacheRecords
Flags
| Flag name | Description |
|---|---|
| --cache string | Specifies a DNS cache |
| --cache-type string | Specifies a cache type. One of rrset, msg, or nameserver |
| --result-limit int | Specifies the limit of number of records |
Flags to filter RRSet records
| Flag name | Description |
|---|---|
| --owner string | Domain name |
| --type string | Resource type |
| --class string | Resource class |
| --ttl_range int:int | TTL range. Either min or max may be omitted |
Flags to filter DNS messages
| Flag name | Description |
|---|---|
| --qname string | Domain name |
| --rcode int | DNS return code |
Flags to filter nameserver records
| Flag name | Description |
|---|---|
| --address stringg | Nameserver IP address |
| --has_edns | Supports EDNS |
| --has_lame | Nameserver is lame for one or more items |
| --rtt_range int:int | RTT range. Either min or max may be omitted |
| --ttl_range int:int | TTL range. Either min or max may be omitted |
| --zone_name string | Zone name |
Command Example for bdt_cli Tool:
Connect to the debug sidecar.
oc exec -it deploy/f5-tmm -c debug -n <project> -- bash
In this example, the debug sidecar is in the cnf-gateway Project:
oc exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash
Connect to TMM.
bdt_cli -u -s tmm0:8850 [command]
Example routes.
bdt_cli -u -s tmm0:8850 route
routeType:1 isIpv6:false destNet:{ip:{addr:<none>, rd:0} pl:0} gw:{ip:{addr:10.59.147.121, rd:0}} gwType:1 interface:external routeType:1 isIpv6:false destNet:{ip:{addr:10.19.148.120, rd:0} pl:29} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:external routeType:1 isIpv6:false destNet:{ip:{addr:192.168.202.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:internal routeType:0 isIpv6:false destNet:{ip:{addr:169.254.1.1, rd:0} pl:32} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:eth0 routeType:1 isIpv6:false destNet:{ip:{addr:169.254.0.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:tmm
To set the logging level of f5-tmm container to Error, run the following command.
bdt_cli logLevel -l 5
Following are the logging levels listed in the order of message severity.
1-Debug, 2-Informational, 3-Notice (Default), 4-Warning, 5-Error, 6-Critical, 7-Alert, 8-EmergencyNote: The logging levels generally log messages from the lower severity levels.
mrfdb¶
The mrfdb utility enables reading and writing dSSM database records. The mrfdb tool queries the dSSM Database Sentinel Pod, sending commands to the dssmmaster DB, and relaying the response back to the debug sidecar.
The mrfdb command uses these four subcomands.
The IP address of the dSSM Sentinel service to be queried.
The serverName designating the dSSM server-farm controlled by the dssmmaster DB.
The type designating the command category: dns46, cgnat, custom.
The command that is specific to the chosen type (category).
Command Example:
Login to the debug sidecar container.
In this example, the debug sidecar is in the cnf-gateway namespace.
oc exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash
Run the mrfdb utility.
In this example, the mrfdb utility queries for all DB records.
mrfdb -ipport=f5-dssm-sentinel:26379 -serverName=server -displayAllBins
configview¶
Use the configview utility to show configuration objects created by the installed CNF CRs.
View the TMM deployment logs, and grep for UUID events.
In this example, TMM is in the cnf-gateway Project:
oc logs deploy/f5-tmm -c f5-tmm -n cnf-gateway | grep UUID
In this example, the first log UUID cnf-gateway-net-external-vlan will be used to query with configview.
<134>Jan 1 1:10:11 f5-tmm-7d5b489c5b-fffgt tmm1[36]: 01010058:6: audit log: action: CREATE; UUID: cnf-gateway-net-external-vlan; event: declTmm.vlan; Error: No error
Connect to the debug sidecar.
In this example, the debug sidecar is in the cnf-gateway Project:
oc exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash
Execute the configview utility.
configview uuid cnf-gateway-net-external-vlan
The example output displays the CR parameters and values.
request:[declTmm.vlan]:{name:"external" id:"cnf-gateway-net-external-vlan" tag:3350 mtu:9000 tagged_interfaces:"1.2"}
ipint_dump¶
The dwbld daemon compiles the binary blob file that holds policy details or context association. Each policy contains categories and action configurations and are encoded in the blob.
If there is no feedlist, the blob only includes a top-level header section with category actions, but does not have any Feedlist IP Addresses encoded in the blob.
Run the following command to inspect IP intelligence policy.
ipint_dump /shared/dwblclass/dwbl_blob
Sample Output:
#### Categories Section. ####
Bit pos Category Name Category Changed
------- ------------- ----------------
0 whitelist No
1 spam_sources No
2 windows_exploits No
3 web_attacks No
4 botnets No
5 scanners No
6 denial_of_service No
7 infected_sources No
8 phishing No
9 proxy No
10 network No
11 cloud_provider_networks No
12 mobile_threats No
14 tor_proxy No
15 application_denial_of_service No
16 attacked_ips No
17 appiq_badactors No
63 additional No
#### Assignments Section. ####
Context OID Context Type Context Name Container Index Policy Name
----------- ------------ ------------ --------------- -----------
2 Virtual Server f5local-dns-vs-udp-virtual_serv0 my-ipi-policy-ipipolicy
#### Containers (policy) Section. ####
Container ID 0
Preamble
Magic: 30330003 (OK)
Version: 0.0
Generation: 0
Start time: 0
Commit time: 1739223562
Uses iprep_src: yes
Uses iprep_dst: no
Masks
Action: 0xfffffffffffffffe
LOG_BL: 0x000000000000401e
LOG_BL_WL: 0x0000000000000000
SRC_IP_ENABLE: 0x000000000000401e
DST_IP_ENABLE: 0x0000000000000000
LPM Section
Count: 3
Bytes: 224
iprep_dump¶
The iprep_dump lists all IPv4 addresses and categories present in the IP Reputation database.
The iprep_dump -6 command prints the IPv6 addresses in the database file.
Sample Output:
iprep_dump | more
opening database in /var/IpRep/F5IpRep.dat
size of IPv4 reputation database = 0
1.0.0.2 32 256 Proxy
1.0.0.10 32 128 Phishing
1.0.0.20 32 128 Phishing
1.0.0.69 32 2048 Mobile Threats
1.0.1.0 32 2048 Mobile Threats
1.0.1.1 32 128 Phishing
1.0.1.2 32 128 Phishing
1.0.1.3 32 128 Phishing
1.0.1.4 32 128 Phishing
1.0.1.5 32 128 Phishing
1.0.1.21 32 2048 Mobile Threats
1.0.5.4 32 2048 Mobile Threats
1.0.28.4 32 2048 Mobile Threats
1.0.152.14 32 16 Scanners
1.0.178.120 32 256 Proxy
1.0.186.215 32 256 Proxy
1.0.212.180 32 256 Proxy
1.0.230.156 32 16 Scanners
1.0.244.179 32 16 Scanners
1.0.249.14 32 16 Scanners
1.0.252.4 32 1 Spam Sources
1.0.252.152 32 256 Proxy
1.0.254.5 32 16 Scanners
1.0.255.5 32 16 Scanners
1.0.255.22 32 16 Scanners
1.0.255.73 32 16 Scanners
1.1.1.12 32 128 Phishing
1.1.4.92 32 2048 Mobile Threats
1.1.10.4 32 2048 Mobile Threats
1.1.11.1 32 128 Phishing
1.1.166.127 32 256 Proxy
1.1.168.59 32 256 Proxy
1.1.229.206 32 1 Spam Sources
1.1.230.41 32 2 Windows Exploits
1.2.1.0 32 2048 Mobile Threats
1.2.3.8 32 128 Phishing
...
iprep_dump -6 | more
opening database in /var/IpRep/F5IpV6Rep.dat
size of IPv6 reputation database = 0
:: 64 202 Windows Exploits,BotNets,Infected Sources,Phishing
0:ffff:c0a9:bd35:: 64 128 Phishing
1:: 64 130 Windows Exploits,Phishing
1be:2db0:dec0:f528:: 64 1 Spam Sources
e80:: 64 128 Phishing
1111:2222:3333:: 64 128 Phishing
1234:1234:1234:1234:: 64 128 Phishing
2001:: 64 3 Spam Sources,Windows Exploits
2001:0:53aa:64c:: 64 67 Spam Sources,Windows Exploits,Infected Sources
2001:0:5ef5:79fd:: 64 128 Phishing
2001:0:9d38:6ab8:: 64 2 Windows Exploits
2001:0:9d38:6abd:: 64 2 Windows Exploits
2001:0:9d38:90d7:: 64 1 Spam Sources
2001:0:9d38:953c:: 64 9 Spam Sources,BotNets
2001:df:465:77:: 64 2 Windows Exploits
2001:200:dff:fff1:: 64 128 Phishing
2001:208:: 64 130 Windows Exploits,Phishing
2001:240:2401:15ac:: 64 2 Windows Exploits
2001:240:2401:ae11:: 64 2 Windows Exploits
2001:240:2403:1a08:: 64 2 Windows Exploits
2001:240:2403:3408:: 64 2 Windows Exploits
2001:240:2403:3702:: 64 2 Windows Exploits
2001:240:2403:667c:: 64 2 Windows Exploits
2001:240:2404:d050:: 64 2 Windows Exploits
2001:240:2405:202a:: 64 2 Windows Exploits
2001:240:2405:da28:: 64 138 Windows Exploits,BotNets,Phishing
...
iprep_lookup¶
The iprep_lookup ip-address command can be used to look for an address in the IP Reputation third-party database file.
Sample Output:
iprep_lookup 1.2.2.1
opening database in /var/IpRep/F5IpRep.dat, /var/IpRep/F5IpV6Rep.dat
size of IP reputation database = 24010644, 1629744
iprep threats list for ip = 1.2.2.1 is:
bit 11 - Mobile Threats
iprep_lookup 2001:240:240e:e3f5::
opening database in /var/IpRep/F5IpRep.dat, /var/IpRep/F5IpV6Rep.dat
size of IP reputation database = 24010756, 1629744
iprep threats list for ip = 2001:240:240e:e3f5:: is:
bit 1 - Windows Exploits
netkvest¶
Note: The netkvest utility supports only the ping and traceroute diagnostic utilities.
Use the netkvest utility to check connectivity to a remote host from a specified source SNAT pool
Connect to the debug sidecar.
oc exec -it deploy/f5-tmm -c debug -n <project> -- bash
In this example, the debug sidecar is in the spk-ingress Project
oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
To check the connectivity to a remote host from a specified source SNAT pool using the ping diagnostic utility, run the following command.
oc exec -it deploy/f5-tmm -c debug -- netkvest -s <source_SNAT_pool_name> -d <remote_host> -u <diagnostic utility>
In this example, the netkvest utility checks for destination 22.22.22.100 from egress-snatpool source SNAT pool using ping diagnostic utility.
oc exec -it deploy/f5-tmm -c debug -- netkvest -s egress-snatpool -d 22.22.22.100 -u ping
Sample Output
PING 22.22.22.100 (22.22.22.100) 64 data bytes 64 bytes from 22.22.22.100: icmp_seq=0 ttl=63 64 bytes from 22.22.22.100: icmp_seq=1 ttl=63 64 bytes from 22.22.22.100: icmp_seq=2 ttl=63 64 bytes from 22.22.22.100: icmp_seq=3 ttl=63 64 bytes from 22.22.22.100: icmp_seq=4 ttl=63 64 bytes from 22.22.22.100: icmp_seq=5 ttl=63 64 bytes from 22.22.22.100: icmp_seq=6 ttl=63 64 bytes from 22.22.22.100: icmp_seq=7 ttl=63 64 bytes from 22.22.22.100: icmp_seq=8 ttl=63 64 bytes from 22.22.22.100: icmp_seq=9 ttl=63 64 bytes from 22.22.22.100: icmp_seq=10 ttl=63 PING 22.22.22.100 (22.22.22.100) 64 data bytes 64 bytes from 22.22.22.100: icmp_seq=0 ttl=63 64 bytes from 22.22.22.100: icmp_seq=1 ttl=63 64 bytes from 22.22.22.100: icmp_seq=2 ttl=63 64 bytes from 22.22.22.100: icmp_seq=3 ttl=63 64 bytes from 22.22.22.100: icmp_seq=4 ttl=63 64 bytes from 22.22.22.100: icmp_seq=5 ttl=63 64 bytes from 22.22.22.100: icmp_seq=6 ttl=63 64 bytes from 22.22.22.100: icmp_seq=7 ttl=63 64 bytes from 22.22.22.100: icmp_seq=8 ttl=63 64 bytes from 22.22.22.100: icmp_seq=9 ttl=63 64 bytes from 22.22.22.100: icmp_seq=10 ttl=63 2025-06-18 14:21:12 [info]: main.main: Execution is successful
To check the connectivity to a remote host from a specified source SNAT pool using the traceroute diagnostic utility, run the following command.
oc exec -it deploy/f5-tmm -c debug -- netkvest -s <source_SNAT_pool_name> -d <remote_host> -u <diagnostic utility>
In this example, the netkvest utility checks for destination 22.22.22.100 from source SNAT pool using the traceroute diagnostic utility.
oc exec -it deploy/f5-tmm -c debug -- netkvest -s egress-snatpool -d 22.22.22.100 -u traceroute
Sample Output
traceroute to 22.22.22.100 (22.22.22.100), 64 hops max, 64 byte packets 1 33.33.33.254 2 22.22.22.100 traceroute to 22.22.22.100 (22.22.22.100), 64 hops max, 64 byte packets 1 33.33.33.254 2 22.22.22.100 2025-06-18 14:21:12 [info]: main.main: Execution is successful
Limitations:
The netkvest utility has limitations based on the IP version, as shown below:
Note: When using the netkvest utility, make sure the source and destination IP addresses are of the same type—either both IPv4 or both IPv6. Mixing them will cause the command to fail.
If the user specifies a diagnostic command with an IPv4 source, but provides an IPv6 destination, the command will fail with an error.
Example 1 – IPv4 source with IPv6 destination.
oc exec -it deploy/tmm -c debug -- netkvest -s 11.11.11.11 -d 2002::22:22:22:100 -u ping
Sample Output
2025-06-18 12:01:06 [error] main.main: Execution failed: Destination type is IPv6, but no IPv6 addresses found in source. Command terminated with exit code 2.
Similarly, if the user specifies a diagnostic command with an IPv6 source, but provides an IPv4 destination, the command will also fail:
Example 2 – IPv6 source with IPv4 destination.
oc exec -it deploy/tmm -c debug -- netkvest -s 2002::11:11:11:11 -d 22.22.22.100 -u ping
Sample Output
2025-06-18 12:05:45 [error] main.main: Execution failed: Destination type is IPv4, but no IPv4 addresses found in source. Command terminated with exit code 2.
Disabling the sidecar¶
The TMM debug sidecar installs by default with the CNFs Controller. You can disable the debug sidecar by setting the debug.enabled parameter to false in the BIG-IP Controller Helm values file:
debug:
enabled: false
Viewing DNS Cache Derived Stats¶
The Viewing DNS Cache Stats allows you to analyze the performance and activity of DNS caches by using the dns-cache-stats utility. You can access detailed statistics about DNS cache behavior such as query counts, cache hits/misses, traffic patterns, and historical response data across various time intervals.
Command Example:
Login to the debug sidecar container.
In this example, the debug sidecar is in the cnf-gateway namespace.
kubectl exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash
Run the dns-cache-stats utility.
In this example, the dns-cache-stats utility queries for all DNS cache records.
$ kubectl -n f5-cnf exec -it deploy/f5-tmm -c debug -- dns-cache-stats addr:10.43.0.10_53_0 --------------------------------------------------------------------------- Net::DNS-Resolver: addr:10.43.0.10_53_0 --------------------------------------------------------------------------- Client Summary Total Total Queries 40656 Total Responses 40656 Answered Locally 40590 Stale Record Answers 0 Using Nameservers 66 Client History Last 1 sec Last 5 min Last 1 hr Last 1 day Total Responses 0 0 0 0 Local Zones Total Responses 0 Response Policy Zones Total Rewrites 0 Response Client Cache Total % Time (us) Hits 37118 91 0 Misses 3538 8 47946331 Forward Zones Total Responses 3481 Client Queries Total Resolving 0 Maximum Resolving 0 Overflowed 0 Timed Out 0 Nameserver Summary Total Total Queries 26 Prefetched 0 Total Responses 0 Queries/Second 0 Component Caches Total % or Rate Complete Message Hits 37118 84% Misses 6962 15% Evictions 0 0/s Modifications 3481 0/s Resource Record Set Hits 0 0% Misses 21738 100% Evictions 0 0/s Modifications 0 0/s Internet Nameserver Hits 3540 99% Misses 11 0% Evictions 0 0/s Modifications 1 0/s Nameserver Traffic Total UDP TCP Bits In 0 0 0 Bits Out 19680 19680 0 Packets In 0 0 0 Packets Out 26 26 0 Queries Waiting to Send 0 0 0 Concurrent Flows 0 0 0 Maximum Flows 2 2 0 Unsolicited Replies 0 0 - Nameserver Errors Total Memory Allocation 0 No Free Port IPv4 0 No Free Port IPv6 0 No Route IPv4 0 No Route IPv6 0 SERVFAIL Response 0 FORMERR Response 0 Other Error Response 0
Feedback
Provide feedback to improve this document by emailing cnfdocs@f5.com.