Debug Sidecar¶
Overview¶
The TMM Proxy Pod’s debug sidecar provides a set of command line tools for obtaining low-level, diagnostic data and statistics about the Service Proxy Traffic Management Microkernel (TMM). The debug sidecar deploys by default with the BIG-IP Controller.
Command line tools¶
The table below lists and describes the available command line tools:
| Tool | Description |
|---|---|
| tmctl | Displays various TMM traffic processing statistics, such as pool and virtual server connections. |
| bdt_cli | Displays TMM networking information such as ARP and route entries. See the bdt_cli section below. |
| mrfdb | Enables reading and writing dSSM database records. See the mrfdb section below. |
| configview | Displays Custom Resource (CR) configuration objects using their logged UUID. |
| qkview | Creates a diagnostic data TAR file for F5 support. See the Qkview section below. |
| ipint_dump | Inspects the IP Intelligence policy binary blob for diagnostics. |
| iprep_dump | Lists IP Addresses in IP Reputation database. |
| iprep_lookup | Validate IP Address entry in IP Reputation database. |
Connecting to the sidecar¶
To connect to the debug sidecar and begin gathering diagnostic information, use the commands below.
Connect to the debug sidecar:
In this example, the debug sidecar is in the cnf-gateway Project:
kubectl exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash
Execute one of the available diagnostic commands:
In this example, ping is used to test connectivity to a remote host with IP address 192.168.10.100:
ping 192.168.10.100
PING 192.168.10.100 (192.168.10.100): 56 data bytes 64 bytes from 192.168.10.100: icmp_seq=0 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=2 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=3 ttl=64 time=0.067 ms
Type Exit to leave the debug sidecar.
Command examples¶
tmctl¶
Use the tmctl tool to query Service Proxy TMM for application traffic processing statistics.
Virtual server connections
To view virtual server connection statistics run the following command:
Client side statstics
tmctl -d blade virtual_server_stat -s name,clientside.tot_conns
Server side statstics
tmctl -d blade virtual_server_stat -s name,serverside.tot_conns
bdt_cli¶
Use the bdt_cli tool to query the Service Proxy TMM for networking data.
Commands:
- arp - Get ARP routes and their status
- check - Get TMM Check Magic
- completion - Generate the autocompletion script for the specialized shell
- connection - Get Connection List
- help - Help about any command
- l2forward - Get L2 Forwarding entries
- logLevel - Set the TMM log level
- route - Get Route List
Command example:
Connect to the debug sidecar:
kubectl exec -it deploy/f5-tmm -c debug -n <project> -- bash
In this example, the debug sidecar is in the cnf-gateway Project:
kubectl exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash
Connect to TMM:
bdt_cli -u -s tmm0:8850 [command]
Example routes:
bdt_cli -u -s tmm0:8850 route
routeType:1 isIpv6:false destNet:{ip:{addr:<none>, rd:0} pl:0} gw:{ip:{addr:10.59.147.121, rd:0}} gwType:1 interface:external routeType:1 isIpv6:false destNet:{ip:{addr:10.19.148.120, rd:0} pl:29} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:external routeType:1 isIpv6:false destNet:{ip:{addr:192.168.202.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:internal routeType:0 isIpv6:false destNet:{ip:{addr:169.254.1.1, rd:0} pl:32} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:eth0 routeType:1 isIpv6:false destNet:{ip:{addr:169.254.0.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:tmm
To set the logging level of f5-tmm container to Error, run the following command:
bdt_cli logLevel -l 5
Following are the logging levels listed in the order of message severity.
1-Debug, 2-Informational, 3-Notice (Default), 4-Warning, 5-Error, 6-Critical, 7-Alert, 8-Emergency
Note: The logging levels generally log messages from the lower severity levels.
mrfdb¶
The mrfdb utility enables reading and writing dSSM database records. The mrfdb tool queries the dSSM Database Sentinel Pod, sending commands to the dssmmaster DB, and relaying the response back to the debug sidecar. The mrfdb command uses these four subcomands:
- The IP address of the dSSM Sentinel service to be queried.
- The serverName designating the dSSM server-farm controlled by the dssmmaster DB.
- The type designating the command category: dns46, cgnat, custom.
- The command that is specific to the chosen type (category).
Command example:
Login to the debug sidecar container:
In this example, the debug sidecar is in the cnf-gateway namespace.
kubectl exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash
Run the mrfdb utility:
In this example, the mrfdb utility queries for all DB records.
mrfdb -ipport=f5-dssm-sentinel:26379 -serverName=server -displayAllBins
configview¶
Use the configview utility to show configuration objects created by the installed CNF CRs.
View the TMM deployment logs, and grep for UUID events:
In this example, TMM is in the cnf-gateway Project:
kubectl logs deploy/f5-tmm -c f5-tmm -n cnf-gateway | grep UUID
In this example, the first log UUID cnf-gateway-net-external-vlan will be used to query with configview.
<134>Jan 1 1:10:11 f5-tmm-7d5b489c5b-fffgt tmm1[36]: 01010058:6: audit log: action: CREATE; UUID: cnf-gateway-net-external-vlan; event: declTmm.vlan; Error: No error
Connect to the debug sidecar:
In this example, the debug sidecar is in the cnf-gateway Project:
kubectl exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash
Execute the configview utility:
configview uuid cnf-gateway-net-external-vlan
The example output displays the CR parameters and values.
request:[declTmm.vlan]:{name:"external" id:"cnf-gateway-net-external-vlan" tag:3350 mtu:9000 tagged_interfaces:"1.2"}
Qkview¶
The qkview utility collects diagnostic and logging information from the f5-tmm container, and stores the data in a Linux TAR file. If you enabled the Fluentd Logging collector, run the qkview utility on f5-fluentd container to gather log files from all of the CNFs Pods. Qkview files are typically generated and sent to F5 for further analysis. Use the steps below to run the qkview utility, and copy the file to your local workstation.
Obtain the name of the TMM Proxy Pod:
In this example, the TMM Proxy Pod is in the cnf-gateway namespace.
kubectl get pods --selector app=f5-tmm -n cnf-gateway
In this example, the TMM Proxy Pod is named f5-tmm-77b95f699f-5zv8n.
NAME READY STATUS f5-tmm-77b95f699f-5zv8n 5/5 Running
Connect to the debug sidecar:
kubectl exec -it f5-tmm-77b95f699f-5zv8n -c debug -n cnf-gateway -- bash
The shell prompt should contain the name of the TMM Proxy Pod.
debuguser@f5-tmm-77b95f699f-5zv8n:~$
Run the qkview utility:
qkview
The command output should indicate the file was created and saved.
Diagnostic snapshot file saved: qkview.20220511-185024.tar.gz
Copy the file to your workstation:
kubectl cp <namespace>/<podname>:<file> ./<file> -c debug
In this example, the qkview named qkview.20220511-185024.tar.gz is copied to the local workstation.
kubectl cp cnf-gateway/f5-tmm-77b95f699f-k8bfh:qkview.20220511-185024.tar.gz \ ./qkview.20220511-185024.tar.gz -c debug
Obtain the name of the Fluent logging Pod:
kubectl get pods --selector run=f5-fluentd -n cnf-gateway
In this example the Fluentd logging Pod is named f5-toda-fluentd-84f96b6757-v5wj9.
f5-toda-fluentd-84f96b6757-v5wj9 1/1 Running
Connect to the Fluentd logging Pod:
kubectl exec -it f5-toda-fluentd-84f96b6757-v5wj9 -n cnf-gateway -- bash
Run the qkview utility:
qkview
The command output should indicate the file was created and saved.
Diagnostic snapshot file is saved: qkview.20220511-195129.tar.gz
Copy the file to your workstation:
kubectl cp <namespace>/<podname>:<file> ./<file>
In this example, the qkview named qkview.20220511-195129.tar.gz is copied to the local workstation.
kubectl cp cnf-gateway/ff5-toda-fluentd-84f96b6757-v5wj:qkview.20220511-195129.tar.gz \ ./qkview.20220511-195129.tar.gz
ipint_dump¶
The dwbld daemon compiles the binary blob file that holds policy details or context association. Each policy contains categories and action configurations and are encoded in the blob.
If there is no feedlist, the blob only includes a top-level header section with category actions, but does not have any Feedlist IP Addresses encoded in the blob.
Run the following command to inspect IP intelligence policy:
ipint_dump /shared/dwblclass/dwbl_blob
Sample Output:
#### Categories Section. ####
Bit pos Category Name Category Changed
------- ------------- ----------------
0 whitelist No
1 spam_sources No
2 windows_exploits No
3 web_attacks No
4 botnets No
5 scanners No
6 denial_of_service No
7 infected_sources No
8 phishing No
9 proxy No
10 network No
11 cloud_provider_networks No
12 mobile_threats No
14 tor_proxy No
15 application_denial_of_service No
16 attacked_ips No
17 appiq_badactors No
63 additional No
#### Assignments Section. ####
Context OID Context Type Context Name Container Index Policy Name
----------- ------------ ------------ --------------- -----------
2 Virtual Server f5local-dns-vs-udp-virtual_serv0 my-ipi-policy-ipipolicy
#### Containers (policy) Section. ####
Container ID 0
Preamble
Magic: 30330003 (OK)
Version: 0.0
Generation: 0
Start time: 0
Commit time: 1739223562
Uses iprep_src: yes
Uses iprep_dst: no
Masks
Action: 0xfffffffffffffffe
LOG_BL: 0x000000000000401e
LOG_BL_WL: 0x0000000000000000
SRC_IP_ENABLE: 0x000000000000401e
DST_IP_ENABLE: 0x0000000000000000
LPM Section
Count: 3
Bytes: 224
iprep_dump¶
The iprep_dump lists all IPv4 addresses and categories present in the IP Reputation database.
The iprep_dump -6 command prints the IPv6 addresses in the database file.
Sample Output:
iprep_dump | more
opening database in /var/IpRep/F5IpRep.dat
size of IPv4 reputation database = 0
1.0.0.2 32 256 Proxy
1.0.0.10 32 128 Phishing
1.0.0.20 32 128 Phishing
1.0.0.69 32 2048 Mobile Threats
1.0.1.0 32 2048 Mobile Threats
1.0.1.1 32 128 Phishing
1.0.1.2 32 128 Phishing
1.0.1.3 32 128 Phishing
1.0.1.4 32 128 Phishing
1.0.1.5 32 128 Phishing
1.0.1.21 32 2048 Mobile Threats
1.0.5.4 32 2048 Mobile Threats
1.0.28.4 32 2048 Mobile Threats
1.0.152.14 32 16 Scanners
1.0.178.120 32 256 Proxy
1.0.186.215 32 256 Proxy
1.0.212.180 32 256 Proxy
1.0.230.156 32 16 Scanners
1.0.244.179 32 16 Scanners
1.0.249.14 32 16 Scanners
1.0.252.4 32 1 Spam Sources
1.0.252.152 32 256 Proxy
1.0.254.5 32 16 Scanners
1.0.255.5 32 16 Scanners
1.0.255.22 32 16 Scanners
1.0.255.73 32 16 Scanners
1.1.1.12 32 128 Phishing
1.1.4.92 32 2048 Mobile Threats
1.1.10.4 32 2048 Mobile Threats
1.1.11.1 32 128 Phishing
1.1.166.127 32 256 Proxy
1.1.168.59 32 256 Proxy
1.1.229.206 32 1 Spam Sources
1.1.230.41 32 2 Windows Exploits
1.2.1.0 32 2048 Mobile Threats
1.2.3.8 32 128 Phishing
...
iprep_dump -6 | more
opening database in /var/IpRep/F5IpV6Rep.dat
size of IPv6 reputation database = 0
:: 64 202 Windows Exploits,BotNets,Infected Sources,Phishing
0:ffff:c0a9:bd35:: 64 128 Phishing
1:: 64 130 Windows Exploits,Phishing
1be:2db0:dec0:f528:: 64 1 Spam Sources
e80:: 64 128 Phishing
1111:2222:3333:: 64 128 Phishing
1234:1234:1234:1234:: 64 128 Phishing
2001:: 64 3 Spam Sources,Windows Exploits
2001:0:53aa:64c:: 64 67 Spam Sources,Windows Exploits,Infected Sources
2001:0:5ef5:79fd:: 64 128 Phishing
2001:0:9d38:6ab8:: 64 2 Windows Exploits
2001:0:9d38:6abd:: 64 2 Windows Exploits
2001:0:9d38:90d7:: 64 1 Spam Sources
2001:0:9d38:953c:: 64 9 Spam Sources,BotNets
2001:df:465:77:: 64 2 Windows Exploits
2001:200:dff:fff1:: 64 128 Phishing
2001:208:: 64 130 Windows Exploits,Phishing
2001:240:2401:15ac:: 64 2 Windows Exploits
2001:240:2401:ae11:: 64 2 Windows Exploits
2001:240:2403:1a08:: 64 2 Windows Exploits
2001:240:2403:3408:: 64 2 Windows Exploits
2001:240:2403:3702:: 64 2 Windows Exploits
2001:240:2403:667c:: 64 2 Windows Exploits
2001:240:2404:d050:: 64 2 Windows Exploits
2001:240:2405:202a:: 64 2 Windows Exploits
2001:240:2405:da28:: 64 138 Windows Exploits,BotNets,Phishing
...
iprep_lookup¶
The iprep_lookup ip-address command can be used to look for an address in the IP Reputation third-party database file.
Sample Output:
iprep_lookup 1.2.2.1
opening database in /var/IpRep/F5IpRep.dat, /var/IpRep/F5IpV6Rep.dat
size of IP reputation database = 24010644, 1629744
iprep threats list for ip = 1.2.2.1 is:
bit 11 - Mobile Threats
iprep_lookup 2001:240:240e:e3f5::
opening database in /var/IpRep/F5IpRep.dat, /var/IpRep/F5IpV6Rep.dat
size of IP reputation database = 24010756, 1629744
iprep threats list for ip = 2001:240:240e:e3f5:: is:
bit 1 - Windows Exploits
Disabling the sidecar¶
The TMM debug sidecar installs by default with the CNFs Controller. You can disable the debug sidecar by setting the debug.enabled parameter to false in the BIG-IP Controller Helm values file:
debug:
enabled: false
Feedback¶
Provide feedback to improve this document by emailing cnfdocs@f5.com.