Debug Sidecar¶
The Debug sidecar provides a set of command line tools for obtaining low-level, diagnostic data and statistics about the Traffic Management Microkernel (TMM).
Command line tools¶
The table below lists and describes the available command line tools:
| Tool | Description |
|---|---|
| tmctl | Displays various TMM traffic processing statistics, such as pool and virtual server connections. |
| core-tmm | Creates a diagnostic core file of the TMM process. |
| bdt_cli | Displays TMM networking information such as ARP and route entries. See the bdt_cli section below. |
| mrfdb | Enables reading and writing dSSM database records. See the mrfdb section below. |
| configview | Displays Custom Resource (CR) configuration objects using their logged UUID. |
| tcpdump | Displays packets sent and received on the specified network interface. |
| ping | Send ICMP ECHO_REQUEST packets to remote hosts. |
| traceroute | Displays the packet route in hops to a remote host. |
Note: Type man f5-tools in the debug container to get a full list of TMM specific commands.
Connecting to the debug sidecar container¶
To connect to the debug sidecar and begin gathering diagnostic information, use the commands below.
Connect to the debug sidecar container:
In this example, the debug sidecar is a container in TMM pod in default Project:
kubectl exec -it f5-tmm-6cdbc6bb65-j2r7d -c debug -n default -- bash
Execute one of the available diagnostic commands:
In this example, ping is used to test connectivity to a remote host with IP address 192.168.10.100:
ping 192.168.10.100
PING 192.168.10.100 (192.168.10.100): 56 data bytes 64 bytes from 192.168.10.100: icmp_seq=0 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=2 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=3 ttl=64 time=0.067 ms
Type exit to leave the debug sidecar.
Command examples¶
tmctl¶
Use the tmctl tool to query TMM for application traffic processing statistics.
Connect to the debug sidecar container:
kubectl exec -it f5-tmm-6cdbc6bb65-j2r7d -c debug -n <project> -- bash
In this example, the debug sidecar is in the default Project:
kubectl exec -it f5-tmm-6cdbc6bb65-j2r7d -c debug -n default -- bash
To view virtual server connection statistics run the following command:
tmctl -d blade virtual_server_stat -s name,clientside.tot_conns
To view pool member connection statistics run the following command:
tmctl -d blade pool_member_stat -s pool_name,serverside.tot_conns
bdt_cli¶
Use the bdt_cli tool to query the TMM for networking data.
Commands:
- arp - Get ARP routes and their status.
- check - Check the grpc connection to TMM.
- completion - Generate the autocompletion script for the specialized shell.
- connection or connection list - Get the list of connections.
- help - Help about any command.
- l2forward - Get L2 Forwarding entries.
- route - Get Route List.
- logLevel - Set the TMM log level.
- connection delete - Delete the connections based on filter operations.
- tcpdumpSSLProvider - Enables a tcpdump option to decrypt pcap data in packet capture.
Supported flags to filter connections for both list and delete commands:
- cs_client_addr - Clientside client IP address
- cs_client_port - Clientside client port
- cs_server_addr - Clientside server IP address
- cs_server_port - Clientside server port
- ss_server_addr - Serverside server IP address
- ss_server_port - Serverside server port
- ss_client_addr - Serverside client IP address
- ss_client_port - Serverside client port
- type - Connection Type
- protocol - Protocol
- idle_time - Idle Time
- connection_id - Connection ID
- vs_name - Virtual Server Name
- cs_client_prefix - Clientside client prefix
- cs_server_prefix - Clientside server prefix
- vlan_name - Vlan Name
Command example:
Connect to the debug sidecar container:
kubectl exec -it f5-tmm-6cdbc6bb65-j2r7d -c debug -n <project> -- bash
In this example, the debug sidecar is in the default Project:
kubectl exec -it f5-tmm-6cdbc6bb65-j2r7d -c debug -n default -- bash
Connect to TMM:
bdt_cli -u -s tmm0:8850 [command]
Example of showing routes:
bdt_cli -u -s tmm0:8850 route
routeType:1 isIpv6:false destNet:{ip:{addr:<none>, rd:0} pl:0} gw:{ip:{addr:10.59.147.121, rd:0}} gwType:1 interface:external routeType:1 isIpv6:false destNet:{ip:{addr:10.19.148.120, rd:0} pl:29} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:external routeType:1 isIpv6:false destNet:{ip:{addr:192.168.202.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:internal routeType:0 isIpv6:false destNet:{ip:{addr:169.254.1.1, rd:0} pl:32} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:eth0 routeType:1 isIpv6:false destNet:{ip:{addr:169.254.0.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:tmm
To set the f5-tmm container’s logging level to Error, run the following command:
The logging levels are listed below in the order of message severity. More severe levels generally log messages from the lower severity levels as well.
1-Debug, 2-Informational, 3-Notice (Default), 4-Warning, 5-Error, 6-Critical, 7-Alert, 8-Emergency
bdt_cli logLevel -l 5
List all connections:
bdt_cli -u -s tmm0:8850 connection
(or)
bdt_cli -u -s tmm0:8850 connection list
List Connection with a filter:
Note: The system supports both filter and wildcard operations for retrieving the list of connections.
bdt_cli -u -s tmm0:8850 connection list --flag
In this example, listing a connection with a filter like Clientside client port is
5506:bdt_cli -u -s tmm0:8850 connection delete --cs_client_port 5506
Delete Connection with a filter:
Note: Currently, the system only supports filter operations but not wildcard for deleting connections.
bdt_cli -u -s tmm0:8850 connection delete --flag
In this example, deleting a connection with a filter like Serverside server port is
8051:bdt_cli -u -s tmm0:8850 connection delete --cs_server_port 8051
mrfdb¶
The mrfdb utility enables reading and writing dSSM database records. The mrfdb tool queries the dSSM Database Sentinel Pod, sending commands to the dssmmaster DB, and relaying the response back to the debug sidecar. The mrfdb command uses these four subcomands:
- The IP address of the dSSM Sentinel service to be queried.
- The serverName designating the dSSM server-farm controlled by the dssmmaster DB.
- The type designating the command category: cgnat, custom.
- The command that is specific to the chosen type (category).
Command example:
Obtain the IP address of the dSSM Sentinel:
In this example, dSSM is installed in the f5-utils Project.
kubectl get svc -n f5-utils
In this example, the Sentinel IP address is 10.203.180.204.
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) f5-dssm-db ClusterIP 10.108.254.57 <none> 6379/TCP f5-dssm-sentinel ClusterIP 10.103.180.204 <none> 26379/TCP
Login to the debug sidecar container:
In this example, the debug sidecar is in the default Project.
kubectl exec -it f5-tmm-6cdbc6bb65-j2r7d -c debug -n default -- bash
Run the mrfdb utility:
In this example, the mrfdb utility queries for all DB records.
mrfdb -ipport=10.103.180.204:26379 -serverName=dssm-svc -displayAllBins
Detailed examples:
For detailed examples using mrfdb, refer to the following:
- The Persistence records section of the F5SPKIngressTCP and F5SPKIngressUDP CR overviews.
configview¶
Use the configview utility to show configuration object created by the installed BIG-IP Next for Kubernetes CRs.
View the TMM deployment logs, and grep for UUID events:
In this example, TMM is in the default Project:
kubectl logs f5-tmm-6cdbc6bb65-j2r7d -n default | grep UUID
In this example, the first log UUID default-net-external-vlan will be used to query with configview.
<134>Jan 1 1:10:11 f5-tmm-7d5b489c5b-fffgt tmm1[36]: 01010058:6: audit log: action: CREATE; UUID: default-net-external-vlan; event: declTmm.vlan; Error: No error
Connect to the debug sidecar container:
In this example, the debug sidecar is in the default Project:
kubectl exec -it f5-tmm-6cdbc6bb65-j2r7d -c debug -n default -- bash
Execute the configview utility:
configview uuid default-net-external-vlan
The example output displays the CR parameters and values.
request:[declTmm.vlan]:{name:"external" id:"default-net-external-vlan" tag:3350 mtu:1500 tagged_interfaces:"1.2"}