Debug Sidecar¶
Overview¶
The Service Proxy Pod’s debug sidecar provides a set of command line tools for obtaining low-level, diagnostic data and statistics about the Service Proxy Traffic Management Microkernel (TMM). The debug sidecar deploys by default with the SPK Controller.
Command line tools¶
The table below lists and describes the available command line tools:
Tool | Description |
---|---|
tmctl | Displays various TMM traffic processing statistics, such as pool and virtual server connections. |
core-tmm | Creates a diagnostic core file of the TMM process. |
bdt_cli | Displays TMM networking information such as ARP and route entries. See the bdt_cli section below. |
tmm_cli | Sets the TMM logging level. See the tmm_cli section below. |
mrfdb | Enables reading and writing dSSM database records. See the mrfdb section below. |
qkview | Creates a diagnostic data TAR file for F5 support. See the Qkview section below. |
configviewer | Displays a log of the configuration objects created and deleted using SPK Custom Resources (CRs). See the configviewer section below. |
tcpdump | Displays packets sent and received on the specified network interface. |
ping | Send ICMP ECHO_REQUEST packets to remote hosts. |
traceroute | Displays the packet route in hops to a remote host. |
Note: Type man f5-tools in the debug container to get a full list of TMM specific commands.
Connecting to the sidecar¶
To connect to the debug sidecar and begin gathering diagnostic information, use the commands below.
Connect to the debug sidecar:
In this example, the debug sidecar is in the spk-ingress Project:
oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
Execute one of the available diagnostic commands:
In this example, ping is used to test connectivity to a remote host with IP address 192.168.10.100:
ping 192.168.10.100
PING 192.168.10.100 (192.168.10.100): 56 data bytes 64 bytes from 192.168.10.100: icmp_seq=0 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=2 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=3 ttl=64 time=0.067 ms
Type Exit to leave the debug sidecar.
Command examples¶
tmctl¶
Use the tmctl tool to query Service Proxy TMM for application traffic processing statistics.
Connect to the debug sidecar:
oc exec -it deploy/f5-tmm -c debug -n <project> -- bash
In this example, the debug sidecar is in the spk-ingress Project:
oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
To view virtual server connection statistics run the following command:
tmctl -d blade virtual_server_stat -s name,clientside.tot_conns
To view pool member connection statistics run the following command:
tmctl -d blade pool_member_stat -s pool_name,serverside.tot_conns
bdt_cli¶
Use the bdt_cli tool to query the Service Proxy TMM for networking data.
Connect to the debug sidecar:
oc exec -it deploy/f5-tmm -c debug -n <project> -- bash
In this example, the debug sidecar is in the spk-ingress Project:
oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
Connect to TMM referencing the gRPC channel SSL/TL certificates and key:
bdt_cli -tls=true -use_fqdn=true -server_addr=tmm0:8850 \ -ca_file=/etc/ssl/certs/ca_root.crt \ -client_crt=/etc/ssl/certs/f5-ing-demo-f5ingress.crt \ -client_key=/etc/ssl/private/f5-ing-demo-f5ingress.key
Once connected, enter a number representing the network data of interest:
Enter the request type(number or string): 1. check 2. arp 3. connection 4. route 5. exit
The output will resemble the following:
"2" looks like a number. Enter ArpRequest(override fields as necessary, defaults are listed here): e.g. {}
Select the Enter key again to view the networking data:
name:169.254.0.254 ipAddr:169.254.0.254 macAddr:00:01:23:45:67:fe vlan:tmm expire:0 status:permanent name:169.254.0.253 ipAddr:169.254.0.253 macAddr:00:98:76:54:32:10 vlan:tmm expire:0 status:permanent name:169.254.0.1 ipAddr:169.254.0.1 macAddr:00:01:23:45:67:00 vlan:tmm expire:0 status:permanent name:10.244.1.98 ipAddr:10.244.1.98 macAddr:22:22:fe:6d:59:e1 vlan:eth0 expire:0 status:permanent name:10.20.200.210 ipAddr:10.20.200.210 macAddr:96:b3:23:d4:7c:69 vlan:net1 expire:0 status:permanent
tmm_cli¶
By default, the f5-tmm container logs events at the Notice level. You can use the tmm_cli command to modify the logging level. The logging levels are listed below in the order of message severity. More severe levels generally log messages from the lower severity levels as well.
1-Debug, 2-Informational, 3-Notice, 4-Warning, 5-Error, 6-Critical, 7-Alert, 8-Emergency
Connect to the debug sidecar:
oc exec -it deploy/f5-tmm -c debug -n <project> -- bash
In this example, the debug sidecar is in the spk-ingress Project:
oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
To set the f5-tmm container’s logging level to Debug, run the following command:
tmm_cli -logLevel 1
ok
The f5-tmm container will log an event message simlilar to the following:
Set bigdb var 'log.tmm.level'='Debug'
configviewer¶
Use the configviewer utility to show events related to installing SPK CRs.
You must set the
CONFIG_VIEWER_ENABLE
parameter totrue
when deploying the SK Controller. For example:tmm: customEnvVars: - name: CONFIG_VIEWER_ENABLE value: "true"
Connect to the debug sidecar:
oc exec -it deploy/f5-tmm -c debug -n <project> -- bash
In this example, the debug sidecar is in the spk-ingress Project:
oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
After deploying a Custom Resource (CR), you can view the current configuration event with the following command:
Note: The example respresents a portion of the TMM configuration.
configviewer --ipport=tmm0:11211 --displayall
GetAll Connect! GetAll Connect Complete! pattern: 006f40782e* binlookup config_viewer_bin Query: get/th /6552fc31.0/* -------------------------------------------------------------------------------------------------- Config for pool_member_list updated at <some date / time> { "name": "apps-nginx-crd-pool-member-list", "id": "apps-nginx-crd-pool-member-list", "members": [ "apps-nginx-crd-pool-member-10.244.1.22", "apps-nginx-crd-pool-member-10.244.1.23", "apps-nginx-crd-pool-member-10.244.2.21" ] }
mrfdb¶
The mrfdb utility enables reading and writing dSSM database records. Use the steps below to add an F5SPKEgress Custom Resource (CR) DNS46 record.
Obtain the name of the first dSSM Sentinel:
In this example, the dSSM Sentinel is in the spk-utilities Project:
oc get pods -n spk-utilities | grep sentinel-0
In this example, the dSSM Sentinel is named f5-dssm-sentinel-0.
f5-dssm-sentinel-0 1/1 Running
Obtain the IP address of the master dSSM database:
oc logs f5-dssm-sentinel-0 -n spk-utilities | grep master | tail -1
In this example, the master dSSM DB IP address is 10.128.0.221.
Apr 2022 21:02:43.543 * +slave slave 10.131.1.152:6379 10.131.1.152 6379 @ dssmmaster 10.128.0.221 6379
Connect to the TMM debug sidecar:
In this example, the debug sidecar is in the spk-ingress Project:
oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
Add the DNS46 record to the dSSM DB:
In this example, the DB entry maps IPv4 address 10.1.1.1 to IPv6 address 2002::10:1:1:1.
mrfdb -ipport=10.128.0.221:6379 -serverName=server -type=dns46 -set -key=10.1.1.1 -val=2002::10:1:1:1
View the new DNS46 record entry:
mrfdb -ipport=10.128.0.221:6379 -serverName=server -type=dns46 -display=all
t_dns462002::10:1:1:1 10.1.1.1 t_dns4610.1.1.1 2002::10:1:1:1
Delete the DNS46 entry from the dSSM DB:
mrfdb -ipport=10.128.0.221:6379 -serverName=server -type=dns46 -delete -key=10.1.1.1 -val=2002::10:1:1:1
Persisting files¶
Some diagnostic tools such as qkview and tcpdump produce files that require further analysis by F5. When you install the SPK Controller, you can configure the debug.persistence
Helm parameter to ensure diagnostic files created in the debug sidecar container are saved to a filesystem. Use the steps below to verify a PersistentVolume is available, and to configure persistence.
Verify a StoraceClass is available for the debug container:
oc get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE managed-nfs-storage storage.io/nfs Delete Immediate
Set the
persistence.enabled
parameter totrue
, and configure thestorageClass
name:Note: In this example,
managed-nfs-storage
value is obtained from the NAME field in step 1:debug: persistence: enabled: true storageClass: "managed-nfs-storage" accessMode: ReadWriteOnce size: 1Gi
After you deploy the Controller and Service Proxy Pods, find the bound PersistentVolume:
oc get pv | grep f5-debug-sidecar
In this example, the pv is Bound in the spk-ingress Project as expected:
pvc-42a5ef7-5c5f-4518-930f-851abf32c67 1Gi Bound spk-ingress/f5-debug-sidecar managed-nfs-storage
Use the PersistentVolume ID to find the Server name and the Path, or location on the cluster node where diagnostic files are storeed.
Important: Files must be placed in the the debug sidecar’s /shared directory to be persisted.
oc describe pv <pv_id> | grep -iE 'path|server'
In this example, the PersistentVolume ID is pvc-42a5ef7-5c5f-4518-930f-851abf32c67:
oc describe pv pvc-42a5ef7-5c5f-4518-930f-851abf32c67 | grep -iE 'path|server'
The Server and Path information will resemble the following:
Server: provisioner.ocp.f5.com Path: /opt/local-path-provisioner/pvc-42a5ef7-5c5f-4518-930f-851abf32c67_ingress_f5-debug-sidecar
Qkview¶
The qkview utility collects diagnostic and logging information from the f5-tmm container, and stores the data in a Linux TAR file. If you enabled the Fluentd Logging collector, run the qkview utility on f5-fluentd container to gather log files from all of the SPK Pods. Qkview files are typically generated and sent to F5 for further analysis. Use the steps below to run the qkview utility, and copy the file to your local workstation.
Switch to the Service Proxy TMM Pod Project:
In this example, the spk-ingress Project is selected.
oc project spk-ingress
Obtain the name of the Service Proxy TMM Pod:
oc get pods --selector app=f5-tmm
In this example, the Service Proxy TMM Pod name is f5-tmm-79df567d45-ssjv9.
NAME READY STATUS f5-tmm-79df567d45-ssjv9 5/5 Running
Set the Service Proxy TMM Pod name as an environment variable:
In this example, the environment variable is named TMM_POD.
TMM_POD=f5-tmm-79df567d45-ssjv9
Open a remote shell to the TMM Pod’s debug container:
oc rsh -c debug $TMM_POD bash
The shell will display the name of the Service Proxy TMM Pod.
[root@f5-tmm-79df567d45-ssjv9 /]#
Change into the /shared directory mapped to the persistent volume:
cd /shared
Run the qkview utility:
qkview
The qkview file appears similar to the following:
qkview.20210219-223559.tar.gz
Type Exit to exit the debug container.
Copy the Qkview file to your local workstation:
oc rsync -c debug $TMM_POD:/shared/<file> .
In this example, the /shared/qkview.20210219-223559.tar.gz Qkview file is copied to the local workstation.
oc rsync -c debug $TMM_POD:/shared/qkview.20210219-223559.tar.gz .
Switch to the Fluentd logging Pod project:
In this example, the spk-utilities Project is selected.
oc project spk-utilities
Obtain the name of the Fluentd logging Pod:
oc get pods --selector run=f5-fluentd
In this example, the Fluentd logging Pod is named f5-toda-fluentd-768b475dc-pk6bp.
NAME READY STATUS f5-toda-fluentd-768b475dc-pk6bp 1/1 Running
Set the Fluentd logging Pod name as an environment variable:
FLUENTD_POD=f5-toda-fluentd-768b475dc-pk6bp
Connect to the f5-fluentd container:
oc rsh deploy/f5-toda-fluentd bash
Change into the /var/log/f5 directory mapped to the persistent volume:
cd /var/log/f5
Run the qkview utility:
qkview
The Qkview file appears similar to the following, on the worker node’s mapped filesystem:
qkview.20210219-273529.tar.gz
Type Exit to exit the f5-fluent container.
Copy the Qkview file to the local filesystem:
In this example, the file /shared/qkview.20210730-231942.tar.gz is copied to the local workstation.
oc rsync $FLUENTD_POD:/shared/qkview.20210730-231942.tar.gz .
Disabling the sidecar¶
The TMM debug sidecar installs by default with the Controller. You can disable the debug sidecar by setting the debug.enabled
parameter to false
in the Controller Helm values file:
debug:
enabled: false
Feedback¶
Provide feedback to improve this document by emailing spkdocs@f5.com.