Debug Sidecar¶
Overview¶
The Service Proxy Pod’s debug sidecar provides a set of command line tools for obtaining low-level, diagnostic data and statistics about the Service Proxy Traffic Management Microkernel (TMM). The debug sidecar deploys by default with the SPK Controller.
Command line tools¶
The table below lists and describes the available command line tools:
Tool | Description |
---|---|
tmctl | Displays various TMM traffic processing statistics, such as pool and virtual server connections. |
core-tmm | Creates a diagnostic core file of the TMM process. |
bdt_cli | Displays TMM networking information such as ARP and route entries. See the bdt_cli section below. |
tmm_cli | Sets the TMM logging level. See the tmm_cli section below. |
mrfdb | Enables reading and writing dSSM database records. See the mrfdb section below. |
configviewer | Displays a log of the configuration objects created and deleted using SPK Custom Resources (CRs). See the configviewer section below. |
tcpdump | Displays packets sent and received on the specified network interface. |
ping | Send ICMP ECHO_REQUEST packets to remote hosts. |
traceroute | Displays the packet route in hops to a remote host. |
Note: Type man f5-tools in the debug container to get a full list of TMM specific commands.
Connecting to the sidecar¶
To connect to the debug sidecar and begin gathering diagnostic information, use the commands below.
Connect to the debug sidecar:
In this example, the debug sidecar is in the spk-ingress Project:
oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
Execute one of the available diagnostic commands:
In this example, ping is used to test connectivity to a remote host with IP address 192.168.10.100:
ping 192.168.10.100
PING 192.168.10.100 (192.168.10.100): 56 data bytes 64 bytes from 192.168.10.100: icmp_seq=0 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=1 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=2 ttl=64 time=0.067 ms 64 bytes from 192.168.10.100: icmp_seq=3 ttl=64 time=0.067 ms
Type Exit to leave the debug sidecar.
Command examples¶
tmctl¶
Use the tmctl tool to query Service Proxy TMM for application traffic processing statistics.
Connect to the debug sidecar:
oc exec -it deploy/f5-tmm -c debug -n <project> -- bash
In this example, the debug sidecar is in the spk-ingress Project:
oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
To view virtual server connection statistics run the following command:
tmctl -d blade virtual_server_stat -s name,clientside.tot_conns
To view pool member connection statistics run the following command:
tmctl -d blade pool_member_stat -s pool_name,serverside.tot_conns
bdt_cli¶
Use the bdt_cli tool to query the Service Proxy TMM for networking data.
Commands:
- arp - Get ARP routes and their status.
- check - Get TMM Check Magic.
- completion - Generate the autocompletion script for the specialized shell.
- connection - Get Connection List.
- help - Help about any command.
- l2forward - Get L2 Forwarding entries.
- route - Get Route List.
Command example:
Connect to the debug sidecar:
oc exec -it deploy/f5-tmm -c debug -n <project> -- bash
In this example, the debug sidecar is in the spk-ingress Project:
oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
Connect to TMM:
bdt_cli -u -s tmm0:8850 [command]
Example of showing routes:
bdt_cli -u -s tmm0:8850 route
routeType:1 isIpv6:false destNet:{ip:{addr:<none>, rd:0} pl:0} gw:{ip:{addr:10.59.147.121, rd:0}} gwType:1 interface:external routeType:1 isIpv6:false destNet:{ip:{addr:10.19.148.120, rd:0} pl:29} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:external routeType:1 isIpv6:false destNet:{ip:{addr:192.168.202.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:internal routeType:0 isIpv6:false destNet:{ip:{addr:169.254.1.1, rd:0} pl:32} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:eth0 routeType:1 isIpv6:false destNet:{ip:{addr:169.254.0.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:tmm
tmm_cli¶
By default, the f5-tmm container logs events at the Notice level. You can use the tmm_cli command to modify the logging level. The logging levels are listed below in the order of message severity. More severe levels generally log messages from the lower severity levels as well.
1-Debug, 2-Informational, 3-Notice, 4-Warning, 5-Error, 6-Critical, 7-Alert, 8-Emergency
Connect to the debug sidecar:
oc exec -it deploy/f5-tmm -c debug -n <project> -- bash
In this example, the debug sidecar is in the spk-ingress Project:
oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
To set the f5-tmm container’s logging level to Debug, run the following command:
tmm_cli -logLevel 1
ok
The f5-tmm container will log an event message simlilar to the following:
Set bigdb var 'log.tmm.level'='Debug'
mrfdb¶
The mrfdb utility enables reading and writing dSSM database records. The mrfdb tool queries the dSSM Database Sentinel Pod, sending commands to the dssmmaster DB, and relaying the response back to the debug sidecar. The mrfdb command uses these four subcomands:
- The IP address of the dSSM Sentinel service to be queried.
- The serverName designating the dSSM server-farm controlled by the dssmmaster DB.
- The type designating the command category: dns46, cgnat, custom.
- The command that is specific to the chosen type (category).
Command example:
Obtain the IP address of the dSSM Sentinel:
In this example, dSSM is installed in the spk-utilities Project.
oc get svc -n spk-utilities
In this example, the Sentinel IP address is 10.203.180.204.
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) f5-dssm-db ClusterIP 10.108.254.57 <none> 6379/TCP f5-dssm-sentinel ClusterIP 10.103.180.204 <none> 26379/TCP
Login to the debug sidecar container:
In this example, the debug sidecar is in the spk-ingress Project.
oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
Run the mrfdb utility:
In this example, the mrfdb utility queries for all DB records.
mrfdb -ipport=10.103.180.204:26379 -serverName=server -displayAllBins
Detailed examples:
For detailed mrfdb usage examples, refer to the following guides:
- The Manual DNS46 Entry section of the F5SPKEgress CR overview.
- The Persistence records section of the F5SPKIngressTCP and F5SPKIngressUDP CR overviews.
configviewer¶
Use the configviewer utility to show events related to installing SPK CRs.
You must set the
CONFIG_VIEWER_ENABLE
parameter totrue
when deploying the SK Controller. For example:tmm: customEnvVars: - name: CONFIG_VIEWER_ENABLE value: "true"
Connect to the debug sidecar:
oc exec -it deploy/f5-tmm -c debug -n <project> -- bash
In this example, the debug sidecar is in the spk-ingress Project:
oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
After deploying a Custom Resource (CR), you can view the current configuration event with the following command:
Note: The example respresents a portion of the TMM configuration.
configviewer --ipport=tmm0:11211 --displayall
GetAll Connect! GetAll Connect Complete! pattern: 006f40782e* binlookup config_viewer_bin Query: get/th /6552fc31.0/* -------------------------------------------------------------------------------------------------- Config for pool_member_list updated at <some date / time> { "name": "apps-nginx-crd-pool-member-list", "id": "apps-nginx-crd-pool-member-list", "members": [ "apps-nginx-crd-pool-member-10.244.1.22", "apps-nginx-crd-pool-member-10.244.1.23", "apps-nginx-crd-pool-member-10.244.2.21" ] }
Persisting files¶
Some diagnostic tools such as tcpdump produce files that require further analysis by F5. When you install the SPK Controller, you can configure the debug.persistence
Helm parameter to ensure diagnostic files created in the debug sidecar container are saved to a filesystem. Use the steps below to verify a PersistentVolume is available, and to configure persistence.
Verify a StoraceClass is available for the debug container:
oc get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE managed-nfs-storage storage.io/nfs Delete Immediate
Set the
persistence.enabled
parameter totrue
, and configure thestorageClass
name:Note: In this example,
managed-nfs-storage
value is obtained from the NAME field in step 1:debug: persistence: enabled: true storageClass: "managed-nfs-storage" accessMode: ReadWriteOnce size: 1Gi
After you deploy the Controller and Service Proxy Pods, find the bound PersistentVolume:
oc get pv | grep f5-debug-sidecar
In this example, the pv is Bound in the spk-ingress Project as expected:
pvc-42a5ef7-5c5f-4518-930f-851abf32c67 1Gi Bound spk-ingress/f5-debug-sidecar managed-nfs-storage
Use the PersistentVolume ID to find the Server name and the Path, or location on the cluster node where diagnostic files are storeed.
Important: Files must be placed in the the debug sidecar’s /shared directory to be persisted.
oc describe pv <pv_id> | grep -iE 'path|server'
In this example, the PersistentVolume ID is pvc-42a5ef7-5c5f-4518-930f-851abf32c67:
oc describe pv pvc-42a5ef7-5c5f-4518-930f-851abf32c67 | grep -iE 'path|server'
The Server and Path information will resemble the following:
Server: provisioner.ocp.f5.com Path: /opt/local-path-provisioner/pvc-42a5ef7-5c5f-4518-930f-851abf32c67_ingress_f5-debug-sidecar
Disabling the sidecar¶
The TMM debug sidecar installs by default with the Controller. You can disable the debug sidecar by setting the debug.enabled
parameter to false
in the Controller Helm values file:
debug:
enabled: false
Feedback¶
Provide feedback to improve this document by emailing spkdocs@f5.com.
Supplemental¶
- The Debug API can run diagnostic commands on a targeted TMM from your local workstation.
- Persistence Volumes