Debug API¶
Overview¶
The Service Proxy for Kubernetes (SPK) Cluster Wide Controller (CWC) has been enhanced to support the debug API to run the diagnostic commands on any of the targeted TMM Pods, sending and receiving diagnostic data between the CWC and debug sidecar via RabbitMQ.
This document guides you through understanding and running the CWC debug API.
Diagnostic utilities¶
The CWC API supports the following diagnostic utilities:
Utility | Description |
---|---|
ping | Used to perform a ping for the specified IPv4 address. |
ping6 | Used to perform a ping for the specified IPv6 address. |
traceroute | Used to perform a traceroute to the specified IPv4 address. |
traceroute6 | Used to perform a traceroute to the specified IPv6 address. |
tmctl | Used to retrieve a table from tmctl with the corresponding statistics (ex: virtual_server_stat) |
tcpdump | Used to perform a tcpdump with the specified filters. Command will timeout after 30 seconds. |
configview | Used to perform a configview query either using UUID or CRD type. Send configview --help for usage. |
mrfdb | Used to perform an mrfdb query. Send mrfdb --help for usage. NOTE: the equals sign between flags and values is not supported. Use spaces to separate arguments. |
lsndb | Used to perform an lsndb query. Send lsndb with no arguments for usage. |
showconn | Used to perform a showconn command with specified filter and wildcard operations to displays the list of connections. |
killconn | Used to perform a killconn with specified filter operation to delete connections. |
Alternate namespaces¶
When the SPK CWC controller.f5_lic_helper.cwcNamespace
parameter is set to a namespace other than default, the debug.cwcNamespace
parameter must be set to the same value in the SPK Controller Helm values file. In the example below, the parameter is set to spk-telemetry.
debug:
rabbitmqNamespace: "spk-telemetry"
CWC Debug REST APIs¶
The sections below describe the CWC REST APIs. Use the Procedure section to see how the CWC API can be used to run a diagnostic command.
POST Request¶
Use the POST request to create new job IDs. You can request the CWC to run a utility on a debug container by POSTing a request using /debug API.
POST request CWC API Endpoint
https://f5-spk-cwc.spk-telemetry:30881/debug
POST request Body
{
namespace: <kubernetes namespace>,
command: <ping/ping6/traceroute/trceroute6/tmctl/tcpdump>,
params: <arguments to the utility>,
tmm-id: <TMM in which the utility is to be run>
}
Command examples¶
The commands below are run in the same directory as the cert-gen directory created when setting up the SPK CWC.
PING
curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"ping\",\"params\":\"10.44.0.6\",\"tmmId\":\"Tf5-tmm-6cdbc6bb65-j2r7d\"}"
TRACEROUTE
curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"traceroute\",\"params\":\"10.44.0.6\",\"tmmId\":\"\f5-tmm-6cdbc6bb65-j2r7d"}
TCPDUMP
curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"tcpdump\",\"params\":\"-nni,internal,-c,4\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}
TMCTL
curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"tmctl\",\"params\":\"virtual_server_stat\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}"
CONFIGVIEW
Configview can be triggered either by UUID or CRD type.
Using UUID:
Note: Use the following command to obtain UUIDs from TMM: oc logs deploy/f5-tmm -c f5-tmm | grep -i uuid
.
curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"configview\",\"params\":\"uuid spk-ingress-dia-test-app-f5ing-dia-test-app-tcp-profile\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}"
Using CRD:
curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw '{"namespace": "spk-ingress","command":"configview","params":"f5-spk-ingresstcp --name f5ing-testapp","tmmId":"f5-tmm-7bb9bc4994-8t492"}'
Available objects (CRD) for viewing include:
- virtual-server
- f5-spk-ingressdiameter
- f5-spk-ingressegressudp
- f5-spk-ingressgtp
- f5-spk-staticroute
- f5-spk-vlan
- f5-spk-ingresshttp2
- f5-spk-ingressngap
- f5-spk-ingresssbi
- f5-spk-ingresstcp
- f5-spk-ingressudp
MRFDB
curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"mrfdb\",\"params\":\"-ipport memcachedhost:11211 -serverName server -displayAllBins\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}"
LSNDB
curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"configview\",\"params\":\"list all\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}"
SHOWCONN - For listing all connections
curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-connection\",\"command\":\"showconn\",\"params\":\"\",\"tmmId\":\"f5-tmm-68c58b8dfc-hr4qx\"}"
SHOWCONN - For listing a connection with a filter
curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-connection\",\"command\":\"showconn\",\"params\":\"--cs_server_port,8051\",\"tmmId\":\"f5-tmm-68c58b8dfc-hr4qx\"}"
KILLCONN - For deleting a connection with a filter
curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-connection\",\"command\":\"killconn\",\"params\":\"--cs_server_port,8051\",\"tmmId\":\"f5-tmm-68c58b8dfc-hr4qx\"}"
GET Requests¶
Use the Debug API to run a utility, and add jobs to the queue. The job handler routine handles jobs by sending the request to the designated debug sidecar and waits for its reply. The job goes through its life cycle and comes to completion when the task is successfully executed by debug sidecar and replies the response to CWC.
Note: You can create a maximum of 5 jobs, each lasting for up to 6 minutes.
To retrieve the required Job ID response, Jobs list, and TMMs list details from the debug sidecar to CWC, use GET Request.
GET API Request for getting the list of TMMs in a specified namespace
Can query to get the list of TMMs in a specified namespace as shown below:
curl -s https://f5-spk-cwc.spk-telemetry:30881/debug/<namespace>/tmms \
--cert client_certificate.pem \
--key client_key.pem \
--cacert ca_certificate.pem
The list of TMMs are displayed with their Running status.
GET API Request for getting the Job ID Response
Can query to get the Job ID response to the completed job as shown below:
curl https://f5-spk-cwc.spk-telemetry:30881/debug/<JOB ID> \
--cert client_certificate.pem \
--key client_key.pem \
--cacert ca_certificate.pem
GET API Request for getting the list of Jobs
Can query to get the list of Jobs as shown below:
curl https://f5-spk-cwc.spk-telemetry:30881/debug/jobs \
--cert client_certificate.pem \
--key client_key.pem \
--cacert ca_certificate.pem
The list of jobs are displayed with status (In Progress / Complete) and description of the commands.
Procedure¶
Use this procedure to ping a remote host from the debug sidecar using the CWC Debug API.
Change into the directory with the SPK Software files.
As described in the SPK Licensing guide, create a new directory for the CWC REST API certificates:
mkdir cwc_api
Copy each of the certificates into the new directory:
cp api-server-secrets/ssl/client/certs/client_certificate.pem cwc_api
cp api-server-secrets/ssl/ca/certs/ca_certificate.pem cwc_api
cp api-server-secrets/ssl/client/secrets/client_key.pem cwc_api
Obtain the name of the TMM Pod(s) in the Project:
In this example, the CWC is in the spk-telemetry namespace, and the TMM Pod(s) are in the spk-ingress namespace.
curl -s https://f5-spk-cwc.spk-telemetry:30881/debug/spk-ingress/tmms \ --cert cwc_api/client_certificate.pem \ --key cwc_api/client_key.pem \ --cacert cwc_api/ca_certificate.pem
In this example, the TMM Pod name is f5-tmm-595985589b-shxx2.
TMM ID: f5-tmm-595985589b-shxx2 STATUS: Running
Use the TMM Pod name to create the ping diagnostic job:
The following example shows the POST request query using curl:
curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \ --cert cwc_api/client_certificate.pem \ --key cwc_api/client_key.pem \ --cacert cwc_api/ca_certificate.pem \ --data-raw '{"namespace":"spk-ingress","command":"ping","params":"192.168.10.10","tmmId":"f5-tmm-595985589b-shxx2"}'
In this example, the job ID is 4625993b-31e2-4570-8b41-2c1296026c16.
JobID: 4625993b-31e2-4570-8b41-2c1296026c16
Query the list of available jobs:
Note: There can be a maximum of 5 jobs, each lasting up to 6 minutes.
curl -s https://f5-spk-cwc.spk-telemetry:30881/debug/jobs \ --cert cwc_api/client_certificate.pem \ --key cwc_api/client_key.pem \ --cacert cwc_api/ca_certificate.pem \
In the example below, Ping is the most recent job added to the queue.
JobId: 5775e5e2-1c26-43c9-bf4d-50a87a6ae188 Status: Complete Desc: Running Tmctl JobId: 13ec0601-366e-45ff-859b-6c51b321ce84 Status: Complete Desc: Running Traceroute JobId: e1fba37b-0a6d-4e16-ae69-db86244f1721 Status: Complete Desc: Running Tcpdump JobId: 4625993b-31e2-4570-8b41-2c1296026c16 Status: Complete Desc: Running Ping
Use the job ID to run the ping diagnostic:
curl https://f5-spk-cwc.spk-telemetry:30881/debug/4625993b-31e2-4570-8b41-2c1296026c16 \ --cert cwc_api/client_certificate.pem \ --key cwc_api/client_key.pem \ --cacert cwc_api/ca_certificate.pem \
Job ID: 4625993b-31e2-4570-8b41-2c1296026c16 Namespace: spk-ingress Status: Complete ... 1/1 responses received Pending Responses: None Start Time: October/18/2022 - 17:54:55 Last Updated: October/18/2022 - 17:54:55 End Time: October/18/2022 - 17:54:55 TmmID: f5-tmm-595985589b-shxx2 Tmm Job Data: ping 192.168.10.10 PING 192.168.10.10 (192.168.10.10) 56(84) bytes of data. 64 bytes from 192.168.10.10: icmp_seq=1 ttl=255 time=2.40 ms 64 bytes from 192.168.10.10: icmp_seq=2 ttl=255 time=6.09 ms 64 bytes from 192.168.10.10: icmp_seq=3 ttl=255 time=2.17 ms 64 bytes from 192.168.10.10: icmp_seq=4 ttl=255 time=1.42 ms
Feedback¶
Provide feedback to improve this document by emailing spkdocs@f5.com.