Debug API¶

The Service Proxy for Kubernetes (SPK) Cluster Wide Controller (CWC) has been enhanced to support the debug API, allowing diagnostic utilities to be executed on any targeted TMM Pod. Diagnostic data is sent and received between the CWC and the debug sidecar via RabbitMQ.

Warning: The RabbitMQ namespace must be specified during installation. If the correct namespace is not provided, the debug API call will fail. For more information, see TMM values.

Diagnostic Utilities¶

The CWC API supports the following diagnostic utilities.

Utility	Description
ping	Used to perform a ping for the specified IPv4 address.
ping6	Used to perform a ping for the specified IPv6 address.
traceroute	Used to perform a traceroute to the specified IPv4 address.
traceroute6	Used to perform a traceroute to the specified IPv6 address.
netkvest	Used to perform a netkvest to check connectivity to a remote host from the specified source SNAT pool using the ping and traceroute diagnostic utilities.
tmctl	Used to retrieve a table from `tmctl` with the corresponding statistics (ex: virtual_server_stat)
tcpdump	Used to perform a `tcpdump` with the specified filters. The utility will timeout after 30 seconds.
configview	Used to perform a `configview` query either using UUID or CRD type. Send `configview --help` for usage.
mrfdb	Used to perform an `mrfdb` query. Send `mrfdb --help` for usage. NOTE: The equals sign between flags and values is not supported. Use spaces to separate arguments.
lsndb	Used to perform an `lsndb` query. Send `lsndb` with no arguments for usage.
showconn	Used to perform a `showconn` utility with specified filter and wildcard operations to displays the list of connections.
killconn	Used to perform a `killconn` utility with specified filter operation to delete connections.

CWC Debug REST APIs¶

The sections below describe the CWC REST APIs. Use the Procedure section to see how the CWC API can be used to run a diagnostic utilities.

POST Request¶

Use the POST request to create new job IDs. You can request the CWC to run a utility on a debug container by posting a request using debug API.

POST request CWC API Endpoint

https://f5-spk-cwc.spk-telemetry:30881/debug

POST request Body

{
 namespace: <kubernetes namespace>, 
 command: <ping/ping6/traceroute/trceroute6/tmctl/tcpdump>, 
 params: <arguments to the utility>, 
 tmm-id: <TMM in which the utility is to be run> 
}

Diagnostic Utilities Examples¶

The diagnostic utilities below are run in the same directory as the cert-gen directory created when setting up the SPK CWC.

PING

curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"ping\",\"params\":\"10.44.0.6\",\"tmmId\":\"Tf5-tmm-6cdbc6bb65-j2r7d\"}"

TRACEROUTE

curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"traceroute\",\"params\":\"10.44.0.6\",\"tmmId\":\"\f5-tmm-6cdbc6bb65-j2r7d"}

NETKVEST

Note: The netkvest utility supports only the ping and traceroute diagnostic utilities.

Using ping:

curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"netkvest\",\"params\":\"-s,egress-snatpool,-d,22.22.22.100,-u,ping\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}

Using traceroute:

curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"netkvest\",\"params\":\"-s,egress-snatpool,-d,22.22.22.100,-u,traceroute\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}

TCPDUMP

curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"tcpdump\",\"params\":\"-nni,internal,-c,4\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}

TMCTL

curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"tmctl\",\"params\":\"virtual_server_stat\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}"

CONFIGVIEW

The configview can be triggered using UUID or CRD types, as shown below:

Using UUID:

Note: Use the following command to obtain UUIDs from TMM: oc logs deploy/f5-tmm -c f5-tmm | grep -i uuid.

curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"configview\",\"params\":\"uuid spk-ingress-dia-test-app-f5ing-dia-test-app-tcp-profile\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}"

Using CRD:

curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw '{"namespace": "spk-ingress","command":"configview","params":"f5-spk-ingresstcp --name f5ing-testapp","tmmId":"f5-tmm-7bb9bc4994-8t492"}'

The Available objects (CRD) for viewing include:

virtual-server
f5-spk-ingressdiameter
f5-spk-ingressegressudp
f5-spk-ingressgtp
f5-spk-staticroute
f5-spk-vlan
f5-spk-ingresshttp2
f5-spk-ingressngap
f5-spk-ingresssbi
f5-spk-ingresstcp
f5-spk-ingressudp

MRFDB

curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"mrfdb\",\"params\":\"-ipport memcachedhost:11211 -serverName server -displayAllBins\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}"

LSNDB

curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"configview\",\"params\":\"list all\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}"

SHOWCONN - For listing all connections

curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-connection\",\"command\":\"showconn\",\"params\":\"\",\"tmmId\":\"f5-tmm-68c58b8dfc-hr4qx\"}"

SHOWCONN - For listing a connection with a filter

curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-connection\",\"command\":\"showconn\",\"params\":\"--cs_server_port,8051\",\"tmmId\":\"f5-tmm-68c58b8dfc-hr4qx\"}"

KILLCONN - For deleting a connection with a filter

curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-connection\",\"command\":\"killconn\",\"params\":\"--cs_server_port,8051\",\"tmmId\":\"f5-tmm-68c58b8dfc-hr4qx\"}"

GET Requests¶

Use the Debug API to run a utility, and add jobs to the queue. The job handler routine handles jobs by sending the request to the designated debug sidecar and waits for its reply. The job goes through its life cycle and comes to completion when the task is successfully executed by debug sidecar and replies the response to CWC.

Note: You can create a maximum of 5 jobs, each lasting for up to 6 minutes.

To retrieve the required Job ID response, Jobs list, and TMMs list details from the debug sidecar to CWC, use GET Request.

GET API Request for getting the list of TMMs in a specified namespace

Can query to get the list of TMMs in a specified namespace as shown below:

curl -s https://f5-spk-cwc.spk-telemetry:30881/debug/<namespace>/tmms \
--cert client_certificate.pem \
--key client_key.pem \
--cacert ca_certificate.pem

The list of TMMs are displayed with their Running status.

GET API Request for getting the Job ID Response

Can query to get the Job ID response to the completed job as shown below:

curl https://f5-spk-cwc.spk-telemetry:30881/debug/<JOB ID> \
--cert client_certificate.pem \
--key client_key.pem \
--cacert ca_certificate.pem

GET API Request for getting the list of Jobs

Can query to get the list of Jobs as shown below:

curl https://f5-spk-cwc.spk-telemetry:30881/debug/jobs \
--cert client_certificate.pem \
--key client_key.pem \
--cacert ca_certificate.pem

The list of jobs are displayed with status (In Progress / Complete) and description of the commands.

Requirements¶

Ensure you have:

Installed the SPK software.
Installed the SPK CWC.

Procedure¶

Use this procedure to ping a remote host from the debug sidecar using the CWC Debug API.

Change into the directory with the SPK Software files.
As described in the SPK Licensing guide, create a new directory for the CWC REST API certificates.
```
mkdir cwc_api
```

Copy each of the certificates into the new directory.

cp api-server-secrets/ssl/client/certs/client_certificate.pem cwc_api

cp api-server-secrets/ssl/ca/certs/ca_certificate.pem cwc_api

cp api-server-secrets/ssl/client/secrets/client_key.pem cwc_api

Obtain the name of the TMM Pod(s) in the Project.

In this example, the CWC is in the spk-telemetry namespace, and the TMM Pod(s) are in the spk-ingress namespace.

curl -s https://f5-spk-cwc.spk-telemetry:30881/debug/spk-ingress/tmms \
--cert cwc_api/client_certificate.pem  \
--key cwc_api/client_key.pem \
--cacert cwc_api/ca_certificate.pem

In this example, the TMM Pod name is f5-tmm-595985589b-shxx2.

TMM ID: f5-tmm-595985589b-shxx2	 STATUS: Running

Use the TMM Pod name to create the ping diagnostic job.

The following example shows the POST request query using curl:

curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
--cert cwc_api/client_certificate.pem  \
--key cwc_api/client_key.pem \
--cacert cwc_api/ca_certificate.pem \
--data-raw '{"namespace":"spk-ingress","command":"ping","params":"192.168.10.10","tmmId":"f5-tmm-595985589b-shxx2"}'

In this example, the job ID is 4625993b-31e2-4570-8b41-2c1296026c16.

JobID: 4625993b-31e2-4570-8b41-2c1296026c16

Query the list of available jobs.

Note: There can be a maximum of 5 jobs, each lasting up to 6 minutes.

curl -s https://f5-spk-cwc.spk-telemetry:30881/debug/jobs \
--cert cwc_api/client_certificate.pem  \
--key cwc_api/client_key.pem \
--cacert cwc_api/ca_certificate.pem \

In the example below, Ping is the most recent job added to the queue.

JobId: 5775e5e2-1c26-43c9-bf4d-50a87a6ae188	Status: Complete	Desc: Running Tmctl
JobId: 13ec0601-366e-45ff-859b-6c51b321ce84	Status: Complete	Desc: Running Traceroute
JobId: e1fba37b-0a6d-4e16-ae69-db86244f1721	Status: Complete	Desc: Running Tcpdump
JobId: 4625993b-31e2-4570-8b41-2c1296026c16	Status: Complete	Desc: Running Ping

Use the job ID to run the ping diagnostic.

curl https://f5-spk-cwc.spk-telemetry:30881/debug/4625993b-31e2-4570-8b41-2c1296026c16 \
--cert cwc_api/client_certificate.pem  \
--key cwc_api/client_key.pem \
--cacert cwc_api/ca_certificate.pem \

Job ID: 4625993b-31e2-4570-8b41-2c1296026c16
Namespace: spk-ingress
Status: Complete ... 1/1 responses received
Pending Responses: None
Start Time: October/18/2022 - 17:54:55

Last Updated: October/18/2022 - 17:54:55

End Time: October/18/2022 - 17:54:55

TmmID: f5-tmm-595985589b-shxx2
Tmm Job Data: ping 192.168.10.10
PING 192.168.10.10 (192.168.10.10) 56(84) bytes of data.
64 bytes from 192.168.10.10: icmp_seq=1 ttl=255 time=2.40 ms
64 bytes from 192.168.10.10: icmp_seq=2 ttl=255 time=6.09 ms
64 bytes from 192.168.10.10: icmp_seq=3 ttl=255 time=2.17 ms
64 bytes from 192.168.10.10: icmp_seq=4 ttl=255 time=1.42 ms

Feedback

Provide feedback to improve this document by emailing spkdocs@f5.com.

Previous Next