Debug API

The BIG-IP Next for Kuberneters Cluster Wide Controller (CWC) has been enhanced to support the debug API, allowing diagnostic utilities to be executed on any targeted TMM Pod. Diagnostic data is sent and received between the CWC and the debug sidecar via RabbitMQ.

Warning: The RabbitMQ namespace must be specified during installation. If the correct namespace is not provided, the debug API call will fail.

Note: All requests to the Debug API require authentication using the Admin Token. For more information on retrieving and using the token, see the Admin Access and API Restriction.

Diagnostic Utilities

The CWC Debug APIs supports the following diagnostic utilities.

Utility Description
ping Used to perform a ping for the specified IPv4 address.
ping6 Used to perform a ping for the specified IPv6 address.
traceroute Used to perform a traceroute to the specified IPv4 address.
traceroute6 Used to perform a traceroute to the specified IPv6 address.
netkvest Used to perform a netkvest to check connectivity to a remote host from the specified source SNAT pool using the ping and traceroute diagnostic utilities.
tmctl Used to retrieve a table from tmctl with the corresponding statistics (ex: virtual_server_stat)
tcpdump Used to perform a search `tcpdump with the specified filters. The utility will timeout after 30 seconds.
configview Used to perform a configview query either using UUID or CRD type. Send configview --help for usage.
mrfdb Used to perform an mrfdb query. Send mrfdb --help for usage. NOTE: The equals sign between flags and values is not supported. Use spaces to separate arguments.
lsndb Used to perform an lsndb query. Send lsndb with no arguments for usage.
showconn Used to perform a showconn utility with a specified filter and wildcard operations to display the list of connections.
killconn Used to perform a killconn utility with a specified filter operation to delete connections.

CWC Debug REST APIs

The sections below describe the CWC REST APIs. Follow the Procedure section to see how the CWC API can be used to run diagnostic utilities.

POST Request

Follow the POST request to create new job IDs. You can request the CWC to run an utility on a debug container by POSTing a request using /debug API.

POST request CWC API Endpoint

https://f5-spk-cwc.f5-utils:30881/debug

POST request Body

{
 namespace: <kubernetes namespace>, 
 command: <ping/ping6/traceroute/trceroute6/tmctl/tcpdump>, 
 params: <arguments to the utility>, 
 tmm-id: <TMM on which utility to be run> 
}

POST response results

Retrieve the job ID from the POST response, which will provide the Debug API result:

curl --request POST https://f5-spk-cwc.f5-utils:30881/debug \
--cert ./client_certificate.pem \
--key ./client_key.pem \
--cacert ./ca_certificate.pem \
-H "Authorization: Bearer [admin_token]" \
--data-raw '{"namespace": "f5-utils","command":"configview","params":"uuid f5-utils-f5ing-vlan4-vlan","tmmId":"f5-tmm-wgrvm"}'
JobID: bd1ddd07-fbac-48fd-bdca-9843f1c65636

Diagnostic Utilities Examples

The diagnostic utilities below are run in the same directory as the cert-gen directory created when setting up the BIG-IP Next for Kubernetes CWC.

PING

curl --request POST https://f5-spk-cwc.f5-utils:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
-H "Authorization: Bearer [admin_token]" \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"ping\",\"params\":\"10.44.0.6\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}"

TRACEROUTE

curl --request POST https://f5-spk-cwc.f5-utils:30881/debug \ 
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \ 
--key api-server-secrets/ssl/client/secrets/client_key.pem \ 
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
-H "Authorization: Bearer [admin_token]" \
-data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"traceroute\",\"params\":\"10.44.0.6\",\"tmmId\":\"f5-tmm-kbvng\"}"

NETKVEST

Note: The netkvest utility supports only the ping and traceroute diagnostic utilities.

  1. Using ping:

    curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
-H "Authorization: Bearer [admin_token]" \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"netkvest\",\"params\":\"-s,egress-snatpool,-d,22.22.22.100,-u,ping\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}

  2. Using traceroute:

     curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
 --cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
 --key api-server-secrets/ssl/client/secrets/client_key.pem \
 --cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
 -H "Authorization: Bearer [admin_token]" \
 --data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"netkvest\",\"params\":\"-s,egress-snatpool,-d,22.22.22.100,-u,ptraceroute\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}

TCPDUMP

curl --request POST https://f5-spk-cwc.f5-utils:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
-H "Authorization: Bearer [admin_token]" \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"tcpdump\",\"params\":\"-nni,internal,-c,4\",\"tmmId\":\"f5-tmm-kbvng\"}"

TMCTL

curl --request POST https://f5-spk-cwc.f5-utils:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
-H "Authorization: Bearer [admin_token]" \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"tmctl\",\"params\":\"virtual_server_stat\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}"

CONFIGVIEW

The configview can be triggered using UUID or CRD types, as shown below:

  1. Using UUID:

    Procedure for configview to get object by UUID:

    a. Log into the Debug Container:

    kubectl exec -it f5-tmm-qg2mn -n spk-ingress -c debug -- bash

    b. List All Objects and their ID’s by running the below command:

    configview show all | grep id

    c. Use the Debug API to Retrieve an Object by UUID:

     curl --request POST https://f5-spk-cwc.f5-utils:30881/debug \ 
 --cert ./client_certificate.pem \ 
 --key ./client_key.pem -\
 -cacert ./ca_certificate.pem \
 -H "Authorization: Bearer [admin_token]" \
 --data-raw '{"namespace": "spk-ingress","command":"configview","params":"uuid f5-utils-f5ing-vlan4-vlan","tmmId":"f5-tmm-qg2mn"}'

  2. Using CRD:

    a. Listing a specific CRD, for example f5-spk-vlan with name f5ing-vlan2:

     curl --request POST https://f5-spk-cwc.f5-utils:30881/debug \
 --cert api-server-secrets/ssl/client/certs/client_certificate.pem \
 --key api-server-secrets/ssl/client/secrets/client_key.pem \
 --cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
 -H "Authorization: Bearer [admin_token]" \
 --data-raw '{"namespace": "spk-ingress","command":"configview","params":"f5-spk-vlan --name f5ing-vlan2","tmmId":"f5-tmm-7bb9bc4994-8t492"}'

    b. Listing all CRDs, for example f5-spk-vlan in f5-utils namespace:

    curl --request POST https://f5-spk-cwc.f5-utils:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
-H "Authorization: Bearer [admin_token]" \
--data-raw '{"namespace": "spk-ingress","command":"configview","params":"f5-spk-vlan --namespace spk-ingress","tmmId":"f5-tmm-7bb9bc4994-8t492"}'

    Available objects (CRD) for viewing include:

    • f5-spk-staticroute

    • f5-spk-vlan

    • f5-spk-vxlan

    • virtual-server

MRFDB

curl --request POST https://f5-spk-cwc.f5-utils:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
-H "Authorization: Bearer [admin_token]" \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"mrfdb\",\"params\":\"-ipport=10.103.180.204:26379 -serverName=dssm-svc -displayAllBins\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}"

LSNDB

curl --request POST https://f5-spk-cwc.f5-utils:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
-H "Authorization: Bearer [admin_token]" \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"lsndb\",\"params\":\"list all\",\"tmmId\":\"f5-tmm-kbvng\"}"

SHOWCONN - For listing all connections

curl --request POST https://f5-spk-cwc.f5-utils:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
-H "Authorization: Bearer [admin_token]" \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"showconn\",\"params\":\"\",\"tmmId\":\"f5-tmm-68c58b8dfc-hr4qx\"}"

SHOWCONN - For listing a connection with a filter

curl --request POST https://f5-spk-cwc.f5-utils:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
-H "Authorization: Bearer [admin_token]" \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"showconn\",\"params\":\"--cs_server_port 8051\",\"tmmId\":\"f5-tmm-68c58b8dfc-hr4qx\"}"

KILLCONN - For deleting a connection with a filter

curl --request POST https://f5-spk-cwc.f5-utils:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
-H "Authorization: Bearer [admin_token]" \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"killconn\",\"params\":\"--cs_server_port 8051\",\"tmmId\":\"f5-tmm-68c58b8dfc-hr4qx\"}"

GET Requests

Use the Debug API to run a utility, and add jobs to the queue. The job handler routine handles jobs by sending the request to the designated debug sidecar and waits for its reply. The job goes through its life cycle and comes to completion when the task is successfully executed by debug sidecar and replies the response to CWC.

Note: You can create a maximum of 5 jobs, each lasting for up to 6 minutes._

To retrieve the required Job ID response, Jobs list, and TMMs list details from the debug sidecar to CWC, use GET Request.

GET API Request for getting the list of TMMs in a specified namespace

Use the below command to query the list of TMMs in a specified namespace as shown below:

curl -s https://f5-spk-cwc.f5-utils:30881/debug/<namespace>/tmms \
--cert client_certificate.pem \
--key client_key.pem \
--cacert ca_certificate.pem \ 
-H "Authorization: Bearer [admin_token]"

The list of TMMs are displayed with their Running status.

GET API Request for getting the Job ID Response

Use the below command to get the Job ID response to the completed job as shown below:

curl https://f5-spk-cwc.f5-utils:30881/debug/<JOB ID> \
--cert client_certificate.pem \
--key client_key.pem \
--cacert ca_certificate.pem \
-H "Authorization: Bearer [admin_token]"

GET API Request for getting the list of Jobs

Use the below command to get the list of Jobs as shown below:

curl https://f5-spk-cwc.f5-utils:30881/debug/jobs \
--cert client_certificate.pem \
--key client_key.pem \
--cacert ca_certificate.pem \
-H "Authorization: Bearer [admin_token]"

The list of jobs are displayed with status (In Progress / Complete) and description of the commands.

Procedure

Follow these procedure to ping a remote host from the debug sidecar using the CWC Debug API.

  1. As described in the BIG-IP Next for Kubernetes Licensing guide, create a new directory for the CWC REST API certificates:

    mkdir cwc_api

  2. Copy each of the certificates into the new directory (see Create CWC REST API certificates section in Create Cluster Wide Controller Certificates):

    cp api-server-secrets/ssl/client/certs/client_certificate.pem cwc_api

    cp api-server-secrets/ssl/ca/certs/ca_certificate.pem cwc_api

    cp api-server-secrets/ssl/client/secrets/client_key.pem cwc_api

  3. Obtain the name of the TMM Pod(s) in the Project:

    In this example, the CWC is in the f5-utils namespace, and the TMM Pod(s) are in the spk-ingress namespace.

    curl -s https://f5-spk-cwc.f5-utils:30881/debug/spk-ingress/tmms \
--cert cwc_api/client_certificate.pem  \
--key cwc_api/client_key.pem \
--cacert cwc_api/ca_certificate.pem \
-H "Authorization: Bearer [admin_token]"

    In this example, the TMM Pod name is f5-tmm-595985589b-shxx2.

    TMM ID: f5-tmm-595985589b-shxx2	 STATUS: Running

  4. Use the TMM Pod name to create the ping diagnostic job:

    The following example shows the POST request query using curl:

    curl --request POST https://f5-spk-cwc.f5-utils:30881/debug \
--cert cwc_api/client_certificate.pem  \
--key cwc_api/client_key.pem \
--cacert cwc_api/ca_certificate.pem \
-H "Authorization: Bearer [admin_token]" \  
--data-raw '{"namespace":"spk-ingress","command":"ping","params":"192.168.10.10","tmmId":"f5-tmm-595985589b-shxx2"}'

    In this example, the job ID is 4625993b-31e2-4570-8b41-2c1296026c16.

    JobID: 4625993b-31e2-4570-8b41-2c1296026c16

  5. Query the list of available jobs:

    Note: There can be a maximum of 5 jobs, each lasting up to 6 minutes.

    curl -s https://f5-spk-cwc.f5-utils:30881/debug/jobs \
--cert cwc_api/client_certificate.pem  \
--key cwc_api/client_key.pem \
--cacert cwc_api/ca_certificate.pem \  
-H "Authorization: Bearer [admin_token]"

    In the example below, Ping is the most recent job added to the queue.

    JobId: 5775e5e2-1c26-43c9-bf4d-50a87a6ae188	Status: Complete	Desc: Running Tmctl
JobId: 13ec0601-366e-45ff-859b-6c51b321ce84	Status: Complete	Desc: Running Traceroute
JobId: e1fba37b-0a6d-4e16-ae69-db86244f1721	Status: Complete	Desc: Running Tcpdump
JobId: 4625993b-31e2-4570-8b41-2c1296026c16	Status: Complete	Desc: Running Ping

  6. Use the job ID to run the ping diagnostic:

    curl https://f5-spk-cwc.f5-utils:30881/debug/4625993b-31e2-4570-8b41-2c1296026c16 \
--cert cwc_api/client_certificate.pem  \
--key cwc_api/client_key.pem \
--cacert cwc_api/ca_certificate.pem \
-H "Authorization: Bearer [admin_token]"    

    Job ID: 4625993b-31e2-4570-8b41-2c1296026c16
Namespace: spk-ingress
Status: Complete ... 1/1 responses received
Pending Responses: None
Start Time: November/28/2024 - 15:7:32

Last Updated: November/28/2024 - 15:7:32

End Time: November/28/2024 - 15:7:32

TmmID: f5-tmm-595985589b-shxx2
Tmm Job Data: ping 192.168.10.10
PING 192.168.10.10 (192.168.10.10) 56(84) bytes of data.
64 bytes from 192.168.10.10: icmp_seq=1 ttl=255 time=2.40 ms
64 bytes from 192.168.10.10: icmp_seq=2 ttl=255 time=6.09 ms
64 bytes from 192.168.10.10: icmp_seq=3 ttl=255 time=2.17 ms
64 bytes from 192.168.10.10: icmp_seq=4 ttl=255 time=1.42 ms