Debug API

Overview

The Service Proxy for Kubernetes (SPK) Cluster Wide Controller (CWC) has been enhanced to support the debug API to run the diagnostic commands on any of the targeted TMM Pods, sending and receiving diagnostic data between the CWC and debug sidecar via RabbitMQ.

This document guides you through understanding and running the CWC debug API.

Diagnostic utilities

The CWC API supports the following diagnostic utilities:

Utility Description
ping Used to perform a ping for the specified IPv4 address.
ping6 Used to perform a ping for the specified IPv6 address.
traceroute Used to perform a traceroute to the specified IPv4 address.
traceroute6 Used to perform a traceroute to the specified IPv6 address.
tmctl Used to retrieve a table from tmctl with the corresponding statistics (ex: virtual_server_stat)
tcpdump Used to perform a tcpdump with the specified filters. Command will timeout after 30 seconds.
configview Used to perform a configview query either using UUID or CRD type. Send configview --help for usage.
mrfdb Used to perform an mrfdb query. Send mrfdb --help for usage. NOTE: the equals sign between flags and values is not supported. Use spaces to separate arguments.
lsndb Used to perform an lsndb query. Send lsndb with no arguments for usage.
showconn Used to perform a showconn command with specified filter and wildcard operations to displays the list of connections.
killconn Used to perform a killconn with specified filter operation to delete connections.

Alternate namespaces

When the SPK CWC controller.f5_lic_helper.cwcNamespace parameter is set to a namespace other than default, the debug.cwcNamespace parameter must be set to the same value in the SPK Controller Helm values file. In the example below, the parameter is set to spk-telemetry.

debug:
  rabbitmqNamespace: "spk-telemetry"

CWC Debug REST APIs

The sections below describe the CWC REST APIs. Use the Procedure section to see how the CWC API can be used to run a diagnostic command.

POST Request

Use the POST request to create new job IDs. You can request the CWC to run a utility on a debug container by POSTing a request using /debug API.

POST request CWC API Endpoint

https://f5-spk-cwc.spk-telemetry:30881/debug 

POST request Body

{
 namespace: <kubernetes namespace>, 
 command: <ping/ping6/traceroute/trceroute6/tmctl/tcpdump>, 
 params: <arguments to the utility>, 
 tmm-id: <TMM in which the utility is to be run> 
}

Command examples

The commands below are run in the same directory as the cert-gen directory created when setting up the SPK CWC.

PING

curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"ping\",\"params\":\"10.44.0.6\",\"tmmId\":\"Tf5-tmm-6cdbc6bb65-j2r7d\"}"

TRACEROUTE

curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"traceroute\",\"params\":\"10.44.0.6\",\"tmmId\":\"\f5-tmm-6cdbc6bb65-j2r7d"}

TCPDUMP

curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"tcpdump\",\"params\":\"-nni,internal,-c,4\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}

TMCTL

curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"tmctl\",\"params\":\"virtual_server_stat\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}"

CONFIGVIEW

Configview can be triggered either by UUID or CRD type.

Using UUID:

Note: Use the following command to obtain UUIDs from TMM: oc logs deploy/f5-tmm -c f5-tmm | grep -i uuid.

curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"configview\",\"params\":\"uuid spk-ingress-dia-test-app-f5ing-dia-test-app-tcp-profile\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}"

Using CRD:

curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw '{"namespace": "spk-ingress","command":"configview","params":"f5-spk-ingresstcp --name f5ing-testapp","tmmId":"f5-tmm-7bb9bc4994-8t492"}'

Available objects (CRD) for viewing include:

  • virtual-server
  • f5-spk-ingressdiameter
  • f5-spk-ingressegressudp
  • f5-spk-ingressgtp
  • f5-spk-staticroute
  • f5-spk-vlan
  • f5-spk-ingresshttp2
  • f5-spk-ingressngap
  • f5-spk-ingresssbi
  • f5-spk-ingresstcp
  • f5-spk-ingressudp

MRFDB

curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"mrfdb\",\"params\":\"-ipport memcachedhost:11211 -serverName server -displayAllBins\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}"

LSNDB

curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-ingress\",\"command\":\"configview\",\"params\":\"list all\",\"tmmId\":\"f5-tmm-6cdbc6bb65-j2r7d\"}"

SHOWCONN - For listing all connections

curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-connection\",\"command\":\"showconn\",\"params\":\"\",\"tmmId\":\"f5-tmm-68c58b8dfc-hr4qx\"}"

SHOWCONN - For listing a connection with a filter

curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-connection\",\"command\":\"showconn\",\"params\":\"--cs_server_port,8051\",\"tmmId\":\"f5-tmm-68c58b8dfc-hr4qx\"}"

KILLCONN - For deleting a connection with a filter

curl --request POST https://f5-spk-cwc.default:30881/debug \
--cert api-server-secrets/ssl/client/certs/client_certificate.pem  \
--key api-server-secrets/ssl/client/secrets/client_key.pem \
--cacert api-server-secrets/ssl/ca/certs/ca_certificate.pem \
--data-raw "{\"namespace\": \"spk-connection\",\"command\":\"killconn\",\"params\":\"--cs_server_port,8051\",\"tmmId\":\"f5-tmm-68c58b8dfc-hr4qx\"}"

GET Requests

Use the Debug API to run a utility, and add jobs to the queue. The job handler routine handles jobs by sending the request to the designated debug sidecar and waits for its reply. The job goes through its life cycle and comes to completion when the task is successfully executed by debug sidecar and replies the response to CWC.

Note: You can create a maximum of 5 jobs, each lasting for up to 6 minutes.

To retrieve the required Job ID response, Jobs list, and TMMs list details from the debug sidecar to CWC, use GET Request.

GET API Request for getting the list of TMMs in a specified namespace

Can query to get the list of TMMs in a specified namespace as shown below:

curl -s https://f5-spk-cwc.spk-telemetry:30881/debug/<namespace>/tmms \
--cert client_certificate.pem \
--key client_key.pem \
--cacert ca_certificate.pem

The list of TMMs are displayed with their Running status.

GET API Request for getting the Job ID Response

Can query to get the Job ID response to the completed job as shown below:

curl https://f5-spk-cwc.spk-telemetry:30881/debug/<JOB ID> \
--cert client_certificate.pem \
--key client_key.pem \
--cacert ca_certificate.pem

GET API Request for getting the list of Jobs

Can query to get the list of Jobs as shown below:

curl https://f5-spk-cwc.spk-telemetry:30881/debug/jobs \
--cert client_certificate.pem \
--key client_key.pem \
--cacert ca_certificate.pem

The list of jobs are displayed with status (In Progress / Complete) and description of the commands.

Requirements

Ensure you have:

Procedure

Use this procedure to ping a remote host from the debug sidecar using the CWC Debug API.

  1. Change into the directory with the SPK Software files.

  2. As described in the SPK Licensing guide, create a new directory for the CWC REST API certificates:

    mkdir cwc_api
    
  3. Copy each of the certificates into the new directory:

    cp api-server-secrets/ssl/client/certs/client_certificate.pem cwc_api
    
    cp api-server-secrets/ssl/ca/certs/ca_certificate.pem cwc_api
    
    cp api-server-secrets/ssl/client/secrets/client_key.pem cwc_api
    
  4. Obtain the name of the TMM Pod(s) in the Project:

    In this example, the CWC is in the spk-telemetry namespace, and the TMM Pod(s) are in the spk-ingress namespace.

    curl -s https://f5-spk-cwc.spk-telemetry:30881/debug/spk-ingress/tmms \
    --cert cwc_api/client_certificate.pem  \
    --key cwc_api/client_key.pem \
    --cacert cwc_api/ca_certificate.pem
    

    In this example, the TMM Pod name is f5-tmm-595985589b-shxx2.

    TMM ID: f5-tmm-595985589b-shxx2	 STATUS: Running
    
  5. Use the TMM Pod name to create the ping diagnostic job:

    The following example shows the POST request query using curl:

    curl --request POST https://f5-spk-cwc.spk-telemetry:30881/debug \
    --cert cwc_api/client_certificate.pem  \
    --key cwc_api/client_key.pem \
    --cacert cwc_api/ca_certificate.pem \
    --data-raw '{"namespace":"spk-ingress","command":"ping","params":"192.168.10.10","tmmId":"f5-tmm-595985589b-shxx2"}'
    

    In this example, the job ID is 4625993b-31e2-4570-8b41-2c1296026c16.

    JobID: 4625993b-31e2-4570-8b41-2c1296026c16
    
  6. Query the list of available jobs:

    Note: There can be a maximum of 5 jobs, each lasting up to 6 minutes.

    curl -s https://f5-spk-cwc.spk-telemetry:30881/debug/jobs \
    --cert cwc_api/client_certificate.pem  \
    --key cwc_api/client_key.pem \
    --cacert cwc_api/ca_certificate.pem \
    

    In the example below, Ping is the most recent job added to the queue.

    JobId: 5775e5e2-1c26-43c9-bf4d-50a87a6ae188	Status: Complete	Desc: Running Tmctl
    JobId: 13ec0601-366e-45ff-859b-6c51b321ce84	Status: Complete	Desc: Running Traceroute
    JobId: e1fba37b-0a6d-4e16-ae69-db86244f1721	Status: Complete	Desc: Running Tcpdump
    JobId: 4625993b-31e2-4570-8b41-2c1296026c16	Status: Complete	Desc: Running Ping
    
  7. Use the job ID to run the ping diagnostic:

    curl https://f5-spk-cwc.spk-telemetry:30881/debug/4625993b-31e2-4570-8b41-2c1296026c16 \
    --cert cwc_api/client_certificate.pem  \
    --key cwc_api/client_key.pem \
    --cacert cwc_api/ca_certificate.pem \
    
    Job ID: 4625993b-31e2-4570-8b41-2c1296026c16
    Namespace: spk-ingress
    Status: Complete ... 1/1 responses received
    Pending Responses: None
    Start Time: October/18/2022 - 17:54:55
    
    Last Updated: October/18/2022 - 17:54:55
    
    End Time: October/18/2022 - 17:54:55
    
    TmmID: f5-tmm-595985589b-shxx2
    Tmm Job Data: ping 192.168.10.10
    PING 192.168.10.10 (192.168.10.10) 56(84) bytes of data.
    64 bytes from 192.168.10.10: icmp_seq=1 ttl=255 time=2.40 ms
    64 bytes from 192.168.10.10: icmp_seq=2 ttl=255 time=6.09 ms
    64 bytes from 192.168.10.10: icmp_seq=3 ttl=255 time=2.17 ms
    64 bytes from 192.168.10.10: icmp_seq=4 ttl=255 time=1.42 ms
    

Feedback

Provide feedback to improve this document by emailing spkdocs@f5.com.