How to: Log in and use the Debug Utility¶
Overview¶
Follow this procedure to turn on the Instance Debug Utility and troubleshoot data plane issues by accessing logs or using the Debug Utility tools available for the BIG-IP Next Instance within BIG-IP Next Central Manager.
The following sections detail the functionality of the Instance Debug Utility:
Procedures¶
Retrieve client SSH public key
Enable Debug Utility using BIG-IP Next Central Manager
Log into the Debug Utility
Disable Debug Utility
Usage Examples
If you are unable to access the Debug Utility using BIG-IP Next Central Manager GUI or API, see Enable the Debug utility on Instance API via Mac or Linux.
Prerequisites¶
BIG-IP Next Central Manager
BIG-IP Next VE or VELOS instance managed by BIG-IP Central Manager
Command-line access on Windows, Mac, or Linux workstation
SSH client
Client SSH public key
Retrieve client SSH public key¶
Retrieve the client workstation public SSH key. The key will be used later configuration steps.
Windows¶
Getting Windows client SSH key
Confirm the SSH key exists by running
type ~/.ssh\id_rsa.pubin Powershell. If there is output it menas the key already exists. If this is true, skip ahead to Retrieving client public SSH key on Windows. Otherwise continue to set 2.Generate SSH keys by running
ssh-keygenin Powershell.
Example
PS C:\Users\bob> ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (C:\Users\bob/.ssh/id_rsa):
Created directory 'C:\Users\bob/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\bob/.ssh/id_rsa.
Your public key has been saved in C:\Users\bob/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:AWfaXFw52E8lPgha0AA1PbJqpGZAP3+icdDAS6vPvSM bob@DESKTOP-CM158QA
The key's randomart image is:
+---[RSA 3072]----+
| . ooB*++.....|
| . + B.=*.+o.. |
| . o =. =o ..+o |
| . * o .. .. |
| o * .S |
| . = * . |
| = * o |
| E o |
| ..o |
+----[SHA256]-----+
Retrieving client public SSH key on Windows¶
In Powershell, output the public SSH key for the user. Save the key. It will be used when enabling the Debug utility.
type ~/.ssh\id_rsa.pub
Example
PS C:\Users\bob> type ~/.ssh\id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2h/MRdduZvE/h15vrf0jKZTYgTUSWmLhijmsFcTNUtYiuRcGmdxmM053r4zWWJe7NcAzYj8m7VSgNVggBWqrKoFhKRZCHHl9pwn1Gb5kekensdLr6H7B4l6xMx5FOMdXzXj98tgD1194pOH3EewwI3fI/QgcGhXULplXjMwtr7zlrGBNVSCnohnGEiLniIw8bR3B/glTOsqQRfU9PT5bPw4yigUcPzt2h9SJNh2qo3TLCZkZXSHOQl9NEvfAsWzw2/0n+jpC6AKq+8uy6iTuhDe7mT/09AwjiPFZ0Zc3lXL9WejP5ED4qwXYjXsI5u/mFU2guTThjqOT4WccCHZznJX6Te9qiBOz0ZUDWzvVAyeKmEci+cKtPUvrpc/zVYyotujNh7p21Pd+0QER+yXKUQDvwGKqWGRKVNb5F1fpNxRkbdtGL/74Rgf8Z+vAH709mSVK1FTzVOJTExMdkZV50Ntxh9X7/FeQJT+2ra0PMG5M386O9Rmkm9Gxq4xpEask= bob@WIN10
Mac and Linux¶
Getting Mac or Linux client SSH key
On the client workstation command prompt, display the public SSH key of the current user.
Example
cat ~/.ssh/id_rsa.pub
Enable Debug Utility using BIG-IP Next Central Manager¶
Log in to the BIG-IP Next Central Manager.
Navigate to the Infrastructure section.
Select the BIG-IP instance in the name column.
Select Debug.
Provide the client SSH public key.
Click the Start Debug Session button.
Prerequisites
Authenticate with the BIG-IP Next Central Manager API. For details refer to How to: Authenticate with the BIG-IP Next Central Manager API.
Note: A username should be created using a combination of uppercase letters (A-Z), lowercase letters (a-z), and numbers (0-9) to enable the debug utility. Although the following are allowed, avoid using them as they will impact enabling the debug utility:
underscores (_), dashes (-) or dots (.).
starting the username with an uppercase letter or number.
Use the following APIs to enable/disable debug utility:
Check the enabled/disabled status of the BIG-IP Next Instance Debug Sidecar by sending GET request to
device/v1/proxy/{{instance_id}}?path=/systems/{{instance-systems-ID}}/dataplane-debugendpoint.
GET https://{{cm_mgmt_ip}}/api/device/v1/proxy/{{instance_id}}?path=/systems/{{instance-systems-ID}}/dataplane-debug
Get the
instance_idby sending GET request tospaces/default/instancesendpoint.
GET https://cm_mgmt_ip/api/v1/spaces/default/instances
Get the
instance_system_idby sending GET request todevice/v1/proxy/${instanceId}?path=systemsendpoint.
GET https://{{cm_mgmt_ip}}/api/device/v1/proxy/${instanceId}?path=systems
Enable Debug utility by sending a PUT request to
api/device/v1/proxy/{{instace_id}}?path=/actions/systems/{{instance_system_id}}/dataplane-debug/enableendpoint.
PUT https://{{ip}}/api/device/v1/proxy/{{instace_id}}?path=/actions/systems/{{instance_system_id}}/dataplane-debug/enable
For the request payload, use the following example, modifying the values as required.
{
"sshPublicKey": "{{ssh_key}}",
"allowedIps": [
"0.0.0.0/0"
],
"username": "{{username}}",
"port": "{{port}}"
}
Note: Users with usernames that do not follow the pattern ‘^[a-z][a-z0-9]’—specifically those starting with an uppercase letter, number, or containing underscores (_), dashes (-), or dots (.)—will need to enable the utility via OpenAPI.
Port value will always be default to 2222.
Log in to the Debug Utility¶
Copy the SSH Access connection command from the BIG-IP Central Manager.
Use the connection command in a Powershell, Linux, or Mac command prompt to log in to the Debug utility.
Example
ssh admin@192.0.2.131 -p 1234
Disable Debug Utility¶
Disable the Debug utility after troubleshooting is completed.
Log in to BIG-IP Next Central Manager.
Navigate to the Infrastructure section.
Select the BIG-IP instance in the name column.
Select Debug.
Click the Stop Debug Session button to disable the utility.
Prerequisites
Authenticate with the BIG-IP Next Central Manager API. For details refer to How to: Authenticate with the BIG-IP Next Central Manager API.
Use the following APIs to enable/disable debug utility:
Check the enabled/disabled status of the BIG-IP Next Instance Debug Sidecar by sending GET request to
device/v1/proxy/{{instance_id}}?path=/systems/{{instance-systems-ID}}/dataplane-debugendpoint.
GET https://{{cm_mgmt_ip}}/api/device/v1/proxy/{{instance_id}}?path=/systems/{{instance-systems-ID}}/dataplane-debug
Get the
instance_idby sending GET request tospaces/default/instancesendpoint.
GET https://cm_mgmt_ip/api/v1/spaces/default/instances
Get the
instance_system_idby sending GET request todevice/v1/proxy/${instanceId}?path=systemsendpoint.
GET https://{{cm_mgmt_ip}}/api/device/v1/proxy/${instanceId}?path=systems
Disable Debug utility by sending a PUT request to
/api/device/v1/proxy/{{instance_id}}?path=/actions/systems/{{instance-systems-ID}}/dataplane-debug/disableendpoint.
PUT https://{{CM}}/api/device/v1/proxy/{{instance_id}}?path=/actions/systems/{{instance-systems-ID}}/dataplane-debug/disable
Access to the Debug Utility Tools¶
F5 custom tools¶
Tool |
Description |
|---|---|
Displays various TMM traffic processing statistics |
|
Displays TMM networking information such as ARP and route entries with various sub-commands |
General networking tools¶
Tool |
Description |
|---|---|
Captures and replays packets sent and received on network interfaces |
|
Tests reachability of remote hosts on IP networks using ICMP |
|
Displays the packet route in hops to a remote host |
|
Retrieves files using HTTP, HTTPS, FTP and FTPS |
|
CLI to OpenSL toolkit |
Access to Instance Logs¶
By running the following commands, you can view all Instance logs available within the debug utility.
Example
cd /logs/ Ls
Note: You can also review the f5-fsm-tmm log using QKView or a 3rd party tool to view the log archive.
You can tail each file or query it for a specific value.
Example
Search for the keyword
TMC ERRORin the f5-fsm-tmm log to find configuration errors.nano f5-fsm-tmm-0.log --> search for specific key word "TMC ERROR" OR /cat f5-fsm-tmm-0.log | grep -i "TMC ERROR"
Example git Tail f5-dssm log to observe real-time logging messages.
Tail -f f5-dssm.log
Sample output
/tail -f f5-dssm.log {"ts":"2024-10-11 20:33:37.214254 UTC","sev":"unknown","ct":"f5-dssm","stream":"","scid":"5546759f9f-t7k9m","sysid":"a0f22942-d2d2-e364-f48b-944eb8db386e","mid":"unknown","args":[],"log":"Existing State from ha JSON is STANDALONE"} {"ts":"2024-10-11 20:33:37.214403 UTC","sev":"unknown","ct":"f5-dssm","stream":"","scid":"5546759f9f-t7k9m","sysid":"a0f22942-d2d2-e364-f48b-944eb8db386e","mid":"unknown","args":[],"log":"Got inotify event: /f5-ha/ MOVED_TO ha.json"} {"ts":"2024-10-11 20:33:37.226358 UTC","sev":"unknown","ct":"f5-dssm","stream":"","scid":"5546759f9f-t7k9m","sysid":"a0f22942-d2d2-e364-f48b-944eb8db386e","mid":"unknown","args":[],"log":"Existing State from ha JSON is STANDALONE"} {"ts":"2024-10-11 20:33:37.226717 UTC","sev":"unknown","ct":"f5-dssm","stream":"","scid":"5546759f9f-t7k9m","sysid":"a0f22942-d2d2-e364-f48b-944eb8db386e","mid":"unknown","args":[],"log":"Got inotify event: /f5-ha/ MODIFY ha.json"} {"ts":"2024-10-11 20:33:37.236320 UTC","sev":"unknown","ct":"f5-dssm","stream":"","scid":"5546759f9f-t7k9m","sysid":"a0f22942-d2d2-e364-f48b-944eb8db386e","mid":"unknown","args":[],"log":"Existing State from ha JSON is STANDALONE"} {"ts":"2024-10-11 21:32:29.074046 UTC","sev":"unknown","ct":"f5-dssm","stream":"","scid":"5546759f9f-t7k9m","sysid":"a0f22942-d2d2-e364-f48b-944eb8db386e","mid":"unknown","args":[],"log":"266:M 11 Oct 2024 21:32:29.073 * 1 changes in 3600 seconds. Saving..."} {"ts":"2024-10-11 21:32:29.074612 UTC","sev":"unknown","ct":"f5-dssm","stream":"","scid":"5546759f9f-t7k9m","sysid":"a0f22942-d2d2-e364-f48b-944eb8db386e","mid":"unknown","args":[],"log":"266:M 11 Oct 2024 21:32:29.074 * Background saving started by pid 12041"} {"ts":"2024-10-11 21:32:29.081845 UTC","sev":"unknown","ct":"f5-dssm","stream":"","scid":"5546759f9f-t7k9m","sysid":"a0f22942-d2d2-e364-f48b-944eb8db386e","mid":"unknown","args":[],"log":"12041:C 11 Oct 2024 21:32:29.081 * DB saved on disk"} {"ts":"2024-10-11 21:32:29.082281 UTC","sev":"unknown","ct":"f5-dssm","stream":"","scid":"5546759f9f-t7k9m","sysid":"a0f22942-d2d2-e364-f48b-944eb8db386e","mid":"unknown","args":[],"log":"12041:C 11 Oct 2024 21:32:29.082 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB"} {"ts":"2024-10-11 21:32:29.174947 UTC","sev":"unknown","ct":"f5-dssm","stream":"","scid":"5546759f9f-t7k9m","sysid":"a0f22942-d2d2-e364-f48b-944eb8db386e","mid":"unknown","args":[],"log":"266:M 11 Oct 2024 21:32:29.174 * Background saving terminated with success"}
Note: If an error has been found, delete the application in the unknown state. Resolve the configuration error and create a new application.
Usage Examples¶
tmctl¶
Note: To get full usage details use tmctl --help.
Example Use tmctl to query for HTTP status codes
Query the database for all HTTP profile table names.
tmctl -d /var/tmstat/blade/ -w 200 profile_http_stat -s name
| Argument | Description |
|---|---|
| -d /var/tmstat/blade/ | Path to DB dir on Debug utility |
| -w 200 | Wrap output at 200 characters |
| profile_http_stat | Table to query |
| -s name | Select values from the name column in the table |
Sample output (tmctl)
/tmctl -d /var/tmstat/blade/ -w 200 profile_http_stat -s name
name
-------------------------------------------------------------------------------
http
_vault_http
_avr_http
_memstat_http
...
_mon_http
/app/tenantCxQ5:prod/stack/tenantCxQ5:prod:vs_29/serverside/http
/app/tenantCxQ5:prod/stack/tenantCxQ5:prod:vs_29/clientside/http
/app/tenantCxQ5:prod/stack/tenantCxQ5:prod:vs_29/clientside/http
/app/tenantCxQ5:prod/stack/tenantCxQ5:prod:vs_29/serverside/http
/app/tenantOu812:dev/stack/tenantOu812:dev:vs_30/clientside/http
/app/tenantOu812:dev/stack/tenantOu812:dev:vs_30/serverside/http
/app/tenantOu812:dev/stack/tenantOu812:dev:vs_30/clientside/http
/app/tenantOu812:dev/stack/tenantOu812:dev:vs_30/serverside/http
Save the clientside HTTP profile name to a variable.
Copy and paste the stats object name from the previous command and save it to a variable. Alternatively, enter the statistics object name when executing the command.
name=<stats_name_key_value>
Example (tmctl)
/name='/app/tenantCxQ5:prod/stack/tenantCxQ5:prod:vs_29/clientside/http'
Use the name saved in the variable to query the table and select which columns to display.
tmctl \
-d /var/tmstat/blade/ \
-w 200 profile_http_stat \
name="${name}" \
-s 'name,vs_name,resp_1xx_cnt,resp_2xx_cnt,resp_3xx_cnt,resp_4xx_cnt,resp_5xx_cnt,number_reqs' \
-P \
--sortby=vs_name
| Argument | Description |
|---|---|
| -d /var/tmstat/blade/ | Directory location of db files to get results of all tmms |
| -w 200 | Wrap output at 200 characters |
| profile_http_stat | Table to query |
| name="${name}" | Retrieve stats the name key matched the value set in variable $name |
| -s name,vs_name... | Columns to select from the table |
| -P | Pivot display as property list |
| --sortby=vs_name | Sort results based on the vs_name key values |
Sample output (tmctl)
/tmctl -d /var/tmstat/blade/ -w 200 profile_http_stat name="${name}" -s name,vs_name,get_reqs,post_reqs,put_reqs,delete_reqs,resp_1xx_cnt,resp_2xx_cnt,resp_3xx_cnt,resp_4xx_cnt,resp_5xx_cnt,number_reqs -P --sortby=vs_name
Name Value
------------ ------------------------------------------------------------------------------------------------------------
name /app/tenantCxQ5:prod/stack/tenantCxQ5:prod:vs_29/clientside/http
vs_name /app/tenantCxQ5:prod/stack/tenantCxQ5:prod:vs_29
get_reqs 6928802
post_reqs 3
put_reqs 0
delete_reqs 0
resp_1xx_cnt 0
resp_2xx_cnt 395780
resp_3xx_cnt 0
resp_4xx_cnt 6533033
resp_5xx_cnt 0
number_reqs 6928822
Name Value
------------ ------------------------------------------------------------------------------------------------------------
name /app/tenantCxQ5:prod/stack/tenantCxQ5:prod:vs_29/clientside/http
vs_name _listener
get_reqs 0
post_reqs 0
put_reqs 0
delete_reqs 0
resp_1xx_cnt 0
resp_2xx_cnt 0
...
bdt_cli¶
Bdt_cli is a multipurpose tool for working with TMM stats.
| sub-commands | Description |
|---|---|
| help | Show command specific usage details |
| arp | Get ARP routes and their statuses |
| connection | List and modify connection table values |
| l2forward | Display L2 Forwarding entries |
| logLevel | Set TMM Log Level |
| route | Display Route List |
| tcpdumpSSLProvider | Enable SSL session capture on tcpdump |
bdt_cli help¶
Use bdt_cli <command> --help to get sub command specific help
bdt_cli arp¶
Get ARP routes and route statuses.
bdt_cli arp
Sample output
/bdt_cli arp
name:fe80::250:56ff:feba:79f9%vlan4094 ipAddr:fe80::250:56ff:feba:79f9%vlan4094 macAddr:00:50:56:ba:79:f9 vlan:internal-vlan expire:2234 status:resolved
name:127.20.1.7 ipAddr:127.20.1.7 macAddr:02:01:23:45:00:06 vlan:tmm_bp expire:0 status:permanent
name:169.254.0.4 ipAddr:169.254.0.4 macAddr:00:01:23:45:67:03 vlan:tmm expire:0 status:permanent
name:fe80::250:56ff:feba:da0a%vlan4094 ipAddr:fe80::250:56ff:feba:da0a%vlan4094 macAddr:00:50:56:ba:da:0a vlan:internal-vlan expire:1674 status:resolved
bdt_cli Connection¶
The bdt_cli connection command shows and deletes connections in the connection table.
List connections and limit output to match client IP, listener adddress, and listener service port.
bdt_cli connection list --cs_client_addr 203.0.113.130 --cs_server_addr 203.0.113.30 --cs_server_port 80
| Argument | Description |
|---|---|
| connection list | Show connection table entries |
| --cs_client_addr 203.0.113.130 | Select client IP 203.0.113.130 |
| --cs_server_addr 203.0.113.30 | Select clientside server with IP 203.0.113.29 |
| --cs_server_port 80 | Select clientside server port 80 |
Sample output
/bdt_cli connection list --cs_client_addr 203.0.113.130 --cs_server_addr 203.0.113.30 --cs_server_port 80
csClientAddr:{ip:{addr:203.0.113.130, rd:0}} csClientPort:49336 csServerAddr:{ip:{addr:203.0.113.30, rd:0}} csServerPort:80 ssServerAddr:{ip:{addr:<none>, rd:0}} ssServerPort:0 type:0 protocol:6 idleTime:153 idleTimeout:300 ssClientAddr:{ip:{addr:<none>, rd:0}} ssClientPort:0 virtualPathAddr:{ip:{addr:203.0.113.30, rd:0}} virtualPathPort:80 haUnitId:0 csBytesIn:112 csBytesOut:60 csPacketsIn:2 csPacketsOut:1 ssBytesIn:0 ssBytesOut:0 ssPacketsIn:0 ssPacketsOut:0 lasthopInfo:external-vlan 00:50:56:ba:0a:4d nexthopInfo:No nexthop ingressMac:<nil>
Delete connection(s) based on the clientside ephemeral port and the client IP address (bdt_cli connection)
bdt_cli connection delete --cs_client_addr 203.0.113.130 --cs_client_port 49336
| Argument | Description |
|---|---|
| connection delete | Delete connection table entry |
| --cs_client_addr 203.0.113.130 | Choose client source IP 203.0.113.130 |
| --cs_client_port 49336 | Choose client ephemeral port 49336 |
Sample output (bdt_cli connection)
/bdt_cli connection delete --cs_client_addr 203.0.113.130 --cs_client_port 49336
csClientAddr:{ip:{addr:203.0.113.130, rd:0}} csClientPort:49336 csServerAddr:{ip:{addr:203.0.113.30, rd:0}} csServerPort:80 ssServerAddr:{ip:{addr:<none>, rd:0}} ssServerPort:0 type:0 protocol:6 idleTime:119 idleTimeout:300 ssClientAddr:{ip:{addr:<none>, rd:0}} ssClientPort:0 virtualPathAddr:{ip:{addr:203.0.113.30, rd:0}} virtualPathPort:80 haUnitId:0 csBytesIn:112 csBytesOut:60 csPacketsIn:2 csPacketsOut:1 ssBytesIn:0 ssBytesOut:0 ssPacketsIn:0 ssPacketsOut:0 lasthopInfo:external-vlan 00:50:56:ba:0a:4d nexthopInfo:No nexthop ingressMac:<nil>
Note:
The details of removed entries are listed in the command output.
No output means no tables matching the flags were found to delete.
bdt_cli l2forward¶
The bdt_cli l2forward gets L2 Forwarding entries.
bdt_cli l2forward --hidden
Sample output
/bdt_cli l2forward --hidden
vlan:tmm_bp mac:02:01:23:45:00:07 member:xchg dynamic:false
vlan:tmm_bp mac:02:01:23:45:00:05 member:xchg dynamic:false
vlan:tmm_bp mac:02:01:23:45:00:02 member:xchg dynamic:false
vlan:tmm_bp mac:02:01:23:45:00:00 member:xchg dynamic:false
vlan:tmm_bp mac:02:01:23:45:00:06 member:xchg dynamic:false
vlan:tmm_bp mac:02:01:23:45:00:04 member:xchg dynamic:false
vlan:tmm_bp mac:02:01:23:45:00:03 member:xchg dynamic:false
vlan:tmm_bp mac:02:01:23:45:00:01 member:xchg dynamic:false
Result key:
<VLAN:name> <MAC:address> <Member:name> <Route type:Dynamic or Static>
Note: --hidden required to have <Member:name>
bdt_cli logLevel¶
logLevel sets TMM Log Level.
Log Levels: 1 - Debug 2 - Informational 3 - Notice 4 - Warning 5 - Error 6 - Critical 7 - Alert 8 - Emergency
Note:
logLevel is camelCase.
There is no response unless `–debug`` is included in the command.
To set TMM log level to Warning:
bdt_cli logLevel --debug -tmmLogLevel 4
| Argument | Description |
|---|---|
| --tmmLogLevel | TMM log setting command |
| --debug | Increase verbosity |
Sample output
/bdt_cli logLevel --debug --tmmLogLevel 4
type.googleapis.com/stats.request { request: {
{
"embedded": {
"@type": "declTmm.createMsgV2",
"embedded": [
{
"@type": "declTmm.sys_variable",
"id": "log.tmm.level",
"name": "log.tmm.level",
"value": "Warning"
}
]
}
} } }
/
bdt_cli route¶
The bdt_cli route command shows the TMM routing table entries.
Get route list¶
bdt_cli route
Sample output
/bdt_cli route
routeType:0 isIpv6:false destNet:{ip:{addr:100.76.0.0, rd:0} pl:14} gw:{ip:{addr:169.254.0.253, rd:0}} gwType:1 interface:tmm
routeType:0 isIpv6:false destNet:{ip:{addr:100.75.0.0, rd:0} pl:16} gw:{ip:{addr:169.254.0.253, rd:0}} gwType:1 interface:tmm
routeType:0 isIpv6:false destNet:{ip:{addr:<none>, rd:0} pl:0} gw:{ip:{addr:169.254.0.253, rd:0}} gwType:1 interface:tmm
routeType:1 isIpv6:false destNet:{ip:{addr:10.14.0.0, rd:0} pl:16} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:external-vlan
routeType:1 isIpv6:false destNet:{ip:{addr:10.13.0.0, rd:0} pl:16} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:internal-vlan
routeType:1 isIpv6:false destNet:{ip:{addr:127.20.0.0, rd:0} pl:16} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:tmm_bp
routeType:1 isIpv6:false destNet:{ip:{addr:169.254.0.0, rd:0} pl:24} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:tmm
routeType:0 isIpv6:true destNet:{ip:{addr:<none>, rd:0} pl:0} gw:{ip:{addr:fc00:f5::1, rd:0}} gwType:1 interface:tmm
routeType:1 isIpv6:true destNet:{ip:{addr:fe80:0:ecff:ffff::, rd:0} pl:64} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:external-vlan
routeType:1 isIpv6:true destNet:{ip:{addr:ff02:0:ecff:ffff::, rd:0} pl:64} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:external-vlan
routeType:1 isIpv6:true destNet:{ip:{addr:fe80:0:edff:ffff::, rd:0} pl:64} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:internal-vlan
routeType:1 isIpv6:true destNet:{ip:{addr:ff02:0:edff:ffff::, rd:0} pl:64} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:internal-vlan
routeType:1 isIpv6:true destNet:{ip:{addr:ff02:0:100::, rd:0} pl:64} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:tmm_bp
routeType:1 isIpv6:true destNet:{ip:{addr:fe80:0:100::, rd:0} pl:64} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:tmm_bp
routeType:1 isIpv6:true destNet:{ip:{addr:ff02::, rd:0} pl:64} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:tmm
routeType:1 isIpv6:true destNet:{ip:{addr:fe80::, rd:0} pl:64} gw:{ip:{addr:<none>, rd:0}} gwType:0 interface:tmm
Understanding the output (bdt_cli route)¶
| Field | Description |
|---|---|
| Route Type | dynamic=1 static=0 |
| IP Version | IPv4 or IPv6 |
| Destination | Route Destination Network |
| Gateway | Gateway to reach Destination Network |
| Gateway Type | 1=gw 0=interface |
| Interface | Outgoing interface |
This table shows the parts of one line of output.
Route Type |
IP Version |
Destination Network |
Gateway |
Gateway Type |
Interface |
|---|---|---|---|---|---|
routeType:1 |
isIpv6:false |
destNet:{ip:{addr:10.14.0.0, rd:0} pl:16} |
gw:{ip:{addr: |
gwType:0 |
interface:external-vlan |
tcpdump¶
Tcpdump is a command-line utility that captures and analyzes network traffic.
Application used in examples¶
| Host | Cientside | Serverside |
|---|---|---|
| Client | 203.0.113.130 | --- |
| BIG-IP Next | 203.0.113.29:80 | 198.51.100.131 |
| app_server1 | --- | 198.51.100.199:31900 |
| app_server2 | --- | 198.51.100.200:31900 |
tcpdump example 1: Capture clientside and serverside for a single application¶
Capture all traffic to a clientside listener and the server-side application servers. Then copy capture file from BIG-IP Next instance to a workstation.
In the Debug utility start the capture listening on all interfaces, filtering whats captured, and saving it to a file.
tcpdump --f5 n -s0 -i 0.0 '(host 203.0.113.29 and port 80) or (host(198.51.100.199 or 198.51.100.200) and port 31900)' -w /home/debuguser/all_app_traffic.pcap
Important: Running tcpdump on interface 0.0 is not rate-limited and has the potential to create very large files. F5 recommends this option only when using filters to limit the size of the capture.
| Argument | Description |
|---|---|
| -s0 | Snaplength captue size unlimited |
| --f5 n | Capture internal TMM information at full detail |
| -i 0.0 | Interface 0.0 captures on all TMM interfaces |
| '(host 203.0.113.29 and port 80) or (host(198.51.100.199 or 198.51.100.200) and port 31900)' | filter: (Destination IP and port 80) or (app_servers IPs and port 31900) |
| -w /home/debuguser/all_app_traffic.pcap | Save capture file named -w all_app_traffic.pcap in directory(/home/debuguser) |
Use the scp command on the user’s workstation to copy the capture file from the Debug utility home directory to the users current directory.
scp -P 1234 admin@192.0.2.131:/home/debuguser/all_app_traffic.pcap .
Tcpdump example 2: Capture single client to listener - clientside only¶
Capture clientside traffic from the client IP to the listener IP and listener service. Include all ICMP and ARP.
/tcpdump -s0 --f5 n -i external-vlan 'host(203.0.113.130 and 203.0.113.29) and port 80 or icmp or arp' -w /home/debuguser/clientside.pcap
| Argument | Description |
|---|---|
| -s0 | Snaplength captue size unlimited |
| --f5 n | capture internal TMM information at full detail |
| -i external-vlan | Capture traffic VLAN |
| 'host(203.0.113.130 and 203.0.113.29) and port 80 or icmp or arp' | filter: client IP, and Destination IP, and port 80, all ARP and ICMP |
| -w /home/debuguser/clientside.pcap | Save capture file named clientside.pcap in directory(/home/debuguser) |
Use the scp command on the user’s workstation to copy the capture file from the Debug utility home directory to the users current directory.
scp -P 1234 admin@192.0.2.131:/home/debuguser/clientside.pcap .
Example 3: Capture clientside from client IP and serverside to application servers¶
Capture on interface 0.0 and limit the capture to invlolve the clientside client IP and destination IP and port, or include application servers IPs on port 31900.
tcpdump -s0 --f5 n -i 0.0 '(host(203.0.113.130 and 203.0.113.29) and port 80) or (host(198.51.100.199 or 198.51.100.200) and port 31900)' -w cs_ss.pcap
Important: Running tcpdump on interface 0.0 is not rate-limited and has the potential to create very large files. F5 recommends this option only when using filters to limit the size of the capture.
| Argument | Description |
|---|---|
| -s0 | Snaplength capture size unlimited |
| --f5 n | Captures internal TMM information at full detail |
| -i 0.0 | Interface 0.0 captures on all TMM interfaces |
| (host(203.0.113.130 and 203.0.113.29) and port 80) or (host(198.51.100.199 or 198.51.100.200) and port 31900) | Capture involved clientside and serverside application traffic |
| -w cs_ss.pcap | Save capture file in current directory named cs_ss.pcap |
Use the scp command on the user’s workstation to copy the capture file from the Debug utility home directory to the users current directory.
scp -P 1234 admin@192.0.2.131:/home/debuguser/cs_ss.pcap .
Example 4: Capture ssl session information using tcpdump¶
To enable ssl for tcpdump
bdt_cli tcpdumpSSLProvider -v enable
To capture the ssl session secrets using tcpdump
tcpdump -i any --f5 ssl -w all_app_with_ssl_traffic.pcap
| Argument | Description |
|---|---|
| -i any | Capture on all interfaces that are available and in use |
| --f5 ssl | Capture the ssl session information |
| -w all_app_with_ssl_traffic.pcap | Save capture file named all_app_with_ssl_traffic.pcap in directory(/home/debuguser) |
Use the scp command on the user’s workstation to copy the capture file from the Debug utility home directory to the users current directory
scp -P 1234 admin@192.0.2.131:/home/debuguser/all_app_with_ssl_traffic.pcap
To disable ssl for tcdump
bdt_cli tcpdumpSSLProvider -v disable
ping¶
Ping is a network diagnostic tool that uses ICMP to test the reachability of a remote host and measures the round-trip times for messages sent.
Use ping to send ICMP Echo Request messages to a remote host and see if an Echo Reponse is returned.
/ping -c 4 host.example.com
| Argument | Description |
|---|---|
| -c 4 | Send four Echo Requests |
| host.example.com | Remote host FQDN or IP |
Sample output
/ping -c 4 host.example.com
PING host.example.com (203.0.113.130) 56(84) bytes of data.
64 bytes from host.example.com (203.0.113.130): icmp_seq=1 ttl=111 time=1.95 ms
64 bytes from host.example.com (203.0.113.130): icmp_seq=2 ttl=111 time=2.14 ms
64 bytes from host.example.com (203.0.113.130): icmp_seq=3 ttl=111 time=2.23 ms
64 bytes from host.example.com (203.0.113.130): icmp_seq=4 ttl=111 time=2.10 ms
--- host.example.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 1.946/2.104/2.228/0.102 ms
traceroute¶
Traceroute is a network diagnostic tool. It tracks the pathway taken by packets from your computer to a destination, and displays transit delays.
Use traceroute to verify the path to example.com.
traceroute example.com
| Argument | Description |
|---|---|
| example.com | Validate path to this remote host |
Sample output
/traceroute example.com
traceroute to example.com (142.251.33.110), 30 hops max, 60 byte packets
1 100.76.0.1 (100.76.0.1) 0.054 ms 0.022 ms 0.018 ms
2 198.51.100.254 (198.51.100.254) 0.269 ms 0.236 ms 0.222 ms
3 198.51.255.252 (198.51.255.252) 0.198 ms 0.145 ms 0.133 ms
4 ipfw-rockon.cxq5net.com (192.0.2.174) 1.217 ms 1.170 ms 1.106 ms
5 204-134-187-130.static.decadelink.net (204.134.187.130) 1.240 ms 50.204.110.2 (50.204.110.2) 1.214 ms 204-134-187-130.static.decadelink.net (204.134.187.130) 1.144 ms
6 204-134-187-190.static.decadelink.net (204.134.187.190) 1.334 ms 1.294 ms 1.429 ms
7 sea-edge-14.inet.mabell.net (67.129.1.145) 1.531 ms 1.206 ms 1.631 ms
8 sea-edge-15.inet.mabell.net (67.14.41.158) 1.655 ms sea-edge-15.inet.mabell.net (67.14.41.162) 1.999 ms sea-edge-15.inet.mabell.net (67.14.41.158) 1.864 ms
9 * * *
10 * 72.14.243.62 (72.14.243.62) 1.777 ms 1.958 ms
11 142.250.167.78 (142.250.167.78) 1.852 ms * *
12 74.125.243.193 (74.125.243.193) 2.097 ms 1.830 ms 142.251.241.136 (142.251.241.136) 1.683 ms
13 142.251.50.177 (142.251.50.177) 1.907 ms 142.251.50.175 (142.251.50.175) 1.878 ms 142.251.55.198 (142.251.55.198) 1.913 ms
14 sea30s10-in-f14
/
curl¶
curl is a command-line utility for working with files using HTTP, HTTPS, and FTP protocols.
Use curl to make an HTTP request and send the output to the screen.
curl http://127.0.0.1:8800/validated/config -o /tmp/output.txt --verbose
| Argument | Description |
|---|---|
| -o/--output | write to file |
| -v / --verbose | make the operation more talkative |
Sample output
/curl http://127.0.0.1:8800/validated/config -o /tmp/output.txt --verbose
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 {}:{}:{} {}:{}:{} {}:{}:{} 0* Trying 127.0.0.1:8800...
Connected to 127.0.0.1 (127.0.0.1) port 8800
> GET /validated/config HTTP/1.1
> Host: 127.0.0.1:8800
> User-Agent: curl/8.5.0
> Accept: /
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Wed, 16 Oct 2024 04:39:42 GMT
< Transfer-Encoding: chunked
<
{ [12677 bytes data]
100 12677 0 12677 0 0 1769k 0 {}:{}:{} {}:{}:{} {}:{}:{} 2063k
Connection #0 to host 127.0.0.1 left intact
openssl¶
The openssl command is the CLI to the OpenSL toolkit. It has various sub-commands and options that provide a wide range of functionalities.
Use openssl to connect to a TLS endpoint for diagnostic purposes.
openssl s_client -connect 192.0.2.194:443
| Argument | Description |
|---|---|
| s_client | Starts the SSL/TLS client |
| -connect 192.0.2.194:443 | Connect to 192.0.2.194 on port 443 |
Sample output
Note: The output is truncated.
/openssl s_client -connect 192.0.2.194:443
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 C = US, ST = UTAH, L = Lehi, O = Cxq5, OU = Sun, CN = test.org
verify error:num=18:self-signed certificate
verify return:1
depth=0 C = US, ST = UTAH, L = Lehi, O = Cxq5, OU = Sun, CN = test.org
verify return:1
---
Certificate chain
0 s:C = US, ST = UTAH, L = Lehi, O = Cxq5, OU = Sun, CN = test.org
i:C = US, ST = UTAH, L = Lehi, O = Cxq5, OU = Sun, CN = test.org
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Sep 22 23:35:21 2023 GMT; NotAfter: Sep 19 23:35:21 2033 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDiTCCAnGgAwIBAgIJAOg8DNLmZ7fWMA0GCSqGSIb3DQEBCwUAMFsxCzAJBgNV
BAYTAlVTMQ0wCwYDVQQIDARVVEFIMQ0wCwYDVQQHDARMZWhpMQ0wCwYDVQQKDARD
eHE1MQwwCgYDVQQLDANTdW4xETAPBgNVBAMMCHRlc3Qub3JnMB4XDTIzMDkyMjIz
MzUyMVoXDTMzMDkxOTIzMzUyMVowWzELMAkGA1UEBhMCVVMxDTALBgNVBAgMBFVU
QUgxDTALBgNVBAcMBExlaGkxDTALBgNVBAoMBEN4cTUxDDAKBgNVBAsMA1N1bjER
MA8GA1UEAwwIdGVzdC5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQCyj4x1TaSUaG8YGlb0oKX9Fe0IiJOdrxuf1hGtfpocch/ObVHfJ15zLvuMibvf
7Hq5tagKEALTUq7W4uhYidpQpe4ZHaBvezV+4qKqj6AUZBRVSPmTlUk5TCbDodTs
FE/rT4SGmakgDzXWbzTh6iZwJB/bbmNBjG4zf9mDdkvRNBTc4zQQhxVKILCoYL8v
+VO8R3gW8Z7GAondc631j73yy5zfpc74puQv6/ehCAyA0+M7rtsjxxwc27S/eFFs
7GggKy7eEUEVXP1fT8eUCyshqQHX8HW7s/3xmbSgYgqp5L+0K46XdwSk0IZ3UaSx
z99AuXFSFAp6fIUaDjyMn1vlAgMBAAGjUDBOMB0GA1UdDgQWBBQvVSe4jk4t737r
oC83MJJ8fEjtajAfBgNVHSMEGDAWgBQvVSe4jk4t737roC83MJJ8fEjtajAMBgNV
HRMEBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBfJgY/8yHJ6IfcFuwgOcoe6q5f
bkD9dBIZfgLFxKwIzORAFafHyxgN/PTR9Xk3a9XcByfTZMBrDFC7RKqgm+Fi3iHr
b1649i4HsEdjwhXgV+dECQqhEjIVaczxXjw0vvmeDDS0SsEWTYce3H8//8f9ZjNP
HM4bTRmo/934zk7FftJQreqtuBtPpPObow4b3hwjv9arqSu99XkPbnKw/kw7cPOc
LuloQURYc4IyJU0KNfzWh8GjRrVo+VrntL78JDdJ4UYpZ33SL17zu9R56reJ1WV7
NftA5yfTuua0KSuxkcQJedZYmiMDPKXBP4ldL3PZ14xlACxx5oiUyK/kgTtf
-----END CERTIFICATE-----
subject=C = US, ST = UTAH, L = Lehi, O = Cxq5, OU = Sun, CN = test.org
issuer=C = US, ST = UTAH, L = Lehi, O = Cxq5, OU = Sun, CN = test.org
---
No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 1595 bytes and written 419 bytes
Verification error: self-signed certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
...
Enable the Debug utility on Instance API via Mac or Linux¶
When the BIG-IP Next Central Manager, which normally manages the instance, is unavailable, follow this procedure to enable the Debug utility by making direct request to the BIG-IP Next instance API.
Recommendation: Use the BIG-IP Next Central manager to enable the Debug utility. For directions on how to enable the Debug utility using BIG-IP Next Central Manager see: How to: Log in and use the Debug utility.
Overview¶
Enable Debug utility Mac and Linux
Connect to the Debug utility from client workstation
Disable the Debug utility
Prerequisites¶
Curl HTTP client installed on workstation with network access to the Instance management IP address
Workstation
Curl HTTP Client installed
Network access to the management IP of Instance
SSH client
Management credentials currently set for the instance. These are set when the BIG-IP instance is added to the BIG-IP Next Central Manager.
Enable Debug Utility¶
Request authentication token from instance API¶
The procedure below is an example of how to request an authentication token and saving the token as a variable.
Set the variables for requesting an authentication token:
Set the username that will be used to authenticate to the BIG-IP Next instance, the default is admin-cm. This username was created when the BIG-IP was discovered by the BIG-IP Next Central Manager
export username=<admin-cm>
Set the password corresponding to the username:
password=<password>
Set the IP address for the BIG-IP Next instance:
instance_ip=<IP_address_BIGIP>
Make an API request for an authentication token to connect directly to the BIG-IP Next instance. Be sure to include the instance port number(5443) within the API URL. This command will request an authentication token, then utilize the jq command to filter the token from the response and store the auth token in a variable.
token=$(curl -sk -u "${username}:${password}" https://${instance_ip}:5443/api/v1/login | jq -r '.token')
Note: Repeat the above request as many time as necessary to get new auth tokens after they expire.
Check that the variable named token is populated: Note: This is optional, to view the token and add it directly to the API request to connect to the instance.
echo $token
Note: The token expires after five minutes.
Having requested the token and populated a variable (or copied it), it is now possible to send API requests to the BIG-IP Next instance.
Make Request to enable Instance Debug Utility¶
To enable the Debug utility make an HTTP PUT request to the BIG-IP instance API. The body of the PUT request will contain the details, in JSON format, that will allow a client to connect to the Debug shell using the SSH public key of the client workstation to authenticate.
Note: A username should be created using a combination of uppercase letters (A-Z), lowercase letters (a-z), and numbers (0-9) to enable the debug utility. Although the following are allowed, avoid using them as they will impact enabling the debug utility:
underscores (_), dashes (-) or dots (.).
starting the username with an uppercase letter or number.
Populate variables that will be used to create a PUT data file¶
Save the client public SSH key to variable.
Example
export client_public_ssh_key=$(cat ~/.ssh/id_rsa.pub)
Set a variable for the port number the SSH client will use to connect to the Debug utility.
export port_number=<port_number>
Important: Substitute <port_number> with the chosen port number for connecting to the Debug utility. Any number not currently in use will suffice.
Example
export port_number=4422
Create a file that contains the following and name it put_template.
{
"sshPublicKey": "${client_public_ssh_key}",
"allowedIps": [
"0.0.0.0/0"
],
"username": "${username}",
"port": ${port_number}
}
Note: Users with usernames that do not follow the pattern ‘^[a-z][a-z0-9]’—specifically those starting with an uppercase letter, number, or containing underscores (_), dashes (-), or dots (.); will need to enable the utility via OpenAPI.
Create a data file based off the template and populated variables using the envsubst command.
Example
envsubst < put_template > put.json
Note: There is no output from the above command. To see resulting file contents use: cat put.json.
Confirm that the put.json file contains the values substituted for the variables, and that the JSON is formatted correctly, use the jq command. If there is a problem with the JSON formatting jq will give an error. If the JSON is correct jq will output the file contents formatted.
cat put.json | jq .
Example output
% cat put.json | jq .
{
"sshPublicKey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmkeRGhRHHZjdIes1GMWbFhW8gDkpUn7fDpPtqRisw7yrzCEoa84eUbK/GilKvSMqlU2plcW8/ANlOn1vHvVR3ULerZksRU+LYVAH7hBSJA6M9hN+u2wz6wW+UfdJkcIXFnTBBQzg1F3HAd5vZR+l5lsv0YniUO0vtiPqfDAe61ksimHOnIwoKfjKhhnA1n3fYLE7vSOIR+1fuQnUsi0wrBD46kDtFVnSevhLrT9DXJVloO8KPf8hL+CXQ5+u2sID0nZlFJZ/j8ztYBYnYTnCHmtlmjMd0KBv78huaO5LqdCyXTp+SKrg4SVFMYwDnz9a0qquKzMvQ97zJWYPDBVfPyMdRDtzYy+EjClYs382ZNOmGr6dDHyEpRt/QnNxXXT9rkFJHkvZ2UlDegs40mDVwL5vYRH2MpCJp+jGxlTzRv5jcq9framwY+bpx1WRD9X6bMs/gliiku6b1pjYe7QPzkK/iTUEFXvo2zJr7g6VYrzdSgHTtAKQPThSahzBjr0= t.thomas@client.local",
"allowedIps": [
"0.0.0.0/0"
],
"username": "admin",
"port": 4422
}
Save the instance system ID to a variable. The ID will be used in the request to enable the Debug utility.
Example
system_id=$(curl -sk -H "Authorization: Bearer ${token}" https://${instance_ip}:5443/api/v1/systems | jq -r '._embedded.systems[].id')
Make request to enable Debug utility¶
Make a PUT request to the instance API using the data file and the populated variables.
Example
curl -sk -H "Authorization: Bearer ${token}" -H "Content-Type: application/json" https://${instance_ip}:5443/api/v1/actions/systems/${system_id}/dataplane-debug/enable -X PUT --upload-file put.json
Connect to the Debug utility from client workstation¶
ssh <username>@<BIG-IP Instance IP> -p <port_number>
Example
ssh admin@10.154.73.131 -p 4422
Example output Debug utility command line
% ssh admin@10.154.73.131 -p 4422
****************************************
* *
* Welcome to BIG-IP Next Debug Console *
* *
****************************************
Last login: Tue Oct 10 19:34:28 2023 from 100.76.0.78
/
See: Debug utility tools for more information on the tools available in the Debug utility.
Disable the Debug utility¶
Once troubleshooting is completed, the Debug utility should be disabled.
Example
curl -sk -H "Authorization: Bearer ${token}" -H "Content-Type: application/json" https://${instance_ip}:5443/api/v1/actions/systems/${system_id}/dataplane-debug/disable -X PUT
Note: If the authentication token expires request a new auth token and try the request again.
token=$(curl -sk -u "${username}:${password}" https://${instance_ip}:5443/api/v1/login | jq -r '.token')
The last request will turn off the Debug utility and will need to be enabled again before it can be accessed.