F5OS-A 1.0.0 - QKView Overview¶
Feature Overview¶
QKView is the F5 standard for collecting data from systems for use in troubleshooting. It is a snapshot in time of the data on the system that can be sent in for review by Support as well as PD for problem and case resolution. This implementation of QKView is different than that used in BIG-IP and will be used moving forward in new products such as F5OS and Modular BIG-IP.
The QkView in F5OS collects information about the rSeries platform, and doesn’t collect statistics and configurations of each of the tenants deployed.
Feature deeper overview¶
For F5OS, QKView has been rearchitected to be a client-server design in order to make it scalable for 4-32 blades and 2 controllers. While that is not strictly necessary for the appliance hardware because there is no controller or blade, the F5OS-A uses the same approach. For further architectural overview, please review the Architecture section below. In this particular design, data is stored in a raw format where possible and packaged by container. Each container runs a qkview-collect client that uses a config file to tell it what to grab for data. That data is in turn packaged into a tar.gz archive and stored in a monitored location on the filesystem. Once the qkview-collect is done, the parent daemon qkviewd that spawned it will grab the archive along with all of the other containers’ archives and package them together into one larger archive. Each archive contains a manifest file which is a JSON description of what is in the archive.
QKView is run on the appliance only once unlike VELOS where it is run on both the active and standby controllers ( not active-standby architecture for the rSeries ).
ConfD CLI related commands are collected from the “system_manager” container.
ConfD CLI commands¶
The following commands are used to run and manage QKView:
Starts a new QKView capture¶
appliance-1# system diagnostics qkview capture
result Warning: Qkview may contain sensitive data such as secrets, passwords and core files. Handle with care. Please send this file to F5 support.
Qkview file appliance-1.qkview is being collected.
return code 200
resultint 0
appliance-1#
Show the status of the currently running QKView¶
This is how to check the status of the currently running QKView:
appliance-1# system diagnostics qkview status
result {"Busy":true,"Percent":16,"Status":"collecting","Message":"Collecting Data","Filename":"appliance-1.qkview"}
resultint 0
appliance-1#
Stop/cancel the currently running QKView¶
This is how to cancel the system diagnostics qkview:
appliance-1# system diagnostics qkview cancel
result Qkview with filename appliance-1.qkview was canceled
return code 200
resultint 0
appliance-1#
List of available qkview files¶
Provides a of QKView files currently on the system ( two file available, one final, one cancelled ).
appliance-1# system diagnostics qkview list
result {"Qkviews":[{"Filename":"appliance-1.qkview","Date":"2021-11-23T10:19:44.939085339Z","Size":41008290},
{"Filename":"appliance-1.qkview.canceled","Date":"2021-11-23T10:16:05.513087947Z","Size":677323}]}
resultint 0
appliance-1#
Delete a QKView file¶
This is how to delete the qkView files:
appliance-1# system diagnostics qkview delete filename appliance-1.qkview.canceled
result Deleted Qkview file appliance-1.qkview.canceled
return code 200
resultint 0
appliance-1# system diagnostics qkview list
result {"Qkviews":[{"Filename":"appliance-1.qkview","Date":"2021-11-23T10:19:44.939085339Z","Size":41008290}]}
resultint 0
appliance-1#
Current location of the created qkview files¶
This is the location of the created qkview files:
[root@appliance-1 qkview]# pwd
/var/shared/qkview
[root@appliance-1 qkview]# ls -ltr
total 56992
-rw-r--r--. 1 root root 41008290 Nov 23 10:19 appliance-1.qkview
[root@appliance-1 qkview]#
GUI Screen Shots¶
The following is a screenshot of the QKView screen in the F5OS webUI. The QKView screen will be the same in both the system controller webUI as well as the chassis partition webUI.
New / Updated logs¶
Currently logs for the qkview collections are availableon the appliance at the following location:
Expand list for full list of logs
[root@appliance-1 log]# pwd
/var/F5/system/log
[root@appliance-1 log]# tail -1000 platform.log | grep qkview
2021-11-23T10:16:38.369038+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000043 msg="Collecting container snapshots" container="qkviewd"
2021-11-23T10:16:38.369431+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000038 msg="Request host to collect qkview" container="qkviewd"
2021-11-23T10:16:38.369475+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000050 msg="Request to collect core-files" container="qkviewd"
2021-11-23T10:16:38.369532+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c8100000000003e msg="Collecting from peers" container="qkviewd" NUMPEERS=0
2021-11-23T10:16:38.369557+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c8100000000003d msg="Info:" container="qkviewd" MESSAGE=No Core Files Found
2021-11-23T10:16:38.369569+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c8100000000003d msg="Info:" container="qkviewd" MESSAGE=Peers have been collected
2021-11-23T10:16:38.381167+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c8100000000003d msg="Info:" container="qkviewd" MESSAGE=Host Info Collected
2021-11-23T10:16:43.508365+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/appliance_orchestration_manager
2021-11-23T10:16:45.369185+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_dagd
2021-11-23T10:16:47.251198+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_L2
2021-11-23T10:16:49.176277+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_lldpd
2021-11-23T10:16:51.133935+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_network_manager
2021-11-23T10:16:53.603715+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system-vconsole
2021-11-23T10:16:55.417077+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_platform-stats-bridge
2021-11-23T10:17:03.618521+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_TPOB
2021-11-23T10:17:05.491399+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system-common
2021-11-23T10:17:07.731128+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_license_service
2021-11-23T10:17:09.633124+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_velocity_rsyslogd
2021-11-23T10:17:11.715095+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_user_manager
2021-11-23T10:17:13.708791+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_image_agent
2021-11-23T10:17:16.275498+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_host_config
2021-11-23T10:17:18.215352+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_api_svc_gateway
2021-11-23T10:17:20.491111+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/diag-agent
2021-11-23T10:17:22.724646+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_platform-mgr
2021-11-23T10:17:24.730825+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/tcpdumpd_manager
2021-11-23T10:17:26.723225+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_datapath_cp_proxy
2021-11-23T10:17:28.522721+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_control
2021-11-23T10:17:30.358763+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/alert-service
2021-11-23T10:17:32.531200+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_stpd
2021-11-23T10:17:34.619448+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_sw_rbcast
2021-11-23T10:17:36.533615+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/ihealth-service
2021-11-23T10:17:38.943618+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/dma-agent
2021-11-23T10:17:41.025952+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_lacpd
2021-11-23T10:17:43.012445+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/line-dma-agent
2021-11-23T10:17:57.464841+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/platform-fwu
2021-11-23T10:18:26.433875+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_manager
2021-11-23T10:18:28.567938+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/upgrade-service
2021-11-23T10:18:36.450483+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/platform-monitor
2021-11-23T10:18:38.424406+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/optics-mgr
2021-11-23T10:18:48.143112+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_fpga
2021-11-23T10:18:49.954357+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/name-service-ldap
2021-11-23T10:18:52.233107+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/platform-hal
2021-11-23T10:18:54.270451+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_lacpd_proxy
2021-11-23T10:18:56.101445+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/snmp-trapd
2021-11-23T10:18:58.073842+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/platform-stats
2021-11-23T10:19:00.172064+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/authentication-mgr
2021-11-23T10:19:02.458655+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/http-server
2021-11-23T10:19:04.400116+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/lcd-webserver
2021-11-23T10:19:06.356559+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/stream-generator
2021-11-23T10:19:08.526595+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_tmstat_zmq
2021-11-23T10:19:10.608542+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_rsyslogd
2021-11-23T10:19:12.652119+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/qkviewd
2021-11-23T10:19:14.475404+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/confd-key-migration-mgr
2021-11-23T10:19:16.644594+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/lopd
2021-11-23T10:19:18.986161+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/utils-agent
2021-11-23T10:19:20.888129+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/platform-diag
2021-11-23T10:19:24.163568+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/qat-support-pod
2021-11-23T10:19:26.302392+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_tmstat_merged
2021-11-23T10:19:28.218143+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/swdiag-agent
2021-11-23T10:19:30.187276+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/fips-service
2021-11-23T10:19:31.985418+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/snmpd
2021-11-23T10:19:33.789646+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_latest_vers
2021-11-23T10:19:35.541664+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/vanquish-gui
2021-11-23T10:19:37.321185+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/firmware-fpga
2021-11-23T10:19:39.630046+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/system_confgen
2021-11-23T10:19:41.709135+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/firmware
2021-11-23T10:19:43.378997+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000056 msg="Container subpackage collected for" container="qkviewd" container=/appliance-services-registry-2000
2021-11-23T10:19:43.622208+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c8100000000003d msg="Info:" container="qkviewd" MESSAGE=Host Containers Collected
2021-11-23T10:19:43.622238+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000044 msg="Collected container snapshots" container="qkviewd"
2021-11-23T10:19:43.622244+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c8100000000002f msg="Collected snapshots" container="qkviewd" num_containers=61
2021-11-23T10:19:43.622251+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000041 msg="Starting collate of container saved to file:" container="qkviewd" COLLATE=appliance-1.qkview
2021-11-23T10:19:44.939924+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000031 msg="Qkview file saved at" container="qkviewd" file_name=/var/shared/qkview/appliance-1.qkview
2021-11-23T10:19:44.939937+00:00 appliance-1 qkviewd[9]: priority="Info" msgid=0x0c81000000000042 msg="Finished collate of container saved to file:" container="qkviewd" COLLATE=/var/shared/qkview/appliance-1.qkview
[root@appliance-1 log]#
Tools¶
There are a small handful of tools available for aiding in reviewing QKView archives. The following is a list and their locations, please refer to the links for further information on each:
verify-qkview- a utility for verifying the integrity of the qkview archive
[root@appliance-1 ~]# verify-qkview --help
Usage of verify-qkview:
-cmd-stats-out string
Write command stats in packages to csv file
-expect-host
If true, verify will fail if no host package is found
-expect-peer int
Specify the number of peers to expect. Zero default indicates no check
-pkg-stats-out string
Write captured packages stats to csv file
-qkview string
qkview tar ball to be verified (default "qkview.tar.gz")
-schema string
Schema file for the manifest (default "/etc/qkviewd/manifest_schema.json")
-verbose
If true, enable verbose logging
-verify-subpackages
If true, subpackages will be verified
[root@appliance-1 ~]#
iHealth- the standard qkview viewing application for Support and Customers that allows searching and heuristics on the qkview archive. This application will be augmented over time, but as of this writing only allows viewing of the file structure and files as well as searching the archive. iHealth is available at the following web address: iHealth
Expected / Possible Problems¶
Currently the QKView archive is a hierarchy of archives that all have a manifest.json file. Manually attempting to find desired information can be difficult as you must traverse multiple levels and extract multiple archives, you also have to know which container the data you are looking for is in and it may be duplicated in multiple containers. It is recommended to use the iHealth system and use the search functionality there to look for the data you require (at the time of this writing).
Collecting a full Qkview can take up to 20-30 minutes on a chassis, but only takes a few minutes to run on an rSeries appliance, since an rSeries appliance is not a bladed system. You can use the QKViewstatus command listed above to check how far along it is.
Architecture¶
The F5OS QKView utility has been changed to a client-server architecture to remain scalable for more than 4-8 blades and system controllers. The following is a diagram of the architecture as it is implemented in F5OS:
The following is a diagram of the archive structure for reference:
QKView collection agent architecture:
