F5OS-A 1.0.0 - System Logging¶
Feature Overview¶
Logging in rSeries appliances works similar toF5OS Loggingon VELOS chassis. The logging architecture enables F5OS software components to log to rsyslogd. F5OS logging does not include BIG-IP tenant logging.
Feature deeper overview¶
What it is:
A common logging infrastructure for F5OS-A
A set of commands for configuring the log level of all system services, configuring log rotation, and configuring remote syslog.
All logs from containers and pods go through a containerized rsyslogd.
What it is not:
It is not telemetry.
It is not a BIG-IP tenant log aggregator. You need to log into the BIG-IP tenant to see the tenant logs.
Log file location¶
On the filesystem, F5OS-A log files are stored to**/var/F5/system/log/**. Common Linux log files are stored in**/var/log/**.
The log file for rsyslogd is**/var/F5/system/log/velos.log**.
Remote syslog server destinations are stored in**/var/F5/system/etc/rsyslog.d/destinations.conf**. Do not edit this file, use the confd commands for remote logging to make changes.
ls -l /var/F5/system/log/
total 270796
-rw-r--r--. 1 root root 9742475 Nov 24 15:32 audit.log
-rw-r--r--. 1 root root 186870 Nov 15 21:33 confd.log
-rw-r--r--. 1 root root 33765884 Nov 24 15:32 devel.log
-rw-r--r--. 1 root root 50624964 Nov 24 15:32 lcd.log
-rw-r--r--. 1 root root 104857973 Nov 15 06:44 lcd.log.1
-rw-r--r--. 1 root root 1396192 Nov 24 15:33 logrotate.log
-rw-r--r--. 1 root root 5245071 Nov 24 00:34 logrotate.log.1
-rw-r--r--. 1 root root 28600 Nov 21 16:17 logrotate.log.2.gz
-rw-r--r--. 1 root root 58408317 Nov 24 15:32 platform.log
-rw-r--r--. 1 root root 7366 Nov 24 15:01 rsyslogd_init.log
-rw-r--r--. 1 root root 0 Oct 26 20:46 snmp.log
-rw-r--r--. 1 root root 330 Nov 12 17:44 startup.log
-rw-r--r--. 1 root root 192 Nov 12 17:40 startup.log.prev
drwxr-xr-x. 2 root root 4096 Oct 26 20:46 trace
-rw-r--r--. 1 root root 966 Nov 12 17:44 vconsole_auth.log
-rw-r--r--. 1 root root 9456 Nov 12 17:44 vconsole_startup.log
-rw-r--r--. 1 root root 12949512 Nov 4 17:54 velos.log
drwxr-xr-x. 2 root root 4096 Nov 12 17:44 webui
Log levels¶
| VALUE | SEVERITY | DESCRIPTION | EXAMPLES | | 0 | Emergency | System is unusable | Not for individual components. | | 1 | Alert | Should be corrected immediately | No network connectivity | | 2 | Critical | Critical conditions | Non recoverable failures in system’s main components. | | 3 | Error | Error conditions | Serious conditions that could cause major problems like storage exhaustion. | | 4 | Warning | May indicate that an error will occur if action is not taken. | Issues that can cause serious problems in the near future. Getting low on storage. | | 5 | Notice | Events that are unusual, but not error conditions. | A component restarted unexpectedly. | | 6 | Informational | Normal operational messages that require no action. | A component has started, paused or ended successfully. | | 7 | Debug | Information useful to developers for debugging the application. | |
Configuring logging via confd¶
**show running-config system logging sw-components sw-component ?**Expand source
vanquish-01# show running-config system logging sw-components sw-component ?
Description: List of platform sw-components
Possible completions:
alert-service api-svc-gateway appliance-orchestration-agent appliance-orchestration-manager authd dagd-service datapath-cp-proxy
diag-agent disk-usage-statd dma-agent fips-service fpgamgr ihealth-upload-service ihealthd
image-agent kubehelper l2-agent lacpd license-service line-dma-agent lldpd
lopd network-manager optics-mgr platform-diag platform-fwu platform-hal platform-mgr
platform-monitor platform-stats-bridge qkviewd rsyslog-configd snmp-trapd stpd sw-rbcast
sys-host-config system-control tcpdumpd-manager tmstat-agent tmstat-merged upgrade-service user-manager
vconsole | <cr>
Possible match completions:
config Configuration data for platform sw-component logging
Check the current severity for a component:
show running-config system logging sw-components sw-component alert-serviceExpand source
vanquish-01# show running-config system logging sw-components sw-component alert-service
system logging sw-components sw-component alert-service
config name alert-service
config description "Alert service"
config severity INFORMATIONAL
!
Check the severity options for a component:
**system logging sw-components sw-component alert-service config severity ?**Expand source
vanquish-01# config
vanquish-01(config)# system logging sw-components sw-component alert-service config severity ?
Description: sw-component logging severity level. Default is INFORMATIONAL.
Possible completions:
[INFORMATIONAL] ALERT CRITICAL DEBUG EMERGENCY ERROR INFORMATIONAL NOTICE WARNING
Change the logging severity:
system logging sw-components sw-component alert-service config severity DEBUGExpand source
vanquish-01(config)# system logging sw-components sw-component alert-service config severity DEBUG
vanquish-01(config-sw-component-alert-service)# commit
Commit complete.
vanquish-01(config-sw-component-alert-service)# exit
vanquish-01(config)# exit
vanquish-01# show running-config system logging sw-components sw-component alert-service
system logging sw-components sw-component alert-service
config name alert-service
config description "Alert service"
config severity DEBUG
!
Add a remote logging destination:
**system logging remote-servers remote-server <remote_ip> config proto <tcp|udp>**Expand source
vanquish-01(config)# system logging remote-servers remote-server 10.10.10.10 config proto tcp
vanquish-01(config-remote-server-10.10.10.10)# commit
Commit complete.
vanquish-01(config-remote-server-10.10.10.10)# exit
vanquish-01(config)# exit
See the remote logging destinations:
show running-config system logging remote-serversExpand source
vanquish-01# show running-config system logging remote-servers
system logging remote-servers remote-server 10.10.10.10
config remote-port 514
config proto tcp
!
Remove a remote logging destination:
**no system logging remote-servers remote-server <remote_ip>**Expand source
vanquish-01# config
Entering configuration mode terminal
vanquish-01(config)# no system logging remote-servers remote-server 10.10.10.10
vanquish-01(config)# commit
Commit complete.
vanquish-01(config)# exit
Add a selector/filter to a remote log destination:
system logging remote-servers remote-server <remote_ip> selectors selector LOCAL0 NOTICEExpand source
vanquish-01(config)# system logging remote-servers remote-server 10.10.10.10 selectors selector LOCAL0 NOTICE
vanquish-01(config-remote-server-10.10.10.10)# commit
Commit complete.
Remote a selector/filter from a remote log destination:
no system logging remote-servers remote-server <remote_ip> selectors selector LOCAL0 NOTICEExpand source
vanquish-01(config)# no system logging remote-servers remote-server 10.10.10.10 selectors selector LOCAL0 NOTICE
vanquish-01(config)# commit
Commit complete.
Configuring logging via the GUI¶
You can configure the log level and view/configure remote syslog servers on the same screen.
Configuring logging via the RESTCONF API¶
List the log settings for all services
curl -sku admin:admin -H 'Content-Type: application/yang-data+json' https://vanquish/api/data/openconfig-system:system/logging/f5-openconfig-system-logging:sw-components
{
"f5-openconfig-system-logging:sw-components": {
"sw-component": [
{
"name": "alert-service",
"config": {
"name": "alert-service",
"description": "Alert service",
"severity": "DEBUG"
}
},
...
{
"name": "vconsole",
"config": {
"name": "vconsole",
"description": "Tenant virtual console",
"severity": "INFORMATIONAL"
}
}
]
}
}
List the log setting for a specific service (upgrade-service in this example)
curl -sku admin:admin -H 'Content-Type: application/yang-data+json' https://vanquish/api/data/openconfig-system:system/logging/f5-openconfig-system-logging:sw-components/sw-component=upgrade-service
{
"f5-openconfig-system-logging:sw-component": [
{
"name": "upgrade-service",
"config": {
"name": "upgrade-service",
"description": "Software upgrade service",
"severity": "INFORMATIONAL"
}
}
]
}
Set the log level to Error
curl -X PATCH -sku admin:admin -H 'Content-Type: application/yang-data+json' \
https://vanquish/api/data/openconfig-system:system/logging/f5-openconfig-system-logging:sw-components/sw-component=upgrade-service/config/severity \
-d '{"f5-openconfig-system-logging:severity":"ERROR"}'
Log file rotation settings¶
You can change the log file rotation settings by manually editing the configuration files. The settings are not intended to be tunable by customers.
ls -l /var/F5/system/etc/logrotate.d/
total 8
-rw-r--r--. 1 root root 140 Oct 26 20:46 appliance.conf
-rw-r--r--. 1 root root 217 Oct 26 20:46 velos.conf
cat appliance.conf
/var/log/audit.log
/var/log/confd.log
/var/log/devel.log
/var/log/lcd.log
/var/log/snmp.log {
rotate 5
size 100M
copytruncate
}
cat velos.conf
/var/log/velos.log {
rotate 10
size 1G
sharedscripts
postrotate
pkill -HUP rsyslogd
endscript
}
/var/log/logrotate.log
/var/log/rsyslogd.log {
rotate 2
size 5M
copytruncate
}
Viewing log files¶
This is part of confd but it’s relevant to logging
List the files in /var/F5/system/log/ (the confd path is “log/system”)
file list path log/system/
entries {
name
audit.log
confd.log
devel.log
lcd.log
lcd.log.1
logrotate.log
logrotate.log.1
logrotate.log.2.gz
platform.log
rsyslogd_init.log
snmp.log
startup.log
startup.log.prev
trace/
vconsole_auth.log
vconsole_startup.log
velos.log
webui/
}
List the files in /var/log/ (the confd path is “log/host”)
file list path log/host
entries {
name
anaconda/
ansible.log
appliance.log
audit/
boot.log
btmp
btmp.1
chrony/
containers/
cron
cron.1
cron.2.gz
dmesg
dmesg.old
flannel-install.log
glusterfs/
grubby
httpd/
journal/
k3s-install-ansible.log
k3s-install.log
k3s-update.log
k3s.log
kubevirt-install.log
lastlog
lcd-webserver/
lopd
messages
messages.1
messages.2.gz
messages.3.gz
messages.4.gz
multus-install.log
omd-iptable.log
omd_ring.log
platform/
pods/
qat_confd.log
rhsm/
run_container.log
run_lopd
run_plugin.log
run_stats.log
samba/
secure
secure.1
secure.2.gz
secure.3.gz
secure.4.gz
sssd/
sw-deport.log
sw-upgrade.log
tally/
tuned/
watchdog/
wtmp
}
Cat out velos.log and search for logs from the tcpdump service
file show log/system/velos.log | include tcpdumpExpand source
2021-10-26T21:12:03.353455+00:00 appliance-1 tcpdumpd-master[8]: priority="Notice" version=1.0 msgid=0x5405000000000017 msg="tcpdumpd-master inited." CLIENT_LISTEN_ADDR="0.0.0.0" CLIENT_LISTEN_PORT=44557.
2021-10-26T21:12:03.353463+00:00 appliance-1 tcpdumpd-master[8]: priority="Notice" version=1.0 msgid=0x5402000000000005 msg="tcpdumpd-master hw_type." HWTYPE="Vanquish-High".
2021-10-26T21:12:03.353729+00:00 appliance-1 tcpdumpd-master[8]: priority="Notice" version=1.0 msgid=0x5406000000000011 msg="Version: .." major=0 minor=0 trivial=1.
2021-10-26T21:12:03.359894+00:00 appliance-1 tcpdumpd-master[8]: priority="Info" version=1.0 msgid=0x5404000000000002 msg="Updating stored interface cache." NUMINTERFACE=22.
2021-11-04T17:53:54.162743+00:00 appliance-1 tcpdumpd-master[8]: priority="Err" version=1.0 msgid=0x5406000000000003 msg="Bad msg from DMAA. rcvlen: sizeoftcpdump_driver_notify_t:" rcvlen=0 sizeof_tcpdump=8.
2021-11-04T17:53:54.162815+00:00 appliance-1 tcpdumpd-master[8]: priority="Err" version=1.0 msgid=0x5406000000000009 msg="Memory Unmap returned:" ret=0.
2021-11-04T17:53:54.162850+00:00 appliance-1 tcpdumpd-master[8]: priority="Notice" version=1.0 msgid=0x5406000000000010 msg="Tcpdump DMAA MSG_CAPCTL sent." TCPDUMP_DMAA_SOCKET=12 ENABLE=0 RET=3.
2021-11-04T17:53:54.162884+00:00 appliance-1 tcpdumpd-master[8]: priority="Crit" version=1.0 msgid=0x5406000000000016 msg="DMAA cleaned up." DMAA_STATE=0 TCPDUMP_DMAA_SOCKET=-1.
2021-11-04T17:53:54.162911+00:00 appliance-1 tcpdumpd-master[8]: priority="Err" version=1.0 msgid=0x5406000000000004 msg="DMAA process failed, ret:" type="cleanup_clients_and_dmaa" ret=3.
2021-11-04T17:54:06.844606+00:00 appliance-1 tcpdumpd-master[8]: priority="Info" version=1.0 msgid=0x6602000000000007 msg="DB state monitor disconnected".
2021-11-04T17:54:06.844708+00:00 appliance-1 tcpdumpd-master[8]: priority="Info" version=1.0 msgid=0x6602000000000005 msg="DB is not ready".
Is it working or is it failing?¶
$ docker ps --filter=NAME=syslog --format='{{ .Status }}\t {{ .Names }}'
Up 11 days system_velocity_rsyslogd
Up 11 days system_rsyslogd
Reboots
cat /var/F5/system/log/platform.log | grep BOOT-MARKER
2021-11-12T15:36:21+00:00 appliance-1.chassis.local notice boot_marker: ---===[ BOOT-MARKER ]===---
2021-11-12T15:36:26+00:00 appliance-1.chassis.local notice boot_marker: ---===[ BOOT-MARKER ]===---
2021-11-12T15:55:44+00:00 appliance-1.chassis.local notice boot_marker: ---===[ BOOT-MARKER ]===---
2021-11-12T15:55:50+00:00 appliance-1.chassis.local notice boot_marker: ---===[ BOOT-MARKER ]===---
2021-11-12T17:42:25+00:00 appliance-1.chassis.local notice boot_marker: ---===[ BOOT-MARKER ]===---
2021-11-12T17:42:43+00:00 appliance-1.chassis.local notice boot_marker: ---===[ BOOT-MARKER ]===---
last | grep reboot
reboot system boot 3.10.0-1160.25.1 Fri Nov 12 17:42 - 18:10 (12+00:28)
reboot system boot 3.10.0-1160.25.1 Fri Nov 12 15:55 - 18:10 (12+02:14)
reboot system boot 3.10.0-1160.25.1 Fri Nov 12 15:36 - 15:51 (00:15)
reboot system boot 3.10.0-1160.25.1 Thu Nov 4 17:56 - 15:34 (7+21:37)
reboot system boot 3.10.0-1160.25.1 Tue Oct 26 21:08 - 17:54 (8+20:45)
reboot system boot 3.10.0-1160.25.1 Tue Oct 26 20:48 - 17:54 (8+21:05)
reboot system boot 3.10.0-1160.25.1 Tue Oct 26 20:42 - 17:54 (8+21:12)
Known Issues¶
Log files from /varF5/system/log are not downloadable in the GUI via the File Utilities screen.
There are no confd commands or GUI pages to view/search the logs.
