Fluentd Logging

The BIG-IP Next for Kubernetes Fluentd logging Pod is an open source data collector that can be configured to receive logging data from the BIG-IP Next for Kubernetes, Traffic Management Microkernel (TMM), and Distributed Session State Management (dSSM) Pods. To create log file directories for each of the BIG-IP Next for Kubernetes Pods, Fluentd must bind to a Kubernetes Persistent Volumes.

Fluentd Service

After installing Fluentd, a Service object is created to receive logging data on TCP service port 54321, and forward the data to Fluentd on TCP service port 24224. Ensure the Service port is available, and the cluster has CoreDNS enabled. In this example, the BIG-IP Next for Kubernetes components will need to resolve the f5-toda-fluentd.f5-utils hostname:

Example Fluentd Service:

Name:              f5-toda-fluentd
Namespace:         f5-utils
IP:                10.109.102.215
Port:              <unset>  54321/TCP
Endpoints:         10.244.1.75:24224

Log file locations

Fluentd collects logging data in the following log files:

Container Log file
f5-dssm-sentinel /var/log/f5/<ns>/f5-dssm-sentinel-0/sentinel.log
f5-dssm-db /var/log/f5/<ns>/f5-dssm-db-0/dssm.log
f5ingress /var/log/f5/<ns>/helm_release-f5ingress/pod_name/f5ingress.log
f5-tmm /var/log/f5/<ns>/f5-tmm/pod_name/f5-fsm-tmm.log
f5-tmm-routing /var/log/f5/<ns>/f5-tmm/pod_name/f5-tmm-routing.log

Note:

  • <ns> - The namespace of the log.

  • To modify the TMM logging level, review the bdt_cli section of the Debug Sidecar overview.

  • When storage usage goes beyond 85%, Fluentd runs a cleanup script that deletes the oldest log files to free space for new ones. To adjust the threshold, add the environment variable DISK_CLEANUP_THRESHOLD: to fluentd deployment to the desired value, for example DISK_CLEANUP_THRESHOLD: 90.

Viewing logs

After installing the BIG-IP Next for Kuberneters with the Controller and dSSM Pods, you can use the following steps to view the logs in the f5-fluentd container:

  1. Run the following command to list the logging directories.

    kubectl -n f5-utils exec -it deploy/f5-toda-fluentd -- ls /var/log/f5

    Sample Output:

    f5-afm f5-crdconversion-6dd74cd5-m6mp9 f5-dssm-sentinel-0 f5-ipam-ctlr-5cb9546595-x42q9 f5-rabbit
f5-cne-controller f5-dssm-db-0 f5-dssm-sentinel-1 f5-observer-0 spk-csrc
f5-coremond f5-dssm-db-1 f5-dssm-sentinel-2 f5-observer-operator f5-spk-cwc
f5-crdconversion-6dd74cd5-dd8r4 f5-dssm-db-2 f5-ipam-ctlr-5cb9546595-6lnvd f5-observer-receiver-0 f5-tmm

  2. View the logs using the more/cat command:

    Example more command to view logs:

    kubectl -n f5-utils exec -it deploy/f5-toda-fluentd -- more -d /var/log/f5/<ns>/f5-dssm-db-0/dssm.log

    Example cat command to view logs:

    kubectl -n f5-utils exec -it deploy/f5-toda-fluentd -- cat /var/log/f5/f5-alpha/f5-dssm-db-0/dssm.log

Cleaning old log folders

You can enable the Folder Cleaner feature to automatically remove the old orphaned log directories. Deleting the old log folders prevents the storage volume from running out of free space. Normally, a scheduled job or cron task (CronJob) is performed within a container.

Note: By default, this feature is disabled.

The folders are deleted based on:

  • Disk usage thresholds

  • Configurable retention policies

  • File modification times

Using the oc edit deployment f5-toda-fluentd command, open the f5-toda-fluentd deployment in edit mode and set the environment variables in the fluentd container.

Variable Description Default Value
DISK_ALERT_THRESHOLD The percentage of disk usage. This triggers alerts (0-100) 85.0
DISK_CLEANUP_THRESHOLD The percentage of disk usage. This triggers cleabyo (0-100) 95.0
ENABLE_DISK_CLEANUP Enable or disable this feature. false
FOLDER_PATH The root folder path to monitor and clean. /var/log/f5/f5-alpha/
LOG_RETENTION_DAYS Number of inactive days before folder is deleted. 30

Sample file

This below sample file sets cleaner thresholds as env vars in the fluentd container.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: f5-toda-fluentd
spec:
  template:
    spec:
      containers:
      - name: f5-fluentd
        image: f5-fluentd:latest
        env:
        - name: FOLDER_PATH
          value: "/var/log/f5/f5-alpha/"
        - name: DISK_CLEANUP_THRESHOLD
          value: "95.0"
        - name: DISK_ALERT_THRESHOLD
          value: "85.0"
        - name: LOG_RETENTION_DAYS
          value: "30"
        - name: ENABLE_DISK_CLEANUP
          value: "true"
...

Supplemental