Fluentd Logging¶

Overview¶

The Cloud-Native Network Functions (CNFs) Fluentd Pod is an open-source data collector that can be configured to receive logging data from the BIG-IP Controller, Traffic Management Microkernel (TMM), Edge Firewall, Distributed Session State Managment (dSSM) Pods, and BGP updates from the TMM routing container. The Fluentd Pod must bind to a Kubernetes persistence volume in order to create the necessary log file directories.

This document guides you through understanding, configuring and deploying the f5-fluentd logging container.

Fluentd Service¶

When installing Fluentd, a Service object is created to receive logging data on TCP service port 54321, and forward the data to Fluentd on TCP service port 24224.

Example Fluentd Service:

Name:              f5-toda-fluentd
Namespace:         cnf-gateway
IP:                10.109.102.215
Port:              <unset>  54321/TCP
Endpoints:         10.244.1.75:24224

Log file locations¶

Fluentd collects logging data in the following log files:

Container	Log file
f5-dssm-sentinel	/var/log/f5/f5-dssm-sentinel-0/sentinel.log
f5-dssm-db	/var/log/f5/f5-dssm-db-0/dssm.log
f5ingress	/var/log/f5/helm_release-f5ingress/pod_name/f5ingress.log
f5-tmm	/var/log/f5/f5-tmm/pod_name/f5-fsm-tmm.log
f5-tmm-routing	/var/log/f5/f5-tmm/pod_name/f5-tmm-routing.log

Note:

To modify the TMM logging level, review the tmm_cli section of the Debug Sidecar overview.
When storage usage goes beyond 85%, Fluentd runs a cleanup script that deletes the oldest log files to free space for new ones. To adjust the threshold, add the environment variable DISK_CLEANUP_THRESHOLD: to fluentd deployment to the desired value, for example DISK_CLEANUP_THRESHOLD: 90.

Requirements¶

Prior to installing Fluentd, ensure you have:

An available persistence volume.
Installed the CNFs software.
Installed the CNFs Cert Manager.
A Linux based workstation with Helm installed.

Procedures¶

Installation¶

Use the following steps to the install the f5-fluentd container.

Change into the directory containing the latest CNFs Software, and obtain the f5-toda-fluentd Helm chart version:

In this example, the CNF files are in the cnfinstall directory:
```
cd cnfinstall
```
```
ls -1 tar | grep f5-toda-fluentd
```
The example output should appear similar to the following:
```
f5-toda-fluentd-2.3.2-0.0.6.tgz
```
Create a new Project for the CNFs Pods using the following command syntax:
```
oc create ns <project name>
```
In this example, a new Project named cnf-gateway is created:
```
oc create ns cnf-gateway
```
Create a Helm values file named fluentd-values.yaml, and set the image.repository, persistence.storageClass parameters:
```
image:
  repository: <registry>

persistence:
  enabled: true
  storageClass: "<name>"
```
In this example, Helm pulls the f5-fluentd image from registry.com, and the container will bind to the storageClass named managed-nfs-storage:
```
image:
  repository: registry.com

persistence:
  enabled: true
  storageClass: "managed-nfs-storage"
```

Required: Add the following parameters to the values file to collect logging data from each of the enabled CNFs Pods:

Important: The Fluentdbit and Fluentd configurations are mandatory for proper log file recovery in the event of pod or container restarts; without this configuration, the users will not able to receover their log files.

f5ingress_logs:
  enabled: true
  stdout: true
dssm_logs:
  enabled: true
  stdout: true
dssm_sentinel_logs:
  enabled: true
  stdout: true
pccd_logs:
  enabled: true
  stdout: true
ipsd_logs:
  enabled: true
  stdout: true
cm_logs:
## Enable Cert manager logging
  enabled: true
  stdout: true

Refer the imagePullSecret in fluentd-values.yaml to pull the Fluentd image from private repository.
```
imageCredentials:
   name: <imagepullsecret name>
```
Add the F5 Toda Fluentd serviceAccount to the Project’s privileged security context constraint (SCC):

Note: The cnf-cwc serviceAccount name is based on the Helm release name. See Step 7.
```
oc adm policy add-scc-to-user privileged -n <project> -z <serviceaccount>
```
In this example, the f5-toda-fluentd serviceAccount is added to the cnf-gateway Project’s privileged SCC:
```
oc adm policy add-scc-to-user privileged -n cnf-gateway -z f5-toda-fluentd
```

Install the Fluentd Pod using the following command syntax:

Note: Be certain to save the Fluentd hostname from the command output for the Controller installation.

helm install <release> tar/<helm-chart>.tgz \
-f <values>.yaml -n <namespace>

For example:

helm install f5-fluentd tar/f5-toda-fluentd-2.3.2-0.0.6.tgz \
-f fluentd-values.yaml -n cnf-gateway

In the example output, the Fluentd hostname is f5-toda-fluentd.cnf-gateway.svc.cluster.local.:

FluentD hostname: f5-toda-fluentd.cnf-gateway.svc.cluster.local.
FluentD port: "54321"

The Fluent Pod should now be successfully installed:

oc get pods -n cnf-gateway

In this example, the Fluentd Pod STATUS is Running:

NAME                              READY   STATUS    
f5-toda-fluentd-8cf96967b-jxckr   1/1     Running  

Fluentd should also be bound to the persistent volume:

oc get pvc -n cnf-gateway

In this example, the Fluentd Pod PVC displays STATUS as Bound:

NAME              STATUS   VOLUME                                     STORAGECLASS
f5-toda-fluentd   Bound    pvc-7d36b530-b718-466c-9b6e-895e8f1079a2   managed-nfs-storage

Viewing logs¶

After installing the BIG-IP Controller and dSSM Pods, you can use the following steps to view the logs in f5-fluentd container:

Note: Since shell is disabled, you cannot use the cd command. Hence, always use the absolute path to the log file.

List all subdirectories that contain logs.

oc exec -it deploy/f5-toda-fluentd -n <project> -- ls /var/log/f5

Example

In this example, the container is in the cnf-gateway Project:

oc exec -it deploy/f5-toda-fluentd -n cnf-gateway

Sample Output:

In this example, the logging directories are present for the f5ingress, f5-tmm, f5-dssm-db, and f5-dssm-sentinel pods.
```
f5-dssm-db-0  f5-dssm-db-1  f5-dssm-db-2  f5-dssm-sentinel-0

f5-dssm-sentinel-1  f5-dssm-sentinel-2  f5-ingress-f5ingress  f5-tmm 
```
Run more command to view the logs.

oc exec -it deploy/f5-toda-fluentd -n cnf-gateway -- more -d /var/log/f5/f5-dssm-db-0/dssm.log

Cleaning old log folders¶

You can enable the Folder Cleaner feature to automatically remove the old orphaned log directories. Deleting the old log folders prevents the storage volume from running out of free space. Normally, a scheduled job or cron task (CronJob) is performed within a container.

Note: By default, this feature is disabled.

The folders are deleted based on:

Disk usage thresholds
Configurable retention policies
File modification times

Using the oc edit deployment f5-toda-fluentd command, open the f5-toda-fluentd deployment in edit mode and set the environment variables in the fluentd container. The required environment variables are

Variable	Description	Default value
FOLDER_PATH	The root folder path to monitor and clean.	/var/log/f5/
DISK_ALERT_THRESHOLD	The percentage of disk usage. This triggers alerts (0-100)	85.0
DISK_CLEANUP_THRESHOLD	The percentage of disk usage. This triggers cleabyo (0-100)	95.0
ENABLE_DISK_CLEANUP	Enable or disable this feature.	false
LOG_RETENTION_DAYS	Number of inactive days before folder is deleted.	30

Sample file

This below sample file sets cleaner thresholds as env vars in the fluentd container.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: f5-toda-fluentd
spec:
  template:
    spec:
      containers:
      - name: f5-fluentd
        image: f5-fluentd:latest
        env:
        - name: FOLDER_PATH
          value: "/var/log/f5/"
        - name: DISK_CLEANUP_THRESHOLD
          value: "95.0"
        - name: DISK_ALERT_THRESHOLD
          value: "85.0"
        - name: LOG_RETENTION_DAYS
          value: "30"
        - name: ENABLE_DISK_CLEANUP
          value: "true"
...

Next step¶

Continue to one of the following steps listed by installation precedence:

Optional: Install the dSSM Database to store session-state information.
Required: Install the BIG-IP Controller, TMM and AFM Pods.

Feedback¶

Provide feedback to improve this document by emailing cnfdocs@f5.com.