Fluentd Logging¶

Overview¶

The Service Proxy for Kubernetes (SPK) Fluentd pod is an open source data collector that can be configured to receive logging data from the SPK Controller, Traffic Management Microkernel (TMM), and Distributed Session State Management (dSSM) pods. The logs collected by Fluentd can be routed to standard output, filesystem, and Elasticsearch. To store the logs on the filesystem, the Fluentd pod must be bound to a Kubernetes persistence volume.

This document guides you through configuring and deploying the Fluentd pod to collect and store logs from different SPK components.

Fluentd Service¶

Upon installing SPK Fluentd, a Kubernetes Service object is created to receive logging data from Fluent bit on TCP service port 54321 and forward the data to Fluentd on TCP target port 24224. Ensure the service port is available, and the cluster has CoreDNS enabled. In the following example setup, Fluentd is deployed in two projects to collect logs from their respective pods.

Example Fluentd Setup:

_images/spk-fluentd-logging-latest.png

Example Fluentd Service:

Name:              f5-toda-fluentd
Namespace:         spk-utilities
Port:              54321/TCP

Name:              f5-toda-fluentd
Namespace:         spk-ingress
Port:              54321/TCP

Log file locations¶

The log data collected by Fluentd is stored in the following locations:

Container	Log file
f5-dssm-sentinel	/var/log/f5/f5-dssm-sentinel-0/sentinel.log
f5-dssm-db	/var/log/f5/f5-dssm-db-0/dssm.log
f5ingress	/var/log/f5/helm_release-f5ingress/pod_name/f5ingress.log
f5-tmm	/var/log/f5/f5-tmm/pod_name/f5-fsm-tmm.log
f5-tmm-routing	/var/log/f5/f5-tmm/pod_name/f5-tmm-routing.log

_images/spk_info.png Note: To modify the TMM logging level, review the bdt_cli section of the Debug Sidecar overview.

Requirements¶

Prior to installing Fluentd, ensure you have:

An OpenShift cluster.
An available persistence volume.
Installed the SPK software.
A Linux based workstation with Helm installed.

Procedures¶

Installation¶

Use the following steps to the install the f5-fluentd pod.

Change into local directory with the SPK files, and list the files in the tar directory:

In this example, the SPK files are in the spkinstall directory:
```
cd spkinstall
```
```
ls -1 tar
```
In this example, Fluentd Helm chart is named f5-toda-fluentd-3.0.25.tgz:
```
csrc-0.4.2.tgz
cwc-2.0.16.tgz
f5-cert-gen-0.5.2.tgz
f5-cert-manager-0.5.12.tgz
f5-dssm-3.0.37.tgz
f5-toda-fluentd-3.0.25.tgz
f5ingress-10.0.104.tgz
rabbitmq-2.0.5.tgz
spk-docker-images.tgz
```
Create two new projects for deploying Fluentd:
```
oc new-project <project>
```
Note:  You could skip project creation step if you already have existing projects to deploy Fluentd, created as part of dSSM Database and SPK Controller.

In this example, two new projects named spk-utilities and spk-ingress are created:
```
oc new-project spk-utilities
oc new-project spk-ingress
```
Create a Helm values file named fluentd-values.yaml, and set the image.repository and the persistence.storageClass parameters:
```
image:
  repository: '<registry>'   

persistence:
  enabled: true
  storageClass: '<name>' 
```
In this example, Helm pulls the f5-fluentd image from registry.com, and the pod will bind to the storageClass named managed-nfs-storage:
```
image:
  repository: 'local.registry.com'

persistence:
  enabled: true
  storageClass: 'managed-nfs-storage'
```
In fluentd-values.yaml file set the serviceAccount.create parameter:

Note: The serviceAccount will not be created by default.
```
serviceAccount:
  create: false
  name: default
```
Required:  Add the following parameters to fluentd-values.yaml file to collect logging data from different SPK components:

Important: The Fluent bit and Fluentd configurations are mandatory for proper log file recovery in the event of pod or container restarts; without this configuration, the users will not be able to recover their log files.
```
# enable controller logging 
f5ingress_logs:
  enabled: true
  stdout: true
# enable dSSM DB logging
dssm_logs:
  enabled: true
  stdout: true
# enable dSSM Sentinel logging   
dssm_sentinel_logs:
  enabled: true
  stdout: true
# enable cert manager logging  
cm_logs:
  enabled: true
  stdout: true
```
Note:  In this example, we are using the same values file for both spk-utilities and spk-ingress projects. It can be split into separate files (fluentd-spk-utilities-values.yaml and fluentd-spk-ingress-values.yaml) if required, to contain only the applicable components of the respective projects.

Install the f5-fluentd pod and reference the fluentd-values.yaml values file. Be certain to make note of the Fluentd hostnames displayed after the installation in order to update Fluent bit sidecar configuration in the last step:

In this example, the Fluentd Pod installs to the spk-utilities and spk-ingress projects.

helm install f5-fluentd tar/f5-toda-fluentd-3.0.25.tgz -f fluentd-values.yaml -n spk-utilities

Fluentd hostname: f5-toda-fluentd.spk-utilities.svc.cluster.local.
Fluentd port: "54321"

helm install f5-fluentd tar/f5-toda-fluentd-3.0.25.tgz -f fluentd-values.yaml -n spk-ingress

Fluentd hostname: f5-toda-fluentd.spk-ingress.svc.cluster.local.
Fluentd port: "54321"

The f5-fluentd pod should now be successfully installed:

In this example, the Fluentd Pod STATUS is Running:

oc get pods -n spk-ingress

NAME                              READY   STATUS    
f5-toda-fluentd-8cf96967b-jxckr   1/1     Running  

oc get pods -n spk-utilities

NAME                              READY   STATUS    
f5-toda-fluentd-9cgg22gc-iuacd    1/1     Running  

Fluentd should also be bound to the persistent volume:

In this example, the Fluentd Pod PVC displays STATUS as Bound:

oc get pvc -n spk-ingress

NAME              STATUS   VOLUME                                     STORAGECLASS
f5-toda-fluentd   Bound    pvc-7d36b530-b718-466c-9b6e-895e8f1079a2   managed-nfs-storage

oc get pvc -n spk-utilities

NAME              STATUS   VOLUME                                     STORAGECLASS
f5-toda-fluentd   Bound    pvc-4e77x331-u529-566w-6c1r-913f4g1116c1   managed-nfs-storage

Update Fluent bit sidecar of all pods to redirect logs to Fluentd pod deployed in their respective projects:

_images/spk_info.png Note:  In this example, the Fluentd hostnames are f5-toda-fluentd.spk-utilities.svc.cluster.local. and f5-toda-fluentd.spk-ingress.svc.cluster.local. The hostname value needs to be updated in dssm-values.yaml and ingress-values.yaml.

dssm-values.yaml

sentinel: 
  fluentbit_sidecar: 
    image: 
      repository: 'local.registry.com' 
    fluentd: 
      host: 'f5-toda-fluentd.spk-utilities.svc.cluster.local.' 

db: 
  fluentbit_sidecar: 
    image: 
      repository: 'local.registry.com' 
    fluentd: 
      host: 'f5-toda-fluentd.spk-utilities.svc.cluster.local.' 

ingress-values.yaml

controller: 
  fluentbit_sidecar:
    enabled: true 
    fluentd: 
      host: 'f5-toda-fluentd.spk-ingress.svc.cluster.local.' 
    image: 
      repository: 'local.registry.com' 

f5-toda-logging: 
 enabled: true
 fluentd: 
   host: 'f5-toda-fluentd.spk-ingress.svc.cluster.local.' 

Viewing logs¶

After installing the Controller and dSSM pods, you can use the following steps to view the logs in the f5-fluentd container:

oc exec -it deploy/f5-toda-fluentd -n <project> -- sh

In this example, the container is in the spk-utilities project:

oc exec -it deploy/f5-toda-fluentd -n spk-utilities -- sh

Change to the main logging directory, and list the subdirectories:
```
cd /var/log/f5; ls
```
In this example, logging directories are present for the f5-tmm, f5-dssm-db, and f5-dssm-sentinel pods:
```
f5-dssm-db-0  f5-dssm-db-1  f5-dssm-db-2  f5-dssm-sentinel-0  
f5-dssm-sentinel-1  f5-dssm-sentinel-2 f5-tmm  
```
Change into one of the subdirectories, for example f5-dssm-db-0:
```
cd f5-dssm-db-0
```
View the logs using the more command:
```
more -d dssm.log
```
Note:  Logs collected by Fluentd are also available as part of QKView diagnostic tarball. To include log files from all Fluentd pods, remember to run the QKView utility with their respective namespaces.

In this example, QKView utility should be invoked as follows:
```
./qkview-wrapper-linux -f ./qkview-collector.sh -n spk-ingress,spk-utilities 
```
(or)
```
./qkview-wrapper-darwin -f ./qkview-collector.sh -n spk-ingress,spk-utilities 
```
In the iHealth website, the log files from different pods would be visible under their respective Fluentd pod’s filesystem subtree:

Next step¶

Continue to one of the following steps listed by installation precedence:

Optional: Install the dSSM Database to store session-state information.
Required: Install the SPK Controller and Service Proxy TMM pods.

Feedback¶

Provide feedback to improve this document by emailing spkdocs@f5.com.

Supplemental¶

Previous Next