CNFs Event Logs¶

Identifying errors while performing CNFs installation procedures will save considerable post-installation troubleshooting time; helping you to avoid complicated configuration reviews and log file analysis. This document demonstrates how to view CNFs event logs using the standard kubectl logs command, and how to look for errors that may occur during installations. The CNFs event logs may not be available when Fluentd has been configured to collect the Controller and Traffic Management Microkernel (TMM) logs. Review the Fluentd Logging overview for more information.

BIG-IP Controller¶

The BIG-IP Controller listens to the Kubernetes API for CNFs’ Custom Resource (CR) installation events. When one of the CNFs CRs is installed, the BIG-IP Controller processes the CR data, logs both CR processing and configuration data, and if the configuration passes validation, configures the TMM Proxy Pod.

To filter the BIG-IP Contoller logs for errors while installing a CNFs CR, and also to gather the CR configuration logged during the installation, you must open two separate shells. Use the steps below to view the CNF BIG-IP Controller logs while installing a simple F5BigContextSecure CR:

Using shell 1, obtain the name of the BIG-IP Controller Pod:

In this example, the BIG-IP Controller is in the cnf-gateway Namespace:
```
kubectl get pods -n cnf-gateway | grep ingress
```
In this example, the BIG-IP Controller name is f5ingress-f5ingress-6c4ffd7886-zxpp7:
```
f5ingress-f5ingress-6c4ffd7886-zxpp7   2/2     Running
```
Stream the Controller events in real time and filter for errors using the -f option and grep command. The kubectl logs command uses the Pod name twice; once with the hash suffix to target the Pod, and once without the hash to target the container:
```
kubectl logs -f f5ingress-f5ingress-6c4ffd7886-zxpp7 -c f5ingress-f5ingress -n cnf-controller | grep -iE 'error|added'
```

Using shell 2, install the CNF CR, and look for error log messages in shell 1. Error log messages begin with a capital E:

kubectl apply -f cnf-contextsecure-cr.yaml

An Error message is logged indicating the F5BigContextSecure CR ipProtocol value any does not match the profile value tcp:

E0127 12:00:00.12345  1 controller_securecontext.go:95] Error with AddOrUpdateSecureContext configuration for cnf-gateway/cnf-context-secure: Error Validating SecureContext: Profile 'tcp' incompatible with ipProtocol 'any'

Fix: The F5BigContextSecure ipProtocol parameter was set to tcp to match the profile parameter value, and the CR was reinstalled:
```
kubectl apply -f cnf-contextsecure-cr.yaml
```
An added/updated message is logged indicating the F5BigContextSecure CR was successfully added to TMM:
```
I0202 12:00:00:1234    1 event.go:282] Event(v1.ObjectReference{Kind:"F5SecureContext",
SecureContext cnf-gateway/cnf-context-secure was added/updated
```
Using shell 2, capture the BIG-IP Controller configuration logs sent to TMM, using the >filename.txt and --since= options to capture and limit the amount of data:

In this example, the last 2 minutes of Controller logs are saved to the controller-logs.txt file:
```
kubectl logs f5ingress-f5ingress-6c4ffd7886-zxpp7 -c f5ingress-f5ingress -n cnf-controller --since=2m > controller-logs.txt
```

Use an editor to view the Controller logging data and the configuration sent to TMM:

I0129 01:00:00.34760     1 controller_securecontext.go:124] Adding SecureContext: cnf-context-secure
I0129 01:00:00.36214     1 grpccfg2.go:97] Create mTLS secure context.
I0129 01:00:00.36300     1 grpccfg2.go:389] gRPC - Send GRPC Message:

                  {
                          "embedded": {
                          "@type": "type.googleapis.com/declTmm.create_msg",
                          "revision": 0,
                          "embedded": {
                                  "@type": "type.googleapis.com/declTmm.virtual_server",
                                  "id": "cnf-gateway-cnf-context-secure-SecureContext_vs",
                                  "enabled": true,
                                 "source_address_translation_type": "SRC_TRANS_NONE",
                                  "default_pool": "",
                                  "traffic_matching_criteria": "cnf-gateway-cnf-context-secure-SecureContext_tmc",
                                  "security_nat_policy": "cnf-nat-policy-natpolicy",
                                  "security_firewall_enforced_policy": "cnf-fw-policy-firewallpolicy",
                                  "security_log_profile": "cnf-log-profile-securitylogprofile",
                                  "no_translate": true,
                                  "irules_reference": [
                                   ]
                          }
                          }
                  }

TMM Proxy Pod¶

The TMM Proxy Pod processes 5G workloads when configured with one or more of the Traffic Management CNFs CRs. To process traffic, the TMM Proxy Pod relies on supporting Pods, for example the dSSM Database. When connectivity failures occur between the PODs, traffic management processing failures may also occur. Use the steps below to filter the TMM logs for errors connecting to the dSSM (redis) Database.

_images/spk_info.png Note: To modify the TMM logging level, review the tmm_cli section of the Debug Sidecar overview.

Obtain the name of the TMM Proxy Pod(s):

In this example, the TMM Proxy Pod is in the cnf-gateway Namespace:

kubectl get pods -n cnf-gateway | grep tmm

The command output shows there are two TMMs in the Namespace:

f5-tmm-dcbc5c4b6-9ldbh                 4/4     Running
f5-tmm-dcbc5c4b6-qkrkh                 4/4     Running

View the TMM Proxy Pod logs, using the grep command to filter for specific events. The kubectl logs command uses the Pod name to target the Pod, and f5-tmm to target the container:

In this case, NAT is not working, so the grep command will filter for dSSM (redis) database connectivity events:
```
kubectl logs f5-tmm-dcbc5c4b6-9ldbh -n cnf-gateway -c f5-tmm | grep -i redis
```
In this example, the TMM Proxy Pod is unable to connect to the dSSM (redis) database:
```
Jan 01 12:00:00.12345 tmm[12]: redis_reconnect/1255: Retrying redis connect: 5
redis_sentinel_conn_fail/1031: Connection establishment with REDIS SENTINEL server failed
redis_reconnect_later/1280: Exceeded max no of retries
```

Fix: The TMM SESSIONDB_EXTERNAL_SERVICE Helm parameter was set to the correct dSSM hostname f5-dssm-sentinel.cnf-gateway, and the BIG-IP Controller was reinstalled. The TMM logs now show successfull connectivity:

kubectl logs f5-tmm-dcbc5c4b6-9ldbh -n cnf-gateway -c f5-tmm | grep -i redis

Jan 01 12:00:00:12345 tmm[12]: redis_sentinel_connected/1006: Connection establishment with REDIS SENTINEL server successful
Jan 01 12:00:00:12346 tmm[12]: redis_pubsub_connected/1418: Pubsub connection established with REDIS server
Jan 01 12:00:00:12347 tmm[12]: redis_connected/4846: Connection established with REDIS server

Use the grep -i <string> option to filter for additional events that may relate to current issues: grpc, vlan, endpoint, etc.

Logging Levels¶

By default, the f5-tmm container logs events at the Notice level. For troubleshooting purposes, you can change the default logging level using the BIG-IP Controller Helm values file, or on a currently installed and running f5-tmm container using the tmm_cli command. The available logging levels are listed below, use either of the procedures to change the default logging level.

1-Debug, 2-Informational, 3-Notice, 4-Warning, 5-Error, 6-Critical, 7-Alert, 8-Emergency

tmm_cli¶

Use the steps below to modify the logging level for a currently installed and running f5-tmm container.

Connect to the debug sidecar:

In this example, the debug sidecar is in the cnf-gateway Project:
```
kubectl exec -it deploy/f5-tmm -c debug -n cnf-gateway -- bash
```
To set the f5-tmm container’s logging level to Debug, run the following command:
```
tmm_cli -logLevel 1
```
```
ok
```
The f5-tmm container will log an event message simlilar to the following:
```
Set bigdb var 'log.tmm.level'='Debug'
```

Helm value¶

Use the steps below to modify the default logging level using the BIG-IP Controller Helm values.yaml file.

Edit the Helm values.yaml file, and add the tmm.logLevel parameter with the desired logging level value:

In the example below, the logging level is set to Debug:
```
tmm:
  logLevel: Debug
```
Install the BIG-IP Controller.

Feedback¶

Provide feedback to improve this document by emailing cnfdocs@f5.com.

Supplemental¶

Review the Troubleshooting Section of the BGP Overview

Previous Next