Troubleshoot your installation¶

Installation Issues¶

Issue	Diagnosis	Resolution
FLO pod not starting	Check image pull secrets (far-pull-secret), verify FAR credentials. Check: `kubectl describe pod <FLO_POD>`	Verify: `kubectl get secret far-pull-secret -n f5-cne-core`
CRDs not installed	FLO’s CRD Installer Job may have failed. Check: `kubectl get jobs	grep crd`
Env-Discovery not labeling DPU nodes	Verify DPU node is joined to cluster. Check: `kubectl get nodes`. Verify Env-Discovery pod logs.	Manually label: `kubectl label node <DPU_NODE> f5-tmm=true`
TMM not scheduling on DPU	Check node labels (`f5-tmm=true`), taints, and resource availability (hugepages).	Verify: `kubectl describe node <DPU_NODE>
DPU node cannot join Kubernetes cluster	DPU node fails to join or rejoin the cluster after `kubeadm join`.	Reset Kubernetes on the DPU node and reinstall. See Reinstall Kubernetes on a DPU node.

Reinstall Kubernetes on a DPU node¶

If a DPU node cannot join the Kubernetes cluster, reset Kubernetes on the DPU and reinstall.

On the DPU node:

Reset Kubernetes and remove CNI configuration.

sudo kubeadm reset
sudo rm -rf /etc/cni/net.d

Remove Kubernetes packages.

sudo apt remove kubeadm kubelet kubectl kubernetes-cni
sudo rm -rf /lib/systemd/system/kubelet.service.d
sudo rm -rf /etc/apt/keyrings/kubernetes.asc
sudo rm -rf /etc/apt/keyrings/kubernetes-apt-keyring.gpg
sudo rm /etc/apt/sources.list.d/kubernetes.*

Reboot the DPU node.
```
sudo reboot
```
Restart containerd.
```
sudo systemctl restart containerd
```

Add the Kubernetes apt repo.

sudo mkdir -p /etc/apt/keyrings && \
  curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key | \
  sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes.asc 2>&1 && \
  echo "Types: deb
URIs: https://pkgs.k8s.io/core:/stable:/v1.34/deb
Suites: /
Signed-By: /etc/apt/keyrings/kubernetes.asc" | \
  sudo tee /etc/apt/sources.list.d/kubernetes.sources && \
  sudo apt-get update -qq 2>&1 | tail -3

If the GPG key fails, remove and re-download it.

sudo rm -f /etc/apt/keyrings/kubernetes.asc && \
  curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key | \
  sudo tee /etc/apt/keyrings/kubernetes.asc > /dev/null 2>&1 && \
  sudo apt-get update -qq 2>&1 | tail -5

Install Kubernetes packages.

sudo apt-get install -y kubeadm=1.34.2-1.1 kubelet=1.34.2-1.1 kubectl=1.34.2-1.1 kubernetes-cni 2>&1 | tail -10

Verify kubelet is running and reload the configuration.

sudo systemctl status kubelet
sudo systemctl daemon-reload

On the host node:

Generate a new join token.

kubeadm token create --print-join-command

Join the DPU node to the cluster.

sudo kubeadm join <CONTROL_PLANE_IP>:6443 --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH> \
  --node-name <DPU_NODE_NAME>

Networking Issues¶

Issue	Diagnosis	Resolution
No external connectivity	Verify OVS bridges, SF/VF connectivity, VLAN CR status. Check Self IPs on TMM.	Exec into TMM debug sidecar: `ip addr`, ping from TMM.
VLAN CR stuck in non-Ready	Check CNE Controller logs for gRPC errors. Verify TMM is running.	`kubectl logs <F5_INGRESS_POD>
Egress SNAT not working	Verify F5SPKEgress and F5SPKSnatpool CRs. Check SNAT pool addresses are routable.	Use netkvest for debugging: ping/traceroute from SNAT pool.
MTU mismatch / fragmentation	Compare MTU across host, OVS, DPU, and VLAN CR. Use tcpdump to check for fragmentation.	Set consistent MTU in F5SPKVlan CR `spec.mtu` field.

FLO and Lifecycle Issues¶

Issue	Diagnosis	Resolution
CR Status: Accepted=False	CR syntax error or validation failure.	`kubectl describe <CR>`, check status message, fix CR YAML.
CR Status: Reconciled=False (Waiting)	Missing dependencies (e.g., CRDs not installed, secrets missing).	Check FLO logs: `kubectl logs <FLO_POD>`
CR Status: Available=False	Component pod is unhealthy or not running.	Check component pod: `kubectl describe pod`, `kubectl logs`
FLO not reconciling changes	FLO watches BnkGatewayClass CR for changes.	Verify CR was modified: `kubectl get <CR> -o yaml`

TMM Issues¶

Issue	Diagnosis	Resolution
TMM pod CrashLoopBackOff	Check TMM logs for startup errors. Verify hugepages, VFIO, and SR-IOV.	`kubectl logs <TMM_POD> -c f5-tmm --previous`
TMM not processing traffic	Verify VLANs, Virtual Server CRs, and backend service endpoints.	Use debug sidecar: tmctl, tmsh commands.
Core dump generated	Coremond captures core files. Check storage availability.	`kubectl logs <COREMOND_POD>`, check PV usage.
High TMM CPU utilization	Check traffic volume, number of connections, active iRules.	Review OTEL metrics: `f5_tmm_cpu_utilization`

Licensing Issues¶

Issue	Diagnosis	Resolution
License activation fails	CWC can’t reach TEEMs (connected mode) or FLP.	Check network connectivity from CWC pod. Verify proxy settings.
Telemetry report download fails	Report may be in wrong state.	Check status API first; report state must be `Telemetry in Progress`.
License switch rejected	Cannot switch eval to eval. Report must be in valid state.	Download and submit pending reports before switching.

Debug Sidecar and netkvest¶

The TMM Debug Sidecar provides CLI access to diagnostic utilities:

# Connect to the debug sidecar
kubectl exec -it <TMM_POD> -c f5-tmm-debug -n <cneInstanceNamespace> -- bash

# Available utilities:
tmsh        # TMM shell for configuration inspection
tmctl       # TMM statistics and counters
tcpdump     # Packet capture on TMM interfaces
ssldump     # SSL/TLS session inspection
netkvest    # Network connectivity testing (ping, traceroute from SNAT pool)

netkvest usage examples:

# Ping from a specific SNAT pool IP
netkvest ping --source-snat-pool egress-snat --destination 10.0.0.1

# Traceroute from SNAT
netkvest traceroute --source-snat-pool egress-snat --destination 10.0.0.1

CWC Debug REST APIs provide remote diagnostics:

curl --key cwc_api/client_key.pem \
  --cert cwc_api/client_certificate.pem \
  --cacert cwc_api/ca_certificate.pem \
  https://f5-spk-cwc.f5-cne-core:30881/debug/diagnostics \
  -H "Authorization: Bearer $CWC_TOKEN"

Log Formats and Analysis¶

BNK components use structured log formats:

Component	Log Format and Content
TMM	JSON structured; includes timestamp, severity, module, and message. Log levels: Emergency through Debug.
CNE Controller	JSON structured; reports CR processing, gRPC communication, and reconciliation events.
CWC	JSON structured; includes licensing events, telemetry, QKView operations.

# View TMM logs (last 50 lines)
kubectl logs daemonset/f5-tmm -c f5-tmm -n <cneInstanceNamespace> --tail=50

# View CNE Controller logs
kubectl logs deployment/f5-ingress -n <cneInstanceNamespace> --tail=50

# View FLO logs
kubectl logs deployment/f5-flo -n f5-cne-core --tail=50

# Filter by severity
kubectl logs daemonset/f5-tmm -c f5-tmm -n <cneInstanceNamespace> | grep '"severity":"error"'

Validating Scalable Function K8s Resources¶

The SR-IOV Network Device Plugin is used to create SF resource on your k8s cluster. This resource is consumed by f5-tmm pod.

This section assumes the SF are created on the DPU. The example logs also assumes the following sriovdp-config ConfigMap used by the SR-IOV Network Device Plugin.

apiVersion: v1
kind: ConfigMap
metadata:
  name: sriovdp-config
data:
  config.json: |
    {
        "resourceList": [
            {
                "resourceName": "bf3_p0_sf",
                "resourcePrefix": "nvidia.com",
                "deviceType": "auxNetDevice",
                "selectors": [{
                    "vendors": ["15b3"],
                    "devices": ["a2dc"],
                    "pciAddresses": ["0000:03:00.0"],
                    "pfNames": ["p0#1"],
                    "auxTypes": ["sf"]
                }]
            },
            {
                "resourceName": "bf3_p1_sf",
                "resourcePrefix": "nvidia.com",
                "deviceType": "auxNetDevice",
                "selectors": [{
                    "vendors": ["15b3"],
                    "devices": ["a2dc"],
                    "pciAddresses": ["0000:03:00.1"],
                    "pfNames": ["p1#1"],
                    "auxTypes": ["sf"]
                }]
            }
        ]
    }

The logs are on the sriov-device-plugin pod deployed on the DPU, find this pod.

kubectl -n kube-system get pods --selector=app=sriovdp -owide

Sample Output:

NAME                              READY   STATUS    RESTARTS       AGE     IP               NODE                       NOMINATED NODE   READINESS GATES
kube-sriov-device-plugin-nstrs    1/1     Running   0              2d2h    <IP address>     dpu.example.com            <none>           <none>
kube-sriov-device-plugin-p8mv5    1/1     Running   0              2d22h   <IP address>     server-system.example.com  <none>           <none>
. . .

The logs below show a proper config, notice the new ResourcePools named bf3_p0_sf and bf3_p1_sf are created.

kubectl logs pod/kube-sriov-device-plugin-nstrs -n kube-system

Sample Output:

I0318 00:01:56.858474       1 manager.go:57] Using Kubelet Plugin Registry Mode
I0318 00:01:56.858881       1 main.go:46] resource manager reading configs
I0318 00:01:56.858984       1 manager.go:86] raw ResourceList: {
    "resourceList": [
        {
            "resourceName": "bf3_p0_sf",
            "resourcePrefix": "nvidia.com",
            "deviceType": "auxNetDevice",
            "selectors": [{
                "vendors": ["15b3"],
                "devices": ["a2dc"],
                "pciAddresses": ["0000:03:00.0"],
                "pfNames": ["p0#1"],
                "auxTypes": ["sf"]
            }]
        },
        {
            "resourceName": "bf3_p1_sf",
            "resourcePrefix": "nvidia.com",
            "deviceType": "auxNetDevice",
            "selectors": [{
                "vendors": ["15b3"],
                "devices": ["a2dc"],
                "pciAddresses": ["0000:03:00.1"],
                "pfNames": ["p1#1"],
                "auxTypes": ["sf"]
            }]
        }
    ]
}
I0318 00:01:56.859175       1 factory.go:203] *types.AuxNetDeviceSelectors for resource bf3_p0_sf is [0x4000256ea0]
I0318 00:01:56.859221       1 factory.go:203] *types.AuxNetDeviceSelectors for resource bf3_p1_sf is [0x4000257040]
I0318 00:01:56.859229       1 manager.go:106] unmarshalled ResourceList: [{ResourcePrefix:nvidia.com ResourceName:bf3_p0_sf DeviceType:auxNetDevice ExcludeTopology:false Selectors:0x4000124420 AdditionalInfo:map[] SelectorObjs:[0x4000256ea0]} {ResourcePrefix:nvidia.com ResourceName:bf3_p1_sf DeviceType:auxNetDevice ExcludeTopology:false Selectors:0x4000124438 AdditionalInfo:map[] SelectorObjs:[0x4000257040]}]
I0318 00:01:56.859296       1 manager.go:217] validating resource name "nvidia.com/bf3_p0_sf"
I0318 00:01:56.859332       1 manager.go:217] validating resource name "nvidia.com/bf3_p1_sf"
I0318 00:01:56.859339       1 main.go:62] Discovering host devices
I0318 00:01:56.929377       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:03:00.0	02          	Mellanox Technolo...	MT43244 BlueField-3 integrated Connec...
I0318 00:01:56.931375       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:03:00.1	02          	Mellanox Technolo...	MT43244 BlueField-3 integrated Connec...
I0318 00:01:56.933649       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:03:00.0	02          	Mellanox Technolo...	MT43244 BlueField-3 integrated Connec...
I0318 00:01:56.933718       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:03:00.1	02          	Mellanox Technolo...	MT43244 BlueField-3 integrated Connec...
I0318 00:01:56.933726       1 main.go:68] Initializing resource servers
I0318 00:01:56.933736       1 manager.go:117] number of config: 2
I0318 00:01:56.933753       1 manager.go:121] Creating new ResourcePool: bf3_p0_sf
I0318 00:01:56.933759       1 manager.go:122] DeviceType: auxNetDevice
I0318 00:01:56.943349       1 manager.go:138] initServers(): selector index 0 will register 1 devices
I0318 00:01:56.943399       1 factory.go:124] device added: [identifier: mlx5_core.sf.4, vendor: 15b3, device: a2dc, driver: mlx5_core]
I0318 00:01:56.943449       1 manager.go:156] New resource server is created for bf3_p0_sf ResourcePool
I0318 00:01:56.943462       1 manager.go:121] Creating new ResourcePool: bf3_p1_sf
I0318 00:01:56.943469       1 manager.go:122] DeviceType: auxNetDevice
I0318 00:01:56.951885       1 manager.go:138] initServers(): selector index 0 will register 1 devices
I0318 00:01:56.951930       1 factory.go:124] device added: [identifier: mlx5_core.sf.5, vendor: 15b3, device: a2dc, driver: mlx5_core]
I0318 00:01:56.951963       1 manager.go:156] New resource server is created for bf3_p1_sf ResourcePool
I0318 00:01:56.951979       1 main.go:74] Starting all servers...
I0318 00:01:56.952034       1 server.go:259] starting bf3_p0_sf device plugin endpoint at: nvidia.com_bf3_p0_sf.sock
I0318 00:01:56.952604       1 server.go:259] starting bf3_p1_sf device plugin endpoint at: nvidia.com_bf3_p1_sf.sock
I0318 00:01:56.952749       1 main.go:79] All servers started.
I0318 00:01:56.952769       1 main.go:80] Listening for term signals
I0318 00:01:57.191927       1 server.go:167] ListAndWatch(bf3_p0_sf) invoked
I0318 00:01:57.192116       1 server.go:120] Plugin: nvidia.com_bf3_p1_sf.sock gets registered successfully at Kubelet
I0318 00:01:57.191983       1 server.go:180] ListAndWatch(bf3_p0_sf): send devices devices:{ID:"mlx5_core.sf.4" health:"Healthy"}
I0318 00:01:57.191953       1 server.go:120] Plugin: nvidia.com_bf3_p0_sf.sock gets registered successfully at Kubelet
I0318 00:01:57.192268       1 server.go:167] ListAndWatch(bf3_p1_sf) invoked
I0318 00:01:57.192303       1 server.go:180] ListAndWatch(bf3_p1_sf): send devices devices:{ID:"mlx5_core.sf.5" health:"Healthy"}

Examine the DPU node to ensure the new SF resources are available in the cluster.

kubectl describe node dpu.example.com

Sample Output:

Name:               dpu.example.com
Capacity:
nvidia.com/bf3_p0_sf:  1
nvidia.com/bf3_p1_sf:  1
Allocatable:
nvidia.com/bf3_p0_sf:  1
nvidia.com/bf3_p1_sf:  1

Multus out of memory after CNE installation¶

When bringing up a CNEInstance and the CNE pods are failing to make progress, look at the status of all multus pods. Multus pods running out of memory can be a reason for this. Below is an example output what you’d see.

$ kubectl -n kube-system get po --selector=app=multus -owide
NAME                   READY   STATUS      RESTARTS        AGE     IP              NODE                       NOMINATED NODE   READINESS GATES
kube-multus-ds-2g47t   1/1     Running     0               5h39m   10.155.217.33   k8s-control                <none>           <none>
kube-multus-ds-2t4gs   1/1     Running     0               5h39m   10.155.217.35   k8s-worker-1               <none>           <none>
kube-multus-ds-69klt   1/1     Running     0               5h39m   10.155.217.34   k8s-worker-0               <none>           <none>
kube-multus-ds-9nt79   1/1     Running     0               5h39m   10.154.34.41    titan5-dpu.npi.f5net.com   <none>           <none>
kube-multus-ds-9x6q5   0/1     OOMKilled   8 (5m33s ago)   5h39m   10.154.34.40    titan5.npi.f5net.com       <none>           <none>

The memory for multus need to be increased to resolve this issue.

Increase the Multus DaemonnSet memory resource requests and limits.

kubectl -n kube-system patch ds kube-multus-ds \
--type='json' \
-p='[
    {"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/memory","value":"256Mi"},
    {"op":"replace","path":"/spec/template/spec/containers/0/resources/limits/memory","value":"256Mi"}
]'

The MULTUS pod that failed should now be back up.

If that did not work, take the following steps below.

Log into the DPU node using SSH.
Delete the multus-shim.
```
sudo rm -f /opt/cni/bin/multus-shim
```

Delete the failing pod.

kubectl --namespace kube-system delete pod kube-multus-ds-9nt79

The MULTUS pod that failed should now be back up.

CRD Quick Reference¶

CRD	Category	Purpose
F5SPKVlan	Networking	VLAN configuration: interfaces, Self IPs, MTU, bonding
F5SPKStaticRoute	Networking	Static route table management for TMM
L4Route	Traffic (F5)	L4 TCP/UDP virtual server and load balancing pool
Gateway	Traffic (Gateway API)	Entry point for external traffic with listeners
HTTPRoute	Traffic (Gateway API)	HTTP routing rules (path, header matching)
GRPCRoute	Traffic (Gateway API)	gRPC routing rules
GatewayClass	Traffic (Gateway API)	Infrastructure-level gateway class definition
ReferenceGrant	Traffic (Gateway API)	Cross-namespace resource references
F5SPKEgress	Traffic (F5)	Egress SNAT, flow forwarding, PseudoCNI, VXLAN
F5SPKSnatpool	Traffic (F5)	SNAT pool IP address allocation
F5SPKGlobalOptions	Protection/NAT	Global options including DPU crypto acceleration
F5BigFwPolicy	Firewall	Stateful firewall policy with rule lists
F5BigFwRulelist	Firewall	Firewall rule definitions (addresses, ports, actions)
F5BigCneAddresslist	Firewall	IP address lists for firewall rules
F5BigCnePortlist	Firewall	Port lists for firewall rules
F5BigDdosGlobal	DDoS	DDoS detection vectors, thresholds, allow lists
F5BigLogHslpub	Logging	HSL publisher for remote log servers
F5BigLogProfile	Logging	Log profile configuration
F5BigCneIrule	Extensibility	iRule definitions for custom traffic handling
SecPolicy	Gateway Extension	Security policy attachment (Firewall, DDoS to Gateway)
NetPolicy	Gateway Extension	Network policy attachment (iRules, HSL, TCP to Gateway)
BnkGatewayClass	Lifecycle (FLO)	Triggers FLO to deploy all BNK components