Administration Guide

DPU Health Monitoring

Monitor DPU health as part of your ongoing cluster operations.

1. Check DPU node status in Kubernetes.

   # Verify DPU node is Ready
   kubectl get nodes -o wide
   
   # Check node conditions
   kubectl describe node <DPU_NODE> | grep -A 10 Conditions
   

2. Monitor DPU resource consumption.

   # Check resource usage on DPU node
   kubectl top node <DPU_NODE>
   
   # Check TMM pod resource usage
   kubectl top pod -l app=f5-tmm -n <cneInstanceNamespace>
   

3. Access the DPU console for hardware diagnostics.

   # Via RShim (from host)
   ssh ubuntu@192.168.100.2
   
   # Check DPU firmware version
   mlxfwmanager --query
   
   # Check DPU temperature and health
   mst status
   mlxlink -d /dev/mst/mt41692_pciconf0 -m
   

4. Monitor DPU network links.

   # Check OVS bridge status
   ovs-vsctl show
   
   # Check SF/VF link status
   ip link show
   
   # Check interface statistics
   ethtool -S <INTERFACE_NAME>
   

Component Status Checks

Run regular health checks on all BNK components.

1. Check all pods status.

   kubectl get pods -n <cneInstanceNamespace> -o wide
   

2. Check TMM DaemonSet status.

   kubectl get daemonset f5-tmm -n <cneInstanceNamespace>
   

3. Check dSSM Sentinel health.

   kubectl exec -it <DSSM_POD> -n f5-cne-core -- redis-cli -p 26379 sentinel masters
   

4. Check CWC license and health status.

   curl --key cwc_api/client_key.pem --cert cwc_api/client_certificate.pem \
     --cacert cwc_api/ca_certificate.pem \
     https://f5-spk-cwc.f5-cne-core:30881/status \
     -H "Authorization: Bearer $CWC_TOKEN"
   

5. Check OTEL Collector exporting.

   kubectl logs -l app=otel-collector -n <cneInstanceNamespace> --tail=20
   

6. Check CRD installation count.

   kubectl get crds | grep f5 | wc -l
   

FLO CR Status Monitoring

FLO reports three status conditions for each component custom resource:

Condition	Status	Meaning	Action
Accepted	True	FLO received a valid CR	Normal state
Accepted	False (Failed)	FLO rejected the CR	Check CR syntax and FLO logs
Reconciled	True	FLO applied configuration	Normal state
Reconciled	False (Waiting)	Waiting for dependencies	Check the referenced resources
Available	True	Component is healthy and running	Normal state
Available	False	Component is unhealthy	Check logs and events

# Check FLO status for all components
kubectl get bnkgatewayclass -o yaml | grep -A 5 'conditions'

# Detailed status for a specific component
kubectl describe <COMPONENT_CR_TYPE> <CR_NAME> -n <NAMESPACE>

Scaling TMM Instances

In daemon-set mode, the TMM workload runs as a daemon set with one TMM pod on each DPU node.

To scale TMM capacity:

• Add DPU nodes: Add new hardware, join the node to the cluster, and label it with app=f5-tmm. FLO deploys TMM automatically.

• Remove DPU nodes: Drain the node, remove the label, and then remove it from the cluster.

• Add Self IPs: Update F5SPKVlan custom resources to include Self IPs for new DPU nodes.

• Verify scaling: Use kubectl get daemonset f5-tmm to confirm that the number of TMM pods matches the number of DPU nodes.

Configuration Backup and Export

Back up BNK configuration on a regular schedule.

1. Export all F5 custom resources.

   # Export all F5 CRs to YAML files
   for crd in $(kubectl get crds | grep f5 | awk '{print $1}'); do
     resource=$(echo $crd | cut -d. -f1)
     kubectl get $resource -A -o yaml > backup_${resource}.yaml
   done
   
   # Export Gateway API CRs
   kubectl get gatewayclasses,gateways,httproutes,grpcroutes -A -o yaml > backup_gateway_api.yaml
   

2. Export Helm values.

   helm get values flo -n f5-cne-core > backup_flo_values.yaml
   

3. Export secrets and certificates.

   # Export FAR secret (base64-encoded)
   kubectl get secret far-pull-secret -n <NAMESPACE> -o yaml > backup_far_secret.yaml
   
   # Export TLS certificates
   kubectl get certificates -A -o yaml > backup_certificates.yaml
   

4. Store backups securely.

   Store your backups in a version-controlled or secure backup system.

Certificate Rotation

BNK uses cert-manager to manage TLS certificates for internal communication. cert-manager renews certificates before expiration.

# Check certificate status
kubectl get certificates -A

# View certificate details
kubectl describe certificate <CERT_NAME> -n <NAMESPACE>

# Force renewal if needed
kubectl cert-manager renew <CERT_NAME> -n <NAMESPACE>