Troubleshooting

This page provides an initial CLI-first troubleshooting workflow for F5 Insight. Use this guide to quickly identify platform, service, Kubernetes, networking, and certificate issues before contacting support.

Purpose and Scope

This troubleshooting guide currently focuses on:

• CLI-based diagnostics from the F5 Insight host
• System health and first-boot/service status
• Kubernetes workload and event checks
• Network and certificate validation commands

Quick Triage (2-5 Minutes)

Run this command set first to collect a fast health snapshot:

hostname
date
uptime
df -h
free -h
sudo systemctl --failed
sudo systemctl status f5insight-first-boot --no-pager
sudo kubectl get nodes -o wide
sudo kubectl get pods -A
sudo kubectl get events -A --sort-by=.metadata.creationTimestamp | tail -n 50

Interpretation:

• If system services are failed, continue with service and journal checks.
• If pods are not Running or repeatedly restarting, continue with Kubernetes diagnostics.
• If everything appears healthy but UI/API is unavailable, continue with network and certificate sections.

Relevant Logs

Primary startup log:

• /opt/f5insight/logs/f5insight-startup.log

Useful commands:

sudo tail -n 200 /opt/f5insight/logs/f5insight-startup.log
sudo tail -f /opt/f5insight/logs/f5insight-startup.log
sudo journalctl -u f5insight-first-boot -n 200 --no-pager
sudo journalctl -p err -n 200 --no-pager

Core Utilities and Their Purpose

• systemctl: Verify service state (active, failed, restarting)
• journalctl: Inspect detailed service failure reasons and system errors
• kubectl: Inspect cluster objects, pod health, events, and logs
• ip and ss: Validate interface addressing, routes, and listening ports
• curl: Validate endpoint reachability and TLS handshake behavior
• openssl: Validate certificate dates, issuer, SAN, and chain details
• jq: Parse JSON output from Kubernetes resources and secrets

Kubernetes Health Checks

Start with workload state and restarts:

sudo kubectl get pods -A -o wide
sudo kubectl get deploy,statefulset,daemonset -A
sudo kubectl get pods -A --field-selector=status.phase!=Running
sudo kubectl get events -A --sort-by=.metadata.creationTimestamp | tail -n 100

Investigate unhealthy workloads:

sudo kubectl describe pod <pod-name> -n <namespace>
sudo kubectl logs <pod-name> -n <namespace> --tail=200
sudo kubectl logs <pod-name> -n <namespace> --previous --tail=200

Monitor Backend Service Logs

For live backend troubleshooting, stream recent backend container logs from the deployment:

sudo kubectl logs -c backend -n f5-insight deploy/backend --since 5m -f

Additional backend checks:

sudo kubectl get deploy backend -n f5-insight
sudo kubectl describe deploy backend -n f5-insight
sudo kubectl get pods -n f5-insight -l app=backend -o wide

Resource pressure checks:

sudo kubectl top pods -A

Service-Specific Troubleshooting

Startup Failures

Symptoms:

• First boot does not complete
• Services remain in failed state

Checks:

sudo systemctl status f5insight-first-boot --no-pager
sudo journalctl -u f5insight-first-boot -n 300 --no-pager
sudo tail -n 300 /opt/f5insight/logs/f5insight-startup.log

UI/API Unreachable

Symptoms:

• Browser cannot load UI
• TLS or connection timeout errors

Checks:

ip -4 addr
ip route
ss -tulpn
curl -vk https://127.0.0.1

Backend Connection Errors (UI)

Symptoms:

• UI displays: “Unable to connect to F5 Insight backend”
• API requests fail with connection or backend unavailable errors

First response:

• Try a forced page refresh (Ctrl+Shift+R or Cmd+Shift+R)
• Wait 2-5 minutes for the backend to stabilize

Expected temporary unavailability:

• Node reboots: backend service restarts, may take 1-3 minutes
• Disaster Recovery (DR) setup: backend may be unavailable during failover
• Platform upgrades: backend restarts as part of the upgrade process
• Device certificate changes: ingress/controller may reload briefly

To confirm backend status from CLI:

sudo kubectl logs -c backend -n f5-insight deploy/backend --since 5m -f
sudo kubectl get pods -n f5-insight -l app=backend -o wide
sudo kubectl describe deploy backend -n f5-insight

If the backend pod is not Running or is in CrashLoopBackOff after 5+ minutes, report with backend logs and pod events.

Example: Database Connectivity Failure in Backend Logs

When the backend cannot connect to PostgreSQL, you will see repeated error messages like:

{"level":"warn","timestamp":"2026-03-11T15:09:15.196Z","caller":"db_events_processor/db_listener_impl.go:213","msg":"failed to connect to Postgres, retrying...","pid":1,"attempt":4,"error":"failed to connect to `user=postgres database=f5insight`:\n\t100.75.216.3:5432 (postgres): tls error: EOF\n\t100.75.216.3:5432 (postgres): failed to receive message: read tcp 100.76.0.15:36022->100.75.216.3:5432: read: connection reset by peer"}
{"level":"info","timestamp":"2026-03-11T15:09:15.196Z","caller":"db_events_processor/db_listener_impl.go:224","msg":"Waiting before next retry...","pid":1,"backoff_duration":7}

Key indicators:

• Repeated “failed to connect to Postgres” messages
• Connection reset or TLS errors to the database IP and port (typically 5432)
• Token renewal failures mentioning database connectivity

This is often temporary during database restarts or node maintenance. If errors persist beyond 5 minutes, verify database pod status:

sudo kubectl get statefulset f5-insight-postgres -n f5-insight
sudo kubectl get pods -n f5-insight | grep postgres
sudo kubectl logs -n f5-insight pod/f5-insight-postgres-0 -c postgres --tail=200
sudo kubectl describe pod f5-insight-postgres-0 -n f5-insight

Note

When a pod has multiple containers, specify the container name with -c <container-name>. If you omit -c, kubectl will use the first container or return an error if multiple containers exist.

Certificate/TLS Issues

Symptoms:

• Browser trust warnings
• Certificate name mismatch
• Unexpected issuer or expired certificate

Checks:

sudo kubectl -n kube-system get secret default-ingress-cert
sudo kubectl -n kube-system get secret default-ingress-cert-backup
sudo kubectl -n kube-system get secret default-ingress-cert -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

Adding LLM via UI

Symptoms:

• LLM connection test fails in the UI
• “Unable to connect” or “Invalid endpoint” error when saving LLM settings
• Field validation fails on URL input

Important:

When adding an LLM endpoint in the F5 Insight UI, you must specify the full URL including the protocol and port. Partial URLs or hostnames alone will fail.

Common LLM endpoint URLs:

Ollama (local or remote)

http://ollama.local:11434/v1
http://192.0.2.100:11434/v1
https://ollama.example.com:11434/v1

Azure OpenAI

https://<deployment-name>.openai.azure.com/openai/v1
https://my-org-openai.openai.azure.com/openai/v1

Generic OpenAI-compatible endpoints

https://api.openai.com/v1
https://api.example.com:8000/v1

Troubleshooting failed connections:

1. Verify the endpoint is reachable from the F5 Insight host:

   curl -v <endpoint-url>
   

2. Check that the port is open and the service is running:

   telnet <endpoint-host> <port>
   ss -tulpn | grep <port>
   

3. Verify the URL is complete with protocol and port (A trailing slash is optional and will not affect the configuration.)

4. If using authentication (API keys), ensure the credentials are entered in the appropriate field in the UI

5. Check backend logs for detailed error messages:

   sudo kubectl logs -c backend -n f5-insight deploy/backend --since 10m | grep -i llm
   

Emulating the Backend Validation Check

To manually verify that an LLM endpoint works the same way F5 Insight validates it, you can run these curl commands:

Direct endpoint call (simulates the backend’s outbound health-check request):

curl -X POST "<BASE_URL>/chat/completions" \
  -H "Authorization: Bearer <OPENAI_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"model":"<OPENAI_MODEL>","messages":[{"role":"user","content":"hi, just respond ping."}],"temperature":1.0}'

Example with Ollama endpoint:

curl -X POST "http://198.51.100.50:11434/v1/chat/completions" \
  -H "Authorization: Bearer foobar" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"hi, just respond ping."}],"temperature":1.0}'

Expected successful response:

{"id":"chatcmpl-199","object":"chat.completion","created":1773243818,"model":"gpt-4","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":"ping"},"finish_reason":"stop"}],"usage":{"prompt_tokens":35,"completion_tokens":2,"total_tokens":37}}

Placeholder values

Placeholder	Meaning
<BACKEND_HOST>	Example: localhost
<BACKEND_PORT>	Example: 8080
<ACCESS_TOKEN>	JWT access token from UI login
<OPENAI_API_KEY>	Your LLM provider API key or token
<OPENAI_MODEL>	Example: gpt-4 (for OpenAI), or your provider’s model ID
<BASE_URL>	The full base URL endpoint (for example, https://api.openai.com/v1 or http://ollama.local:11434/v1)

Certificate Chain Import and Order

When importing a CA-issued certificate bundle, order matters.

Required order in the PEM bundle:

1. Device/leaf certificate first
2. Intermediate CA certificate(s) second
3. Root CA certificate last only if explicitly required by your environment

Example bundle creation:

cat device.crt intermediate1.crt intermediate2.crt > /tmp/f5insight-device-chain.pem

Then import:

sudo f5insight-device-cert import-cert \
  --cert-file /tmp/f5insight-device-chain.pem

If replacing both certificate and key:

sudo f5insight-device-cert import-cert \
  --cert-file /tmp/f5insight-device-chain.pem \
  --cert-key /tmp/f5insight-device.key

Important:

• The import workflow validates the first certificate in the bundle against the private key.
• If an intermediate is placed first, import can fail with key mismatch.

Network and Connectivity Checks

Use these commands for routing, DNS, and endpoint validation:

Note

When modifying network settings, use only f5insight-netconfig. Avoid making persistent network changes with direct OS/network manager tools, as that can create configuration drift from the supported F5 Insight workflow.

ip -4 addr
ip route
getent hosts <fqdn>
curl -vk https://<endpoint>

Generating a Support Bundle

The support bundle script collects comprehensive diagnostic data including system info, Kubernetes state, pod configs, logs, and journal entries.

Basic usage:

sudo /opt/f5insight/scripts/support-bundle.sh

The bundle is saved as a compressed tarball in /tmp with a timestamped filename:

f5insight-bundle-f5-insight-YYYYMMDD-HHMMSS.tar.gz

Specify custom output location:

sudo /opt/f5insight/scripts/support-bundle.sh --output=/var/tmp

Monitor bundle generation:

sudo tail -f /opt/f5insight/logs/support-bundle.log

Include the support bundle tarball when escalating to support.

Data to Collect Before Contacting Support

Capture the following before opening a support case:

• Timestamp and timezone of failure
• Symptom summary and exact error text
• Output from Quick Triage commands
• Relevant systemd/journal and startup log excerpts
• kubectl get pods -A and recent cluster events
• Certificate details (subject, issuer, validity, SAN) when TLS is involved
• Support bundle output if collected