Troubleshoot IPsec tunnels

This section helps diagnose and resolve common Internet Protocol Security (IPsec) tunnel issues on CNF.

Check tunnel status

Start by verifying whether the IPsec tunnel is established.

  1. List all active Security Associations (SAs):

    kubectl exec -it <ipsec-pod-name> -- ipsecctl --list-sas
    
  2. If the tunnel is active, you see the Internet Key Exchange (IKE) SA and child (IPsec) SA with status information. If no SAs appear, the tunnel has not been established.

  3. For detailed information including proposal details, and lifetimes:

    kubectl exec -it <ipsec-pod-name> -- ipsecctl --list-sas --details
    

Tunnel does not establish

Symptoms: No SAs appear when you run ipsecctl –list-sas. No encrypted traffic passes between peers.

Possible causes and solutions:

Cause Solution
Mismatched pre-shared keys Verify that the presharedKey value in both the local and remote sections of the F5BigIkePeer CR matches the key configured on the remote peer
Mismatched IKE proposals Verify that the integrity, encryption, prf, and dhGroup values in the proposals section are supported by both peers
Incorrect peer addresses Verify that localAddress and remoteAddress are correct and reachable from the CNF cluster
Firewall blocking IKE traffic Ensure that UDP ports 500 and 4500 are open between the peers
Mismatched IPsec policy proposals Verify that the integrity, encryption, and dhGroup values in the ipsecPolicies proposals match the remote peer's configuration

Tunnel establishes but no traffic flows

Symptoms: SAs appear as active, but data-plane traffic is not encrypted or does not reach the remote network.

Possible causes and solutions:

Cause Solution
Mismatched traffic selectors Verify that the sourceAddress and destinationAddress in the trafficSelectors section match the subnets configured on the remote peer. The local source must match the remote destination, and vice versa
Routing issues Verify that the CNF cluster has routes to the remote subnets through the tunnel endpoint

Traffic flow works and then stops suddenly

Symptoms: IPsec tunnel was established and traffic was flowing, but data-plane traffic stops unexpectedly without any configuration changes.

Check for TMM pod restarts

A TMM pod restart causes the loss of all IPsec SAs owned by that pod. Verify whether any TMM pods have restarted since the tunnel was established:

kubectl get pods -n <namespace> | grep tmm

Check the RESTARTS column and the AGE column. If a TMM pod restarted after the tunnel was created, that pod has lost its IPsec SAs and traffic flowing through it will fail.

To see the exact restart time:

kubectl describe pod <tmm-pod-name> -n <namespace> | grep -A 5 "Last State"

Compare the restart time against the tunnel creation time. If the TMM pod restarted after the tunnel was established, this is the likely cause.

Verify SAs on both endpoints

Check whether IKE and IPsec SAs are still active on the local CNF:

kubectl exec -it <ipsec-pod-name> -n <namespace> -- swanctl --list-sas --details

Also verify the SA state on the remote peer using the equivalent command for your remote peer’s IPsec implementation.

If SAs are missing on the local side but still present on the remote side, the remote peer has not yet detected the failure. Wait for the Dead Peer Detection (DPD) interval to expire, which triggers the remote peer to renegotiate.

Restore traffic

If SAs do not recover automatically after the DPD interval:

  1. Manually initiate renegotiation from the remote peer.

  2. If the issue persists, delete and reapply the F5BigIkePeer CR to force a fresh tunnel establishment:

    kubectl delete f5bigikepeer <cr-name> -n <namespace>
    kubectl apply -f ike-peer.yaml -n <namespace>
    

Note

IPsec pod restarts preserve tunnel state through DSSM and recover automatically. TMM pod restarts, however, cause SA loss and require the remote peer to renegotiate. If TMM pod restarts are frequent, investigate the root cause of the TMM instability separately.

IPsec pod restart disrupts traffic

Symptoms: After a IPsec pod restart, tunnels do not recover automatically.

Expected behavior: BIG-IP Next CNF persists IKE session state to the Distributed Session State Manager (DSSM). When the IPsec pod restarts, it rebuilds all sessions from DSSM without renegotiating with peers. Traffic should remain uninterrupted.

If tunnels do not recover:

  1. Check the IPsec pod logs for error messages:

kubectl logs <IPsec-pod-name>
  1. Verify that DSSM is healthy and accessible from the IPsec pod.

TMM pod restart/down disrupts traffic

Symptoms: After a TMM pod restart, existing IPsec tunnels served by that TMM pod are disrupted. Data-plane traffic stops flowing through the affected tunnels.

Expected behavior: Unlike an IPsec pod restart, a TMM pod restart causes the loss of IPsec Security Associations (SAs) and associated transactional history owned by that TMM pod. The TMM pod does not recover SAs from the Distributed Session State Manager (DSSM).

In most cases, the IPsec pod activates Dead Peer Detection (DPD) and cleans up the affected tunnels after the DPD timeout expires. The IPsec pod then renegotiates new IKE and IPsec SAs automatically. In some cases, the remote peer must initiate renegotiation to restore the tunnels.

To restore traffic after a TMM pod restart:

  1. Verify that the TMM pod has restarted and is in a Running state:

    kubectl get pods -n <namespace> | grep tmm
    
  2. Wait for the DPD timeout to expire. The IPsec pod detects the lost SAs and initiates tunnel cleanup and renegotiation automatically.

  3. Verify that new SAs have been established:

    kubectl exec -it <ipsec-pod-name> -n <namespace> -- swanctl --list-sas
    
  4. If the SAs do not reappear after the DPD timeout, the remote peer may need to initiate renegotiation. Coordinate with the remote peer administrator to restart the IKE negotiation from their side.

Note

IPsec pod restarts preserve tunnel state through DSSM and do not require renegotiation. TMM pod restarts, however, cause SA and transactional history loss on the affected TMM pod. In most cases, the IPsec pod’s DPD mechanism handles cleanup and renegotiation automatically. Plan maintenance windows accordingly when TMM pod restarts are expected.

TCP traffic is reset in multi-pod deployments

Symptoms: IPsec tunnel is established and SAs are active, but TCP connections are reset. UDP traffic may work correctly. The issue occurs only in multi-pod deployments with ECMP routing.

Possible cause: ECMP is routing packets from the same TCP flow to different TMM pods. The TMM pod that receives the SYN-ACK does not have the connection state from the original SYN and sends a TCP RST.

Solution: Configure a FastL4 profile with looseInitiation: true and looseClose: true, and attach it to a Secure Context with no SNAT. See Configure IPsec for multi-pod ECMP deployments for details.

To verify this is the cause, check for TCP resets in the TMM statistics:

tmctl ipsec_data_stat

View TMM statistics

Run the following tmctl commands on a TMM pod to view IPsec-related statistics:

Command What it shows
tmctl ipsec_policy_stat View IPsec policy statistics
tmctl ipsec_data_stat View IPsec data traffic statistics
tmctl ipsec_esp_stat View Encapsulating Security Payload (ESP) statistics
tmctl ike_peer_stat View IKE peer statistics
tmctl pfkey_ike_msg_stat View PF_KEY message statistics (in, out, and drop counts)

View IPsec statistics

Run the following commands on the IPsec pod:

Command What it shows
kubectl exec -it <ipsec-pod-name> -- ipsecctl --stats IKEv2 daemon-level statistics
kubectl exec -it <ipsec-pod-name> -- ipsecctl --list-sas All active IKE and IPsec SAs
kubectl exec -it <ipsec-pod-name> -- ipsecctl --list-conns List all the currently loaded IPsec configurations
kubectl exec -it <ipsec-pod-name> -- ipsecctl --list-sas --details Detailed SA data, including lifetimes, and negotiated proposals
kubectl exec -it <ipsec-pod-name> -- ipsecctl --list-sas <connection_name> SAs for a specific connection