Stateless DAG

In F5 systems, Stateless DAG is a component responsible for distributing network traffic across multiple logical units. This page describes DAG modes for distribution between multiple Traffic Management Microkernel (TMM) pods.

Stateless DAG is a stateless mechanism for efficiently distributing traffic across TMM Pods. It eliminates the need for flow creation and packet proxying, making the solution inherently stateless and optimized for performance. This functionality is tightly integrated within the Container Network Extension (CNE) installation, reducing deployment complexity and providing improved traffic scalability.

The stateless DAG leverages a hash table-based design, where self IPs of TMM Pods are preconfigured and indexed per VLAN for dynamic traffic routing. The hash table contains entries for up to 1024 static slots, ensuring even distribution of packets across active pods. Based on a configurable hashing mode (SRC_ADDR or DEST_ADDR), incoming packets are hashed to determine the target TMM Pod for processing. Failover mechanisms are incorporated through fallback entries in the hash table, allowing traffic to be automatically redirected to other active pods during TMM outages.

Additionally, stateless DAG supports bidirectional traffic routing, ensuring consistent routing of client-to-server and server-to-client packets through the same TMM Pod, preserving packet flow symmetry and simplifying traffic management. This capability is crucial for stateful network functions like NAT, firewall, and load balancing, where symmetrical routing is required to maintain state integrity. By tightly integrating DAG functionality within the CNE framework and enabling stateless packet handling, stateless DAG improves efficiency, simplifies management, and provides a scalable solution for modern CNF deployments.

If you require DAG functionality for both Client-to-Server and Server-to-Client traffic, enable pod_hash for both client and server sides. This ensures the same TMM Pod handles both inbound and outbound traffic.

Note: When enabling pod_hash, ensure the TMM count matches with number of self IPs. For example, if there are three TMMs in your deployment, then ensure the VLANs with enabled pod_hash should have exactly three self IPs.

Packet Flow

In the stateless DAG implementation:

  1. Incoming packets are hashed (based on source IP or destination IP) to select the appropriate TMM Pod.

  2. A hash table with self IP and related entries is used to determine the active TMM Pod for traffic routing.

  3. Traffic received by the correct TMM Pod is processed with features such as NAT, firewall, and forwarded to its destination.

  4. The return traffic is routed symmetrically using bidirectional (optional) hashing configurations or directly to the client via DSR (Direct Source Routing).

Architecture

High-Level Design View

This architecture eliminates flow creation and streamlines traffic distribution to active TMM Pods through a stateless hash table. For every VLAN configured for DAG, a hash table of 1024 static entries ensures even traffic distribution, even during failures.

Single Directional DAG

Bidirectional DAG

Hash Table Operation

The hash table entries are distributed across 1024 static slots so that traffic is evenly allocated among active TMM Pods. Here’s an example illustration of the table structure:

Procedures

Apply the VLAN Configuration for Single Directional DAG

VLAN definitions require F5BigNetVlan to apply the VLAN attributes.

  1. Copy the following example into a tmm-client-vlan-config.yaml file.

    apiVersion: "k8s.f5net.com/v1"
kind: F5BigNetVlan
metadata:
  name: "clientside-vlan"
  namespace: "cnf-gateway"
spec:
  name: clientside
  interfaces:
    - "1.1"
  selfip_v4s:
    - 10.10.10.100
    - 10.10.10.101
  prefixlen_v4: 24
  selfip_v6s:
    - 2002::10:10:10:100
    - 2002::10:10:10:101
  prefixlen_v6: 116
  mtu: 9000
  cmp_hash: SRC_ADDR
  pod_hash: SRC_ADDR

    For the server-side VLAN, copy the following example into a tmm-server-vlan-config.yaml file:

    apiVersion: "k8s.f5net.com/v1"
kind: F5BigNetVlan
metadata:
  name: "serverside-vlan"
  namespace: "cnf-gateway"
spec:
  name: serverside
  interfaces:
    - "1.2"
  selfip_v4s:
    - 192.168.10.100
    - 192.168.10.101
  prefixlen_v4: 24
  selfip_v6s:
    - 2002::192:168:10:100
    - 2002::192:168:10:101
  prefixlen_v6: 116
  mtu: 9000
  cmp_hash: DST_ADDR

  2. Run the following command to apply the VLAN.

    kubectl apply -f tmm-client-vlan-config.yaml

    kubectl apply -f tmm-client-vlan-config.yaml

  3. Verify that the VLAN definitions are applied by checking the CR status.

    Sample Output:

    kubectl apply -f vlan.yaml 
f5bignetvlan.k8s.f5net.com/clientside-vlan created
f5bignetvlan.k8s.f5net.com/serverside-vlan created

Note: When pod_hash is configured on the client-side, ensure that there is route reachability to the client network from the TMM Pod. This is due to auto-lasthop is disabled by default, and return traffic needs routing to reach the client. Routing can be configured either dynamically through BGP or statically. Following is an example of a static route configuration.

apiVersion: "k8s.f5net.com/v1"
kind: F5BigNetStaticroute
metadata:  
  name: "staticroute"  
spec:  
  destination: 11.11.11.100  
  prefixLen: 24  
  type: gateway  
  gateway: 20.1.1.131

For more information on VLAN definitions, see F5BigNetVlan CRD page.

Apply the VLAN Configuration for Bidirectional DAG

VLAN definitions require F5BigNetVlan to apply the VLAN attributes.

  1. Copy the following example into a tmm-client-vlan-config.yaml file.

    apiVersion: "k8s.f5net.com/v1"
kind: F5BigNetVlan
metadata:
  name: "clientside-vlan"
  namespace: "cnf-gateway"
spec:
  name: clientside
  interfaces:
    - "1.1"
  selfip_v4s:
    - 10.10.10.100
    - 10.10.10.101
  prefixlen_v4: 24
  selfip_v6s:
    - 2002::10:10:10:100
    - 2002::10:10:10:101
  prefixlen_v6: 116
  mtu: 9000
  cmp_hash: SRC_ADDR
  pod_hash: SRC_ADDR

    For the server-side VLAN, copy the following example into a tmm-server-vlan-config.yaml file:

    apiVersion: "k8s.f5net.com/v1"
kind: F5BigNetVlan
metadata:
  name: "serverside-vlan"
  namespace: "cnf-gateway"
spec:
  name: serverside
  interfaces:
    - "1.2"
  selfip_v4s:
    - 192.168.10.100
    - 192.168.10.101
  prefixlen_v4: 24
  selfip_v6s:
    - 2002::192:168:10:100
    - 2002::192:168:10:101
  prefixlen_v6: 116
  mtu: 9000
  cmp_hash: DST_ADDR
  pod_hash: DST_ADDR

  2. Run the following command to apply the VLAN.

    kubectl apply -f tmm-client-vlan-config.yaml

    kubectl apply -f tmm-client-vlan-config.yaml

  3. Verify that the VLAN definitions are applied by checking the CR status.

    Sample Output:

    kubectl apply -f vlan.yaml 
f5bignetvlan.k8s.f5net.com/clientside-vlan created
f5bignetvlan.k8s.f5net.com/serverside-vlan created

Note: For the bidirectional DAG, NAT or SNAT automap will not be used. In this scenario, ensure route reachability is configured for both the client and server networks from the TMM Pods.

For more information on VLAN definitions, see F5BigNetVlan CRD page.

DAG statistics

If the TMM Debug sidecar is enabled (default), use the following steps to verify DPI statistics.

  1. Log in to the TMM debug Pod.

    kubectl exec -it f5-tmm -n default -c debug

  2. Verify the DPI statistics.

    tmctl -d blade -w 200 pod_dag_stats

    Sample Output:

    rx_total rx_psudo_dev rx_l2_error rx_l2_tag_error rx_l2_not_dagifc rx_l2_dest_bypass rx_remote_pod rx_non_ip rx_l3_error rx_l3_dest_bypass rx_nat_ip_bypass rx_pick_error
-------- ------------ ----------- --------------- ---------------- ----------------- ------------- --------- ----------- ----------------- ---------------- -------------
  32064            0           0               0              108                65            18     29663           0              2210                0             0

rx_hw_tag rx_sw_tag rx_no_tag tx_nexthop_null tx_self_pod tx_remote_pod tx_fail tx_hw_tag tx_sw_tag tx_no_tag pick_primary pick_fallback mac_hash_collision interfaces
--------- --------- --------- --------------- ----------- ------------- ------- --------- --------- --------- ------------ ------------- ------------------ ----------
    32058         0         6               0           0             0       0         0         0         0            0             0                  0          1

interface_ref vlans pods_resolved pods_unresolved
------------- ----- ------------- ---------------
            1     1             4               1

Stats Supported

The following table lists the supported Stats for DAG CNFs:

Reception Stats

Stat Name Description
total Specifies the total number of packets received for pod dagging.
psudo_dev Specifies tap interface device.
ifc_null Specifies packets with ifc null.
l2_error Specifies the L2 header validation failures.
host Specifies host-marked packets (local processing).
remote_pod Specifies packets received from peer TMM.
l2_dest_bypass Specifies destination MAC addresses not in the DAG table.
non_ip Specifies non-IPv4 or non-IPv6 packets.
l3_error Specifies the L3 key extraction failures.
l3_dest_bypass Specifies loopback, multicast, or self-IP destinations.
pick_error Specifies TMM selection failures.

Transmission Stats

Stat Name Description
nexthop_null Specifies nexthops that are not resolved.
nexthop_err Specifies nexthop resolution failures.
self_pod Specifies the selected local TMM.
remote_pod Specifies packets forwarded to peer TMM.
fail Specifies failures in sending redirected packets.

Selection Stats

Stat Name Description
ifc_ignore Specifies interfaces that are not found/disabled.
primary Specifies the primary pod selected.
fallback Specifies the fallback pod selected.

Hash Efficiency Stats to Track Bucket Collisions

Stat Name Description
mac_hash_collision Number of MAC hash collisions.
ifc_hash_collision Number of interface hash collisions.
vlans Specifies the number of VLANs configured for dagging.

Pods Resolution Stats

Stat Name Description
pods_resolved Number of successful MAC address resolutions.
pods_unresolved Number of unsuccessful MAC address resolutions.