F5SPKEgress

Note

Deprecated as of 2.4.0. Use a GatewaySettings resource instead.

Overview

Function

Declares the egress configuration for a CNE instance, defining source network address translation, DNS/NAT46 and NAT64 translation, and egress traffic handling across TMM instances.

Names

Singular name	Plural name	Short name
f5-spk-egress	f5-spk-egresses	egprof

API group and version

k8s.f5net.com/v3

Scope

Namespace

Specification

metadata

Parameter	Description
name	The unique name identifying the resource. For example, egress-cr
namespace	The namespace of Kubernetes in which the policy is applied.

spec

Field	Required	Default	Description
snatType string	No	SRC_TRANS_AUTOMAP	SNAT type to apply. Accepted values: SRC_TRANS_NONE, SRC_TRANS_SNATPOOL, SRC_TRANS_AUTOMAP. When set to SRC_TRANS_SNATPOOL, you must also set egressSnatpool.
egressSnatpool string	No	—	Name of the SNAT pool. Set this when snatType is SRC_TRANS_SNATPOOL.
egressSnatpoolMode string	No	UNIQUE_SNAT_ADDRESS	SNAT address mode used by TMM pods. UNIQUE_SNAT_ADDRESS assigns each TMM a unique IP address. SHARED_SNAT_ADDRESS_FLOW_FORWARD shares a single IP address across all TMMs. Accepted values: UNIQUE_SNAT_ADDRESS, SHARED_SNAT_ADDRESS_FLOW_FORWARD.
egressSnatpoolFlowForwardVlan string	No	""	VLAN used to forward traffic between TMM pods. Required when egressSnatpoolMode is SHARED_SNAT_ADDRESS_FLOW_FORWARD.
egressSnatpoolProtectionEnabled boolean	No	false	Turns on extra protection for traffic between TMM pods. Applies only when egressSnatpoolMode is SHARED_SNAT_ADDRESS_FLOW_FORWARD.
dualStackEnabled boolean	No	false	Turns on creation of both IPv4 and IPv6 wildcard virtual servers for egress connections.
dnsEnabled boolean	No	false	Turns DNS functionality on or off. When dnsNat46Enabled or nat64Enabled is true, DNS functionality is on even when dnsEnabled is false.
dnsNat46Enabled boolean	No	false	Turns on the DNS46/NAT46 feature.
dnsNat46PoolIps array	No	—	Pool of IP addresses that represent external DNS servers, or gateways to reach the DNS servers.
dnsNat46Ipv4Subnet string	No	10.2.2.0/24	Pool of private IPv4 addresses used to create DNS A records for the internal pods. CIDR format.
dnsNat46SorryIp string	No	192.168.1.1	IP address for the sorry page when the NAT pool is exhausted. IPv4 format.
nat64Enabled boolean	No	false	Turns on DNS64/NAT64 translation for egress connections.
nat64Ipv6Subnet string	No	64:ff9b::/96	IPv6 subnet used for NAT64 translation. CIDR format.
maxDNS46TTL integer	No	120	Ceiling for DNS46 translation TTL, in seconds. AAAA TTLs greater than this value are set to this value. Accepted values: 1 to 2147483647.
debugLogEnabled boolean	No	false	Turns on debug logging for DNS46 translation.
dnsCacheName string	No	""	Name of the required F5SPKDnscache resource. Concatenate the resource’s metadata.namespace and metadata.name with a hyphen, for example dnscache-cr.
maxTmmReplicas integer	No	1	Maximum number of operational TMM pods. This should equal the number of Self IP addresses.
maxReservedStaticIps integer	No	0	Maximum number of IPv4 addresses reserved from dnsNat46Ipv4Subnet for static IPv4-to-IPv6 mapping. Reserved addresses come from the start of the CIDR block. The system distributes the remaining addresses across TMM pods based on maxTmmReplicas.
dnsRateLimit integer	No	0	Rate limit for the DNS virtual server, per pod. To turn off the rate limit, set this to 0. Accepted values: 0 to 4294967295.
vlans object	No	—	VLAN settings for egress traffic.
pseudoCNIConfig object	No	—	Pseudo-CNI settings for the egress configuration.
pvaAccelerationMode string	No	full/assisted	Preferred acceleration mode for the Packet Velocity ASIC (PVA). full/assisted applies partial PVA acceleration when possible, and idle flows can move to software. Accepted values: full/assisted, disabled.
firewallEnforcedPolicy string	No	""	Name of the security firewall enforced policy. Maximum length is 255 characters.
logProfile string	No	""	Name of the security log profile. Maximum length is 255 characters.

spec.vlans

Field	Required	Default	Description
vlanList array	No	[]	List of VLAN names.
disableListedVlans boolean	No	true	When true, the listed VLANs are turned off. When false, only the listed VLANs are turned on.
category string	No	""	VLAN category. The reserved words are internal and external.

spec.pseudoCNIConfig

Field	Required	Default	Description
namespaces array	No	—	List of application pod namespaces to which the egress configuration applies.
appPodInterface string	No	—	Application pod’s network interface name from which egress traffic originates, for example eth0.
appNodeInterface string	No	—	Network interface name on the worker node. Egress traffic is directed to the TMM through this interface on the node where the application pod is deployed.
secureSPKIngressPort integer	No	—	Port number that the ingress virtual server listens on. Required when you configure pseudo-CNI and snatType is SRC_TRANS_NONE. Accepted values: 1025 to 65535.
vxlan object	No	—	VXLAN settings for the pseudo-CNI configuration.

spec.pseudoCNIConfig.vxlan

Field	Required	Default	Description
create boolean	No	false	When true, the system creates a VXLAN. When false, add a VLAN or VXLAN already created on the TMM and worker nodes to vlanList.
tmmInterfaceName string	No	—	TMM interface name on which the VXLAN interface is created. This is the metadata.name from the corresponding VLAN resource, for example internal.
nodeInterfaceName string	No	—	Network interface name on the worker node on which the VXLAN interface is created, for example enP1p1s0f1v0.
mtu integer	No	1430	Maximum transmission unit for the tunnel. Accepted values: 68 to 65515.
port integer	No	4789	UDP port number for VXLAN traffic. Accepted values: 0 to 65535.
key integer	No	0	VNI used for the tunnel. When set to 0, the system generates and assigns a unique VNI. When set to a valid VNI, it must be unique across all egress resources. Accepted values: 0 to 16777215.
ipv4Subnet string	No	—	IPv4 subnet from which addresses are assigned to VXLAN tunnel endpoints in the TMM and on worker nodes. Must be unique across all egress resources, for example 192.168.0.0.
ipv4PrefixLen integer	No	—	Prefix length for the IPv4 subnet. Accepted values: 8 to 18.
ipv6Subnet string	No	—	IPv6 subnet from which addresses are assigned to VXLAN tunnel endpoints in the TMM and on worker nodes. Must be unique across all egress resources, for example fd50::.
ipv6PrefixLen integer	No	—	Prefix length for the IPv6 subnet. Accepted values: 104 to 114.
vrf string	No	""	Name of the VRF (tenant) this VXLAN belongs to. When you don’t set this, the VXLAN is created in the default route domain. This field is immutable after creation.

CR Modifications

Because this resource references a number of additional resources, F5 recommends that you always delete and reapply the resource, rather than using kubectl apply to modify a running resource’s configuration.

Note

This resource is immutable. It is recommended to delete and reapply the resource if you need to make modifications.

Requirements

Make sure you have configured a F5SPKVlan resource.

Egress SNAT

SNATs are used to modify the source IP address of egress packets leaving the cluster. When the BIG-IP Next for Kubernetes Traffic Management Microkernel (TMM) receives an internal packet from an internal Pod, the external (egress) packet source IP address will translate using a configured SNAT IP address. Using the this resource, you can apply SNAT IP addresses using either SNAT pools, or SNAT automap.

snatType

There are three types of SNAT:

Note

Currently, this resource use case with SNAT none type is not supported for GA.

SNAT none

When snatType is configured with SRC_TRANS_NONE, TMM uses application pod’s IP address to communicate with an external server.

SNAT pools

When snatType is configured with SRC_TRANS_SNATPOOL, TMM uses the configured MBIP server-side SNAT pools to communicate with an external server.

SNAT pools are lists of routable IP addresses, used by TMM to translate the source IP address of egress packets. SNAT pools provide a greater number of available IP addresses, and offer more flexibility for defining the SNAT IP addresses used for translation. If snatType is configured with the SRC_TRANS_SNATPOOL type, egressSnatpool will be used as a pool of addresses for source address translation for traffic that matches this F5SPKEgress CR. For more information to enable SNAT pools, see F5SPKSnatpool.

SNAT automap

When snatType is configured with SRC_TRANS_AUTOMAP, TMM uses the configured MBIP server side self IPs to communicate with an external server.

SNAT automap uses TMM’s external F5SPKVlan IP address as the source IP for egress packets. SNAT automap is easier to implement, and conserves IP address allocations. To use SNAT automap, do not include spec.egressSnatpool parameter. Use the installation procedure below to enable egress connectivity using SNAT automap.

Note

• In clusters with multiple BIG-IP Next for Kubernetes Controller instances, ensure the IP addresses defined in each F5SPKSnatpool CR do not overlap.

• Configuration dependencies based on snatType:

  • SRC_TRANS_SNATPOOL requires egressSnatpool; pseudoCNIConfig.secureSPKIngressPort must be omitted.

  • SRC_TRANS_NONE requires pseudoCNIConfig.secureSPKIngressPort; egressSnatpool must be omitted.

  • SRC_TRANS_AUTOMAP requires neither field; both must be omitted.

Configure SNAT Automap

Follow the instructions below to set up the SNAT automap configuration:

1. Copy the below example manifest to a YAML file, then set the namespace parameter to the Controller’s Project and save.

   apiVersion: "k8s.f5net.com/v3"
   kind: F5SPKEgress
   metadata:
     name: egress-cr
     namespace: f5-bnk
   spec:
     dualStackEnabled: <true|false>
     snatType: SRC_TRANS_AUTOMAP
   

2. Apply the manifest.

   kubectl apply -f <file name>
   

   In this example, the resource file is named egress-cr.yaml:

   kubectl apply -f egress-cr.yaml
   

   Sample Output:

   f5spkegress.k8s.f5net.com/egress-cr created
   

Internal Pods can now connect to external resources using the external F5SPKVlan self IP address.

Connection Statistics

Connect to the Debug Sidecar in the BIG-IP Next for Kubernetes to view the traffic processing statistics.

1. Log in to the BIG-IP Next for Kubernetes Debug container.

   kubectl exec -it f5-tmm-wgrvm -c debug -n f5-bnk
   

2. View the traffic processing statistics.

   tmctl -d blade virtual_server_stat -s name,serverside.tot_conns
   

   In this example, 3 IPv4 connections, and 2 IPv6 connections have been initiated by internal Pods:

   name                              serverside.tot_conns
   -----------------                 --------------------
   f5-bnk-egress-crd-egress-ipv6                2
   f5-bnk-egress-crd-egress-ipv4                3
   

Shared SNAT Address and Security Protection

In the default SNAT pool mode, each TMM pod is assigned a unique SNAT address. However, to optimize resource usage and minimize the number of SNAT addresses in large-scale deployments, TMM can be configured to share an SNAT address. Hence, in shared SNAT address mode, multiple TMM pods can use the same set of SNAT addresses to handle egress traffic. In this scenario, there may be instances where response traffic from the router is directed to an alternate TMM pod than the one that originally handled the outgoing traffic. To address this situation, the alternate TMM pod forwards the valid response traffic to original TMM pod so it can be processed properly.

Note

Response Traffic Handling: Only traffic that is valid and belongs to an active TMM pod, or matches an existing outgoing connection, will be forwarded and processed. It is important to follow these rules to ensure smooth traffic management.

Shared SNAT Address with Flow-Forwarding

Flow-Forwarding Architecture: Egress Traffic

Note

ACL and Flow-Forwarding Compatibility: Access Control List (ACL) and Flow-Forwarding cannot be applied simultaneously to the same IP address. In the case of Flow-Forwarding, IP address specifically refers to the Source Network Address Translation (SNAT) address. This means that if Flow-Forwarding is enabled, ACL configurations for the SNAT address are not applicable.

Flow-Forwarding Architecture: Egress Traffic

The above diagram describes how egress traffic from a (Cloud-Native Function) CNF Node is sent to an External Client through TMM with Shared SNAT applied.

Egress Traffic Flow Description:

1. Initiating Egress Traffic:

   • The CNF Node (X:x) initiates egress traffic, sending packets to the external client (Y:y) via the TMM.

2. Applying SNAT (Source Network Address Translation):

   • To send packets to the external client (Y:y), the TMM changes the CNF node’s (X:x) source address to the egress SNAT address (S:s).

   • The packet is then forwarded from the TMM to the router with the SNAT address (S:s).

3. Routing the egress traffic to the External Client:

   • The router forwards the packet to the external client (Y:y).

4. Return Traffic from the External Client:

   • The external client (Y:y) sends a response packet back to the router.

   • Since the SNAT address (S:s) is shared across all TMMs, the router may return the packet to any available TMM (e.g., TMM 1), without knowing which specific TMM handled the initial outgoing packet.

5. Identifying the Original TMM:

   • Upon receiving the packet, the TMM checks which TMM initiated the original packet flow.

   • The packet is then forwarded to the correct TMM (e.g., TMM 2) that originally handled the outgoing traffic.

6. Returning the Packet to the CNF Node:

   • The identified TMM (e.g., TMM 2) sends the packet back to the originating CNF node (X:x), completing the egress traffic flow.

Configuring Shared SNAT Address with Flow-Forwarding

Follow the instructions below to set up the shared SNAT address configuration using the F5SPKEgress and F5SPKSnatpool CRs:

1. Copy the below example F5SPKEgress CR to a YAML file, then modify shared SNAT Address settings as required and save.

   apiVersion: "k8s.f5net.com/v3"
   kind: F5SPKEgress
   metadata:
     name: egress-cr
     namespace: f5-bnk
   spec:
     dualStackEnabled: true
     snatType: SRC_TRANS_SNATPOOL
     egressSnatpool: egress-snatpool
     egressSnatpoolMode: SHARED_SNAT_ADDRESS_FLOW_FORWARD
     egressSnatpoolFlowForwardVlan: vlan-external
     vlans:
       vlanList:
         - vlan-internal
       disableListedVlans: false
     pseudoCNIConfig:
       namespaces:
         - app-ns-1
       appPodInterface: "eth0"
       appNodeInterface: "spk-ms.3001"
   

   Note

   The same egressSnatpool should not be configured across multiple F5SPKEgress CRs.

2. Apply the F5SPKEgress CR that you have created.

   kubectl apply -f egress-cr.yaml
   

   Sample Output

   f5spkegress.k8s.f5net.com/egress-cr created
   

3. Copy the below F5SPKSnatpool CR to a YAML file, then modify shared SNAT Address settings as required and save.

   apiVersion: "k8s.f5net.com/v1"
   kind: F5SPKSnatpool
   metadata:
      name: egress-snatpool
      namespace: f5-bnk
   spec:
      name: egress-snatpool
      sharedSnatAddressEnabled: true
      addressList:
       - - 44.44.44.4
         - 2002::44:44:44:4
   

4. Apply the F5SPKSnatpool CR that you have created.

   kubectl apply -f snatpool-cr.yaml
   

   Sample Output

   f5spksnatpool.k8s.f5net.com/egress-snatpool created
   

Configuring Shared SNAT Address with Security Protection

This section provides the configuration of shared SNAT address with security protection.

Security protection prevents packet forwarding of likely invalid flows. This can help reduce security risk and improve overall performance.

Follow the instructions below to set up the shared SNAT address with security protection using the F5SPKEgress CR:

1. Copy the below example F5SPKEgress CR to a YAML file, then modify security protection settings as required and save.

apiVersion: "k8s.f5net.com/v3"
kind: F5SPKEgress
metadata:
  name: egress-cr
  namespace: f5-bnk
spec:
  dualStackEnabled: true
  snatType: SRC_TRANS_SNATPOOL
  egressSnatpool: egress-snatpool
  egressSnatpoolMode: SHARED_SNAT_ADDRESS_FLOW_FORWARD
  egressSnatpoolFlowForwardVlan: vlan-external
  egressSnatpoolProtectionEnabled: true
  vlans:
    vlanList:
      - vlan-internal
    disableListedVlans: false
  pseudoCNIConfig:
    namespaces:
      - app-ns-1
    appPodInterface: "eth0"
    appNodeInterface: "spk-ms.3001"

2. Apply the F5SPKEgress CR that you have created.

kubectl apply -f egress-cr.yaml

Sample Output:

f5spkegress.k8s.f5net.com/egress-cr created

Connection Statistics

Connect to the Debug Sidecar in the BIG-IP Next for Kubernetes TMM to view the security protection stats.

1. Log in to the BIG-IP Next for Kubernetes Debug container.

   kubectl exec -it f5-tmm-6cdbc6bb65-j2r7d -c debug -n f5-bnk -- bash
   

2. View the security protection stats.

    tmctl -d blade profile_sharedscript_stat -P
   

   Sample Output:

    Name                             Value
    ---------------------------      --------
    cur_deny_listener                0
    tot_alternate_drop               0
    tot_origin_drop                  0
    tot_alternate_invalid_pkt        0
    tot_deny_listener                0
   

PseudoCNI

The Whole Cluster BIG-IP Next for Kubernetes utilises the F5SPKEgress CR to incorporate PseudoCNI configuration, which defines the necessary setting for egress traffic from application pods through TMM pods. The PseudoCNI Controller reads the PseudoCNI configuration from the F5SPKEgress CR and generates the necessary settings to designate TMMs as egress gateways. These mappings are shared with CSRC over GRPC. The CSRC daemonSet, running in privileged mode, configures rules and routes on nodes hosting application pods. The TMM running on the DPU local to the host node is given the highest priority compared to other TMMs.

Note

To ensure proper network communication and management, you must update the Felix configurations to include the TMM internal interface IPv4 Self-IPs. Use the kubectl patch felixconfiguration default --type='merge' -p='{"spec": {"externalNodesList": ["self-ip-address1", "self-ip-address2", "self-ip-address3"]}}' command to update the Felix configuration.

Configuring PseudoCNI

Prerequisites

Ensure you have:

• A Linux-based workstation.

• Installed F5SPKVlan.

Follow the instructions below to set up the PseudoCNI configuration using the F5SPKEgress CR:

1. Copy the below example F5SPKEgress CR into a YAML file, then modify PseudoCNI settings, snatpool and External VLAN Interface Name as required and save.

   apiVersion: "k8s.f5net.com/v3"
   kind: F5SPKEgress
   metadata:
     name: egress-cr
     namespace: f5-bnk
   spec:
     dualStackEnabled: true
     snatType: SRC_TRANS_SNATPOOL
     egressSnatpool: egress-snatpool
     vlans:
       disableListedVlans: false
       vlanList:
         - vlan-external
     pseudoCNIConfig:
       namespaces:
         - app-ns-1
       appPodInterface: "eth0"
       appNodeInterface: "spk-ms.3001"
   

2. Apply the F5SPKEgress CR that you have created.

   kubectl apply -f egress-cr.yaml
   

Note

1. Using only VLAN in the egress CR (without VXLAN) is supported.
2. Make sure to configure one egress CR per namespace and one VLAN per egress CR. A single VLAN cannot be referred to in multiple egress CRs.

Configuring Dynamic Port Selection for CSRC-CNE Controller Communication

Port Conflict Issue from CSRC Logs:

ERROR: Failed to listen: listen tcp :8751: bind: address already in use

Cause:

As the CSRC daemonSet runs in the host network namespace, the CSRC gRPC port (default: 8751) may conflict with other applications using the same port in the host network.

FIX:

If a port conflict occurs during or after BIG-IP Next for Kubernetes installation, perform the following steps to fix this issue:

1. Update the Service Resource.

   kubectl edit svc f5-csrc-grpc-svc
   

   In the csrc-grpc_service.yaml file, modify:

   ports:
     - name: f5-csrc-grpc
       port: 8751       # Change to a new port (e.g., 8875)
       protocol: TCP
       targetPort: 8751 # Update to match the new port (e.g., 8875)
   

2. Update the DaemonSet Configuration.

   kubectl edit daemonset spk-csrc
   

   In the daemonset.yaml file, modify:

   - name: PSEUDOCNI_GRPC_PORT
     value: 8751     # Change to the new port (e.g., 8875)
   

The CSRC pods will automatically restart with the new port configuration, resolving the conflict.

VXLAN with F5SPKEgress CR

The setup process for the F5SPKVXLAN CR is automated to minimize the need for manual input and prevent errors. By using the specified tmmInterfaceName, the node interface is automatically identified and obtains the required underlay network information. It then chooses a unique Virtual Network Identifier (VNI) and subnet, and associates the VXLAN with the F5SPKEgress CR. These improvements simplify the configuration process, improve efficiency, and ensure accurate configurations with minimal user intervention.

Configuring VXLAN with F5SPKEgress CR

Prerequisites:

• The underlying F5SPKVlan CR, referred to in the F5SPKEgress CR as tmmInterfaceName, must be created before creating the F5SPKEgress CR.

• If pseudoCNIConfig is configured and vxlan.create is set to true.

  • vlans.vlanList should be empty.

  • pseudoCNIConfig.appNodeInterface should be empty.

• If pseudoCNIConfig is configured and vxlan.create is set to false.

  • vlans.vlanList should be populated, containing only one element.

• If vxlan.nodeInterfaceName is configured, the specified interface will be used to create the VXLAN interface. If vxlan.nodeInterfaceName is not configured or set to default, the appropriate interface on the node is automatically detected. This detected interface will be the one that can connect to the VLAN interface specified by tmmInterfaceName in the TMM.

• When you configure a subnet for a VXLAN interface manually, ensure that unique VXLAN subnets are configured for all F5SPKEgress (VXLAN) CRs.

• When you configure a VNI manually, ensure that the VNI is configured for all F5SPKEgress (VXLAN) CRs.

Follow the instructions below to set up the VXLAN configuration using the F5SPKEgress CR with default and specific configuration values:

1. Using default values

   Copy the below example F5SPKEgress CR with default values into a YAML file, then modify VXLAN settings as required and save.

   apiVersion: "k8s.f5net.com/v3"
   kind: F5SPKEgress
   metadata:
     name: egress-cr
     namespace: f5-bnk
   spec:
    dualStackEnabled: true
    snatType: SRC_TRANS_SNATPOOL
    egressSnatpool: egress-snatpool
    pseudoCNIConfig:
       namespaces:
         - app-ns-1
       appPodInterface: "eth0"
       vxlan:
         create: true
         tmmInterfaceName: "vlan-internal"
   

2. Using Specific Configuration Values

   Copy the below example F5SPKEgress CR with specific configuration values into a YAML file, then modify VXLAN settings as needed and save.

   apiVersion: "k8s.f5net.com/v3"
   kind: F5SPKEgress
   metadata:
     name: egress-cr
     namespace: f5-bnk
   spec:
    dualStackEnabled: true
    snatType: SRC_TRANS_SNATPOOL
    egressSnatpool: egress-snatpool
    pseudoCNIConfig:
       namespaces:
         - app-ns-1
       appPodInterface: "eth0"
       vxlan:
         create: true
         tmmInterfaceName: "vlan-internal"
         port: 4789
         mtu: 1460
         key: 100
   

3. Apply the F5SPKEgress CR that you have created.

   kubectl apply -f egress-cr.yaml
   

   Sample Output

   f5spkegress.k8s.f5net.com/egress-cr created