Gateway API CRs¶
Overview¶
The Service Proxy for Kubernetes (SPK) usecases such as IngressTCP and IngressUDP can be configured with Gateway API CRs to load balance low-latency TCP and UDP application traffic between networks using a virtual server and load balancing pool.
Gateway API is an open source project managed by the SIG-NETWORK community. It is an API (collection of resources) that model service networking in Kubernetes. These resources - GatewayClass, Gateway, TCPRoute, etc., as well as the Kubernetes Service resource - aim to evolve Kubernetes service networking through expressive, extensible, and role-oriented interfaces that are implemented by many vendors and have broad industry support.
Note: The SPK v1.9.0 Gateway API supports the v1.0.0 version of the Gateway API, which is considered Early Access (EA). EA features are unsupported and are made available to get customer feedback on feature functionality and stability. This is delivered as a separate tarball.
Gateway API CRs¶
Gateway API CRs are as follows:
- GatewayClass CR - GatewayClass is a cluster-scoped resource. It defines a set of Gateways that share a common configuration and behavior.
- Gateway CR - Defines how traffic can be translated and directed to service(s).
- TCPRoute CR - Defines how TCP traffic is managed.
- UDPRoute CR - Defines how UDP traffic is managed.
Gateway API for IngressTCP¶
Based on the configuration of GatewayClass, Gateway, and TCPRoute CRs, an IngressTCP internal data structure is created and this configuration is used by a TMM.
Gateway API for IngressUDP¶
Based on the configuration of GatewayClass, Gateway, and UDPRoute CRs, an IngressUDP internal data structure is created and this configuration is used by a TMM.
CR Parameters¶
GatewayClass¶
The table below describes the CR spec parameters used to configure the GatewayClass CR are:
| Parameter | Description |
|---|---|
controllerName |
Specifies name of the controller that managers the Gateways of this class. The value MUST be domain prefixed path. This is a mandatory field. For example: f5.io/gateway-controller-name. |
parametersRef.group |
ParametersRef is a reference to a resource that contains the configuration parameters corresponding to the GatewayClass. Specifies the group of the referent. |
parametersRef.kind |
Specifies the kind of the referent. |
parametersRef.name |
Specifies the name of the referent. |
parametersRef.namespace |
Specifies the namespace of the referent. This field is required when referring to a Namespace-scoped resource and MUST be unset when referring to a Cluster-scoped resource. |
description |
Specifies the definition of GatewayClass. |
Gateway¶
The table below describes the CR spec parameters used to configure the Gateway CR are:
| Parameter | Description |
|---|---|
gatewayClassName |
Specifies the name of a GatewayClassName object used for this Gateway. For example: f5-gatewayclass |
listeners.name |
Specifies the name of the listener associated with this Gateway to listen for incoming traffic on a specific port. For example: listener-1 |
listeners.port |
Specifies the port on which a listener is configured to listen for incoming traffic. For example: 8080 |
listeners.protocol |
Specifies the protocol being used by the listener. For example: TCP or UDP |
listeners.allowedRoutes.kinds.group |
Specifies the category or type of route that the listener is allowed to use. For example: gateway.networking.k8s.io |
listeners.allowedRoutes.kinds.kind |
Specifies the kind or type of route that the listener is allowed to use. For example: TCPRoute |
listeners.allowedRoutes.namespaces.from |
Specifies from where routes will be selected for this gateway. Possible values are: All or Selector or Same. |
listeners.allowedRoutes.namespaces.selector.matchLabels |
matchLabels is a map of {key,value} pairs. For example: kubernetes.io/metadata.name: f5-apps. Specifies only those namespaces that are labeled ‘kubernetes.io/metadata. name: f5-apps’ will be able to attach their routes to this ‘f5-gateway’ gateway. |
addresses.type |
Specifies the address type for this gateway. Note: only IPAddress type is supported. |
addresses.value |
Specifies at least one valid IP address that has been assigned to this gateway. For example: 11.11.11.154 |
TCPRoute and UDPRoute¶
The table below describes the CR spec parameters used to configure the TCPRoute and UDPRoute CRs are:
| Parameter | Description |
|---|---|
parentRefs.kind |
Specifies the type or category of the parent reference. |
parentRefs.name |
Specifies the name of the parent reference. For example: f5-gateway |
parentRefs.sectionName |
Specifies a specific section or category within the parent reference. (i.e., listener from the gateway). For example: listener-1 |
rules.backendRefs.kind |
BackendRefs defines the backend(s) where matching requests should be sent. Specifies the type or category of the backend reference. |
rules.backendRefs.name |
Specifies the name or identifier of the backend reference. For example: nginx-server |
rules.backendRefs.port |
Specifies the port associated with the backend reference. |
rules.backendRefs.weight |
Specifies the proportion of requests forwarded to the referenced backend. |
Gateway API Support Status¶
GatewayClass CR
The table below describes the support status of GatewayClass CR paremeters:
| Parameter | Supported / Not Supported / Partially Supported |
|---|---|
controllerName |
Supported |
parametersRef.group |
Not Supported |
parametersRef.kind |
Not Supported |
parametersRef.name |
Not Supported |
parametersRef.namespace |
Not Supported |
description |
Supported |
Gateway CR
The table below describes the support status of Gateway CR paremeters:
| Parameter | Supported / Not Supported / Partially Supported |
|---|---|
gatewayClassName |
Supported |
listeners.name |
Supported |
listeners.port |
Supported |
listeners.protocol |
Partially Supported. For example: TCP or UDP |
listeners.allowedRoutes.kinds.group |
Partially Supported |
listeners.allowedRoutes.kinds.kind |
Partially Supported |
listeners.allowedRoutes.namespaces.from |
Partially Supported |
listeners.allowedRoutes.namespaces.selector.matchLabels |
Partially Supported |
addresses.type |
Supported |
addresses.value |
Supported |
TCPRoute and UDPRoute CR
| Parameter | Supported / Not Supported / Partially Supported |
|---|---|
parentRefs.kind |
Partially Supported. Only for Gateway. |
parentRefs.name |
Supported |
parentRefs.sectionName |
Supported |
rules.backendRefs.kind |
Partially Supported |
rules.backendRefs.name |
Supported |
rules.backendRefs.port |
Supported |
rules.backendRefs.weight |
Supported |
CR Examples¶
GatewayClass
apiVersion: gateway.networking.k8s.io/v1alpha
kind: GatewayClass
metadata:
name: bigip-next-SPK
spec:
controllerName: f5.io/gateway-controller-name
parametersRef:
group: f5.io/v1alpha
kind: Config
name: bigip-next-SPK-gateway-config
namespace: spk-ingress
description: This is GatewayClass definition.
Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: f5-gateway
namespace: f5-apps
spec:
gatewayClassName: f5-gatewayclass
listeners:
- name: listener-1
port: 8080
protocol: TCP
allowedRoutes:
kinds:
- group: gateway.networking.k8s.io
kind: TCPRoute
namespaces:
from: Selector
selector:
matchLabels:
kubernetes.io/metadata.name: f5-apps
addresses:
- type: IPAddress
value: 11.11.11.154
TCPRoute
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
name: f5-tcp-route
namespace: f5-apps
spec:
parentRefs:
- kind: Gateway
name: f5-gateway
sectionName: listener-1
rules:
- backendRefs:
- kind: Service
name: nginx-server
port: 8080
weight: 1
UDPRoute
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: UDPRoute
metadata:
name: f5-udp-route
namespace: f5-apps
spec:
parentRefs:
- kind: Gateway
name: f5-gateway
sectionName: listener-2
rules:
- backendRefs:
- kind: Service
name: nginx-server
port: 10001
weight: 1
Requirements¶
Ensure you have:
- Obtained the GW API EA tarball.
- Installed the SPK Cert Manager.
- Installed the SPK CWC
- Installed the RabbitMQ Pod
- A Linux based workstation with Helm installed.
Procedures¶
Modular Controller values
Ensure Helm can obtain the Modular Controller images from the local image registry, add the following Helm values:
Note: The new Modular Controller pod has 4 additional containers, such as f5_infra_controller, f5_k8s_event_subscriber, f5_spk_controller and tmm_pod_manager a part from f5_lic_helper container in F5ingress SPK Controller with additional feature configuration.
controller:
watchNamespace: spk-apps
createTmmLBService: true
enableCustomResources: false
f5_infra_controller:
enabled: true
image:
repository: "local.registry.com"
f5_k8s_event_subscriber:
enabled: true
image:
repository: "local.registry.com"
f5_lic_helper:
enabled: true
image:
repository: "local.registry.com"
f5_spk_controller:
enabled: true
image:
repository: "local.registry.com"
fluentbit_sidecar:
enabled: true
image:
repository: "local.registry.com"
image:
repository: "local.registry.com"
report:
interval: 15
resources:
limits:
memory: 1Gi
requests:
memory: 256Mi
tmm_pod_manager:
enabled: true
image:
repository: "local.registry.com"
vlan_grpc:
enabled: true
f5-stats_collector:
enabled: false
TMM values
To configure the TMM Proxy Helm values for your environment, refer to TMM values section in the SPK Controller guide.
Completed values
The completed Helm values file should appear similar to the following:
f5-tmm:
enabled: true
debug:
enabled: true
image:
repository: "local.registry.com"
f5-toda-logging:
enabled: false
fluentd:
host: f5-toda-fluentd.dp-ingress.svc.cluster.local.
sidecar:
image:
repository: "local.registry.com"
tmstats:
config:
image:
repository: "local.registry.com"
enabled: true
tmm:
customEnvVars:
- name: SSL_SERVERSIDE_STORE
value: /tls/tmm/mds/clt
- name: SSL_TRUSTED_CA_STORE
value: /tls/tmm/mds/clt
- name: CONFIG_VIEWER_ENABLE
value: 'true'
- name: PAL_CPU_SET
value: 0,2
- name: TMM_MAPRES_IGNORE_MEM_LIMIT
value: 'true'
- name: DECL_VERBOSE_CONFIG
value: 'true'
- name: TMM_MAPRES_DELAY_MS
value: '1000'
- name: TMM_MAPRES_VERBOSITY
value: debug
- name: ROUTE_BACKPLANE
value: 'FALSE'
- name: TMM_MAPRES_ADDL_VETHS_ON_DP
value: 'TRUE'
- name: TMM_CALICO_ROUTER
value: default
- name: SESSIONDB_DISCOVERY_SENTINEL
value: 'true'
- name: SESSIONDB_EXTERNAL_SERVICE
value: dssm-f5-dssm-sentinel.default.svc.cluster.local
debug:
enabled: true
dynamicRouting:
enabled: false
egress:
dnsNat46Enabled: false
useSnatpools: false
grpc:
enabled: true
hugepages:
enabled: false
image:
repository: "local.registry.com"
logLevel: Debug
pod:
annotations:
k8s.v1.cni.cncf.io/networks: "[\n {\n \"name\": \"client-net\",\n \"\
ips\": [\"11.11.11.200/24\", \"2002::11:11:11:200/112\"]\n },\n {\n \
\ \"name\": \"server-net\",\n \"ips\": [\"22.22.22.200/24\", \"2002::22:22:22:200/112\"\
]\n }\n]\n"
replicaCount: 1
resources:
limits:
cpu: 2
requests:
cpu: 2
sessiondb:
useExternalStorage: 'true'
blobd:
enabled: false
controller:
watchNamespace: spk-apps
createTmmLBService: true
enableCustomResources: false
f5_infra_controller:
enabled: true
image:
repository: "local.registry.com"
f5_k8s_event_subscriber:
enabled: true
image:
repository: "local.registry.com"
f5_lic_helper:
enabled: true
image:
repository: "local.registry.com"
f5_spk_controller:
enabled: true
image:
repository: "local.registry.com"
fluentbit_sidecar:
enabled: true
image:
repository: "local.registry.com"
image:
repository: "local.registry.com"
report:
interval: 15
resources:
limits:
memory: 1Gi
requests:
memory: 256Mi
tmm_pod_manager:
enabled: true
image:
repository: "local.registry.com"
vlan_grpc:
enabled: true
f5-stats_collector:
enabled: false
Benefits of Gateway API¶
- Role-oriented: Gateway API kinds are modeled after organizational roles that are responsible for managing Kubernetes service networking:
- Infrastructure Provider: Manages infrastructure that allows multiple isolated clusters to serve multiple tenants, e.g. a cloud provider.
- Cluster Operator: Manages clusters and is typically concerned with policies, network access, application permissions, etc.
- Application Developer: Manages an application running in a cluster and is typically concerned with application-level configuration and Service composition.
- Portable: Gateway API specifications are defined as custom resources and are supported by many implementations.
- Expressive: Gateway API kinds support functionality for common traffic routing use cases such as header-based matching, traffic weighting, and others that were only possible in Ingress by using custom annotations.
- Extensible: Gateway allows for custom resources to be linked at various layers of the API. This makes granular customization possible at the appropriate places within the API structure.
Install Containers and CRs - TCP Virtual Server¶
Use the following steps to the install the containers and CRs to establish a connection with TCP Virtual Server.
Note: The new Modular Controller pod has 4 additional containers, such as f5_infra_controller, f5_k8s_event_subscriber, f5_spk_controller and tmm_pod_manager a part from f5_lic_helper container in F5ingress SPK Controller with additional feature configuration.
Change into the local directory with the SPK files, and list the files in the tar directory:
cd <directory>
ls -1 tar
In this example, the SPK files are in the spkinstall directory:
cd spkinstall
ls -1 tar
In this example, Modular Controller and Service Proxy TMM Helm chart is named f5ingress-v0.444.0-spk-mod-controller.8285170:
csrc-v0.4.6.tgz cwc-0.12.17.tgz f5-cert-gen-0.7.3.tgz f5-cert-manager-0.8.0.tgz f5-crdconversion-0.4.8.tgz f5-dssm-0.34.0.tgz f5-toda-fluentd-1.23.4.tgz f5-toda-stats-0.2.6.tgz f5ingress-v0.444.0-spk-mod-controller.8285170.tgz rabbitmq-0.1.3.tgz spk-docker-images.tgz
Switch to the Modular Controller Project:
In this example, the spk-ingress Project is selected:
oc project spk-ingress
Install the Modular Controller and Service Proxy TMM Pods, referencing the Helm values file created in the previous procedure:
helm install <release name> tar/f5ingress-spk-mod-controller-<version>.tgz -f <values>.yaml
In this example, Modular Controller installs using Helm chart version 10.0.1:
helm install modular-controller tar/f5ingress-spk-mod-controller-0.444.0.tgz -f modular-controller-values.yaml
Verify the Pods have installed successfully, and all containers are Running:
oc get pods
In this example, all containers have a STATUS of Running as expected:
NAME READY STATUS modular-controller-f5ingress-596bfbf657-r2rdj 6/6 Running f5-tmm-86894fd76c-p4gdd 2/2 Running
Create a new backend server file named nginx-server.yaml and install in f5-app namespace, and set the parameters as shown in the snippet:
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-server labels: app: nginx-server spec: replicas: 1 selector: matchLabels: app: nginx-server template: metadata: labels: app: nginx-server spec: containers: - name: iperf3 image: 'artifactory.f5net.com/f5-sandbox-docker/my_nginx:6' ports: - name: http containerPort: 80 protocol: TCP - name: https containerPort: 443 protocol: TCP imagePullPolicy: Always securityContext: privileged: true runAsUser: 0 serviceAccount: default --- kind: Service apiVersion: v1 metadata: name: nginx-server labels: app: nginx-server spec: ports: - name: tcp80 protocol: TCP port: 8080 targetPort: 80 - name: tcp443 protocol: TCP port: 8443 targetPort: 443 - name: udp protocol: UDP port: 4445 targetPort: 4445 - name: udp2 protocol: UDP port: 4446 targetPort: 4446 selector: app: nginx-server sessionAffinity: None ipFamilies: - IPv4 - IPv6 ipFamilyPolicy: PreferDualStack
oc create -f <values>.yaml -n <namespace>
For example
oc create -f nginx-server.yaml -n f5-apps
Install the Gateway API CRs:
GatewayClass CR
apiVersion: gateway.networking.k8s.io/v1beta1 kind: GatewayClass metadata: name: f5-gatewayclass namespace: f5-apps spec: controllerName: "f5.com/f5-gateway-controller" description: "F5 BIG-IP Kubernetes Gateway"
oc create -f gateway-class.yaml -n f5-apps
Gateway CR
apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: f5-gateway namespace: f5-apps spec: addresses: - type: IPAddress value: 11.11.11.154 gatewayClassName: f5-gatewayclass listeners: - name: listener-1 port: 8080 protocol: TCP allowedRoutes: kinds: - group: gateway.networking.k8s.io kind: TCPRoute namespaces: from: Selector selector: matchLabels: kubernetes.io/metadata.name: f5-apps
oc create -f gateway.yaml -n f5-apps
TCPRoute CR
apiVersion: gateway.networking.k8s.io/v1alpha2 kind: TCPRoute metadata: name: f5-tcp-route namespace: f5-apps spec: parentRefs: - kind: Gateway name: f5-gateway sectionName: listener-1 rules: - backendRefs: - kind: Service name: nginx-server port: 8080 weight: 1
oc create -f tcproute.yaml -n f5-apps
Now, ensure that a TCP Virtual Server is created.
Verify the TCP traffic from client to TCP virtual server.
Web client should now be able to connect to the application through the Service Proxy TMM.
Install Containers and CRs - UDP Virtual Server¶
Use the following steps to the install the containers and CRs to establish a connection with UDP Virtual Server.
Change into the local directory with the SPK files, and list the files in the tar directory:
cd <directory>
ls -1 tar
In this example, the SPK files are in the spkinstall directory:
cd spkinstall
ls -1 tar
In this example, Modular Controller and Service Proxy TMM Helm chart is named f5ingress-v0.444.0-spk-mod-controller.8285170:
csrc-v0.4.6.tgz cwc-0.12.17.tgz f5-cert-gen-0.7.3.tgz f5-cert-manager-0.8.0.tgz f5-crdconversion-0.4.8.tgz f5-dssm-0.34.0.tgz f5-toda-fluentd-1.23.4.tgz f5-toda-stats-0.2.6.tgz f5ingress-v0.444.0-spk-mod-controller.8285170.tgz rabbitmq-0.1.3.tgz spk-docker-images.tgz
Switch to the Modular Controller Project:
In this example, the spk-ingress Project is selected:
oc project spk-ingress
Install the Modular Controller and Service Proxy TMM Pods, referencing the Helm values file created in the previous procedure:
helm install <release name> tar/f5ingress-spk-mod-controller-<version>.tgz -f <values>.yaml
In this example, Modular Controller installs using Helm chart version 10.0.1:
helm install modular-controller tar/f5ingress-spk-mod-controller-0.444.0.tgz -f modular-controller-values.yaml
Verify the Pods have installed successfully, and all containers are Running:
oc get pods
In this example, all containers have a STATUS of Running as expected:
NAME READY STATUS modular-controller-f5ingress-596bfbf657-r2rdj 6/6 Running f5-tmm-86894fd76c-p4gdd 2/2 Running
Create a new backend server file named nginx-server.yaml and install in f5-app namespace, and set the parameters as shown in the snippet:
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-server labels: app: nginx-server spec: replicas: 1 selector: matchLabels: app: nginx-server template: metadata: labels: app: nginx-server spec: containers: - name: udp-server image: "jpoon/udp-server" ports: - name: httpudp containerPort: 10001 protocol: UDP imagePullPolicy: Always securityContext: privileged: true runAsUser: 0 serviceAccount: default --- kind: Service apiVersion: v1 metadata: name: nginx-server labels: app: nginx-server spec: ports: - name: httpudp protocol: UDP port: 10001 targetPort: 10001 selector: app: nginx-server sessionAffinity: None ipFamilies: - IPv4 - IPv6 ipFamilyPolicy: PreferDualStack
oc create -f <values>.yaml -n <namespace>
For example
oc create -f nginx-server-udp.yaml -n f5-apps
Install the Gateway API CRs:
GatewayClass CR
apiVersion: gateway.networking.k8s.io/v1beta1 kind: GatewayClass metadata: name: f5-gatewayclass namespace: f5-apps spec: controllerName: "f5.com/f5-gateway-controller" description: "F5 BIG-IP Kubernetes Gateway"
oc create -f gateway-class.yaml -n f5-apps
Gateway CR
apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: f5-gateway namespace: f5-apps spec: addresses: - type: IPAddress value: 11.11.11.154 gatewayClassName: f5-gatewayclass listeners: - name: listener-2 port: 8090 protocol: UDP allowedRoutes: kinds: - group: gateway.networking.k8s.io kind: UDPRoute namespaces: from: Selector selector: matchLabels: kubernetes.io/metadata.name: f5-apps
oc create -f gateway.yaml -n f5-apps
UDPRoute CR
apiVersion: gateway.networking.k8s.io/v1alpha2 kind: UDPRoute metadata: name: f5-udp-route namespace: f5-apps spec: parentRefs: - kind: Gateway name: f5-gateway sectionName: listener-2 rules: - backendRefs: - kind: Service name: nginx-server port: 10001 weight: 1
oc create -f udproute.yaml -n f5-apps
Now, ensure that a UDP Virtual Server is created.
Verify the UDP traffic from client to UDP virtual server.
Web client should now be able to connect to the application through the Service Proxy TMM.
Connection statistics¶
If you installed the Modular Controller with the Debug Sidecar enabled, connect to the sidecar to view virtual server and pool member connectivity statistics.
Log in to the Service Proxy Debug container:
oc exec -it f5-tmm-86894fd76c-vznq4 -c debug -- bash
View the virtual server connection statistics:
tmctl -d /var/tmstat/blade virtual_server_stat
For example:
name ------------------------------------------------------ f5-apps-gatewayapi-f5-tcp-route-0-virtual-server destination ----------------------------------------------------------- 00:00:00:00:00:00:00:00:00:00:FF:FF:16:16:16:98:00:00:00:00 source vs_index ----------------------------------------------------------- -------- 00:00:00:00:00:00:00:00:00:00:FF:FF:00:00:00:00:00:00:00:00 0 cs_min_conn_dur cs_max_conn_dur cs_mean_conn_dur no_nodes_errors cycle_count --------------- --------------- ---------------- --------------- ----------- 0 0 0 0 0 delta_busy delta_cycles conf_status no_acl_match no_acl_match_accept ---------- ------------ ----------- ------------ ------------------- 26790597918 0 OK 0 0
Feedback¶
Provide feedback to improve this document by emailing spkdocs@f5.com.
Supplemental¶
- For more information on Gateway API and CRD information, refer to Kubernetes Gateway API guide.
- Gateway API guide.
- Gateway API Specification guide.