F5SPKIngressHTTP2¶
Overview¶
This overview discusses the F5SPKIngressHTTP2 Custom Resource (CR). For the full list of CRs, refer to the SPK CRs overview. The F5SPKIngressHTTP2 CR configures the Service Proxy Traffic Management Microkernel (TMM) to proxy and load balance low-latency 5G Service Based Interface (SBI) messages using an HTTP/2 protocol virtual server, and a load balancing pool consisting of 5G Network Function endpoints. The F5SPKIngressHTTP2 CR fully supports SSL/TLS termination, bi-directional traffic flow, and connection persistence based on network packet headers, footers, and JSON contents.
This document guides you through understanding, configuring and installing a simple F5SPKIngressHTTP2 CR.
CR integration¶
SPK CRs should be integrated into the cluster after the Kubernetes application Deployment and application Service object have been installed. The SPK Controller uses the CR service.name
to discover the application Endpoints, and use them to create TMM’s load balancing pool. The recommended method for installing SPK CRs is to include them in the application’s Helm release. Refer to the Helm CR Integration guide for more information.
CR parameters¶
The table below describes the CR parameters. Each heading below represents the top-level parameter element. For example, to set the Kubernetes services.name
list, use spec.services.name
.
Parameter | Description |
---|---|
sslFileWatchMode |
The type of file watching for any changes in the keys and certs used by client and server profiles in TMM: SSL_FILE_WATCH_MODE_NONE, SSL_FILE_WATCH_MODE_KUBERNETES_SECRET_STORE, or SSL_FILE_WATCH_MODE_FILES_IN_SHARED_VOLUME (default). |
clientssl
Note: Refer to the Managing SSL/TLS certs and keys section prior to configuring the clientssl
parameters.
Parameter | Description |
---|---|
keyCertPairs.cert |
References SSL/TLS certificates and intermediate CA certificates used to terminate secure ingress connections. Certificate names must be appended to the path file://etc/ssl/tls-keys-certs/<name>.crt. |
keyCertPairs.key |
References SSL/TLS private keys. Key names must be appended to the path file://etc/ssl/tls-keys-certs/<name>.key. |
serverNames |
Specifies a list of hostnames, or a fully qualified domain name used to select the profile by server name indication (SNI) matching. |
ciphers |
Specifies the OpenSSL style cipher string: DEFAULT (default) or ALL. |
enableClientAuthentication |
Enables client certificate verification: true or false (default). |
trustedCa |
Specifies list of Root CAs in PEM format used for client certificate verification. |
serverssl¶
Note: Refer to the Managing SSL/TLS certs and keys section prior to configuring the serverssl
parameters.
Parameter | Description |
---|---|
offload |
Specifies if the TMM does TLS offloading: true or false (default). |
keyCertPairs.cert |
References SSL/TLS certificates and intermediate CA certificates used when TMM behaves as an SSL client. Certificate names must be appended to the path file://etc/ssl/tls-keys-certs/<name>.crt. |
keyCertPairs.key |
References the SSL/TLS private keys matching the keyCertPairs.key certificates. Key names must be appended to the path file://etc/ssl/tls-keys-certs/<name>.key. |
ciphers |
Specifies the OpenSSL style cipher string: DEFAULT (default) or ALL. |
enableServerAuthentication |
Enables server certificate verification: true or false (default). |
trustedCa |
Specifies the list of Root CAs used for server certificate verification. |
spec¶
Parameter | Description |
---|---|
services.name |
A list of Service object names for the internal applications (Pods). The Controller creates a load balancing pool using the discovered Service Endpoints. |
services.port |
The exposed port for the service. |
destinationAddress |
Creates an IPv4 virtual server address to receive ingress connections. |
destinationPort |
Creates a service port to receive ingress connections. When the Kubernetes service being load balanced has multiple ports, install one CR per service, or use port 0 for all ports. |
v6destinationAddress |
Creates an IPv6 virtual server address to receive ingress connections. |
loadBalancingMethod |
Specifies the load balancing method used to distribute traffic across pool members: ROUND_ROBIN distributes connections evenly across all pool members (default), and RATIO_LEAST_CONN_MEMBER distributes connections first to members with the least number of active connections. |
ingressVlans.vlanList |
Specifies a list of F5SPKVlan CRs to listen for ingress traffic, using the CR's metadata.name . The list can also be disabled using disableListedVlans . |
ingressVlans.category |
Specifies an F5SPKVlan CR category to listen for ingress traffic. The category can also be disabled using disableListedVlans . |
ingressVlans.disableListedVlans |
Disables the VLANs specified with the vlanList parameter: true (default) or false . Excluding one VLAN may simplify having to enable many VLANS. |
egressVlans.vlanList |
Specifies a list of F5SPKVlan CRs to listen for egress traffic, using the CR's metadata.name . The list can also be disabled using disableListedVlans . |
egressVlans.category |
Specifies an F5SPKVlan CR category to listen for egress traffic. The category can also be disabled using disableListedVlans . |
egressVlans.disableListedVlans |
Disables the VLANs specified with the vlanList parameter: true (default) or false . Excluding one VLAN may simplify having to enable many VLANS. |
idleTimeout |
Specifies the time in seconds that a client connection may remain open without activity before closing. The default is 300. |
ipfamilies |
Specifies the IP version capabilities of the application: IPv4 (default), IPv6, or IPv4andIPv6. |
jsonMaxBytes |
The maximum number of bytes for a JSON payload. The default is 65536. |
jsonMaxEntries |
The maximum number of entries in the JSON payload. The default is 2048. |
irule |
The iRule proc (procedure) called by the staticRoutes.customIruleProc parameter that will apply to matching connections. |
spec.staticRoutes¶
The spec.staticRoutes
parameter defines packet matching conditions
that steer traffic to the configured service
endpoints. The conditions
are applied in the order defined.
Parameter | Description |
---|---|
conditions |
Specifies an array of up to 4 conditions that must be met for the staticRoute to be selected and applied. |
conditions.fieldName |
Specifies a string representing the name of a field in the message, such as an HTTP header or pseudo header. For example, ':m', or JSON path. For example, ':JSON:some:path:in:json:payload'. |
conditions.comparisonOp |
Specifies a comparison operation to perform against the fieldName value to determine whether the condition is met: SR_COMPARE_NONE, SR_COMPARE_EQUALS (default), SR_COMPARE_NOT_EQUALS, SR_COMPARE_STARTS_WITH, SR_COMPARE_ENDS_WITH, SR_COMPARE_CONTAINS, SR_COMPARE_EXISTS, or SR_COMPARE_NOT_EXISTS. |
conditions.values |
Specifies a list of constant values to compare against a fieldName value. When multiple are provided, the condition will be met when any value is matched. |
conditions.caseSensitive |
Enables evaluating conditions using case-sensitivity: true (default) or false. |
customIruleProc |
The name of the spec.irule to call when the condition match occurs. |
persistBidirectional |
Specifies whether persistence should be bidirectional when a packet match occurs: true (default) or false. |
persistField |
Specifies a custom field that matching requests use as a persistence key. The field specifies the name of an HTTP header or pseudo-header such as :m (method), or :u (uri). Paths to values in the JSON payload may also be specified such as :JSON:key1:key2, with each :key in the path navigating one level deeper into the JSON object tree. |
persistTimeout |
Specifies the persistence timeout for matching conditions. This value overrides any global timeout values. The default value 0 indicates this static route applies the global timeout value. |
service |
Specifies the name of the Kubernetes service that matching requests will be routed to. The service is selected from the list defined in the CR's spec.services.name parameter. |
snatType |
Specifies the type of SNAT to use when forwarding requests that match this static route: SRC_TRANS_NONE, SRC_TRANS_SNATPOOL, or SRC_TRANS_AUTOMAP (default). |
snatPool |
If snatType is SRC_TRANS_SNATPOOL, this value is the name of the snatpool to use when forwarding requests that match this static route. |
CR example¶
apiVersion: "k8s.f5net.com/v1"
kind: F5SPKIngressHTTP2
metadata:
name: http2-app
namespace: spk-app
clientssl:
keyCertPairs:
- key: file://etc/ssl/tls-keys-certs/client.key
cert: file://etc/ssl/tls-keys-certs/client.crt
ciphers: "DEFAULT"
spec:
destinationAddress: 10.20.2.206
destinationPort: 80
v6destinationAddress: 2002::10:20:2:206
irule: |
proc insert_http_header_a {} {
log local0. "insert a new HTTP header"
HTTP::header insert "x-custom-proc" "sbi-A"
set acctVal [MESSAGE::field value ":JSON:account"]
log local0. "######### acctValB = $acctVal"
}
proc insert_http_header_b {} {
log local0. "insert a new HTTP header"
HTTP::header insert "x-custom-proc" "sbi-B"
set acctVal [MESSAGE::field value ":JSON:account"]
log local0. "######### acctValB = $acctVal"
}
ipfamilies: IPv4andIPv6
loadBalancingMethod: "ROUND_ROBIN"
services:
- name: web-a
port: 443
services:
- name: web-b
port: 443
staticRoutes:
- customIruleProc: "insert_http_header_a"
persistBidirectional: false
persistField: "${:JSON:account}"
persistTimeout: 60
service: "web-a"
conditions:
- fieldName: ":JSON:account"
comparisonOp: "SR_COMPARE_EQUALS"
values:
- "account1"
caseSensitive: true
- customIruleProc: "insert_http_header_b"
persistBidirectional: false
persistField: "${:JSON:account}"
persistTimeout: 60
service: "web-b"
conditions:
- fieldName: ":JSON:account"
comparisonOp: "SR_COMPARE_EQUALS"
values:
- "account2"
caseSensitive: true
Application Project¶
The SPK Controller and Service Proxy TMM Pods install to a different Project than the TCP application (Pods). When installing the SPK Controller, set the controller.watchNamespace
parameter to the HTTP2 Pod Project(s) in the Helm values file. For example:
Note: The watchNamespace parameter accepts multiple namespaces.
controller:
watchNamespace:
- "spk-app"
- "spk-app2"
Dual-Stack environments¶
Service Proxy TMM’s load balancing pool is created by discovering the Kubernetes Service Endpoints in the Project. In IPv4/IPv6 dual-stack environments, to populate the load balancing pool with IPv6 members, set the Service PreferDualStack
parameter to IPv6
. For example:
kind: Service
metadata:
name: nginx-web-app
namespace: spk-app
labels:
app: http2-web-app
spec:
ipFamilyPolicy: PreferDualStack
ipFamilies:
- IPv6
- IPv4
Important: When enabling PreferDualStack
, ensure TMM’s internal F5SPKVlan interface configuration includes both IPv4 and IPv6 addresses.
Ingress traffic¶
To enable ingress network traffic, Service Proxy TMM must be configured to advertise virtual server IP addresses to external networks using the BGP dynamic routing protocol. Alternatively, you can configure appropriate routes on upstream devices. For BGP configuration assistance, refer to the BGP Overview.
Managing certs and keys¶
Read this section carefully to ensure the SSL/TLS certificates and keys referenced by the F5SPKIngressHTTP2 CR are encoded and installed into the cluster properly. These bullet points are essential:
- When installing the SPK Controller you must set
tmm.tlsStore.enabled
paramter to true. - The SSL/TLS certificates and keys must be Base64 encoded, and stored in a Secret named tls-keys-certs-secret.
- TMM mounts the Secret named tls-keys-certs-secret to the file path file://etc/ssl/tls-keys-certs/.
Important: The tls-keys-certs-secret Secret must be created before the SPK Controller is installed, otherwise the mount will fail and cause the TMM to enter a restart loop.
Use the steps below to generate a new SSL/TLS certficate and key, Base64 encode them, and then create the tls-keys-certs-secret Secret to store them in the cluster. F5 recommends using SSL/TLS certificates signed by a well-known certificate authority (CA) for production application traffic.
Note: Use steps 4 - 6 if you already have an existing SSL/TLS certificate and key pair.
Generate the CA signing certificate and key:
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.crt \ -subj "/C=US/ST=WA/L=Seattle/O=F5/OU=Dev/CN=ca"
Generate the clientssl profile SSL/TLS certificate signing request (CSR):
openssl genrsa -out client.key 4096
openssl req -new -key client.key -out client.csr \ -subj "/C=US/ST=WA/L=Seattle/O=F5/OU=PD/CN=client.com"
Sign the clientssl profile CSR with the CA:
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key \ -set_serial 101 -outform PEM -out client.crt -extensions req_ext -days 365 -sha256
Base64 encode the SSL/TLS certificate and key:
openssl base64 -A -in client.crt -out client-encode.crt openssl base64 -A -in client.key -out client-encode.key
Create the tls-keys-certs-secret Secret that stores the SSL/TLS certificate and key:
echo "apiVersion: v1" > tls-keys-certs-secret.yaml echo "kind: Secret" >> tls-keys-certs-secret.yaml echo "metadata:" >> tls-keys-certs-secret.yaml echo " name: tls-keys-certs-secret" >> tls-keys-certs-secret.yaml echo "data:" >> tls-keys-certs-secret.yaml echo -n " client.crt: " >> tls-keys-certs-secret.yaml cat client-encode.crt >> tls-keys-certs-secret.yaml echo " " >> tls-keys-certs-secret.yaml echo -n " client.key: " >> tls-keys-certs-secret.yaml cat client-encode.key >> tls-keys-certs-secret.yaml
Install the Secret into the SPK Controller Project:
oc apply -f tls-keys-certs-secret.yaml -n spk-ingress
Requirements¶
Ensure you have:
- Installed a K8S Service object and application.
- Installed the SPK Controller.
- A Linux based workstation.
- Installed the dSSM Database to support persistence records.
Installation¶
Use the following steps to obtain the application’s Service object information to configure and install the F5SPKIngressTCP CR.
Switch to the application Project:
oc project <project>
In this example, the application is in the spk-app Project:
oc project spk-app
Use the Service object NAME and PORT to configure the CR
service.name
andservice.port
parameters:oc get service
In this example, the Service object NAMEs are http2-web-a and http2-web-b and the PORT is 443:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) http2-web-a NodePort 10.99.99.91 <none> 443:443/TCP http2-web-b NodePort 10.99.99.99 <none> 443:443/TCP
Copy the example F5SPKIngressHTTP2 CR into a YAML file:
apiVersion: "k8s.f5net.com/v1" kind: F5SPKIngressHTTP2 metadata: name: http2-app namespace: spk-app clientssl: keyCertPairs: - key: file://etc/ssl/tls-keys-certs/client.key cert: file://etc/ssl/tls-keys-certs/client.crt ciphers: "DEFAULT" spec: destinationAddress: 10.20.2.206 destinationPort: 80 v6destinationAddress: 2002::10:20:2:206 irule: | proc insert_http_header_a {} { log local0. "insert a new HTTP header" HTTP::header insert "x-custom-proc" "sbi-A" set acctVal [MESSAGE::field value ":JSON:account"] log local0. "######### acctValA = $acctVal" } proc insert_http_header_b {} { log local0. "insert a new HTTP header" HTTP::header insert "x-custom-proc" "sbi-B" set acctVal [MESSAGE::field value ":JSON:account"] log local0. "######### acctValB = $acctVal" } ipfamilies: IPv4andIPv6 loadBalancingMethod: "ROUND_ROBIN" services: - name: http-web-a port: 443 services: - name: http-web-b port: 443 staticRoutes: - customIruleProc: "insert_http_header_a" persistBidirectional: false persistField: "${:JSON:account}" persistTimeout: 60 service: "web-a" conditions: - fieldName: ":JSON:account" comparisonOp: "SR_COMPARE_EQUALS" values: - "account1" caseSensitive: true - customIruleProc: "insert_http_header_b" persistBidirectional: false persistField: "${:JSON:account}" persistTimeout: 60 service: "web-b" conditions: - fieldName: ":JSON:account" comparisonOp: "SR_COMPARE_EQUALS" values: - "account2" caseSensitive: true
Install the F5SPKIngressHTTP2 CR:
oc apply -f spk-ingress-http2.yaml
Web clients should now be able to connect to the application through the Service Proxy TMM.
Connection statistics¶
If you installed the SPK Controller with the Debug Sidecar enabled, connect to the sidecar to view virtual server and pool member connectivity statistics.
Log in to the Service Proxy Debug container:
oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
View the virtual server connection statistics:
tmctl -d blade virtual_server_stat -s name,clientside.tot_conns
For example:
name serverside.tot_conns --------------------------------------- -------------------- spk-apps-nginx-http2-crd-virtual-server 31
View the load balancing pool connection statistics:
tmctl -d blade pool_member_stat -s pool_name,serverside.tot_conns
For example:
web-apps-http2-crd-pool 15 web-apps-http2-crd-pool 16
Feedback¶
Provide feedback to improve this document by emailing spkdocs@f5.com.