F5SPKIngressHTTP2

Overview

This overview discusses the F5SPKIngressHTTP2 Custom Resource (CR). For the full list of CRs, refer to the SPK CRs overview. The F5SPKIngressHTTP2 CR configures the Service Proxy Traffic Management Microkernel (TMM) to proxy and load balance low-latency 5G Service Based Interface (SBI) messages using an HTTP/2 protocol virtual server, and a load balancing pool consisting of 5G Network Function endpoints. The F5SPKIngressHTTP2 CR fully supports SSL/TLS termination, bi-directional traffic flow, and connection persistence based on network packet headers, footers, and JSON contents.

This document guides you through understanding, configuring and installing a simple F5SPKIngressHTTP2 CR.

CR integration

SPK CRs should be integrated into the cluster after the Kubernetes application Deployment and application Service object have been installed. The SPK Controller uses the CR service.name to discover the application Endpoints, and use them to create TMM’s load balancing pool. The recommended method for installing SPK CRs is to include them in the application’s Helm release. Refer to the Helm CR Integration guide for more information.

CR parameters

The table below describes the CR parameters. Each heading below represents the top-level parameter element. For example, to set the Kubernetes services.name list, use spec.services.name.

Parameter Description
sslFileWatchMode The type of file watching for any changes in the keys and certs used by client and server profiles in TMM: SSL_FILE_WATCH_MODE_NONE, SSL_FILE_WATCH_MODE_KUBERNETES_SECRET_STORE, or SSL_FILE_WATCH_MODE_FILES_IN_SHARED_VOLUME (default).

clientssl

_images/spk_info.png Note: Refer to the Managing SSL/TLS certs and keys section prior to configuring the clientssl parameters.

Parameter Description
keyCertPairs.cert References SSL/TLS certificates and intermediate CA certificates used to terminate secure ingress connections. Certificate names must be appended to the path file://etc/ssl/tls-keys-certs/<name>.crt.
keyCertPairs.key References SSL/TLS private keys. Key names must be appended to the path file://etc/ssl/tls-keys-certs/<name>.key.
serverNames Specifies a list of hostnames, or a fully qualified domain name used to select the profile by server name indication (SNI) matching.
ciphers Specifies the OpenSSL style cipher string: DEFAULT (default) or ALL.
enableClientAuthentication Enables client certificate verification: true or false (default).
trustedCa Specifies list of Root CAs in PEM format used for client certificate verification.

serverssl

_images/spk_info.png Note: Refer to the Managing SSL/TLS certs and keys section prior to configuring the serverssl parameters.

Parameter Description
offload Specifies if the TMM does TLS offloading: true or false (default).
keyCertPairs.cert References SSL/TLS certificates and intermediate CA certificates used when TMM behaves as an SSL client. Certificate names must be appended to the path file://etc/ssl/tls-keys-certs/<name>.crt.
keyCertPairs.key References the SSL/TLS private keys matching the keyCertPairs.key certificates. Key names must be appended to the path file://etc/ssl/tls-keys-certs/<name>.key.
ciphers Specifies the OpenSSL style cipher string: DEFAULT (default) or ALL.
enableServerAuthentication Enables server certificate verification: true or false (default).
trustedCa Specifies the list of Root CAs used for server certificate verification.

spec

Parameter Description
services.name A list of Service object names for the internal applications (Pods). The Controller creates a load balancing pool using the discovered Service Endpoints.
services.port The exposed port for the service.
destinationAddress Creates an IPv4 virtual server address to receive ingress connections.
destinationPort Creates a service port to receive ingress connections. When the Kubernetes service being load balanced has multiple ports, install one CR per service, or use port 0 for all ports.
v6destinationAddress Creates an IPv6 virtual server address to receive ingress connections.
loadBalancingMethod Specifies the load balancing method used to distribute traffic across pool members: ROUND_ROBIN distributes connections evenly across all pool members (default), and RATIO_LEAST_CONN_MEMBER distributes connections first to members with the least number of active connections.
ingressVlans.vlanList Specifies a list of F5SPKVlan CRs to listen for ingress traffic, using the CR's metadata.name. The list can also be disabled using disableListedVlans.
ingressVlans.category Specifies an F5SPKVlan CR category to listen for ingress traffic. The category can also be disabled using disableListedVlans.
ingressVlans.disableListedVlans Disables the VLANs specified with the vlanList parameter: true (default) or false. Excluding one VLAN may simplify having to enable many VLANS.
egressVlans.vlanList Specifies a list of F5SPKVlan CRs to listen for egress traffic, using the CR's metadata.name. The list can also be disabled using disableListedVlans.
egressVlans.category Specifies an F5SPKVlan CR category to listen for egress traffic. The category can also be disabled using disableListedVlans.
egressVlans.disableListedVlans Disables the VLANs specified with the vlanList parameter: true (default) or false. Excluding one VLAN may simplify having to enable many VLANS.
idleTimeout Specifies the time in seconds that a client connection may remain open without activity before closing. The default is 300.
ipfamilies Specifies the IP version capabilities of the application: IPv4 (default), IPv6, or IPv4andIPv6.
jsonMaxBytes The maximum number of bytes for a JSON payload. The default is 65536.
jsonMaxEntries The maximum number of entries in the JSON payload. The default is 2048.
irule The iRule proc (procedure) called by the staticRoutes.customIruleProc parameter that will apply to matching connections.

spec.staticRoutes

The spec.staticRoutes parameter defines packet matching conditions that steer traffic to the configured service endpoints. The conditions are applied in the order defined.

Parameter Description
conditions Specifies an array of up to 4 conditions that must be met for the staticRoute to be selected and applied.
conditions.fieldName Specifies a string representing the name of a field in the message, such as an HTTP header or pseudo header. For example, ':m', or JSON path. For example, ':JSON:some:path:in:json:payload'.
conditions.comparisonOp Specifies a comparison operation to perform against the fieldName value to determine whether the condition is met: SR_COMPARE_NONE, SR_COMPARE_EQUALS (default), SR_COMPARE_NOT_EQUALS, SR_COMPARE_STARTS_WITH, SR_COMPARE_ENDS_WITH, SR_COMPARE_CONTAINS, SR_COMPARE_EXISTS, or SR_COMPARE_NOT_EXISTS.
conditions.values Specifies a list of constant values to compare against a fieldName value. When multiple are provided, the condition will be met when any value is matched.
conditions.caseSensitive Enables evaluating conditions using case-sensitivity: true (default) or false.
customIruleProc The name of the spec.irule to call when the condition match occurs.
persistBidirectional Specifies whether persistence should be bidirectional when a packet match occurs: true (default) or false.
persistField Specifies a custom field that matching requests use as a persistence key. The field specifies the name of an HTTP header or pseudo-header such as :m (method), or :u (uri). Paths to values in the JSON payload may also be specified such as :JSON:key1:key2, with each :key in the path navigating one level deeper into the JSON object tree.
persistTimeout Specifies the persistence timeout for matching conditions. This value overrides any global timeout values. The default value 0 indicates this static route applies the global timeout value.
service Specifies the name of the Kubernetes service that matching requests will be routed to. The service is selected from the list defined in the CR's spec.services.name parameter.
snatType Specifies the type of SNAT to use when forwarding requests that match this static route: SRC_TRANS_NONE, SRC_TRANS_SNATPOOL, or SRC_TRANS_AUTOMAP (default).
snatPool If snatType is SRC_TRANS_SNATPOOL, this value is the name of the snatpool to use when forwarding requests that match this static route.

CR example

apiVersion: "k8s.f5net.com/v1"
kind: F5SPKIngressHTTP2
metadata:
  name: http2-app
  namespace: spk-app
clientssl:
  keyCertPairs:
  - key: file://etc/ssl/tls-keys-certs/client.key
    cert: file://etc/ssl/tls-keys-certs/client.crt
  ciphers: "DEFAULT"
spec:
  destinationAddress: 10.20.2.206
  destinationPort: 80
  v6destinationAddress: 2002::10:20:2:206
  irule: |
    proc insert_http_header_a {} {
      log local0. "insert a new HTTP header"
      HTTP::header insert "x-custom-proc" "sbi-A"
      set acctVal [MESSAGE::field value ":JSON:account"]
      log local0. "#########  acctValB = $acctVal"
    }
    proc insert_http_header_b {} {
      log local0. "insert a new HTTP header"
      HTTP::header insert "x-custom-proc" "sbi-B"
      set acctVal [MESSAGE::field value ":JSON:account"]
      log local0. "#########  acctValB = $acctVal"
    }
  ipfamilies: IPv4andIPv6
  loadBalancingMethod: "ROUND_ROBIN"
  services:
    - name: web-a
      port: 443
  services:
    - name: web-b
      port: 443
  staticRoutes:
  - customIruleProc: "insert_http_header_a"
    persistBidirectional: false
    persistField: "${:JSON:account}"
    persistTimeout: 60
    service: "web-a"
    conditions:
    - fieldName: ":JSON:account"
      comparisonOp: "SR_COMPARE_EQUALS"
      values:
      - "account1"
      caseSensitive: true
  - customIruleProc: "insert_http_header_b"
    persistBidirectional: false
    persistField: "${:JSON:account}"
    persistTimeout: 60
    service: "web-b"
    conditions:
    - fieldName: ":JSON:account"
      comparisonOp: "SR_COMPARE_EQUALS"
      values:
      - "account2"
      caseSensitive: true

Application Project

The SPK Controller and Service Proxy TMM Pods install to a different Project than the TCP application (Pods). When installing the SPK Controller, set the controller.watchNamespace parameter to the HTTP2 Pod Project(s) in the Helm values file. For example:

Note: The watchNamespace parameter accepts multiple namespaces.

controller:
  watchNamespace: 
    - "spk-app"
    - "spk-app2"

Dual-Stack environments

Service Proxy TMM’s load balancing pool is created by discovering the Kubernetes Service Endpoints in the Project. In IPv4/IPv6 dual-stack environments, to populate the load balancing pool with IPv6 members, set the Service PreferDualStack parameter to IPv6. For example:

kind: Service
metadata:
  name: nginx-web-app
  namespace: spk-app
  labels:
    app: http2-web-app
spec:
  ipFamilyPolicy: PreferDualStack
  ipFamilies:
  - IPv6
  - IPv4

_images/spk_warn.png Important: When enabling PreferDualStack, ensure TMM’s internal F5SPKVlan interface configuration includes both IPv4 and IPv6 addresses.

Ingress traffic

To enable ingress network traffic, Service Proxy TMM must be configured to advertise virtual server IP addresses to external networks using the BGP dynamic routing protocol. Alternatively, you can configure appropriate routes on upstream devices. For BGP configuration assistance, refer to the BGP Overview.

Managing certs and keys

Read this section carefully to ensure the SSL/TLS certificates and keys referenced by the F5SPKIngressHTTP2 CR are encoded and installed into the cluster properly. These bullet points are essential:

  • When installing the SPK Controller you must set tmm.tlsStore.enabled paramter to true.
  • The SSL/TLS certificates and keys must be Base64 encoded, and stored in a Secret named tls-keys-certs-secret.
  • TMM mounts the Secret named tls-keys-certs-secret to the file path file://etc/ssl/tls-keys-certs/.

_images/spk_warn.png Important: The tls-keys-certs-secret Secret must be created before the SPK Controller is installed, otherwise the mount will fail and cause the TMM to enter a restart loop.

Use the steps below to generate a new SSL/TLS certficate and key, Base64 encode them, and then create the tls-keys-certs-secret Secret to store them in the cluster. F5 recommends using SSL/TLS certificates signed by a well-known certificate authority (CA) for production application traffic.

_images/spk_info.png Note: Use steps 4 - 6 if you already have an existing SSL/TLS certificate and key pair.

  1. Generate the CA signing certificate and key:

    openssl genrsa -out ca.key 4096
    
    openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.crt \
    -subj "/C=US/ST=WA/L=Seattle/O=F5/OU=Dev/CN=ca"
    
  2. Generate the clientssl profile SSL/TLS certificate signing request (CSR):

    openssl genrsa -out client.key 4096
    
    openssl req -new -key client.key -out client.csr \
    -subj "/C=US/ST=WA/L=Seattle/O=F5/OU=PD/CN=client.com"
    
  3. Sign the clientssl profile CSR with the CA:

    openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key \
    -set_serial 101 -outform PEM -out client.crt -extensions req_ext -days 365 -sha256
    
  4. Base64 encode the SSL/TLS certificate and key:

    openssl base64 -A -in client.crt -out client-encode.crt
    openssl base64 -A -in client.key -out client-encode.key
    
  5. Create the tls-keys-certs-secret Secret that stores the SSL/TLS certificate and key:

    echo "apiVersion: v1" > tls-keys-certs-secret.yaml
    echo "kind: Secret" >> tls-keys-certs-secret.yaml
    echo "metadata:" >> tls-keys-certs-secret.yaml
    echo " name: tls-keys-certs-secret" >> tls-keys-certs-secret.yaml
    echo "data:" >> tls-keys-certs-secret.yaml
    echo -n " client.crt: " >> tls-keys-certs-secret.yaml
    cat client-encode.crt >> tls-keys-certs-secret.yaml
    echo " " >> tls-keys-certs-secret.yaml
    echo -n " client.key: " >> tls-keys-certs-secret.yaml
    cat client-encode.key >> tls-keys-certs-secret.yaml
    
  6. Install the Secret into the SPK Controller Project:

    oc apply -f tls-keys-certs-secret.yaml -n spk-ingress
    

Requirements

Ensure you have:

  • Installed a K8S Service object and application.
  • Installed the SPK Controller.
  • A Linux based workstation.
  • Installed the dSSM Database to support persistence records.

Installation

Use the following steps to obtain the application’s Service object information to configure and install the F5SPKIngressTCP CR.

  1. Switch to the application Project:

    oc project <project>
    

    In this example, the application is in the spk-app Project:

    oc project spk-app
    
  2. Use the Service object NAME and PORT to configure the CR service.name and service.port parameters:

    oc get service 
    

    In this example, the Service object NAMEs are http2-web-a and http2-web-b and the PORT is 443:

    NAME          TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S) 
    http2-web-a   NodePort   10.99.99.91   <none>        443:443/TCP
    http2-web-b   NodePort   10.99.99.99   <none>        443:443/TCP
    
  3. Copy the example F5SPKIngressHTTP2 CR into a YAML file:

    apiVersion: "k8s.f5net.com/v1"
    kind: F5SPKIngressHTTP2
    metadata:
      name: http2-app
      namespace: spk-app
    clientssl:
      keyCertPairs:
      - key: file://etc/ssl/tls-keys-certs/client.key
        cert: file://etc/ssl/tls-keys-certs/client.crt
      ciphers: "DEFAULT"
    spec:
      destinationAddress: 10.20.2.206
      destinationPort: 80
      v6destinationAddress: 2002::10:20:2:206
      irule: |
        proc insert_http_header_a {} {
          log local0. "insert a new HTTP header"
          HTTP::header insert "x-custom-proc" "sbi-A"
          set acctVal [MESSAGE::field value ":JSON:account"]
          log local0. "#########  acctValA = $acctVal"
        }
        proc insert_http_header_b {} {
          log local0. "insert a new HTTP header"
          HTTP::header insert "x-custom-proc" "sbi-B"
          set acctVal [MESSAGE::field value ":JSON:account"]
          log local0. "#########  acctValB = $acctVal"
        }
      ipfamilies: IPv4andIPv6
      loadBalancingMethod: "ROUND_ROBIN"
      services:
        - name: http-web-a
          port: 443
      services:
        - name: http-web-b
          port: 443
      staticRoutes:
      - customIruleProc: "insert_http_header_a"
        persistBidirectional: false
        persistField: "${:JSON:account}"
        persistTimeout: 60
        service: "web-a"
        conditions:
        - fieldName: ":JSON:account"
          comparisonOp: "SR_COMPARE_EQUALS"
          values:
          - "account1"
          caseSensitive: true
      - customIruleProc: "insert_http_header_b"
        persistBidirectional: false
        persistField: "${:JSON:account}"
        persistTimeout: 60
        service: "web-b"
        conditions:
        - fieldName: ":JSON:account"
          comparisonOp: "SR_COMPARE_EQUALS"
          values:
          - "account2"
          caseSensitive: true
    
  4. Install the F5SPKIngressHTTP2 CR:

    oc apply -f spk-ingress-http2.yaml
    
  5. Web clients should now be able to connect to the application through the Service Proxy TMM.

Connection statistics

If you installed the SPK Controller with the Debug Sidecar enabled, connect to the sidecar to view virtual server and pool member connectivity statistics.

  1. Log in to the Service Proxy Debug container:

    oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash
    
  2. View the virtual server connection statistics:

    tmctl -d blade virtual_server_stat -s name,clientside.tot_conns
    

    For example:

    name                                    serverside.tot_conns
    --------------------------------------- --------------------
    spk-apps-nginx-http2-crd-virtual-server                 31
    
  3. View the load balancing pool connection statistics:

    tmctl -d blade pool_member_stat -s pool_name,serverside.tot_conns
    

    For example:

    web-apps-http2-crd-pool                        15
    web-apps-http2-crd-pool                        16
    

Feedback

Provide feedback to improve this document by emailing spkdocs@f5.com.

Supplemental