Custom Resource Definitions

This page describes CIS in CRD Mode.

What are CRDs?

  • Custom resources are extensions of the Kubernetes API.
  • A resource is an endpoint in the Kubernetes API that stores a collection of API objects. For example, the built-in pods resource contains a collection of Pod objects.
  • A custom resource is an extension of the Kubernetes API that is not necessarily available in a default Kubernetes installation. It represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.
  • Custom resources can appear and disappear in a running cluster through dynamic registration, and cluster admins can update custom resources independently of the cluster itself. Once a custom resource is installed, users can create and access its objects using kubectl, just as they do for built-in resources like Pods.
  • CIS supports the following Custom Resources:
    • VirtualServer
    • TLSProfile
    • TransportServer

VirtualServer

VirtualServer resource defines load balancing configuration for a domain name.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
 apiVersion: "cis.f5.com/v1"
 kind: VirtualServer
 metadata:
   name: coffee-virtual-server
   labels:
     f5cr: "true"
 spec:
   host: coffee.example.com
   virtualServerAddress: "172.16.3.4"
   pools:
   - path: /coffee
     service: svc-2
     servicePort: 80

Label

CIS will only process custom resources with f5cr Label as true.

labels:
  f5cr: "true"

Important

The above VirtualServer is insecure. Attach a TLSProfile to make it secure.

TLSProfile

TLSProfile is used to specify the TLS termination for a single/list of services in a VirtualServer Custom Resource. TLS termination relies on SNI. Any non-SNI traffic received on port 443 may result in connection issues. TLSProfile can be created either with certificates stored as k8s secrets or can be referenced by profiles existing in BIG-IP.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
 apiVersion: cis.f5.com/v1
 kind: TLSProfile
 metadata:
   name: reencrypt-tls
   labels:
     f5cr: "true"
 spec:
   tls:
     termination: reencrypt
     clientSSL: /common/clientssl
     serverSSL: /common/serverssl
     reference: bigip             # --> reference profiles created in BIG-IP by User
   hosts:
   - coffee.example.com

VirtualServer with TLSProfile

VirtualServer with TLSProfile is used to specify the TLS termination. TLS termination relies on SNI. Any non-SNI traffic received on port 443 may result in connection issues. The example below shows how to attach a TLSProfile to a VirtualServer.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
 apiVersion: cis.f5.com/v1
 kind: VirtualServer
 metadata:
   name: coffee-virtual-server
   labels:
     f5cr: "true"
   namespace: default
 spec:
   host: coffee.example.com
   tlsProfileName: reencrypt-tls.  # --> This will attach reencrypt-tls TLSProfile
   virtualServerAddress: "172.16.3.4"
   pools:
     - path: /coffee
       service: svc
       servicePort: 80
  • CIS has a 1:1 mapping for a domain (CommonName) and BIG-IP-VirtualServer.
  • You can create any number of custom resources for a single domain. For example, you can create 2 VirtualServers with different terminations (for the same domain): one with edge and another with re-encrypt. To do this you need to create two VirtualServers: one with edge TLSProfile and another with re-encrypt TLSProfile. Both the VirutalServers should be created with same virtualServerAddress.
  • Single or Group of VirtualServers (with the same virtualServerAddress) will be created as one common BIG-IP-VirtualServer.
  • If you want to update secure virtual (TLS Virtual) server to insecure virtual (non-TLS server) server. You need to delete the secure virtual server first and create a new virtual server.

Transport Server

The TransportServer resource exposes the non-HTTP traffic configuration for a virtual server address in BIG-IP.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
apiVersion: "cis.f5.com/v1"
kind: TransportServer
metadata:
   name: transport-server
   labels:
   f5cr: "true"
spec:
   virtualServerAddress: "172.16.3.9"
   virtualServerPort: 8585
   mode: standard
   snat: auto
   pool:
     service: svc-3
     servicePort: 8181
     monitor:
       type: tcp
       interval: 10
       timeout: 10

Important

The TransportServer in this example only forwards the traffic. It is the user’s responsibility to implement secure non-http traffic.


How CIS works with CRDs

  • CIS registers to the Kubernetes client-go using informers to retrieve Virtual Server, TLSProfile, Service, Endpoint and Node create, update and delete events. Resources identified from such events will be pushed to a Resource Queue maintained by CIS.
  • Resource Queue holds the resources to be processed.
  • Virtual Server is the Primary citizen. Any changes in TLSProfile, Service, Endpoint, Node will process their affected Virtual Servers. For example, If svc-a is part of foo-VirtualServer and bar-VirtualServer, any changes in svc-a will put foo-VirtualServer and bar-VirtualServer in resource queue.
  • Worker fetches the affected Virtual Servers from Resource Queue to populate a common structure which holds the configuration of all the Virtual Servers such as TLSProfile, Virtual Server IP, Pool Members and L7 LTM policy actions.
  • VXLAN Manager prepares the BIG-IP NET configuration as AS3 cannot process FDB and ARP entries.
  • LTM Configuration (using AS3) and NET Configuration (using CCCL) will be created in CIS Managed Partition defined by the user.

VirtualServer

Open API Schema Validation for VirtualServer

VirtualServer Components

Parameter Type Required Default Description
host String Required N/A Virtual Host.
pools List of pool Required N/A List of BIG-IP Pool members.
virtualServerAddress String Required N/A IP Address of BIG-IP Virtual Server.
TLSProfileName String Required N/A Describes the TLS configuration for BIG-IP Virtual Server.
waf String Optional N/A Reference to WAF policy on BIG-IP.
snat String Optional auto Reference to SNAT pool on BIG-IP. The other allowed value is: none.

Pool Components

Parameter Type Required Default Description
path String Required N/A Path to access the service.
service String Required N/A Service deployed in Kubernetes cluster.
nodeMemberLabel String Optional N/A List of Nodes to consider in NodePort mode as BIG-IP pool members. This option is only applicable for NodePort mode.
servicePort String Required N/A Port to access service.
monitor String Optional N/A Health Monitor to check the health of Pool Members.

Health Monitor

Parameter Type Required Default Description
type String Required N/A http or https
send String Required GET /rn HTTP request string to send.
recv String Optional N/A String or RegEx pattern to match in first 5,120 bytes of backend response.
interval Int Required 5 Seconds between health queries.
timeout Int Optional 16 Seconds before query fails.

Note

Health Monitor associated with the first path will be considered if multiple paths have the same backend.

TLSProfile

Open API Schema Validation

TLSProfile Components

Parameter Type Required Default Description
termination String Required N/A Termination on BIG-IP Virtual Server. Allowed options are edge, reencrypt, and passthrough.
clientSSL String Required N/A ClientSSL Profile on the BIG-IP. For example /Common/clientssl.
serverSSL String Optional N/A ServerSSL Profile on the BIG-IP. For example /Common/serverssl.
reference String Required N/A Describes the location of profile: BIG-IP, or k8s Secrets. CIS currently supports BIG-IP and secret references. Available options are bigip and secret.

Installation

Since CIS 2.0 uses the AS3 declarative API, you will need to have the AS3 extension version 3.18 or newer installed on BIG-IP. See instructions for installing AS3 on BIG-IP.

Create CIS Controller, BIG-IP Credentials, and RBAC Authentication

  1. Install F5 CRDs. Download customresourcedefinitions.yml and run the following commands.

    kubectl create -f customresourcedefinitions.yml [-n kube-system]
    
  2. Create BIG-IP Credentials:

    kubectl create secret generic bigip-login -n kube-system --from-literal=username=admin --from-literal=password=dummy
    
  3. Create Service Account

    kubectl create serviceaccount bigip-ctlr [-n kube-system]
    
  4. Create Cluster Role and Cluster Role Binding. Download clusterrole.yml and run the following command.

    kubectl create -f clusterrole.yml [-n kube-system]
    

Supported Controller Modes: NodePort and Cluster

Deploy k8s-bigip-ctlr in nodeport and customresource mode.

Download sample-nodeport-k8s-bigip-ctlr-crd-secret.yml and run the following command.

kubectl create -f sample-nodeport-k8s-bigip-ctlr-crd-secret.yml [-n kube-system]

Cluster Mode

Add BIG-IP device to VXLAN

kubectl create -f sample-cluster-k8s-bigip-ctlr-crd-secret.yml [-n kube-system]

Custom Virtual Server Name

CRD allows the user to create a custom name for the virtual servers on BIG-IP using the virtualServerName parameter.

By deploying this yaml file in your cluster, CIS will create a Virtual Server on BIG-IP as “<virtual server name>_<virtual server port>”. For example: cafe_virtual_server_80.

This is optional to use. The default name for a virtual server created on BIG-IP is “crd_<virtual IP address>_<virtual server port>”. For example: crd_172_16_3_4_80.

custom-virtual-name.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
apiVersion: "cis.f5.com/v1"
kind: VirtualServer
metadata:
  name: my-new-virtual-server
  labels:
    f5cr: "true"
spec:
  # This is an insecure virtual, Please use TLSProfile to secure the virtual
  # check out tls examples to understand more.
  host: cafe.example.com
  virtualServerAddress: "172.16.3.4"
  virtualServerName: "cafe-virtual-server"
  pools:
  - path: /coffee
    service: svc-2
    servicePort: 80

Custom Virtual Port in CRD

You can configure the virtual address port number in CRD. This is required if you want to use the same VIP with different port numbers for different domains. There are two options for configuring:

virtualServerHTTPPort

By deploying this yaml file in your cluster, CIS will create a Virtual Server on BIG-IP with VIP custom http port as 500. It will load balance the traffic for domain cafe.example.com.

custom-http-port.yml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
apiVersion: "cis.f5.com/v1"
kind: VirtualServer
metadata:
  name: my-new-virtual-server
  labels:
    f5cr: "true"
spec:
  host: cafe.example.com
  virtualServerAddress: "172.16.3.4"
  virtualServerHTTPPort: 8080
  pools:
  - path: /coffee
    service: svc-2
    servicePort: 80

virtualServerHTTPSPort

By deploying this yaml file in your cluster, CIS will create a Virtual Server on BIG-IP with VIP custom https port as 500. It will load balance the traffic for domain cafe.example.com.

custom-https-port.yml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
apiVersion: "cis.f5.com/v1"
kind: VirtualServer
metadata:
  name: my-new-virtual-server
  labels:
    f5cr: "true"
spec:
  host: cafe.example.com
  virtualServerAddress: "172.16.3.4"
  virtualServerHTTPSPort: 8443
  pools:
  - path: /coffee
    service: svc-2
    servicePort: 80

Virtual Server Custom Resource without Host Parameter

You can create a simple HTTP Virtual Server without the Host parameter. By deploying the following YAML file in your cluster, CIS will create a Virtual Server on BIG-IP with VIP 172.16.3.4 and attach a policy that forwards the traffic to pool svc-1 when the URI path segment is /coffee.

Note

This is an insecure virtual server, please use TLSProfile to secure the virtual.

noHost-single-pool-virtual.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
apiVersion: "cis.f5.com/v1"
kind: VirtualServer
metadata:
  name: hoHost-single-pool
  labels:
    f5cr: "true"
spec:
  # This is an insecure virtual, Please use TLSProfile to secure the virtual
  # check out tls examples to understand more.
  virtualServerAddress: "172.16.3.4"
  pools:
  - path: /coffee
    service: svc-1
    servicePort: 80

Examples Repository

View more examples on GitHub.

Notes

  • --custom-resource-mode=true deploys CIS in Custom Resource Mode.
  • CIS does not watch for Ingress/Routes/ConfigMaps when deployed in CRD Mode.
  • CIS does not support combination of CRDs with any of Ingress/Routes and ConfigMaps.