Custom Resource Definitions

This page describes CIS in CRD Mode. This is an Alpha release that supports limited features.

What are CRDs?

  • Custom resources are extensions of the Kubernetes API.
  • A resource is an endpoint in the Kubernetes API that stores a collection of API objects. For example, the built-in pods resource contains a collection of Pod objects.
  • A custom resource is an extension of the Kubernetes API that is not necessarily available in a default Kubernetes installation. It represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.
  • Custom resources can appear and disappear in a running cluster through dynamic registration, and cluster admins can update custom resources independently of the cluster itself. Once a custom resource is installed, users can create and access its objects using kubectl, just as they do for built-in resources like Pods.
  • CIS supports 2 Custom Resources:
    • VirtualServer
    • TLSProfile

VirtualServer

VirtualServer resource defines load balancing configuration for a domain name.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
 apiVersion: "cis.f5.com/v1"
 kind: VirtualServer
 metadata:
   name: coffee-virtual-server
   labels:
     f5cr: "true"
 spec:
   host: coffee.example.com
   virtualServerAddress: "172.16.3.4"
   pools:
   - path: /coffee
     service: svc-2
     servicePort: 80

Label

CIS will only process custom resources with f5cr Label as true.

labels:
  f5cr: "true"

Important

The above VirtualServer is insecure. Attach a TLSProfile to make it secure.

TLSProfile

TLSProfile is used to specify the TLS termination for a single/list of services in a VirtualServer Custom Resource. TLS termination relies on SNI. Any non-SNI traffic received on port 443 may result in connection issues. TLSProfile can be created either with certificates stored as k8s secrets or can be referenced by profiles existing in BIG-IP.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
 apiVersion: cis.f5.com/v1
 kind: TLSProfile
 metadata:
   name: reencrypt-tls
   labels:
     f5cr: "true"
 spec:
   tls:
     termination: reencrypt
     clientSSL: /common/clientssl
     serverSSL: /common/serverssl
     reference: bigip             # --> reference profiles created in BIG-IP by User
   hosts:
   - coffee.example.com

VirtualServer with TLSProfile

VirtualServer with TLSProfile is used to specify the TLS termination. TLS termination relies on SNI. Any non-SNI traffic received on port 443 may result in connection issues. The example below shows how to attach a TLSProfile to a VirtualServer.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
 apiVersion: cis.f5.com/v1
 kind: VirtualServer
 metadata:
   name: coffee-virtual-server
   labels:
     f5cr: "true"
   namespace: default
 spec:
   host: coffee.example.com
   tlsProfileName: reencrypt-tls.  # --> This will attach reencrypt-tls TLSProfile
   virtualServerAddress: "172.16.3.4"
   pools:
     - path: /coffee
       service: svc
       servicePort: 80
  • CIS has a 1:1 mapping for a domain (CommonName) and BIG-IP-VirtualServer.
  • You can create any number of custom resources for a single domain. For example, you can create 2 VirtualServers with different terminations (for the same domain): one with edge and another with re-encrypt. To do this you need to create two VirtualServers: one with edge TLSProfile and another with re-encrypt TLSProfile. Both the VirutalServers should be created with same virtualServerAddress.
  • Single or Group of VirtualServers (with the same virtualServerAddress) will be created as one common BIG-IP-VirtualServer.
  • If you want to update secure virtual (TLS Virtual) server to insecure virtual (non-TLS server) server. You need to delete the secure virtual server first and create a new virtual server.

How CIS works with CRDs

  • CIS registers to the Kubernetes client-go using informers to retrieve Virtual Server, TLSProfile, Service, Endpoint and Node create, update and delete events. Resources identified from such events will be pushed to a Resource Queue maintained by CIS.
  • Resource Queue holds the resources to be processed.
  • Virtual Server is the Primary citizen. Any changes in TLSProfile, Service, Endpoint, Node will process their affected Virtual Servers. For example, If svc-a is part of foo-VirtualServer and bar-VirtualServer, any changes in svc-a will put foo-VirtualServer and bar-VirtualServer in resource queue.
  • Worker fetches the affected Virtual Servers from Resource Queue to populate a common structure which holds the configuration of all the Virtual Servers such as TLSProfile, Virtual Server IP, Pool Members and L7 LTM policy actions.
  • VXLAN Manager prepares the BIG-IP NET configuration as AS3 cannot process FDB and ARP entries.
  • LTM Configuration (using AS3) and NET Configuration (using CCCL) will be created in CIS Managed Partition defined by the user.

VirtualServer

Open API Schema Validation

VirtualServer Components

Parameter Type Required Default Description
host String Required N/A Virtual Host
pools List of pool Required N/A List of BIG-IP Pool members
virtualServerAddress String Required N/A IP Address of BIG-IP Virtual Server
TLSProfile String Required N/A Describes the TLS configuration for BIG-IP Virtual Server

Pool Components

Parameter Type Required Default Description
path String Required N/A Path to access the service.
service String Required N/A Service deployed in Kubernetes cluster
nodeMemberLabel String Optional N/A List of Nodes to consider in NodePort mode as BIG-IP pool members. This option is only applicable for NodePort mode.
servicePort String Required N/A Port to access service
monitor String Optional N/A Health Monitor to check the health of Pool Members

Health Monitor

Parameter Type Required Default Description
type String Required N/A http or https
send String Optional GET /rn HTTP request string to send
recv String Optional N/A String or RegEx pattern to match in first 5,120 bytes of backend response.
interval Int Required 5 Seconds between health queries
timeout Int Optional 16 Seconds before query fails

Note

Health Monitor associated with the first path will be considere if multiple path has same backend

TLSProfile

Open API Schema Validation

TLSProfile Components

Parameter Type Required Default Description
termination String Required N/A Termination on BIG-IP Virtual Server. Allowed options are [edge, reencrypt, passthrough]
clientSSL String Required N/A ClientSSL Profile on the BIG-IP. Example /Common/clientssl
serverSSL String Optional N/A ServerSSL Profile on the BIG-IP. Example /Common/serverssl
reference String Required N/A Describes the location of profile, BIG-IP or k8s Secrets. We currently support BIG-IP profiles only

Installation

Since CIS 2.0 uses the AS3 declarative API, you will need to have the AS3 extension version 3.18 or newer installed on BIG-IP. See instructions for installing AS3 on BIG-IP.

Create CIS Controller, BIG-IP Credentials, and RBAC Authentication

  1. Install F5 CRDs. Download this YAML file and run the following commands.

    kubectl create -f customresourcedefinitions.yml [-n kube-system]
    
  2. Create BIG-IP Credentials:

    kubectl create secret generic bigip-login -n kube-system --from-literal=username=admin --from-literal=password=dummy
    
  3. Create Service Account

    kubectl create serviceaccount bigip-ctlr [-n kube-system]
    
  4. Create Cluster Role and Cluster Role Binding. Download this YAML file and run the following command.

    kubectl create -f clusterrole.yml [-n kube-system]
    

Supported Controller Modes: NodePort and Cluster

Deploy k8s-bigip-ctlr in nodeport and customresource mode.

Download this YAML file and run the following command.

kubectl create -f sample-nodeport-k8s-bigip-ctlr-crd-secret.yml [-n kube-system]

Cluster Mode

Add BIG-IP device to VXLAN

kubectl create -f sample-cluster-k8s-bigip-ctlr-crd-secret.yml [-n kube-system]

Examples Repository

View more examples on GitHub.

Notes

  • --custom-resource-mode=true deploys CIS in Custom Resource Mode.
  • CIS does not watch for Ingress/Routes/ConfigMaps when deployed in CRD Mode.
  • CIS does not support combination of CRDs with any of Ingress/Routes and ConfigMaps.