Custom Resource Definitions

This page describes CIS in CRD Mode.

What are CRDs?

  • Custom resources are extensions of the Kubernetes API.
  • A resource is an endpoint in the Kubernetes API that stores a collection of API objects. For example, the built-in pods resource contains a collection of Pod objects.
  • A custom resource is an extension of the Kubernetes API that is not necessarily available in a default Kubernetes installation. It represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.
  • Custom resources can appear and disappear in a running cluster through dynamic registration, and cluster admins can update custom resources independently of the cluster itself. Once a custom resource is installed, users can create and access its objects using kubectl, just as they do for built-in resources like Pods.
  • CIS supports the following Custom Resources:

How CIS works with CRDs

  • CIS registers to the Kubernetes client-go using informers to retrieve Virtual Server, TLSProfile, Service, Endpoint and Node create, update and delete events. Resources identified from such events will be pushed to a Resource Queue maintained by CIS.
  • Resource Queue holds the resources to be processed.
  • Virtual Server is the Primary citizen. Any changes in TLSProfile, Service, Endpoint, Node will process their affected Virtual Servers. For example, If svc-a is part of foo-VirtualServer and bar-VirtualServer, any changes in svc-a will put foo-VirtualServer and bar-VirtualServer in resource queue.
  • Worker fetches the affected Virtual Servers from Resource Queue to populate a common structure which holds the configuration of all the Virtual Servers such as TLSProfile, Virtual Server IP, Pool Members and L7 LTM policy actions.
  • VXLAN Manager prepares the BIG-IP NET configuration as AS3 cannot process FDB and ARP entries.
  • LTM Configuration (using AS3) and NET Configuration (using CCCL) is created in CIS Managed Partition defined by the user.

Important

  • You must configure the CRD schema before creating CIS. Run the command kubectl create -f customresourcedefinitions.yml [-n kube-system] after you install F5 CRDs.
  • --custom-resource-mode=true deploys CIS in Custom Resource Mode.
  • CIS does not watch for Ingress/Routes/ConfigMaps when deployed in CRD Mode.
  • CIS does not support combination of CRDs with any of Ingress/Routes and ConfigMaps.

Examples Repository

View more examples on GitHub.

Valid Configuration Parameters for CRDs

AS3 Parameters

Parameter Type Required Default Description
as3-validation Boolean Optional true When set to false, this disables AS3 template validation on the controller.
insecure Boolean Optional false When set to true, this enables insecure SSL communication to the BIG-IP system.
trusted-certs-cfgmap String Required N/A When certificates are provided, adds them to controller’s trusted certificate store.
log-as3-response Boolean Optional false When set to true, adds the body of AS3 API response in Controller logs.

Kubernetes Parameters

Parameter Type Required Default Description Allowed Values Agent
kubeconfig String Optional ./config Path to the kubeconfig file   Both AS3 and CCCL
namespace String Optional All

Kubernetes namespace(s) to watch

  • To specify multiple namespace, use multiple --namespace flags.
  • Watches all namespaces by default.
   Both AS3 and CCCL
namespace-label String Optional N/A Tells the k8s-bigip-ctlr to watch any namespace with this label   Both AS3 and CCCL
node-label-selector String Optional N/A Tells the k8s-bigip-ctlr to watch only nodes with this label   Both AS3 and CCCL
pool-member-type String Optional nodeport

The type of BIG-IP pool members you want to create.

Use cluster to create pool members for each of the endpoints for the Service (the pod’s InternalIP)

Use nodeport to create pool members for each schedulable node using the Service’s NodePort.

 cluster, nodeport Both AS3 and CCCL
running-in-cluster Boolean Optional true Indicates whether or not a kubernetes cluster started k8s-bigip-ctlr true, false Both AS3 and CCCL
use-node-internal Boolean Optional true filter Kubernetes InternalIP addresses for pool members true, false Both AS3 and CCCL

General Parameters

Parameter Type Required Default Description Allowed Values Agent
http-listen-address String Optional “0.0.0.0:8080” Address at which to serve HTTP-based information (for example, /metrics, health) to `Prometheus`_.   Both AS3 and CCCL
log-level String Optional INFO Log level INFO, DEBUG, CRITICAL, WARNING, ERROR Both AS3 and CCCL
node-poll-interval Integer Optional 30 In seconds, the interval at which the CIS polls the cluster to find all node members.   Both AS3 and CCCL
python-basedir String Optional /app/python Path to the python utilities directory.   CCCL
schema-db-base-dir String Optional file:///app/vendor/src/f5/schemas Path to the directory containing the F5 schema db.   CCCL
verify-interval Integer N/A 30 In seconds, the interval at which the CIS verifies that the BIG-IP configuration matches the state of the orchestration system.  

For CCCL, LTM and NET

For AS3, only NET
agent String Optional AS3 You can also change the value to CCCL for CCCL mode. AS3, CCCL Both AS3 and CCCL
version Boolean Optional false Print CIS version   Both AS3 and CCCL

BIG-IP system Parameters

Parameter Type Required Default Description Allowed Values Agent
bigip-partition String Required N/A The BIG-IP partition in which to configure objects.   Both AS3 and CCCL
bigip-password String Required N/A

BIG-IP iControl REST password

You can secure your BIG-IP credentials using a Kubernetes Secret.

   Both AS3 and CCCL
bigip-url String Required N/A BIG-IP admin IP address   Both AS3 and CCCL
bigip-username String Required N/A

BIG-IP iControl REST username

The BIG-IP user account must have the appropriate role defined:

For nodeport type pool members, the role must be Administrator.

For cluster type pool members, the role must be Administrator.

   Both AS3 and CCCL
credentials-directory String Optional N/A Directory that contains the BIG-IP username, password, or url files   Both AS3 and CCCL

VXLAN Parameters

Parameter Type Required Default Description Allowed Values Agent
openshift-sdn-name String Optional N/A

Name of the VXLAN tunnel on the BIG-IP system that corresponds to an OpenShift SDN HostSubnet.

Only applicable in OpenShift.

   Both AS3 and CCCL
flannel-name String Optional N/A Name of the VXLAN tunnel on the BIG-IP system that corresponds to a Flannel subnet.   CCCL

AS3 ConfigMap

The AS3 ConfigMap hosts AS3 extensions, in JSON format, as a configuration artifact. CIS can manage and orchestrate BIG-IP declaratively through this ConfigMap.

In agent AS3 mode, CIS handles Ingress or Route resources by converting them into AS3 declarations before posting to BIG-IP. When AS3 ConfigMap is configured along with Ingress or Routes, CIS manages ConfigMap and Ingress (or) Routes AS3 declarations separately. While sending an AS3 declaration to BIG-IP, CIS will combine both of these AS3 declarations as a single declaration and POST it to BIG-IP.

CIS does not watch for Ingress/Routes/ConfigMaps when deployed in CRD Mode.

CIS does not support the combination of CRDs with any of Ingress/Routes and ConfigMaps. So the migration from AS3 CM to CRD can cause some downtime as well.

Below is an example of how to do use AS3 CM and CRD for the same set of requirements.

configmap.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
 # Please edit the object below. Lines beginning with a '#' will be ignored,
 # and an empty file will abort the edit. If an error occurs while saving this file will be
 # reopened with the relevant failures.
 apiVersion: v1
 data:
   template: |
     {"class": "AS3", "action": "deploy", "persist": true, "declaration": {"class": "ADC", "schemaVersion": "3.23.0", "id": "urn:uuid:33045210-3ab8-4636-9b2a-c98d22ab915d-pytest2", "label": "pytest2", "remark": "AS3 Declation for pytest2", "pytest2_tnt_0": {"class": "Tenant", "pytest2_tnt_0_https0": {"class": "Application", "template": "https", "serviceMain": {"class": "Service_HTTPS", "virtualAddresses": ["172.16.3.9"], "pool": "pytest2_tnt_0_https0_p0", "serverTLS": "pytest2_tnt_0_https0_ssl_0", "persistenceMethods": []}, "pytest2_tnt_0_https0_p0": {"class": "Pool", "monitors": ["http"], "members": [{"servicePort": 80, "shareNodes": false, "serverAddresses": []}]}, "pytest2_tnt_0_https0_ssl_0": {"class": "TLS_Server", "certificates": [{"certificate": "pytest2_tnt_0_https0_cert_0"}]}, "pytest2_tnt_0_https0_cert_0": {"class": "Certificate", "remark": "in practice we recommend using a passphrase", "certificate": "", "privateKey": "", "passphrase": {"ciphertext": "ZjVmNQ==", "protected": "eyJhbGciOiJkaXIiLCJlbmMiOiJub25lIn0", "ignoreChanges": true}}}}}}
 kind: ConfigMap
 metadata:
   labels:
     as3: "true"
     f5type: virtual-server
   managedFields:
   - apiVersion: v1
     fieldsType: FieldsV1
     fieldsV1:
       f:data:
         .: {}
         f:template: {}
       f:metadata:
         f:labels:
           .: {}
           f:as3: {}
           f:f5type: {}
     manager: kubectl
     operation: Update
   name: pytest2
   namespace: default

service.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
 # Please edit the object below. Lines beginning with a '#' will be ignored,
 # and an empty file will abort the edit. If an error occurs while saving this file will be
 # reopened with the relevant failures.
 #
 apiVersion: v1
 kind: Service
 metadata:
   creationTimestamp: "2020-11-02T11:17:35Z"
   labels:
     cis.f5.com/as3-app: pytest2_tnt_0_https0
     cis.f5.com/as3-pool: pytest2_tnt_0_https0_p0
     cis.f5.com/as3-tenant: pytest2_tnt_0
   managedFields:
   - apiVersion: v1
     fieldsType: FieldsV1
     fieldsV1:
       f:metadata:
         f:labels:
           .: {}
           f:cis.f5.com/as3-app: {}
           f:cis.f5.com/as3-pool: {}
           f:cis.f5.com/as3-tenant: {}
       f:spec:
         f:externalTrafficPolicy: {}
         f:ports:
           .: {}
           k:{"port":80,"protocol":"TCP"}:
             .: {}
             f:name: {}
             f:port: {}
             f:protocol: {}
             f:targetPort: {}
         f:selector:
           .: {}
           f:app: {}
         f:sessionAffinity: {}
         f:type: {}
     manager: kubectl
     operation: Update
     time: "2020-11-02T11:17:35Z"
   name: pytest2-tnt-0-https0-p0
   namespace: default
   resourceVersion: "1489990"
   selfLink: /api/v1/namespaces/default/services/pytest2-tnt-0-https0-p0
   uid: bfa981a1-b88f-4ad4-a337-0314dcdf0692
 spec:
   clusterIP: 10.105.213.231
   externalTrafficPolicy: Cluster
   ports:
   - name: pytest2-tnt-0-https0-p0-80
     nodePort: 32519
     port: 80
     protocol: TCP
     targetPort: 80
   selector:
     app: pytest2-tnt-0-https0-p0
   sessionAffinity: None
   type: NodePort
 status:
   loadBalancer: {}

deployment.yaml
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
 # Please edit the object below. Lines beginning with a '#' will be ignored,
 # and an empty file will abort the edit. If an error occurs while saving this file will be
 # reopened with the relevant failures.
 #
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   annotations:
     deployment.kubernetes.io/revision: "1"
   creationTimestamp: "2020-11-02T11:17:34Z"
   generation: 1
   labels:
     app: pytest2-tnt-0-https0-p0
   managedFields:
   - apiVersion: apps/v1
     fieldsType: FieldsV1
     fieldsV1:
       f:metadata:
         f:labels:
           .: {}
           f:app: {}
       f:spec:
         f:progressDeadlineSeconds: {}
         f:replicas: {}
         f:revisionHistoryLimit: {}
         f:selector:
           f:matchLabels:
             .: {}
             f:app: {}
         f:strategy:
           f:rollingUpdate:
             .: {}
             f:maxSurge: {}
            f:maxUnavailable: {}
           f:type: {}
         f:template:
           f:metadata:
             f:labels:
               .: {}
               f:app: {}
           f:spec:
             f:containers:
               k:{"name":"pytest2-tnt-0-https0-p0"}:
                 .: {}
                 f:env:
                   .: {}
                   k:{"name":"service_name"}:
                     .: {}
                     f:name: {}
                     f:value: {}
                 f:image: {}
                 f:imagePullPolicy: {}
                 f:name: {}
                 f:ports:
                   .: {}
                   k:{"containerPort":80,"protocol":"TCP"}:
                     .: {}
                     f:containerPort: {}
                     f:protocol: {}
                 f:resources: {}
                 f:terminationMessagePath: {}
                 f:terminationMessagePolicy: {}
             f:dnsPolicy: {}
             f:restartPolicy: {}
             f:schedulerName: {}
             f:securityContext: {}
             f:terminationGracePeriodSeconds: {}
     manager: kubectl
     operation: Update
     time: "2020-11-02T11:17:34Z"
   - apiVersion: apps/v1
     fieldsType: FieldsV1
     fieldsV1:
       f:metadata:
         f:annotations:
           .: {}
           f:deployment.kubernetes.io/revision: {}
       f:status:
         f:availableReplicas: {}
         f:conditions:
           .: {}
           k:{"type":"Available"}:
             .: {}
             f:lastTransitionTime: {}
             f:lastUpdateTime: {}
             f:message: {}
             f:reason: {}
             f:status: {}
             f:type: {}
           k:{"type":"Progressing"}:
             .: {}
             f:lastTransitionTime: {}
             f:lastUpdateTime: {}
             f:message: {}
             f:reason: {}
             f:status: {}
             f:type: {}
         f:observedGeneration: {}
         f:readyReplicas: {}
         f:replicas: {}
         f:updatedReplicas: {}
     manager: kube-controller-manager
     operation: Update
     time: "2020-11-02T11:17:37Z"
   name: pytest2-tnt-0-https0-p0
   namespace: default
   resourceVersion: "1490014"
   selfLink: /apis/apps/v1/namespaces/default/deployments/pytest2-tnt-0-https0-p0
   uid: 5b51a71b-d464-4a77-bb86-ef84e9946818
 spec:
   progressDeadlineSeconds: 600
   replicas: 2
   revisionHistoryLimit: 10
   selector:
     matchLabels:
       app: pytest2-tnt-0-https0-p0
   strategy:
     rollingUpdate:
       maxSurge: 25%
       maxUnavailable: 25%
     type: RollingUpdate
   template:
     metadata:
       creationTimestamp: null
       labels:
         app: pytest2-tnt-0-https0-p0
     spec:
       containers:
       - env:
  - name: service_name
           value: pytest2-tnt-0-https0-p0
         image: f5networksdevel/test-nginx:latest
         imagePullPolicy: Always
         name: pytest2-tnt-0-https0-p0
        ports:
         - containerPort: 80
           protocol: TCP
         resources: {}
         terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
       dnsPolicy: ClusterFirst
       restartPolicy: Always
       schedulerName: default-scheduler
       securityContext: {}
       terminationGracePeriodSeconds: 30