Custom Resource Definitions¶
This page describes CIS in CRD Mode.
What are CRDs?¶
- Custom resources are extensions of the Kubernetes API.
- A resource is an endpoint in the Kubernetes API that stores a collection of API objects. For example, the built-in pods resource contains a collection of Pod objects.
- A custom resource is an extension of the Kubernetes API that is not necessarily available in a default Kubernetes installation. It represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.
- Custom resources can appear and disappear in a running cluster through dynamic registration, and cluster admins can update custom resources independently of the cluster itself. Once a custom resource is installed, users can create and access its objects using kubectl, just as they do for built-in resources like Pods.
- CIS supports the following Custom Resources:
How CIS works with CRDs¶
- CIS registers to the Kubernetes client-go using informers to retrieve Virtual Server, TLSProfile, Service, Endpoint and Node create, update and delete events. Resources identified from such events will be pushed to a Resource Queue maintained by CIS.
- Resource Queue holds the resources to be processed.
- Virtual Server is the Primary citizen. Any changes in TLSProfile, Service, Endpoint, Node will process their affected Virtual Servers. For example, If svc-a is part of foo-VirtualServer and bar-VirtualServer, any changes in svc-a will put foo-VirtualServer and bar-VirtualServer in resource queue.
- Worker fetches the affected Virtual Servers from Resource Queue to populate a common structure which holds the configuration of all the Virtual Servers such as TLSProfile, Virtual Server IP, Pool Members and L7 LTM policy actions.
- VXLAN Manager prepares the BIG-IP NET configuration as AS3 cannot process FDB and ARP entries.
- LTM Configuration (using AS3) and NET Configuration (using CCCL) is created in CIS Managed Partition defined by the user.
Important
- You must configure the CRD schema before creating CIS. Run the command
kubectl create -f customresourcedefinitions.yml [-n kube-system]
after you install F5 CRDs. --custom-resource-mode=true
deploys CIS in Custom Resource Mode.- CIS does not watch for Ingress/Routes/ConfigMaps when deployed in CRD Mode.
- CIS does not support combination of CRDs with any of Ingress/Routes and ConfigMaps.
Examples Repository¶
Valid Configuration Parameters for CRDs¶
AS3 Parameters¶
Parameter | Type | Required | Default | Description |
---|---|---|---|---|
as3-validation | Boolean | Optional | true | When set to false, this disables AS3 template validation on the controller. |
insecure | Boolean | Optional | false | When set to true, this enables insecure SSL communication to the BIG-IP system. |
trusted-certs-cfgmap | String | Required | N/A | When certificates are provided, adds them to controller’s trusted certificate store. |
log-as3-response | Boolean | Optional | false | When set to true, adds the body of AS3 API response in Controller logs. |
Kubernetes Parameters¶
Parameter | Type | Required | Default | Description | Allowed Values | Agent |
---|---|---|---|---|---|---|
kubeconfig | String | Optional | ./config | Path to the kubeconfig file | Both AS3 and CCCL | |
namespace | String | Optional | All | Kubernetes namespace(s) to watch
|
Both AS3 and CCCL | |
namespace-label | String | Optional | N/A | Tells the k8s-bigip-ctlr to watch
any namespace with this label |
Both AS3 and CCCL | |
node-label-selector | String | Optional | N/A | Tells the k8s-bigip-ctlr to watch
only nodes with this label |
Both AS3 and CCCL | |
pool-member-type | String | Optional | nodeport | The type of BIG-IP pool members you want to create. Use Use |
cluster, nodeport | Both AS3 and CCCL |
running-in-cluster | Boolean | Optional | true | Indicates whether or not a
kubernetes cluster started
k8s-bigip-ctlr |
true, false | Both AS3 and CCCL |
use-node-internal | Boolean | Optional | true | filter Kubernetes InternalIP addresses for pool members | true, false | Both AS3 and CCCL |
General Parameters¶
Parameter | Type | Required | Default | Description | Allowed Values | Agent |
---|---|---|---|---|---|---|
http-listen-address | String | Optional | “0.0.0.0:8080” | Address at which to serve HTTP-based
information (for example, /metrics ,
health ) to `Prometheus`_. |
Both AS3 and CCCL | |
log-level | String | Optional | INFO | Log level | INFO, DEBUG, CRITICAL, WARNING, ERROR | Both AS3 and CCCL |
node-poll-interval | Integer | Optional | 30 | In seconds, the interval at which the CIS polls the cluster to find all node members. | Both AS3 and CCCL | |
python-basedir | String | Optional | /app/python | Path to the python utilities directory. | CCCL | |
schema-db-base-dir | String | Optional | file:///app/vendor/src/f5/schemas | Path to the directory containing the F5 schema db. | CCCL | |
verify-interval | Integer | N/A | 30 | In seconds, the interval at which the CIS verifies that the BIG-IP configuration matches the state of the orchestration system. | For CCCL, LTM and NET For AS3, only NET |
|
agent | String | Optional | AS3 | You can also change the value to CCCL
for CCCL mode. |
AS3, CCCL | Both AS3 and CCCL |
version | Boolean | Optional | false | Print CIS version | Both AS3 and CCCL |
BIG-IP system Parameters¶
Parameter | Type | Required | Default | Description | Allowed Values | Agent |
---|---|---|---|---|---|---|
bigip-partition | String | Required | N/A | The BIG-IP partition in which to configure objects. | Both AS3 and CCCL | |
bigip-password | String | Required | N/A | BIG-IP iControl REST password You can secure your BIG-IP credentials using a Kubernetes Secret. |
Both AS3 and CCCL | |
bigip-url | String | Required | N/A | BIG-IP admin IP address | Both AS3 and CCCL | |
bigip-username | String | Required | N/A | BIG-IP iControl REST username The BIG-IP user account must have the appropriate role defined: For For |
Both AS3 and CCCL | |
credentials-directory | String | Optional | N/A | Directory that contains the BIG-IP username, password, or url files | Both AS3 and CCCL |
VXLAN Parameters¶
Parameter | Type | Required | Default | Description | Allowed Values | Agent |
---|---|---|---|---|---|---|
openshift-sdn-name | String | Optional | N/A | Name of the VXLAN tunnel on the BIG-IP system that corresponds to an OpenShift SDN HostSubnet. Only applicable in OpenShift. |
Both AS3 and CCCL | |
flannel-name | String | Optional | N/A | Name of the VXLAN tunnel on the BIG-IP system that corresponds to a Flannel subnet. | CCCL |
AS3 ConfigMap¶
The AS3 ConfigMap hosts AS3 extensions, in JSON format, as a configuration artifact. CIS can manage and orchestrate BIG-IP declaratively through this ConfigMap.
In agent AS3 mode, CIS handles Ingress or Route resources by converting them into AS3 declarations before posting to BIG-IP. When AS3 ConfigMap is configured along with Ingress or Routes, CIS manages ConfigMap and Ingress (or) Routes AS3 declarations separately. While sending an AS3 declaration to BIG-IP, CIS will combine both of these AS3 declarations as a single declaration and POST it to BIG-IP.
CIS does not watch for Ingress/Routes/ConfigMaps when deployed in CRD Mode.
CIS does not support the combination of CRDs with any of Ingress/Routes and ConfigMaps. So the migration from AS3 CM to CRD can cause some downtime as well.
Below is an example of how to do use AS3 CM and CRD for the same set of requirements.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | # Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
apiVersion: v1
data:
template: |
{"class": "AS3", "action": "deploy", "persist": true, "declaration": {"class": "ADC", "schemaVersion": "3.23.0", "id": "urn:uuid:33045210-3ab8-4636-9b2a-c98d22ab915d-pytest2", "label": "pytest2", "remark": "AS3 Declation for pytest2", "pytest2_tnt_0": {"class": "Tenant", "pytest2_tnt_0_https0": {"class": "Application", "template": "https", "serviceMain": {"class": "Service_HTTPS", "virtualAddresses": ["172.16.3.9"], "pool": "pytest2_tnt_0_https0_p0", "serverTLS": "pytest2_tnt_0_https0_ssl_0", "persistenceMethods": []}, "pytest2_tnt_0_https0_p0": {"class": "Pool", "monitors": ["http"], "members": [{"servicePort": 80, "shareNodes": false, "serverAddresses": []}]}, "pytest2_tnt_0_https0_ssl_0": {"class": "TLS_Server", "certificates": [{"certificate": "pytest2_tnt_0_https0_cert_0"}]}, "pytest2_tnt_0_https0_cert_0": {"class": "Certificate", "remark": "in practice we recommend using a passphrase", "certificate": "", "privateKey": "", "passphrase": {"ciphertext": "ZjVmNQ==", "protected": "eyJhbGciOiJkaXIiLCJlbmMiOiJub25lIn0", "ignoreChanges": true}}}}}}
kind: ConfigMap
metadata:
labels:
as3: "true"
f5type: virtual-server
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:template: {}
f:metadata:
f:labels:
.: {}
f:as3: {}
f:f5type: {}
manager: kubectl
operation: Update
name: pytest2
namespace: default
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | # Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2020-11-02T11:17:35Z"
labels:
cis.f5.com/as3-app: pytest2_tnt_0_https0
cis.f5.com/as3-pool: pytest2_tnt_0_https0_p0
cis.f5.com/as3-tenant: pytest2_tnt_0
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.: {}
f:cis.f5.com/as3-app: {}
f:cis.f5.com/as3-pool: {}
f:cis.f5.com/as3-tenant: {}
f:spec:
f:externalTrafficPolicy: {}
f:ports:
.: {}
k:{"port":80,"protocol":"TCP"}:
.: {}
f:name: {}
f:port: {}
f:protocol: {}
f:targetPort: {}
f:selector:
.: {}
f:app: {}
f:sessionAffinity: {}
f:type: {}
manager: kubectl
operation: Update
time: "2020-11-02T11:17:35Z"
name: pytest2-tnt-0-https0-p0
namespace: default
resourceVersion: "1489990"
selfLink: /api/v1/namespaces/default/services/pytest2-tnt-0-https0-p0
uid: bfa981a1-b88f-4ad4-a337-0314dcdf0692
spec:
clusterIP: 10.105.213.231
externalTrafficPolicy: Cluster
ports:
- name: pytest2-tnt-0-https0-p0-80
nodePort: 32519
port: 80
protocol: TCP
targetPort: 80
selector:
app: pytest2-tnt-0-https0-p0
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 | # Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2020-11-02T11:17:34Z"
generation: 1
labels:
app: pytest2-tnt-0-https0-p0
managedFields:
- apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.: {}
f:app: {}
f:spec:
f:progressDeadlineSeconds: {}
f:replicas: {}
f:revisionHistoryLimit: {}
f:selector:
f:matchLabels:
.: {}
f:app: {}
f:strategy:
f:rollingUpdate:
.: {}
f:maxSurge: {}
f:maxUnavailable: {}
f:type: {}
f:template:
f:metadata:
f:labels:
.: {}
f:app: {}
f:spec:
f:containers:
k:{"name":"pytest2-tnt-0-https0-p0"}:
.: {}
f:env:
.: {}
k:{"name":"service_name"}:
.: {}
f:name: {}
f:value: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:ports:
.: {}
k:{"containerPort":80,"protocol":"TCP"}:
.: {}
f:containerPort: {}
f:protocol: {}
f:resources: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:dnsPolicy: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext: {}
f:terminationGracePeriodSeconds: {}
manager: kubectl
operation: Update
time: "2020-11-02T11:17:34Z"
- apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:deployment.kubernetes.io/revision: {}
f:status:
f:availableReplicas: {}
f:conditions:
.: {}
k:{"type":"Available"}:
.: {}
f:lastTransitionTime: {}
f:lastUpdateTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"Progressing"}:
.: {}
f:lastTransitionTime: {}
f:lastUpdateTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
f:observedGeneration: {}
f:readyReplicas: {}
f:replicas: {}
f:updatedReplicas: {}
manager: kube-controller-manager
operation: Update
time: "2020-11-02T11:17:37Z"
name: pytest2-tnt-0-https0-p0
namespace: default
resourceVersion: "1490014"
selfLink: /apis/apps/v1/namespaces/default/deployments/pytest2-tnt-0-https0-p0
uid: 5b51a71b-d464-4a77-bb86-ef84e9946818
spec:
progressDeadlineSeconds: 600
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
app: pytest2-tnt-0-https0-p0
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: pytest2-tnt-0-https0-p0
spec:
containers:
- env:
- name: service_name
value: pytest2-tnt-0-https0-p0
image: f5networksdevel/test-nginx:latest
imagePullPolicy: Always
name: pytest2-tnt-0-https0-p0
ports:
- containerPort: 80
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
|