Custom Resource Definitions¶
Overview of CRDs¶
This page describes CIS in CRD Mode.
What are CRDs?
- Custom resources are extensions of the Kubernetes API.
- A resource is an endpoint in the Kubernetes API that stores a collection of API objects. For example, the built-in pods resource contains a collection of Pod objects.
- A custom resource is an extension of the Kubernetes API that is not necessarily available in a default Kubernetes installation. It represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.
- Custom resources can appear and disappear in a running cluster through dynamic registration, and cluster admins can update custom resources independently of the cluster itself. Once a custom resource is installed, users can create and access its objects using kubectl, just as they do for built-in resources like Pods.
- CIS supports the following Custom Resources:
How CIS works with CRDs
- CIS registers to the Kubernetes client-go using informers to retrieve Virtual Server, TLSProfile, Service, Endpoint and Node create, update, and delete events. Resources identified from such events are pushed to a Resource Queue maintained by CIS.
- The Resource Queue holds the resources to be processed.
- Virtual Server is the Primary citizen. Any changes in TLSProfile, Service, Endpoint, or Node will process their affected Virtual Servers. For example, If svc-a is part of foo-VirtualServer and bar-VirtualServer, any changes in svc-a will put foo-VirtualServer and bar-VirtualServer in the resource queue.
- Worker fetches the affected Virtual Servers from the Resource Queue to populate a common structure which holds the configuration of all the Virtual Servers such as TLSProfile, Virtual Server IP, Pool Members, and L7 LTM policy actions.
- VXLAN Manager prepares the BIG-IP NET configuration as AS3 cannot process FDB and ARP entries.
- LTM Configuration (using AS3) and NET Configuration (using CCCL) is created in CIS Managed Partition defined by the user.
- Migrating from AS3 ConfigMaps to CRDs
Important
- You must configure the CRD schema before creating CIS. Run the following commands:
export CIS_VERSION=<cis-version>
# For example
# export CIS_VERSION=v2.12.0
# or
# export CIS_VERSION=2.x-master
# the latter if using a CIS image with :latest label
kubectl create -f https://raw.githubusercontent.com/F5Networks/k8s-bigip-ctlr/${CIS_VERSION}/docs/config_examples/customResourceDefinitions/customresourcedefinitions.yml
--custom-resource-mode=true
deploys CIS in Custom Resource Mode.- CIS does not watch for Ingress/Routes/ConfigMaps when deployed in CRD Mode.
- CIS does not support combination of CRDs with any of Ingress/Routes and ConfigMaps.
- CRDs are supported in Kubernetes from versions 1.16+.
Valid Configuration Parameters for CRDs¶
AS3 Parameters¶
Parameter | Type | Required | Default | Description |
---|---|---|---|---|
as3-validation | Boolean | Optional | true | When set to false, this disables AS3 template validation on the controller. |
insecure | Boolean | Optional | false | When set to true, this enables insecure SSL communication to the BIG-IP system. |
trusted-certs-cfgmap | String | Required | N/A | When certificates are provided, adds them to controller’s trusted certificate store. |
log-as3-response | Boolean | Optional | false | When set to true, adds the body of AS3 API response in Controller logs. |
enable-ipv6 | Boolean | Optional | false | When set to true, it enables IPv6 network support. Available with CIS version 2.7.0+. |
Kubernetes Parameters¶
Parameter | Type | Required | Default | Description | Allowed Values | Agent |
---|---|---|---|---|---|---|
kubeconfig | String | Optional | ./config | Path to the kubeconfig file | Both AS3 and CCCL | |
namespace | String | Optional | All | Kubernetes namespace(s) to watch
|
Both AS3 and CCCL | |
namespace-label | String | Optional | N/A | Tells the k8s-bigip-ctlr to watch
any namespace with this label |
Both AS3 and CCCL | |
node-label-selector | String | Optional | N/A | Tells the k8s-bigip-ctlr to watch
only nodes with this label |
Both AS3 and CCCL | |
pool-member-type | String | Optional | nodeport | The type of BIG-IP pool members you want to create. Use Use |
cluster, nodeport | Both AS3 and CCCL |
running-in-cluster | Boolean | Optional | true | Indicates whether or not a
Kubernetes cluster started
k8s-bigip-ctlr |
true, false | Both AS3 and CCCL |
use-node-internal | Boolean | Optional | true | filter Kubernetes InternalIP addresses for pool members | true, false | Both AS3 and CCCL |
General Parameters¶
Parameter | Type | Required | Default | Description | Allowed Values | Agent |
---|---|---|---|---|---|---|
http-listen-address | String | Optional | “0.0.0.0:8080” | Address at which to serve HTTP-based
information (for example, /metrics ,
health ) to Prometheus. |
Both AS3 and CCCL | |
log-level | String | Optional | INFO | Log level | INFO, DEBUG, CRITICAL, WARNING, ERROR | Both AS3 and CCCL |
node-poll-interval | Integer | Optional | 30 | In seconds, the interval at which the CIS polls the cluster to find all node members. | Both AS3 and CCCL | |
python-basedir | String | Optional | /app/python | Path to the python utilities directory. | CCCL | |
schema-db-base-dir | String | Optional | file:///app/vendor/src/f5/schemas | Path to the directory containing the F5 schema db. | CCCL | |
verify-interval | Integer | N/A | 30 | In seconds, the interval at which the CIS verifies that the BIG-IP configuration matches the state of the orchestration system. | For CCCL, LTM and NET For AS3, only NET |
|
agent | String | Optional | AS3 | You can also change the value to CCCL
for CCCL mode. |
AS3, CCCL | Both AS3 and CCCL |
version | Boolean | Optional | false | Print CIS version | Both AS3 and CCCL |
BIG-IP system Parameters¶
Parameter | Type | Required | Default | Description | Allowed Values | Agent |
---|---|---|---|---|---|---|
bigip-partition | String | Required | N/A | The BIG-IP partition in which to configure objects. | Both AS3 and CCCL | |
bigip-password | String | Required | N/A | BIG-IP iControl REST password You can secure your BIG-IP credentials using a Kubernetes Secret. |
Both AS3 and CCCL | |
bigip-url | String | Required | N/A | BIG-IP admin IP address | Both AS3 and CCCL | |
bigip-username | String | Required | N/A | BIG-IP iControl REST username The BIG-IP user account must have the appropriate role defined: For For |
Both AS3 and CCCL | |
credentials-directory | String | Optional | N/A | Directory that contains the BIG-IP username, password, or url files | Both AS3 and CCCL |
VXLAN Parameters¶
Parameter | Type | Required | Default | Description | Allowed Values | Agent |
---|---|---|---|---|---|---|
openshift-sdn-name | String | Optional | N/A | Name of the VXLAN tunnel on the BIG-IP system that corresponds to an OpenShift SDN HostSubnet. Only applicable in OpenShift. |
Both AS3 and CCCL | |
flannel-name | String | Optional | N/A | Name of the VXLAN tunnel on the BIG-IP system that corresponds to a Flannel subnet. | CCCL |
Using IP Addresses and F5 IPAM Controller with CIS Custom Resources¶
F5 IPAM Controller provides flexibility when it comes to automatic IP address allocation to CIS custom resources and service type load-balancer. However, IP address allocation using F5 IPAM Controller is one to one with respect to custom resources. This means one IP address is allocated on a per hostGroup or hostname basis for virtual server custom resource and per resource one IP Address for transport server custom resource or service type load balancer. Currently there is no support for sharing the F5 IPAM Controller-provided IP address with multiple transport servers or service type load-balancer. Similarly, this IP address cannot be shared between resources such as transport server, virtual server, and service type load-balancer. In scenarios where users want to share the IP address with multiple resources, F5 recommends assigning these IP addresses manually to the respective virtual server and transport server. CIS fully supports assigning the IP address directly in virtual server and transport server custom resource as well as using F5 IPAM controller (or combination of both).
Note
Currently CIS does not provide a way to allocate the IP address manually to a service type load-balancer.
Use Cases:
1: Single Hostname per IP Address
You can create multiple Virtual Server and Transport Server custom resources with each having a unique IP address per host using the virtualServerAddress
property in the respective custom resource spec. By default, all virtual servers with the same hostname are grouped under a single Virtual Server IP address. This is why F5 recommends providing the same virtualServerAddress if the hostname is the same for multiple virtual server resources. You can use also use F5 IPAM Controller to assign the IP addresses using the ipamLabel
property in the virtual server/transport server custom resource spec. If the virtualServerAddress
and ipamLabel
property are both provided in the spec, virtualServerAddress
takes the priority over the ipamLabel
.
2: Multiple Hostnames per IP Address
You can create multiple virtual servers with different hostnames and group them under a single virtual server using the hostGroup
property in the custom resource spec. You can use also use F5 IPAM Controller to assign the IP addresses using the ipamLabel
property in the virtual server/transport server custom resource spec. If virtualServerAddress
and ipamLabel
property both are provided in spec virtualServerAddress
takes the priority over the ipamLabel
.
3: Multiple http/https ports per IP Address
You can create multiple virtual servers with different virtualServerHTTPPort/virtualServerHTTPSPort and group them under a single IP address. CIS will create a virtual server on BIG IP for each Port-IP address combination. You can also use F5 IPAM Controller to assign the IP addresses using the ipamLabel
property in the virtual server/transport server custom resource spec. If virtualServerAddress
and ipamLabel
property both are provided in the spec virtualServerAddress
takes the priority over the ipamLabel
.
4: Multiple http/https and tcp ports per IP Address
You can create multiple virtual servers with different virtualServerHTTPPort/virtualServerHTTPSPort along with multiple transport servers and group them under a single IP address. CIS will create virtual servers on BIG-IP for each Port-IP address combination. F5 IPAM Controller can not be used in this case.
Examples Repository¶
Note
To provide feedback on Container Ingress Services or this documentation, please file a GitHub Issue.