Last updated on: 2023-12-04 07:13:44.

Configuration Parameters¶

All of the configuration parameters below are global.

General¶

Parameter	Type	Required	Default	Description	Allowed Values	Agent	Minimum supported version
http-listen-address	String	Optional	“0.0.0.0:8080”	Address at which to serve HTTP-based information (for example, `/metrics`, `health`) to Prometheus.		Both AS3 and CCCL
node-poll-interval	Integer	Optional	30	Interval at which CIS monitors node member to update BIG-IP. This helps in add/delete node info in BIG-IP.		Both AS3 and CCCL
python-basedir	String	Optional	/app/python	Path to the python utilities directory.		CCCL
schema-db-base-dir	String	Optional	file:///app/vendor/src/f5/schemas	Path to the directory containing the F5 schema db.		CCCL
verify-interval	Integer	N/A	30	In AS3 mode: Interval at which NET config is synced to BIG-IP. In CCCL mode: Interval at which both LTM and NET config is synced to BIG-IP.		For CCCL, LTM and NET For AS3, only NET
vs-snat-pool-name	String	Optional	N/A	Name of the SNAT pool that all virtual servers will reference (format: `/Common/<SNAT pool>`). If it is not set, virtual servers use automap SNAT.		CCCL
agent	String	Optional	AS3	Specify the agent for CIS to communicate with BIG-IP.	AS3, CCCL
version	Boolean	Optional	false	Print CIS version.		Both AS3 and CCCL
custom-resource-mode	Boolean	Optional	false	Specify to process CRD resources. When `true` ConfigMaps, Routes, and Ingress are not processed by CIS.		AS3	2.0.0
ipam	Boolean	Optional	false	Specify if CIS provides the ability to interface with F5 IPAM Controller (FIC).		AS3	2.3.0
disable-teems	Boolean	Optional	false	If true, analytics data is not sent to F5.		Both AS3 and CCCL	2.5.0
hubmode	Boolean	Optional	false	If true, ConfigMaps with Services in same and different namespace are processed. Available with CIS version 2.5.0+.	true, false	AS3	2.5.0
default-route-domain	Int	Optional	0	Set default Route Domain for Custom resources.		AS3	2.4.0
filter-tenants	Boolean	Optional	false	Specify to use tenant filtering API for AS3 declaration. This allows CIS to process each AS3 Tenant separately. Compatible with ConfigMap only.	true, false	AS3	2.7.0
enable-ipv6	Boolean	Optional	false	When set to true, it enables IPv6 network support. Available with CIS version 2.7.0+.	true, false	AS3	2.7.0
cccl-gtm-agent	String	Optional	true	Option to configure GTM objects using CCCL or AS3 Agent. Default Agent is CCCL.	true, false	AS3 and CCCL	2.11.0
http-client-metrics	Boolean	Optional	false	Adds HTTP client metric (Prometheus) instrumentation for the k8s-bigip-ctlr.	true, false	AS3	2.12.0
static-routing-mode	Boolean	Optional	false	Adds Static Routes on the BIGIP so that traffic can be directly route to the pods. (Without tunnels)	true, false	AS3	2.13.0
orchestration-cni	String	Optional	flannel	Kubernetes Cluster CNI Name	cilium-k8s, flannel,ovn-k8s, antrea	AS3	2.13.0
shared-static-routes	Boolean	Optional	false	When set to true, static routes are created on the `/Common` partition, which can be valid only when `static-routing-mode` is enabled.	true, false	AS3	2.14.0
static-route-node-cidr	String	Optional	N/A	Specify the node network cidr to be used for static routing when the node has multiple interfaces This is supported only with CNI ovn-k8s	Any valid cidr eg: 10.4.0.0/14	AS3	2.15.0
controller-mode	String	Optional		Required parameter for NextGen route processing	openshift	AS3	2.12.0

Note

The python-basedir setting lets you specify the path to an alternate python agent that can bridge between the CIS and F5 CCCL.
The time it takes for the CIS to reapply the system configurations to the BIG-IP device is normally low (a few ms) and won’t cause service disruption.
Use vs-snat-pool-name if you want virtual servers to reference a SNAT pool that already exists in the /Common partition on the BIG-IP device. See Overview of SNAT features on AskF5 for more information.
CIS must be configured with --agent=as3 and --custom-resource-mode=true to interface with F5 IPAM Controller.
--custome-resource-mode=true is required to process the custom resources and --controller-mode=openshift is required to process the route and policy resources. Both are mutually exclusive parameters.

Logging¶

Parameter	Type	Required	Default	Description	Allowed Values	Agent
log-level	String	Optional	INFO	Log level	INFO, DEBUG, AS3DEBUG CRITICAL, WARNING, ERROR	Both AS3 and CCCL
cccl-log-level	String	Optional	N/A	CCCL log level	INFO	AS3 (CIS > 2.6
log-as3-response	Boolean	Optional	false	When set to true, adds the body of AS3 API response in Controller logs.	true or false	AS3
log-file	String	Optional	N/A	File path to store the CIS logs.		Both AS3 and CCCL

Note

AS3DEBUG should only be used for debugging purposes, as it may impact CIS performance. AS3DEBUG should only be used with an AS3 agent.
log-as3-response parameter only logs the AS3 response, while AS3DEBUG log-level logs both AS3 requests and responses.

BIG-IP system¶

Parameter	Type	Required	Default	Description	Allowed Values	Agent
bigip-partition	String	Required	N/A	The BIG-IP partition in which to configure objects.	BIG-IP AS3 agent does not support write to “common”	Both AS3 and CCCL
bigip-password	String	Required	N/A	BIG-IP iControl REST password. You can secure your BIG-IP credentials using a Kubernetes Secret.		Both AS3 and CCCL
bigip-url	String	Required	N/A	BIG-IP admin IP address. Examples: URL with non-standard port `--bigip-url= https://x.x.x.x:8443` IP address `--bigip-url= x.x.x.x` IP address with port `--bigip-url= x.x.x.x:8080` IPv6 address `--bigip-url= '[2001:db8::6]'`	IP address URL:PORT IP-addr:PORT For IPv6 address as string inside []	Both AS3 and CCCL
bigip-username	String	Required	N/A	BIG-IP iControl REST username The BIG-IP user account must have the appropriate role defined: For `nodeport` type pool members, the role must be `Administrator`. For `cluster` type pool members, the role must be `Administrator`.		Both AS3 and CCCL
credentials-directory	String	Optional	N/A	Directory that contains the BIG-IP username, password, or url files.		Both AS3 and CCCL

Important

The credentials-directory option is an alternative to using the bigip-username, bigip-password, or bigip-url arguments.

When you use this argument, the controller looks for three files in the specified directory:

“username”, “password”, and “url”

If any of these files do not exist, the controller falls back to using the CLI arguments as parameters.

Each file should contain only the username, password, and url, respectively. You can create and mount the files as Kubernetes Secrets.

It is important to not project the Secret keys to specific paths, as the controller looks for the “username”, “password”, and “url” files directly within the credentials directory.

See example-bigip-credentials-directory.yaml for a deployment example.

GTM BIG-IP system¶

Parameter	Type	Required	Default	Description
gtm-bigip-password	String	Required	N/A	Password for the GMT BIG-IP user account. You can secure your GTM BIG-IP credentials using a Kubernetes Secret.
gtm-bigip-url	String	Required	N/A	URL for the GTM BIG-IP
gtm-bigip-username	String	Required	N/A	Username for the GTM BIG-IP user account.
gtm-credentials-directory	String	Optional	N/A	The directory that contains the GTM BIG-IP username, password, and/or URL files. To be used instead of username, password, and/or URL arguments.

Important

The credentials-directory option is an alternative to using the gtm-bigip-username, gtm-bigip-password, or gtm-bigip-url arguments.

When you use this argument, the controller looks for three files in the specified directory:

“username”, “password”, and “url”

If any of these files do not exist, the controller falls back to using the CLI arguments as parameters.

Each file should contain only the username, password, and url, respectively. You can create and mount the files as Kubernetes Secrets.

It is important to not project the Secret keys to specific paths, as the controller looks for the “username”, “password”, and “url” files directly within the credentials directory.

See example-gtm-bigip-credentials-directory.yaml for a deployment example.

VXLAN¶

Parameter	Type	Required	Default	Description	Agent
openshift-sdn-name	String	Optional	N/A	Name of the VXLAN tunnel on the BIG-IP system that corresponds to an OpenShift SDN HostSubnet. Only applicable in OpenShift.	Both AS3 and CCCL
flannel-name	String	Optional	N/A	Name of the VXLAN tunnel on the BIG-IP system that corresponds to a Flannel subnet.	CCCL
cilium-name	String	Optional	N/A	Name of the VXLAN tunnel on the BIG-IP system that corresponds to a Cilium Subnet.	CCCL

Parameter

Type

Required

Default

Description

Allowed Values

Agent

openshift-sdn-name

String

Optional

N/A

Name of the VXLAN tunnel on the BIG-IP system that corresponds to an OpenShift SDN HostSubnet.

Only applicable in OpenShift.

Both AS3 and CCCL

flannel-name

String

Optional

N/A

Name of the VXLAN tunnel on the BIG-IP system that corresponds to a Flannel subnet.

CCCL

cilium-name

String

Optional

N/A

Name of the VXLAN tunnel on the BIG-IP system that corresponds to a Cilium Subnet.

CCCL

Note

CIS allows only one of the above VxLAN parameters in its deployment.

Kubernetes¶

Parameter	Type	Required	Default	Description	Allowed Values	Agent	Minimum supported version
default-ingress-ip	String	Optional	N/A	The controller configures a virtual server at this IP address for all Ingresses with the annotation: `virtual-server.f5.com/ip: 'controller-default'`		Both AS3 and CCCL
kubeconfig	String	Optional	./config	Path to the kubeconfig file		Both AS3 and CCCL
manage-configmaps	Boolean	Optional	true	Tells the controller whether or not to watch Kubernetes ConfigMaps and apply their configuration. If false, the controller will ignore ConfigMap events.	true, false	Both AS3 and CCCL
namespace	String	Optional	All	Kubernetes namespace(s) to watch To specify multiple namespace, use multiple `--namespace` flags. Watches all namespaces by default.		Both AS3 and CCCL
namespace-label	String	Optional	N/A	Tells the `k8s-bigip-ctlr` to watch any namespace with this label		Both AS3 and CCCL
node-label-selector	String	Optional	N/A	Tells the `k8s-bigip-ctlr` to watch only nodes with this label		Both AS3 and CCCL
pool-member-type	String	Optional	nodeport	The type of BIG-IP pool members you want to create. Use `cluster` to create pool members for each of the endpoints for the Service (the pod’s InternalIP). Use `nodeport` to create pool members for each schedulable node using the Service’s NodePort. Use `nodeportlocal` to create pool members using Antrea CNI NodePortLocal (NPL) feature. This option is applicable only for clusters deployed with Antrea CNI.	cluster, nodeport, nodeportlocal	Both AS3 and CCCL
resolve-ingress-names	String	Optional	N/A	Tells the controller to resolve the first Host in an Ingress resource to an IP address. This IP address will be used as the virtual server address for the Ingress resource. A value of “LOOKUP” will use local DNS to resolve the Host. Any other value is a custom DNS server and the controller sends resolution queries through that server instead. Specifying the flag with no argument will default to LOOKUP.		Not Supported
running-in-cluster	Boolean	Optional	true	Indicates whether or not a Kubernetes cluster started `k8s-bigip-ctlr`	true, false	Both AS3 and CCCL
use-node-internal	Boolean	Optional	true	filter Kubernetes InternalIP addresses for pool members	true, false	Both AS3 and CCCL
use-secrets	Boolean	Optional	true	Tells the controller whether or not to load SSL profiles from Kubernetes Secrets for Ingresses and ConfigMaps. If false, the controller will only use profiles from the BIG-IP system.	true, false	Both AS3 and CCCL
manage-ingress	Boolean	Optional	true	Tells the controller whether or not to watch Kubernetes ingress and apply their configuration. If false, the controller will ignore ingress events.	true, false	Both AS3 and CCCL
manage-ingress-class- only	Boolean	Optional	true	Process all ingress resources without `kubernetes.io/ingress.class` + “annotation and ingresses with annotation `kubernetes.io/ingress.class=f5`”.	true, false	AS3
ingress-class	String	Optional	F5	A class of the Ingress controller. The Ingress controller only processes Ingress resources that belong to its class–i.e. have the annotation `kubernetes.io/ingress.class` equal to the class. Additionally, the Ingress controller processes Ingress resources that do not have that annotation, which can be disabled by setting the `-manage-ingress-class-only` flag.	true, false	AS3
virtualServerHTTPPort	Integer	Optional	N/A	Creates a Virtual Server on BIG-IP with VIP custom HTTP port.		AS3
virtualServerHTTPSPort	Integer	Optional	N/A	Creates a Virtual Server on BIG-IP with VIP custom HTTPS port.		Both AS3 and CCCL
httpTraffic	String	Optional	N/A	Configure the behavior of traffic on HTTP Virtual Server.	`httpTraffic = allow -> Allows HTTP` `httpTraffic = none -> Only HTTPS` `httpTraffic = redirect -> redirects HTTP to HTTPS`	AS3
periodic-sync-interval	Integer	Optional	30	Interval (in seconds) at which to queue resources.		AS3	2.5.0

Note

Use the node-label-selector parameter if you only want the controller to manage specific nodes from the cluster. For example, the BIG-IP device may not be able to reach certain nodes, or the BIG-IP device already manages certain nodes. Therefore, the controller should only watch the nodes that match the environmental constraints (by using a label).

OpenShift Routes¶

The following configuration parameters only apply to OpenShift.

Parameter	Type	Required	Default	Description	Allowed Values	Agent
custom-client-ssl	String	Optional	N/A	Specifies the name of a custom client SSL profile attached to the route HTTPS virtual server and used as default for SNI. This profile must have the Default for SNI field enabled.		CCCL
custom-server-ssl	String	Optional	N/A	Specifies the name of a custom server SSL profile attached to the route HTTPS virtual server and used as default for SNI. This profile must have the Default for SNI field enabled.		CCCL
manage-routes	Boolean	Optional	false	Indicates if `k8s-bigip-ctlr` should handle OpenShift Route objects.	true, false	Both AS3 and CCCL
route-http-vserver	String	Optional	ose-vserver	The name of the http virtual server for OpenShift Routes.		Both AS3 and CCCL
route-https-vserver	String	Optional	https-ose-vserver	The name of the https virtual server for OpenShift Routes.		Both AS3 and CCCL
route-label	String	Optional	N/A	Tells the `k8s-bigip-ctlr` to only watch for OpenShift Route objects with the `f5type` label set to this value.		Both AS3 and CCCL
route-vserver-addr	String	Optional	N/A	Bind address for virtual server for OpenShift Route objects.		Both AS3 and CCCL
default-client-ssl	String	Optional	N/A	Name of the default client ssl profile.		Both AS3 and CCCL
default-server-ssl	String	Optional	N/A	Configuring CIS with –default-server-ssl for OpenShift routes is not recommended.		Both AS3 and CCCL
tls-version	String	Optional	1.2	Configures TLS version to be enabled on BIG-IP. TLS 1.3 is only supported on TMOS version 14.0+.		Only AS3
cipher-group	String	Optional	N/A	Configures a cipher group in BIG-IP and reference it here. Cipher group and ciphers are mutually exclusive, only use one.		Only AS3
ciphers	String	Optional	N/A	Configures a ciphersuite selection string. Cipher-group and ciphers are mutually exclusive, only use one.		Only AS3

Note

If the custom-client-ssl or custom-server-ssl parameters are not provided, then the controller creates default clientssl and serverssl profiles for the OpenShift Route HTTPS virtual server. The controller sets these profiles as Default for SNI.

NextGen Routes¶

The following configuration parameters only apply to NextGen Routes.

Note

For NextGen Routes, controller-mode must be set as openshift.

OpenShift Routes and Kubernetes¶

AS3 Parameters¶

Parameter	Type	Required	Default	Description	Minimum supported version
as3-validation	Boolean	Optional	true	When set to false, this disables AS3 template validation on the controller.
insecure	Boolean	Optional	false	When set to true, this enables insecure SSL communication to the BIG-IP system.
trusted-certs-cfgmap	String	Required	N/A	When certificates are provided, adds them to controller’s trusted certificate store.
as3-post-delay	int	Optional	N/A	Specifies the time, in seconds, that CIS waits to post the available AS3 declaration.	2.0.0
override-as3- declaration	String	Optional	N/A	Provide Namespace and Name of the ConfigMap as <namespace>/<configmap-name>. The JSON key/values from this ConfigMap will oerride key/values from internally generated AS3 declaration.
filter-tenants	Boolean	Optional	false	Specifies whether or not the use tenant filtering API for AS3 declaration.	2.7.0
share-nodes	Boolean	Optional	false	When set to true, the pool member node is created on `/Common` partition, which can be shared among multiple tenants. This argument is only valid for Ingress, Routes, and CRDs.	2.2.0

AS3 Response code handling in CIS¶

AS3 Response Code	HTTP Status	CIS Handling (NextGen Mode CRDs/Routes/ServiceTypeLB)	CIS Handling (Legacy Mode)
200	OK	CIS does not try to repost AS3 declaration.	CIS does not try to repost AS3 declaration.
201	Created	CIS polls for its status continuously and blocks incoming requests.	CIS does not try to repost AS3 declaration.
202	Accepted	CIS polls for its status continuously and blocks incoming requests.	CIS does not try to repost AS3 declaration.
207	Multi Status	CIS retries every 30 seconds for failed tenants and successful tenant declarations will not be posted.	N/A
404	Not Found	If CIS receives 404 at the start-up it immediately exits and the container will be restarted continuously. One of the reasons for a 404 error in this case is that AS3 RPM is not installed on BIG-IP. If CIS receives 404 while posting the AS3 declaration, CIS retries every 30 sec for failed tenants.	If CIS receives 404 at the start-up it immediately exits and the container will be restarted continuously. One of the reasons for a 404 error in this case is that AS3 RPM is not installed on the BIG-IP system. If CIS receives a 404 while posting the AS3 declaration, CIS does not try to repost AS3 declaration.
422	Unprocessable Entity	CIS retries every 30 seconds for failed tenants.	CIS retries every 30 seconds for failed tenants.
503	Service Unavailable	CIS waits for 30 seconds before trying to repost.	CIS waits for 30 seconds before trying to repost.
Others		CIS retries every 30 seconds for failed tenants.	CIS retries every 30 seconds for failed tenants.

CIS Health Checks¶

Kubernetes has two types of health checks:

Readiness Probes: To determine when a pod is ready
Liveness Probes: To determine when a pod is healthy or unhealthy after it has become ready

Readiness Probes Kubernetes uses readiness probes to decide when the container is available for accepting traffic. The readiness probe controls which pods to use as the backend for a service. A pod is considered ready when all of its containers are ready. If a pod is not ready, it is removed from service load balancers. For example, if a container loads a large cache at start-up and takes minutes to start, you should not send requests to this container until it is ready, or the requests will fail. Instead, route requests to other pods, which are capable of servicing requests.

Liveness Probes Kubernetes uses liveness probes to know when to restart a container. If a container is unresponsive, the application could be deadlocked due to a multi-threading defect. Restarting the container can make the application more available.

These are the methods you can use to check container status:

HTTP request to the pod
Command execution to the pod
TCP request to the pod

Probes are defined on a container in a deployment.

Here is an example of the deployment using HTTP method:

Parameter	Description
periodSeconds	specifies that the kubelet should perform a liveness probe every 3 seconds.
initialDelaySeconds	tells the kubelet that it should wait 3 seconds before performing the first probe
timeOutSeconds	how long to wait for the probe to finish. If this time is exceeded, OpenShift Container Platform considers the probe to have failed.

The kubelet uses a web hook to determine the healthiness of the container. The check is successful if the HTTP response code is between 200 and 399.

To perform a probe, the kubelet sends an HTTP GET request to the server that is running in the Container and listening on port 8080. The handler for the server’s /health path returns a success code. For example:

livenessProbe:
   failureThreshold: 3
   httpGet:
      path: /health
      port: 8080
      scheme: HTTP
   initialDelaySeconds: 15
   periodSeconds: 15
   successThreshold: 1
   timeoutSeconds: 15
readinessProbe:
   failureThreshold: 3
   httpGet:
      path: /health
      port: 8080
      scheme: HTTP
   initialDelaySeconds: 30
   periodSeconds: 30
   successThreshold: 1
   timeoutSeconds: 15

To view the liveness and readiness of the deployed pod, run the command Kubectl describe pod <pod_name> -n kube-system. Here is the example output:

        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        - --log-level=debug
        terminationMessagePolicy: Fileermination-log
      --log-level=debug
      --namespace=default
      --route-label=systest
      --insecure=true
      --agent=cccl
   Liveness:      http-get http://:8080/health delay=15s timeout=15s period=15s #success=1 #failure=3
   Readiness:     http-get http://:8080/health delay=30s timeout=15s period=30s #success=1 #failure=3
   Environment:   <none>
   Mounts:        <none>
Volumes:          <none>

curl http://<self-ip>:<port no>/health shows a response of OK.

/ready endpoint Advanced endpoint for monitoring the health of CIS. The endpoint works as follows:

curl http://<self-ip>:<port no>/ready

If everything is working as expected, a successful response of 200 OK is returned.
If the Kubernetes APIs are unreachable, down, or not accessible (ie: permissions problem), then health checks fail resulting in HTTP error code 500 returning “kube-api server is not reachable.”
If the BIG-IPs are unreachable, down, or not accessible (ie: permissions problem), then health checks fail resulting in HTTP error code 500 returning “big-ip server is not reachable.”

Prometheus Metrics¶

Name	Type	Default Status	Description	Labels
bigip_monitored_nodes	Gauge	Enabled	Total count of monitored nodes by the BIG-IP k8s CTLR.	`["nodeselector"]`
bigip_current_errors	Gauge	Enabled	Total count of errors occured parsing the configuration.	`[]`
bigip_monitored_services	Gauge	Enabled	Total count of monitored services by the BIG-IP k8s CTLR.	`["namespace", "name", "status"]`
bigip_http_client_api_requests_total	Counter	Disabled	A counter for requests from the wrapped HTTP client.	`["code", "method"]`
bigip_http_client_in_flight_requests	Gauge	Disabled	Total count of in-flight requests for the wrapped HTTP client.	`[]`
bigip_http_client_dns_duration_seconds	Histogram	Disabled	Trace DNS latency histogram.	`["event"]`
bigip_http_client_tls_duration_seconds	Histogram	Disabled	Trace TLS latency histogram.	`["event"]`
bigip_http_client_request_duration_seconds	Histogram	Disabled	Trace HTTP request latencies histogram.	`[]`

Note

Use the --http-client-metrics deployment parameter to export the AS3 HTTP client prometheus metrics, which are disabled by default.

Rest Methods with CIS¶

CIS uses the following Rest methods along with AS3 and CCCL agents to interact with BIG-IP.

HTTP Method	AS3	CCCL
GET	Yes	Yes
POST	Yes	Yes
PUT	Yes	Yes
DELETE	Yes	Yes

Note

When CIS is deployed in cluster mode with an AS3 agent, CIS still uses the CCCL agent to update the FDB and ARP entries on BIG-IP.

Note

To provide feedback on Container Ingress Services or this documentation, please file a GitHub Issue.

Previous Next