Configuration Parameters¶
All of the configuration parameters below are global.
General¶
Parameter | Type | Required | Default | Description | Allowed Values | Agent | Minimum supported version |
---|---|---|---|---|---|---|---|
http-listen-address | String | Optional | “0.0.0.0:8080” | Address at which to serve HTTP-based
information (for example, /metrics ,
health ) to Prometheus. |
Both AS3 and CCCL | ||
node-poll-interval | Integer | Optional | 30 | Interval at which CIS monitors node member to update BIG-IP. This helps in add/delete node info in BIG-IP. | Both AS3 and CCCL | ||
python-basedir | String | Optional | /app/python | Path to the python utilities directory. | CCCL | ||
schema-db-base-dir | String | Optional | file:///app/vendor/src/f5/schemas | Path to the directory containing the F5 schema db. | CCCL | ||
verify-interval | Integer | N/A | 30 | In AS3 mode: Interval at which NET config is synced to BIG-IP. In CCCL mode: Interval at which both LTM and NET config is synced to BIG-IP. | For CCCL, LTM and NET For AS3, only NET |
||
vs-snat-pool-name | String | Optional | N/A | Name of the SNAT pool that all virtual
servers will reference
(format: /Common/<SNAT pool> ). If it
is not set, virtual servers use automap
SNAT. |
CCCL | ||
agent | String | Optional | AS3 | Specify the agent for CIS to communicate with BIG-IP. | AS3, CCCL | ||
version | Boolean | Optional | false | Print CIS version. | Both AS3 and CCCL | ||
custom-resource-mode | Boolean | Optional | false | Specify to process CRD resources. When |
AS3 | 2.0.0 | |
ipam | Boolean | Optional | false | Specify if CIS provides the ability to interface with F5 IPAM Controller (FIC). | AS3 | 2.3.0 | |
ipam-cluster-label | String | Optional | N/A | Valid for Infoblox IPAM provider only. Prepends the value of this label to form the key. Generally advised to use in MultiCluster Environment. | AS3 | 2.3.0 | |
ipam-namespace | String | Optional | kube-system | Specify the namespace of ipam custom resource. Default value is kube-system. | AS3 | 2.3.0 | |
disable-teems | Boolean | Optional | false | If true, analytics data is not sent to F5. | Both AS3 and CCCL | 2.5.0 | |
hubmode | Boolean | Optional | false | If true, ConfigMaps with Services in same and different namespace are processed. Available with CIS version 2.5.0+. | true, false | AS3 | 2.5.0 |
default-route-domain | Int | Optional | 0 | Set default Route Domain for Custom resources. | AS3 | 2.4.0 | |
filter-tenants | Boolean | Optional | false | Specify to use tenant filtering API for AS3 declaration. This allows CIS to process each AS3 Tenant separately. Compatible with ConfigMap only. | true, false | AS3 | 2.7.0 |
enable-ipv6 | Boolean | Optional | false | When set to true, it enables IPv6 network support. Available with CIS version 2.7.0+. | true, false | AS3 | 2.7.0 |
cccl-gtm-agent | String | Optional | true | Option to configure GTM objects using CCCL or AS3 Agent. Default Agent is CCCL. | true, false | AS3 and CCCL | 2.11.0 |
http-client-metrics | Boolean | Optional | false | Adds HTTP client metric (Prometheus) instrumentation for the k8s-bigip-ctlr. | true, false | AS3 | 2.12.0 |
static-routing-mode | Boolean | Optional | false | Adds Static Routes on the BIGIP so that traffic can be directly route to the pods. (Without tunnels) | true, false | AS3 | 2.13.0 |
orchestration-cni | String | Optional | flannel | Kubernetes Cluster CNI Name | cilium-k8s, flannel,ovn-k8s, antrea | AS3 | 2.13.0 |
shared-static-routes | Boolean | Optional | false | When set to true, static routes are created on
the /Common partition, which can be valid
only when static-routing-mode is enabled. |
true, false | AS3 | 2.14.0 |
static-route-node-cidr | String | Optional | N/A | Specify the node network cidr to be used for static routing when the node has multiple interfaces This is supported only with CNI ovn-k8s | Any valid cidr eg: 10.4.0.0/14 | AS3 | 2.15.0 |
controller-mode | String | Optional | Required parameter for NextGen route processing | openshift | AS3 | 2.12.0 | |
pod-graceful-shutdown | Boolean | Optional | false | Specify whether the pod should undergo a graceful shutdown or not. | true, false | AS3 | 2.17.1 |
load-balancer-class | String | Optional | “” | CIS considers services only that matches the specified class. CIS will ignore services that have this field set and does not match with the provided load-balancer-class. | AS3 | 2.18.0 | |
manage-load-balancer -class-only | Boolean | Optional | false | If set to true, CIS processes all load balancer services with loadBalancerClass only. If set to false, CIS process all the load balancer service without loadBalancerClass and service that have the loadBalancerClass specified by the load-balancer-class parameter. |
true, false | AS3 | 2.18.0 |
Note
- The
python-basedir
setting lets you specify the path to an alternate python agent that can bridge between the CIS and F5 CCCL. - The time it takes for the CIS to reapply the system configurations to the BIG-IP device is normally low (a few ms) and won’t cause service disruption.
- Use
vs-snat-pool-name
if you want virtual servers to reference a SNAT pool that already exists in the/Common
partition on the BIG-IP device. See Overview of SNAT features on AskF5 for more information. - CIS must be configured with
--agent=as3
and--custom-resource-mode=true
to interface with F5 IPAM Controller. --custome-resource-mode=true
is required to process the custom resources and--controller-mode=openshift
is required to process the route and policy resources. Both are mutually exclusive parameters.
Logging¶
Parameter | Type | Required | Default | Description | Allowed Values | Agent |
---|---|---|---|---|---|---|
log-level | String | Optional | INFO | Log level | INFO, DEBUG, AS3DEBUG CRITICAL, WARNING, ERROR | Both AS3 and CCCL |
cccl-log-level | String | Optional | N/A | CCCL log level | INFO | AS3 (CIS > 2.6 |
log-as3-response | Boolean | Optional | false | When set to true, adds the body of AS3 API response in Controller logs. | true or false | AS3 |
log-file | String | Optional | N/A | File path to store the CIS logs. | Both AS3 and CCCL |
Note
- AS3DEBUG should only be used for debugging purposes, as it may impact CIS performance. AS3DEBUG should only be used with an AS3 agent.
log-as3-response
parameter only logs the AS3 response, while AS3DEBUG log-level logs both AS3 requests and responses.
BIG-IP system¶
Parameter | Type | Required | Default | Description | Allowed Values | Agent |
---|---|---|---|---|---|---|
bigip-partition | String | Required | N/A | The BIG-IP partition in which to configure objects. | BIG-IP AS3 agent does not support write to “common” | Both AS3 and CCCL |
bigip-password | String | Required | N/A | BIG-IP iControl REST password. You can secure your BIG-IP credentials using a Kubernetes Secret. |
Both AS3 and CCCL | |
bigip-url | String | Required | N/A | BIG-IP admin IP address. Examples:
|
|
Both AS3 and CCCL |
bigip-username | String | Required | N/A | BIG-IP iControl REST username The BIG-IP user account must have the appropriate role defined: For For |
Both AS3 and CCCL | |
credentials-directory | String | Optional | N/A | Directory that contains the BIG-IP username, password, or url files. | Both AS3 and CCCL |
Important
The credentials-directory
option is an alternative to using the bigip-username
, bigip-password
, or
bigip-url
arguments.
When you use this argument, the controller looks for three files in the specified directory:
- “username”, “password”, and “url”
If any of these files do not exist, the controller falls back to using the CLI arguments as parameters.
Each file should contain only the username, password, and url, respectively. You can create and mount the files as Kubernetes Secrets.
It is important to not project the Secret keys to specific paths, as the controller looks for the “username”, “password”, and “url” files directly within the credentials directory.
See example-bigip-credentials-directory.yaml
for a deployment example.
GTM BIG-IP system¶
Parameter | Type | Required | Default | Description |
---|---|---|---|---|
gtm-bigip-password | String | Required | N/A | Password for the GMT BIG-IP user account. You can secure your GTM BIG-IP credentials using a Kubernetes Secret. |
gtm-bigip-url | String | Required | N/A | URL for the GTM BIG-IP |
gtm-bigip-username | String | Required | N/A | Username for the GTM BIG-IP user account. |
gtm-credentials-directory | String | Optional | N/A | The directory that contains the GTM BIG-IP username, password, and/or URL files. To be used instead of username, password, and/or URL arguments. |
Important
The credentials-directory
option is an alternative to using the gtm-bigip-username
, gtm-bigip-password
, or
gtm-bigip-url
arguments.
When you use this argument, the controller looks for three files in the specified directory:
- “username”, “password”, and “url”
If any of these files do not exist, the controller falls back to using the CLI arguments as parameters.
Each file should contain only the username, password, and url, respectively. You can create and mount the files as Kubernetes Secrets.
It is important to not project the Secret keys to specific paths, as the controller looks for the “username”, “password”, and “url” files directly within the credentials directory.
See example-gtm-bigip-credentials-directory.yaml
for a deployment example.
VXLAN¶
Parameter | Type | Required | Default | Description | Allowed Values | Agent |
---|---|---|---|---|---|---|
openshift-sdn-name | String | Optional | N/A | Name of the VXLAN tunnel on the BIG-IP system that corresponds to an OpenShift SDN HostSubnet. Only applicable in OpenShift. |
Both AS3 and CCCL | |
flannel-name | String | Optional | N/A | Name of the VXLAN tunnel on the BIG-IP system that corresponds to a Flannel subnet. | CCCL | |
cilium-name | String | Optional | N/A | Name of the VXLAN tunnel on the BIG-IP system that corresponds to a Cilium Subnet. | CCCL |
Note
CIS allows only one of the above VxLAN parameters in its deployment.
Kubernetes¶
Parameter | Type | Required | Default | Description | Allowed Values | Agent | Minimum supported version |
---|---|---|---|---|---|---|---|
default-ingress-ip | String | Optional | N/A | The controller configures a virtual
server at this IP address for all
Ingresses with the annotation:
virtual-server.f5.com/ip:
'controller-default' |
Both AS3 and CCCL | ||
kubeconfig | String | Optional | ./config | Path to the kubeconfig file | Both AS3 and CCCL | ||
manage-configmaps | Boolean | Optional | true | Tells the controller whether or not to watch Kubernetes ConfigMaps and apply their configuration. If false, the controller will ignore ConfigMap events. | true, false | Both AS3 and CCCL | |
namespace | String | Optional | All | Kubernetes namespace(s) to watch
|
Both AS3 and CCCL | ||
namespace-label | String | Optional | N/A | Tells the k8s-bigip-ctlr to watch
any namespace with this label |
Both AS3 and CCCL | ||
node-label-selector | String | Optional | N/A | Tells the k8s-bigip-ctlr to watch
only nodes with this label |
Both AS3 and CCCL | ||
pool-member-type | String | Optional | nodeport | The type of BIG-IP pool members you want to create. Use Use Use Use |
cluster, nodeport, nodeportlocal, auto. auto mode is supported in CRD and NextGen mode only | Both AS3 and CCCL | |
resolve-ingress-names | String | Optional | N/A | Tells the controller to resolve the first Host in an Ingress resource to an IP address. This IP address will be used as the virtual server address for the Ingress resource. A value of “LOOKUP” will use local DNS to resolve the Host. Any other value is a custom DNS server and the controller sends resolution queries through that server instead. Specifying the flag with no argument will default to LOOKUP. |
Not Supported | ||
running-in-cluster | Boolean | Optional | true | Indicates whether or not a
Kubernetes cluster started
k8s-bigip-ctlr |
true, false | Both AS3 and CCCL | |
use-node-internal | Boolean | Optional | true | filter Kubernetes InternalIP addresses for pool members | true, false | Both AS3 and CCCL | |
use-secrets | Boolean | Optional | true | Tells the controller whether or not to load SSL profiles from Kubernetes Secrets for Ingresses and ConfigMaps. If false, the controller will only use profiles from the BIG-IP system. | true, false | Both AS3 and CCCL | |
manage-ingress | Boolean | Optional | true | Tells the controller whether or not to watch Kubernetes ingress and apply their configuration. If false, the controller will ignore ingress events. | true, false | Both AS3 and CCCL | |
manage-ingress-class- only | Boolean | Optional | true | Process all ingress resources without
kubernetes.io/ingress.class +
“annotation and ingresses with
annotation
kubernetes.io/ingress.class=f5 ”. |
true, false | AS3 | |
ingress-class | String | Optional | F5 | A class of the Ingress controller. The
Ingress controller only processes
Ingress resources that belong to its
class–i.e. have the annotation
kubernetes.io/ingress.class equal
to the class. Additionally, the Ingress
controller processes Ingress resources
that do not have that annotation, which
can be disabled by setting the
-manage-ingress-class-only flag. |
true, false | AS3 | |
virtualServerHTTPPort | Integer | Optional | N/A | Creates a Virtual Server on BIG-IP with VIP custom HTTP port. | AS3 | ||
virtualServerHTTPSPort | Integer | Optional | N/A | Creates a Virtual Server on BIG-IP with VIP custom HTTPS port. | Both AS3 and CCCL | ||
httpTraffic | String | Optional | N/A | Configure the behavior of traffic on HTTP Virtual Server. | httpTraffic = allow -> Allows HTTP
httpTraffic = none -> Only HTTPS
httpTraffic = redirect -> redirects HTTP to HTTPS |
AS3 | |
periodic-sync-interval | Integer | Optional | 30 | Interval (in seconds) at which to queue resources. | AS3 | 2.5.0 |
Note
Use the node-label-selector
parameter if you only want the controller to manage specific nodes from the cluster.
For example, the BIG-IP device may not be able to reach certain nodes, or the BIG-IP device already manages certain
nodes. Therefore, the controller should only watch the nodes that match the environmental constraints (by using a label).
OpenShift Routes¶
The following configuration parameters only apply to OpenShift.
Parameter | Type | Required | Default | Description | Allowed Values | Agent |
---|---|---|---|---|---|---|
custom-client-ssl | String | Optional | N/A | Specifies the name of a custom client SSL profile attached to the route HTTPS virtual server and used as default for SNI. This profile must have the Default for SNI field enabled. | CCCL | |
custom-server-ssl | String | Optional | N/A | Specifies the name of a custom server SSL profile attached to the route HTTPS virtual server and used as default for SNI. This profile must have the Default for SNI field enabled. | CCCL | |
manage-routes | Boolean | Optional | false | Indicates if k8s-bigip-ctlr should
handle OpenShift Route objects. |
true, false | Both AS3 and CCCL |
route-http-vserver | String | Optional | ose-vserver | The name of the http virtual server for OpenShift Routes. | Both AS3 and CCCL | |
route-https-vserver | String | Optional | https-ose-vserver | The name of the https virtual server for OpenShift Routes. | Both AS3 and CCCL | |
route-label | String | Optional | N/A | Tells the k8s-bigip-ctlr to only
watch for OpenShift Route objects with
the f5type label set to this value. |
Both AS3 and CCCL | |
route-vserver-addr | String | Optional | N/A | Bind address for virtual server for OpenShift Route objects. | Both AS3 and CCCL | |
default-client-ssl | String | Optional | N/A | Name of the default client ssl profile. | Both AS3 and CCCL | |
default-server-ssl | String | Optional | N/A | Configuring CIS with –default-server-ssl for OpenShift routes is not recommended. | Both AS3 and CCCL | |
tls-version | String | Optional | 1.2 | Configures TLS version to be enabled on BIG-IP. TLS 1.3 is only supported on TMOS version 14.0+. | Only AS3 | |
cipher-group | String | Optional | N/A | Configures a cipher group in BIG-IP and reference it here. Cipher group and ciphers are mutually exclusive, only use one. | Only AS3 | |
ciphers | String | Optional | N/A | Configures a ciphersuite selection string. Cipher-group and ciphers are mutually exclusive, only use one. | Only AS3 |
Note
If the custom-client-ssl
or custom-server-ssl
parameters are not provided, then the controller creates default
clientssl and serverssl profiles for the OpenShift Route HTTPS virtual server. The controller sets these profiles as
Default for SNI.
NextGen Routes¶
The following configuration parameters only apply to NextGen Routes.
Note
For NextGen Routes, controller-mode must be set as openshift
.
OpenShift Routes and Kubernetes¶
AS3 Parameters¶
Parameter | Type | Required | Default | Description | Minimum supported version |
---|---|---|---|---|---|
as3-validation | Boolean | Optional | true | When set to false, this disables AS3 template validation on the controller. | |
insecure | Boolean | Optional | false | When set to true, this enables insecure SSL communication to the BIG-IP system. | |
trusted-certs-cfgmap | String | Required | N/A | When certificates are provided, adds them to controller’s trusted certificate store. | |
as3-post-delay | int | Optional | N/A | Specifies the time, in seconds, that CIS waits to post the available AS3 declaration. | 2.0.0 |
override-as3- declaration | String | Optional | N/A | Provide Namespace and Name of the ConfigMap as <namespace>/<configmap-name>. The JSON key/values from this ConfigMap will oerride key/values from internally generated AS3 declaration. | |
filter-tenants | Boolean | Optional | false | Specifies whether or not the use tenant filtering API for AS3 declaration. | 2.7.0 |
share-nodes | Boolean | Optional | false | When set to true, the pool member node
is created on /Common partition,
which can be shared among multiple
tenants. This argument is only valid
for Ingress, Routes, and CRDs. |
2.2.0 |
AS3 Response code handling in CIS¶
AS3 Response Code | HTTP Status | CIS Handling (NextGen Mode CRDs/Routes/ServiceTypeLB) | CIS Handling (Legacy Mode) |
---|---|---|---|
200 | OK | CIS does not try to repost AS3 declaration. | CIS does not try to repost AS3 declaration. |
201 | Created | CIS polls for its status continuously and blocks incoming requests. | CIS does not try to repost AS3 declaration. |
202 | Accepted | CIS polls for its status continuously and blocks incoming requests. | CIS does not try to repost AS3 declaration. |
207 | Multi Status | CIS retries every 30 seconds for failed tenants and successful tenant declarations will not be posted. | N/A |
404 | Not Found | If CIS receives 404 at the start-up it immediately exits and the container will be restarted continuously. One of the reasons for a 404 error in this case is that AS3 RPM is not installed on BIG-IP. If CIS receives 404 while posting the AS3 declaration, CIS retries every 30 sec for failed tenants. | If CIS receives 404 at the start-up it immediately exits and the container will be restarted continuously. One of the reasons for a 404 error in this case is that AS3 RPM is not installed on the BIG-IP system. If CIS receives a 404 while posting the AS3 declaration, CIS does not try to repost AS3 declaration. |
422 | Unprocessable Entity | CIS retries every 30 seconds for failed tenants. | CIS retries every 30 seconds for failed tenants. |
503 | Service Unavailable | CIS waits for 30 seconds before trying to repost. | CIS waits for 30 seconds before trying to repost. |
Others | CIS retries every 30 seconds for failed tenants. | CIS retries every 30 seconds for failed tenants. |
CIS Health Checks¶
Kubernetes has two types of health checks:
- Readiness Probes: To determine when a pod is ready
- Liveness Probes: To determine when a pod is healthy or unhealthy after it has become ready
Readiness Probes Kubernetes uses readiness probes to decide when the container is available for accepting traffic. The readiness probe controls which pods to use as the backend for a service. A pod is considered ready when all of its containers are ready. If a pod is not ready, it is removed from service load balancers. For example, if a container loads a large cache at start-up and takes minutes to start, you should not send requests to this container until it is ready, or the requests will fail. Instead, route requests to other pods, which are capable of servicing requests.
Liveness Probes Kubernetes uses liveness probes to know when to restart a container. If a container is unresponsive, the application could be deadlocked due to a multi-threading defect. Restarting the container can make the application more available.
These are the methods you can use to check container status:
- HTTP request to the pod
- Command execution to the pod
- TCP request to the pod
Probes are defined on a container in a deployment.
Here is an example of the deployment using HTTP method:
Parameter | Description |
---|---|
periodSeconds | specifies that the kubelet should perform a liveness probe every 3 seconds. |
initialDelaySeconds | tells the kubelet that it should wait 3 seconds before performing the first probe |
timeOutSeconds | how long to wait for the probe to finish. If this time is exceeded, OpenShift Container Platform considers the probe to have failed. |
The kubelet uses a web hook to determine the healthiness of the container. The check is successful if the HTTP response code is between 200
and 399
.
To perform a probe, the kubelet sends an HTTP GET request to the server that is running in the Container and listening on port 8080. The handler for the server’s /health path returns a success code. For example:
livenessProbe:
failureThreshold: 3
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 15
periodSeconds: 15
successThreshold: 1
timeoutSeconds: 15
readinessProbe:
failureThreshold: 3
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 15
To view the liveness and readiness of the deployed pod, run the command Kubectl describe pod <pod_name> -n kube-system
. Here is the example output:
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
- --log-level=debug
terminationMessagePolicy: Fileermination-log
--log-level=debug
--namespace=default
--route-label=systest
--insecure=true
--agent=cccl
Liveness: http-get http://:8080/health delay=15s timeout=15s period=15s #success=1 #failure=3
Readiness: http-get http://:8080/health delay=30s timeout=15s period=30s #success=1 #failure=3
Environment: <none>
Mounts: <none>
Volumes: <none>
curl http://<self-ip>:<port no>/health
shows a response of OK.
/ready endpoint Advanced endpoint for monitoring the health of CIS. The endpoint works as follows:
curl http://<self-ip>:<port no>/ready
- If everything is working as expected, a successful response of 200 OK is returned.
- If the Kubernetes APIs are unreachable, down, or not accessible (ie: permissions problem), then health checks fail resulting in HTTP error code 500 returning “kube-api server is not reachable.”
- If the BIG-IPs are unreachable, down, or not accessible (ie: permissions problem), then health checks fail resulting in HTTP error code 500 returning “big-ip server is not reachable.”
Prometheus Metrics¶
Name | Type | Default Status | Description | Labels |
---|---|---|---|---|
bigip_monitored_nodes | Gauge | Enabled | Total count of monitored nodes by the BIG-IP k8s CTLR. | ["nodeselector"] |
bigip_current_errors | Gauge | Enabled | Total count of errors occured parsing the configuration. | [] |
bigip_monitored_services | Gauge | Enabled | Total count of monitored services by the BIG-IP k8s CTLR. | ["namespace", "name", "status"] |
bigip_http_client_api_requests_total | Counter | Disabled | A counter for requests from the wrapped HTTP client. | ["code", "method"] |
bigip_http_client_in_flight_requests | Gauge | Disabled | Total count of in-flight requests for the wrapped HTTP client. | [] |
bigip_http_client_dns_duration_seconds | Histogram | Disabled | Trace DNS latency histogram. | ["event"] |
bigip_http_client_tls_duration_seconds | Histogram | Disabled | Trace TLS latency histogram. | ["event"] |
bigip_http_client_request_duration_seconds | Histogram | Disabled | Trace HTTP request latencies histogram. | [] |
Note
Use the --http-client-metrics
deployment parameter to export the AS3 HTTP client prometheus metrics, which are disabled by default.
Rest Methods with CIS¶
CIS uses the following Rest methods along with AS3 and CCCL agents to interact with BIG-IP.
HTTP Method | AS3 | CCCL |
---|---|---|
GET | Yes | Yes |
POST | Yes | Yes |
PUT | Yes | Yes |
DELETE | Yes | Yes |
Note
When CIS is deployed in cluster mode with an AS3 agent, CIS still uses the CCCL agent to update the FDB and ARP entries on BIG-IP.
Note
To provide feedback on Container Ingress Services or this documentation, please file a GitHub Issue.