OpenShift/Kubernetes Multi-Cluster¶
This page documents a new feature in CIS for Multi-Cluster.
Note
To provide feedback on Container Ingress Services or this documentation, please file a GitHub Issue.
Overview¶
Multi-Cluster Support in CIS allows users to expose multiple apps spread across OpenShift/Kubernetes clusters using a single BIG-IP Virtual Server. An app can be deployed in different OpenShift clusters exposing them using a route resource. Using a Multi-Cluster implementation, CIS can be deployed in an HA topology or a standalone CIS to expose the apps spread across OpenShift clusters.
Note
- CIS supports processing of routes in the traditional way as well as with NextGen Controller and multi-cluster support.
- Currently,
nodePort
is supported. Cluster mode is supported with static route configuration on BIG-IP (no tunnels).
Prerequisites¶
- Cluster node, where CIS is deployed, should be able to reach the API server of all OpenShift/Kubernetes clusters.
extended ConfigMap
needs to be created to run CIS in multi-cluster mode.- kube-config files for each cluster should be available for CIS to access resources such as pods, services, endpoints and nodes.
Topologies¶
Standalone CIS¶
In a Standalone deployment of CIS, CIS is only deployed in one cluster. Next, create a route resource with a multi-cluster annotation, or CRD resource, with extendedServiceReferences to expose the apps in different OpenShift/Kubernetes clusters.

Below is the sample multi-cluster config in an extended ConfigMap.
extendedSpec: |
externalClustersConfig: -------------------------------------|----------------------------| |
- clusterName: cluster3 | |
secret: default/kubeconfig3 |---> Cluster configs for |
- clusterName: cluster4 | all other clusters |
secret: default/kubeconfig4 | except HA clusters |
- clusterName: cluster5 | |
secret: default/kubeconfig5 ------------------------------|----------------------------|
extendedRouteSpec:
- namespace: foo -------------------------------------|
vserverAddr: 10.8.0.4 |
vserverName: nextgenroutes |----------------> RouteGroup with namespace
allowOverride: true |
bigIpPartition: MultiTenant |
policyCR: default/sample-policy _____________________|
- namespace: bar -------------------------------------|
vserverAddr: 10.8.0.5 |----------------> RouteGroup with namespace
allowOverride: false _____________________|
Note
extendedRouteSpec
is only applicable in case of OpenShift route resources, not for CRD resources.
High Availability CIS¶
Prerequisites¶
- A pair of high availability OpenShift/Kubernetes clusters should be available, which have the same applications running in both clusters.
- HealthCheck endpoint should be available to check the health of the primary cluster. Currently, TCP/HTTP Health endpoints are supported.
In an HA deployment of CIS, CIS needs to be deployed in both the primary and secondary cluster. Also, the same extended ConfigMap
needs to be deployed in both the primary and secondary cluster.
CIS will look for the same service name in both primary and secondary clusters to expose the application via routes. Additionally, a multi-cluster annotation is created in the route definition exposing the applications in the other clusters.
Deploying CIS HA in Two Modes:
- active-active and active-standby modes are used if CIS is running in high availability mode.
- Based on the provided value the CIS running on Primary cluster decides whether to monitor the workloads running on Secondary Cluster or not.
- If CIS is running in active-active mode on the primary cluster, the pool members from both virtual servers’ clusters (primary and secondary) are part of the HA cluster, as well as members from other remotely monitored clusters.
- If CIS is running in active-standby mode on the primary cluster, the pool members are updated for virtual servers only from the primary cluster, and from all the other remotely monitored clusters except the secondary cluster.
- If the primary cluster is down, and CIS on the secondary cluster has taken control, then pool members from the secondary cluster, as well as all other remotely monitored clusters, are populated for the virtual servers irrespective of the value of HA mode.
Note
- Additionally, there is another supported mode called
ratio
, in which pool members from all the Kubernetes/Openshift clusters are updated for the virtual server. However, the traffic distribution is done based on the ratio values defined for each cluster. - Ratio doesn’t require CIS to be running in an HA environment. It is supported in both CIS HA and non-HA environments.
The tables below represent the scenarios mentioned above.
HA Mode Values | Primary Cluster Pool Members | Secondary Cluster Pool Members | Other Remotely Monitored Cluster Pool Members | Traffic Distribution Based on Ratio |
---|---|---|---|---|
active-active | Yes | Yes | Yes | No |
active-standby | Yes | No | Yes | No |
ratio | Yes | Yes | Yes | Yes |
HA Mode Values | Primary Cluster Pool Members | Secondary Cluster Pool Members | Other Remotely Monitored Cluster Pool Members | Traffic Distribution Based on Ratio |
---|---|---|---|---|
active-active | No | Yes | Yes | No |
active-standby | No | Yes | Yes | No |
ratio | Yes | Yes | Yes | Yes |

Below is an example of multi-cluster configs, with HA, in an extended ConfigMap.
extendedSpec: |
mode: active-active -----------------------------------|----> HA Mode |
highAvailabilityCIS: --------------------------------------|----------------------------|
primaryEndPoint: http://10.145.72.114:8001 | |
probeInterval: 30 | |
retryInterval: 3 | |
primaryCluster: |---> Cluster configs for |
clusterName: cluster1 | High availability |
secret: default/kubeconfig1 | clusters |---> Multi-Cluster configs
secondaryCluster: | |
clusterName: cluster2 | |
secret: default/kubeconfig2 | |
externalClustersConfig: -------------------------------------| |
- clusterName: cluster3 | |
secret: default/kubeconfig3 |---> Cluster configs for |
- clusterName: cluster4 | all other clusters |
secret: default/kubeconfig4 | except HA clusters |
- clusterName: cluster5 | |
secret: default/kubeconfig5 ------------------------------|----------------------------|
extendedRouteSpec:
- namespace: foo -------------------------------------|
vserverAddr: 10.8.0.4 |
vserverName: nextgenroutes |----------------> RouteGroup with namespace
allowOverride: true |
bigIpPartition: MultiTenant |
policyCR: default/sample-policy _____________________|
- namespace: bar -------------------------------------|
vserverAddr: 10.8.0.5 |----------------> RouteGroup with namespace
allowOverride: false _____________________|
Below is the sample multi-cluster configs, with HA and ratio, in an extended ConfigMap.
extendedSpec: |
mode: ratio ------------------------------------------|----------------------------|
localClusterRatio: 4 | |
highAvailabilityCIS: --------------------------------------| |
primaryEndPoint: http://10.145.72.114:8001 | |
probeInterval: 30 | |
retryInterval: 3 | |
primaryCluster: |---> Cluster configs for |
clusterName: cluster1 | High availability |
secret: default/kubeconfig1 | clusters |---> Multi-Cluster configs
ratio: 3 | |
secondaryCluster: | |
clusterName: cluster2 | |
secret: default/kubeconfig2 | |
ratio: 2 | |
externalClustersConfig: -------------------------------------| |
- clusterName: cluster3 | |
secret: default/kubeconfig3 |---> Cluster configs for |
ratio: 2 | all other clusters |
- clusterName: cluster4 | except HA clusters |
secret: default/kubeconfig4 | |
- clusterName: cluster5 | |
secret: default/kubeconfig5 | |
ratio: 1 ------------------------------|----------------------------|
extendedRouteSpec:
- namespace: foo -------------------------------------|
vserverAddr: 10.8.0.4 |
vserverName: nextgenroutes |----------------> RouteGroup with namespace
allowOverride: true |
bigIpPartition: MultiTenant |
policyCR: default/sample-policy _____________________|
- namespace: bar -------------------------------------|
vserverAddr: 10.8.0.5 |----------------> RouteGroup with namespace
allowOverride: false _____________________|
Note
extendedRouteSpec
is only applicable in the case of openshift route resources, not for CRD resources.
Below is the sample Multi-Cluster Configs with HA and cluster AdminState in extended ConfigMap.
extendedSpec: |
mode: active-active --------------------------------------|----------------------------|
highAvailabilityCIS: --------------------------------------| |
primaryEndPoint: http://10.145.72.114:8001 | |
probeInterval: 30 | |
retryInterval: 3 | |
primaryCluster: |---> Cluster configs for |
clusterName: cluster1 | High availability |
secret: default/kubeconfig1 | clusters |---> Multi-Cluster configs
secondaryCluster: | |
clusterName: cluster2 | |
secret: default/kubeconfig2 | |
adminState: enable | |
externalClustersConfig: ----------------------------------| |
- clusterName: cluster3 | |
secret: default/kubeconfig3 |---> Cluster configs for |
adminState: disable | all other clusters |
- clusterName: cluster4 | except HA clusters |
secret: default/kubeconfig4 | |
- clusterName: cluster5 | |
secret: default/kubeconfig5 | |
adminState: offline ------------------------------|----------------------------|
extendedRouteSpec:
- namespace: foo -------------------------------------|
vserverAddr: 10.8.0.4 |
vserverName: nextgenroutes |----------------> RouteGroup with namespace
allowOverride: true |
bigIpPartition: MultiTenant |
policyCR: default/sample-policy _____________________|
- namespace: bar -------------------------------------|
vserverAddr: 10.8.0.5 |----------------> RouteGroup with namespace
allowOverride: false _____________________|
Note
extendedRouteSpec
is only applicable for OpenShift route resources, not for CRD resources.
Below is the sample Multi-Cluster Configs with standalone CIS and cluster AdminState in extended ConfigMap.
extendedSpec: |
localClusterAdminState: disable ----------------------------|AdminState for local cluster|
externalClustersConfig: ----------------------------------|----------------------------|
- clusterName: cluster3 | |
secret: default/kubeconfig3 |---> Cluster configs for |
adminState: enable | all other clusters |
- clusterName: cluster4 | except HA clusters |
secret: default/kubeconfig4 | |
- clusterName: cluster5 | |
secret: default/kubeconfig5 | |
adminState: offline ------------------------------|----------------------------|
extendedRouteSpec:
- namespace: foo -------------------------------------|
vserverAddr: 10.8.0.4 |
vserverName: nextgenroutes |----------------> RouteGroup with namespace
allowOverride: true |
bigIpPartition: MultiTenant |
policyCR: default/sample-policy _____________________|
- namespace: bar -------------------------------------|
vserverAddr: 10.8.0.5 |----------------> RouteGroup with namespace
allowOverride: false _____________________|
Note
localClusterAdminState
is only applicable in case of standalone CIS. It’s ignored if specified in HA CIS mode.
Configuration¶
OpenShift Routes with multi-cluster¶
CIS Deployment Parameter¶
If you are using multi-cluster mode, the --multi-cluster-mode
parameter is required.
Parameter | Type | Required | Description | Allowed Values |
---|---|---|---|---|
multi-cluster mode | String | Required | Specify whether CIS is running standalone or as primary/secondary in the case of a high availability topology | standalone, primary, or secondary |
Note
Standalone refers to the standalone topology of CIS deployment. See Standalone CIS
Following is the example for a primary CIS deployment:
spec:
containers:
- args:
- --bigip-partition
- <partition>
- --bigip-url
- <ip-address>
- --bigip-username
- <user-name>
- --bigip-password
- <password>
- --log-level
- DEBUG
- --insecure
- --controller-mode=openshift
- --extended-spec-configmap=kube-system/extended-spec-config
- --route-label=systest
- --pool-member-type
- nodeport
- --multi-cluster-mode=primary
command:
- /app/bin/k8s-bigip-ctlr
image: <image-name>
Note
- Update the
multi-cluster-mode
to secondary for secondary CIS deployments in high availablility topology. See High Availability CIS - Update the
multi-cluster-mode
to standalone for standalone topology. See Standalone CIS
Extended ConfigMap Parameters¶
Parameter | Type | Required | Description | Default | Examples |
---|---|---|---|---|---|
clusterName | String | Mandatory | Name of the cluster | cluster1 | |
secret | String | Mandatory | Name of the secret created for kubeconfig. Format: namespace/secretname | test/secret- kubeconfig1 | |
ratio | Integer | Optional | Ratio at which the traffic has to be distributed over clusters. | 1 | 3 |
adminState | String | Optional | adminState can be used to disable/enable/offline clusters | enable | disable |
Note
Avoid specifying HA cluster (primary/secondary cluster) configs in externalClustersConfig.
Parameter | Type | Required | Description | Default | Examples |
---|---|---|---|---|---|
mode | Object | Optional | Type of high availability mode (active-active/active-standby/ratio) | active-standby | active-active |
Specifies whether the CIS HA cluster is configured with active-active mode, active-standby mode or ratio mode.
- If mode Type: active-active, then CIS fetches services from both the HA clusters whenever it’s referenced in Route Spec.
- If mode Type: active-standby, the default, CIS fetches service from only the local cluster whenever it’s referenced in a Route Spec.
- If mode Type: ratio, then CIS works in active-active mode and, splits the traffic according to the ratio specified for each cluster.
Parameter | Type | Required | Description | Default | Examples |
---|---|---|---|---|---|
localClusterRatio | Int | Optional | Ratio for the local cluster where CIS is running. Specify only when using ratio in a CIS non-HA environment. | active-standby | active-active |
Note: It is not needed in the case of using ratio in a CIS HA environment as ratio of the primary cluster does the same thing. If specified in this scenario, then it will be ignored. |
Parameter | Type | Required | Description | Default | Examples |
---|---|---|---|---|---|
primaryClusterEndPoint | String | Mandatory | Endpoint to check the health of the primary cluster. | http://10.145.72.114:8001 | |
probeInterval | Integer | Optional | Time interval between health checks, in seconds. | 60 | 30 |
retryInterval | Integer | Optional | Time interval between recheck when primary cluster is down, in seconds. | 15 | 3 |
primaryCluster | Object | Mandatory | Primary cluster config. | ||
secondaryCluster | Object | Mandatory | Secondary cluster config. |
Parameter | Type | Required | Description | Default | Examples |
---|---|---|---|---|---|
clusterName | String | Mandatory | Name of the cluster. | cluster1 | |
secret | String | Mandatory | Name of the secret created for kubeconfig. Format: namespace/secret-name. | test/secret-kubeconfig1 | |
ratio | Integer | Optional | Ratio at which the traffic has to be distributed over clusters. | 1 | 3 |
adminState | String | Optional | adminState can be used to disable/enable/offline clusters. | enable | disable |
Note
In order to run CIS in high availability mode, multi-cluster-mode parameter (primary/secondary) needs to be set in the CIS deployment arguments.
- It is recommended to provide both primaryCluster and secondaryCluster configs in the extended ConfigMap.
- If no traffic has to be forwarded to a specific cluster, then set the ratio field to 0.
CIS Primary Cluster Endpoint
Health probe parameters are provided in highAvailabilityCIS in extended ConfigMap, helping to ensure high availability of CIS. CIS running in the secondary cluster continuously monitors the health of the primary cluster. If it is down, then the secondary CIS takes the responsibility of posting declarations to BIG-IP.
Note
primaryEndPoint
is a mandatory parameter if CIS is intended to run in multi-cluster HA mode. If this is not specified, then the secondary CIS will not run.
Route Annotation for Multi-ClusterServices¶
Services running in any other OpenShift clusters, apart from the HA cluster pair, can be referenced in the route annotations as mentioned below:
virtual-server.f5.com/multiClusterServices:
'[
{
"clusterName": "cluster2",
"serviceName": "svc-pytest-foo-1-com",
"namespace": "foo",
"port": 80,
"weight": 30,
}
]'
CRD Resources with Multi-Cluster¶
CIS Deployment Parameter¶
Note
In this example, standalone refers to standalone topology of CIS deployment, See Standalone CIS
Sample deployment for primary CIS deployment:
spec:
containers:
- args:
- --bigip-partition
- <partition>
- --bigip-url
- <ip-address>
- --bigip-username
- <user-name>
- --bigip-password
- <password>
- --log-level
- DEBUG
- --insecure
- --custom-resource-mode=true
- --extended-spec-configmap=kube-system/extended-spec-config
- --pool-member-type
- nodeport
- --multi-cluster-mode=primary
command:
- /app/bin/k8s-bigip-ctlr
image: <image-name>
Note
- Update the
multi-cluster-mode
to secondary for secondary CIS deployments in high availability topology, See High Availability CIS - Update the
multi-cluster-mode
to standalone for standalone topology, See Standalone CIS - weight needs to be specified only in A/B scenario.
Virtual Server Pool with Multi-ClusterServices¶
Services running in any other OpenShift/Kubernetes clusters, apart from the HA cluster pair, can be referenced in the VS Pool as mentioned below:
pools:
- path: /tea
serviceNamespace: tea
service: svc-2
servicePort: 80
extendedServiceReferences:
- clusterName: cluster2
namespace: ns1
port: 8080
serviceName: svc-1
- clusterName: cluster3
namespace: ns2
port: 80
serviceName: svc-ext-1
Transport Server Pool with Multi-ClusterServices¶
Services running in any other OpenShift/Kubernetes clusters, apart from the HA cluster pair, can be referenced in the TS Pool as mentioned below:
pool:
service: svc-1
servicePort: 8181
extendedServiceReferences:
- clusterName: cluster2
serviceName: svc-1
namespace: ns1
port: 8181
- clusterName: cluster3
serviceName: svc-ext-1
namespace: ns2
port: 8282
Static Routing Mode¶
CIS supports configuring static routes in BIG-IP with node subnets assigned for the nodes in the OpenShift/Kubernetes cluster. This enables direct routing from BIG-IP to k8s pods in cluster mode without vxaln tunnel configuration on BIG-IP.
Configuration¶
To enable the static route configuration, set --static-routing-mode
to true and --orchestration-cni
to CNI configured in the cluster.
Parameter | Type | Required | Description | Allowed Values | Default | Agent |
---|---|---|---|---|---|---|
static-routing-mode | Boolean | Optional | Adds static routes on the BIG-IP so traffic can be directly routed to the pods, without tunnels. | true/false | false | AS3 |
orchestration-cni | String | Optional | Kubernetes cluster CNI name. | cilium-k8s, flannel, ovn-k8s, antrea | flannel | AS3 |
shared-static-routes | Boolean | Optional | Flag to enable configuration of static routes on BIG-IP in common partition. | true/false | false | AS3 |
CIS Deployment Parameter¶
Note
In this section, standalone refers to the standalone topology of CIS deployment, See Standalone CIS
Sample deployment for primary CIS deployment:
spec:
containers:
- args:
- --bigip-partition
- <partition>
- --bigip-url
- <ip-address>
- --bigip-username
- <user-name>
- --bigip-password
- <password>
- --log-level
- DEBUG
- --insecure
- --custom-resource-mode=true
- --static-routing-mode=true
- --orchestration-cni=ovn-k8s
- --shared-static-routes=true
- --extended-spec-configmap=kube-system/extended-spec-config
- --pool-member-type
- cluster
- --multi-cluster-mode=primary
command:
- /app/bin/k8s-bigip-ctlr
image: <image-name>
Parameter | Type | Required | Description | Default | Examples |
---|---|---|---|---|---|
clusterName | String | Mandatory | Name of the cluster. | cluster1 | |
serviceName | String | Mandatory | Name of the service. | svc-1 | |
namespace | String | Mandatory | Namespace where the service is created. | test | |
port | String/Int | Mandatory | Port of the service. For named port, use the string value. | 80 | |
weight | Integer | Optional | Weight used for traffic splitting. | 0 | 20 |
Cluster Wise Ratio for Traffic Distribution¶
CIS supports distribution of traffic across clusters as per the ratio configured for each cluster in the extended ConfigMap. It works even along with A/B where different weights are defined for each service. In such a case the ratio of traffic distribution is computed taking into consideration both the service weights and cluster ratio. However, the ratio of the clusters, those haven’t hosted any services linked to the concerned route are not taken into consideration while computing the final ratio.
Note
- Cluster wise ratio for traffic distribution is supported in HA as well as non-HA CIS environments.
- Ratio is only supported for NextGen Routes and Virtual Server CR.
- Setting cluster adminState, in conjunction with cluster ratio, will affect the overall traffic distribution across clusters. As the clusters marked as disable or offline, they will not receive traffic. Any ratio defined for these clusters will be rendered ineffective. Thus, in such a scenario it’s recommended to set the cluster ratio to 0 for all clusters marked with disable/offline.
Cluster adminState to enable/disable/offline a cluster¶
adminState
can be provided for a cluster to dictate the state of the particular cluster.
Supported values for adminState are enable, disable, and offline. By default the clusters are in an enabled state.
adminState enable: all new connections are allowed to the pool members from the cluster.
adminState disable: all new connections, except those which match an existing persistence session, are not allowed for the pool members from the cluster.
adminState offline: no new connections are allowed to the pool members from the cluster, even if they match an existing persistence session.
Examples¶
For more examples, see Config Examples
Known issues¶
- Pool members are not getting populated for extended service in ratio mode.
- CIS doesn’t update pool members if the service doesn’t exist in the primary cluster, but exists in the secondary cluster for route.
- CIS, on start up in multiCluster mode, if any external cluster kube-api server is down/not reachable, CIS is stuck and not processing any valid clusters config. Workaround is to remove the unreachable cluster config from ConfigMap and restart CIS.
- CIS fails to post a VS declaration with health monitors in ratio mode. Issue is observed intermittently.
- Route status is not updated in other HA clusters. For example: Active Primary CIS cluster doesn’t update the route status in the Secondary HA cluster and vice-versa.
- CIS, on switch over from ratio to active-standby mode, doesn’t add the external cluster services. For any change to the mode, it is always recommended to restart CIS.
FAQ¶
Is –multi-cluster-mode is a required parameter for Multi-Cluster support?
Yes. Multi-Cluster support only works if --multi-cluster-mode
is defined in the CIS deployment.
Is extended ConfigMap mandatory for Multi-Cluster support?
Yes. Multi-Cluster support only works with extended ConfigMap.
How do you add a new cluster?
To add a new cluster, create a kube-config file with read-only permissions. Next, create a Kubernetes secret using the kube-config file. Refer this in secret in the extended ConfigMap to add the new cluster. CIS dynamically reads the new kube-config of the new cluster and starts listening to the services and endpoints in the new cluster when a route refers this new cluster.
Where do you manage the manifest or configuration objects such as routes, configmaps, etc.?
Manifests or configuration objects are managed centralized in the primary cluster and, if HA is desired, the same manifests are expected to be in the secondary cluster.
What are the supported CNIs?
Currently, NodePort mode is supported. For cluster mode, static routing mode is supported to enable configuration of static routes on BIG-IP for pod network subnets for direct routing from BIG-IP to k8s pods
What kind of providers are supported?
CIS supports hybrid cloud, and any public cloud providers such as; AWS, Azure, GCP, On-Prem, VmWare, Tanzu etc., which are in the same network/datacenter and can communicate with each other.
What kind of clusters are supported?
CIS multicluster solution is currently validated with OpenShift clusters and k8s clusters
How does CIS start as a secondary cluster?
CIS recognizes the cluster as secondary when it starts with a deployment parameter i.e. --multi-cluster-mode=secondary
How does the secondary CIS learn about the primary cluster endpoint state in HA mode?
Both of the CIS’ will communicate with both k8s API servers and prepares the AS3 declaration. The secondary CIS only sends the declaration when the primary cluster’s health probe fails. As soon as primary cluster comes up, secondary CIS stops sending the declaration.
What kind of permission is required for HA or standalone deployment of CIS?
No RBAC change for CIS deployment with multiCluster support. Only an additional kube-config configuration with read only permission is required to access the endpoints from an external cluster.
What kind of permission is required to access external clusters, apart from HA and standalone?
CIS requires read-only permissions in kube-config of external clusters to access resources like pods, services, endpoints and nodes.
Can CIS manage multiple BIG-IPs?
No. CIS can manage only standalone BIG-IP or an HA BIG-IP. In other words, CIS acts as a single point of BIG-IP orchestrator and supports multi-cluster.
Is traffic splitting with cluster ratio supported?
Yes. CIS supports traffic splitting as per the ratio specified for each cluster and also works with A/B as well.
Is A/B supported in multiCluster mode?
Yes. CIS supports A/B with multiCluster.
Is A/B custom persistence supported in all the modes?
No. A/B persistence is supported in ratio mode and pool member type as cluster.
Does extended ConfigMap update require CIS restart?
No. It’s recommended to restart CIS if any HA configuration, or external cluster configurations, are updated in extended ConfigMap. However, CIS restart is not required when updating ratio in the extended ConfigMap.
Does mode update require CIS restart?
Yes. CIS has to be restarted when there is a change in the mode.