OpenShift/Kubernetes Multi-Cluster¶
This page documents a new feature in CIS for Multi-Cluster.
Note
To provide feedback on Container Ingress Services or this documentation, please file a GitHub Issue.
Contents¶
Overview¶
Multi-Cluster Support in CIS allows users to expose multiple apps spread across OpenShift/Kubernetes clusters using a single BIG-IP Virtual Server. An app can be deployed in different OpenShift/Kubernetes clusters exposing them using a Route resource and VS/TS CR. Using a Multi-Cluster implementation, CIS can be deployed in an HA topology or a standalone CIS to expose the apps spread across OpenShift/Kubernetes clusters.
Note
- CIS supports processing of routes in the traditional way as well as with NextGen Controller and multi-cluster support.
- At present,
nodePort
mode is supported and Cluster mode is available with static route configuration on BIG-IP (No tunnels).
Prerequisites¶
- Cluster node, where CIS is deployed, should be able to reach the API server of all OpenShift/Kubernetes clusters.
extended ConfigMap
needs to be created to run CIS in multi-cluster mode.- kube-config files for each cluster should be available for CIS to access resources such as pods, services, endpoints and nodes.
Topologies¶
Standalone CIS¶
In a Standalone deployment of CIS, CIS is only deployed in one cluster. Next, create a route resource with a multi-cluster annotation, or CRD resource, with extendedServiceReferences to expose the apps in different OpenShift/Kubernetes clusters.
Below is the sample multi-cluster config in an extended ConfigMap.
extendedSpec: |
externalClustersConfig: -------------------------------------|----------------------------| |
- clusterName: cluster3 | |
secret: default/kubeconfig3 |---> Cluster configs for |
- clusterName: cluster4 | all other clusters |
secret: default/kubeconfig4 | except HA clusters |
- clusterName: cluster5 | |
secret: default/kubeconfig5 ------------------------------|----------------------------|
extendedRouteSpec:
- namespace: foo -------------------------------------|
vserverAddr: 10.8.0.4 |
vserverName: nextgenroutes |----------------> RouteGroup with namespace
allowOverride: true |
bigIpPartition: MultiTenant |
policyCR: default/sample-policy _____________________|
- namespace: bar -------------------------------------|
vserverAddr: 10.8.0.5 |----------------> RouteGroup with namespace
allowOverride: false _____________________|
Note
extendedRouteSpec
is only applicable in case of OpenShift route resources, not for CRD resources.
High Availability CIS¶
Prerequisites¶
- A pair of high availability OpenShift/Kubernetes clusters should be available, which have the same applications running in both clusters.
- HealthCheck endpoint should be available to check the health of the primary cluster. Currently, TCP/HTTP Health endpoints are supported.
In an HA deployment of CIS, CIS needs to be deployed in both the primary and secondary cluster. Also, the same extended ConfigMap
needs to be deployed in both the primary and secondary cluster.
CIS will look for the same service name in both primary and secondary clusters to expose the application via routes. Additionally, a multi-cluster annotation is created in the route definition exposing the applications in the other clusters.
Deploying CIS HA in Two Modes:
- active-active and active-standby modes are used if CIS is running in high availability mode.
- Based on the provided value the CIS running on Primary cluster decides whether to monitor the workloads running on Secondary Cluster or not.
- If CIS is running in active-active mode on the primary cluster, the pool members from both virtual servers’ clusters (primary and secondary) are part of the HA cluster, as well as members from other remotely monitored clusters.
- If CIS is running in active-standby mode on the primary cluster, the pool members are updated for virtual servers only from the primary cluster, and from all the other remotely monitored clusters except the secondary cluster.
- If the primary cluster is down, and CIS on the secondary cluster has taken control, then pool members from the secondary cluster, as well as all other remotely monitored clusters, are populated for the virtual servers irrespective of the value of HA mode.
Note
- Additionally, there is another supported mode called
ratio
, in which pool members from all the Kubernetes/Openshift clusters are updated for the virtual server. However, the traffic distribution is done based on the ratio values defined for each cluster. - Ratio does not require CIS to be running in an HA environment. It is supported in both CIS HA and non-HA environments.
Note
- For HA mode [Active-Standby, Active-Active, Ratio], CIS monitored resource manifests(such as routes, CRDs, extendedConfigmaps) must be available in both the clusters.
- The CIS monitored resource manifests must be identical on both primary and Secondary Clusters
- So, In case of CIS failover, CIS on Secondary Cluster will take control and will start processing the CIS monitored resource manifests.
- CIS on Secondary Cluster will not process the CIS monitored resource manifests if they are not available in Secondary Cluster.
- MakeSure to have identical resource manifests in both the clusters to avoid any issues during CIS failover.
The tables below represent the scenarios mentioned above.
HA Mode Values | Primary Cluster Pool Members | Secondary Cluster Pool Members | Other Remotely Monitored Cluster Pool Members | Traffic Distribution Based on Ratio |
---|---|---|---|---|
active-active | Yes | Yes | Yes | No |
active-standby | Yes | No | Yes | No |
ratio | Yes | Yes | Yes | Yes |
HA Mode Values | Primary Cluster Pool Members | Secondary Cluster Pool Members | Other Remotely Monitored Cluster Pool Members | Traffic Distribution Based on Ratio |
---|---|---|---|---|
active-active | No | Yes | Yes | No |
active-standby | No | Yes | Yes | No |
ratio | Yes | Yes | Yes | Yes |
Below is an example of multi-cluster configs, with HA, in an extended ConfigMap.
extendedSpec: |
mode: active-active -----------------------------------|----> HA Mode |
highAvailabilityCIS: --------------------------------------|----------------------------|
primaryEndPoint: http://10.145.72.114:8001 | |
probeInterval: 30 | |
retryInterval: 3 | |
primaryCluster: |---> Cluster configs for |
clusterName: cluster1 | High availability |
secret: default/kubeconfig1 | clusters |---> Multi-Cluster configs
secondaryCluster: | |
clusterName: cluster2 | |
secret: default/kubeconfig2 | |
externalClustersConfig: -------------------------------------| |
- clusterName: cluster3 | |
secret: default/kubeconfig3 |---> Cluster configs for |
- clusterName: cluster4 | all other clusters |
secret: default/kubeconfig4 | except HA clusters |
- clusterName: cluster5 | |
secret: default/kubeconfig5 ------------------------------|----------------------------|
extendedRouteSpec:
- namespace: foo -------------------------------------|
vserverAddr: 10.8.0.4 |
vserverName: nextgenroutes |----------------> RouteGroup with namespace
allowOverride: true |
bigIpPartition: MultiTenant |
policyCR: default/sample-policy _____________________|
- namespace: bar -------------------------------------|
vserverAddr: 10.8.0.5 |----------------> RouteGroup with namespace
allowOverride: false _____________________|
Below is the sample multi-cluster configs, with HA and ratio, in an extended ConfigMap.
extendedSpec: |
mode: ratio ------------------------------------------|----------------------------|
localClusterRatio: 4 | |
highAvailabilityCIS: --------------------------------------| |
primaryEndPoint: http://10.145.72.114:8001 | |
probeInterval: 30 | |
retryInterval: 3 | |
primaryCluster: |---> Cluster configs for |
clusterName: cluster1 | High availability |
secret: default/kubeconfig1 | clusters |---> Multi-Cluster configs
ratio: 3 | |
secondaryCluster: | |
clusterName: cluster2 | |
secret: default/kubeconfig2 | |
ratio: 2 | |
externalClustersConfig: -------------------------------------| |
- clusterName: cluster3 | |
secret: default/kubeconfig3 |---> Cluster configs for |
ratio: 2 | all other clusters |
- clusterName: cluster4 | except HA clusters |
secret: default/kubeconfig4 | |
- clusterName: cluster5 | |
secret: default/kubeconfig5 | |
ratio: 1 ------------------------------|----------------------------|
extendedRouteSpec:
- namespace: foo -------------------------------------|
vserverAddr: 10.8.0.4 |
vserverName: nextgenroutes |----------------> RouteGroup with namespace
allowOverride: true |
bigIpPartition: MultiTenant |
policyCR: default/sample-policy _____________________|
- namespace: bar -------------------------------------|
vserverAddr: 10.8.0.5 |----------------> RouteGroup with namespace
allowOverride: false _____________________|
Note
extendedRouteSpec
is only applicable in the case of openshift route resources, not for CRD resources.
Below is the sample Multi-Cluster Configs with HA and cluster AdminState in extended ConfigMap.
extendedSpec: |
mode: active-active --------------------------------------|----------------------------|
highAvailabilityCIS: --------------------------------------| |
primaryEndPoint: http://10.145.72.114:8001 | |
probeInterval: 30 | |
retryInterval: 3 | |
primaryCluster: |---> Cluster configs for |
clusterName: cluster1 | High availability |
secret: default/kubeconfig1 | clusters |---> Multi-Cluster configs
secondaryCluster: | |
clusterName: cluster2 | |
secret: default/kubeconfig2 | |
adminState: enable | |
externalClustersConfig: ----------------------------------| |
- clusterName: cluster3 | |
secret: default/kubeconfig3 |---> Cluster configs for |
adminState: disable | all other clusters |
- clusterName: cluster4 | except HA clusters |
secret: default/kubeconfig4 | |
- clusterName: cluster5 | |
secret: default/kubeconfig5 | |
adminState: offline ------------------------------|----------------------------|
extendedRouteSpec:
- namespace: foo -------------------------------------|
vserverAddr: 10.8.0.4 |
vserverName: nextgenroutes |----------------> RouteGroup with namespace
allowOverride: true |
bigIpPartition: MultiTenant |
policyCR: default/sample-policy _____________________|
- namespace: bar -------------------------------------|
vserverAddr: 10.8.0.5 |----------------> RouteGroup with namespace
allowOverride: false _____________________|
Note
extendedRouteSpec
is only applicable for OpenShift route resources, not for CRD resources.
Below is the sample Multi-Cluster Configs with standalone CIS and cluster AdminState in extended ConfigMap.
extendedSpec: |
localClusterAdminState: disable ----------------------------|AdminState for local cluster|
externalClustersConfig: ----------------------------------|----------------------------|
- clusterName: cluster3 | |
secret: default/kubeconfig3 |---> Cluster configs for |
adminState: enable | all other clusters |
- clusterName: cluster4 | except HA clusters |
secret: default/kubeconfig4 | |
- clusterName: cluster5 | |
secret: default/kubeconfig5 | |
adminState: offline ------------------------------|----------------------------|
extendedRouteSpec:
- namespace: foo -------------------------------------|
vserverAddr: 10.8.0.4 |
vserverName: nextgenroutes |----------------> RouteGroup with namespace
allowOverride: true |
bigIpPartition: MultiTenant |
policyCR: default/sample-policy _____________________|
- namespace: bar -------------------------------------|
vserverAddr: 10.8.0.5 |----------------> RouteGroup with namespace
allowOverride: false _____________________|
Note
localClusterAdminState
is only applicable in case of standalone CIS. It’s ignored if specified in HA CIS mode.
extendedRouteSpec
is only applicable for OpenShift route resources, not for CRD resources.
Configuration¶
OpenShift Routes with multi-cluster¶
CIS Deployment Parameter for NextGenRoutes¶
If you are using multi-cluster mode, the --multi-cluster-mode
parameter is required.
Parameter | Type | Required | Description | Allowed Values |
---|---|---|---|---|
multi-cluster-mode | String | Required | Specify whether CIS is running standalone or as primary/secondary in the case of a high availability topology | standalone, primary, or secondary |
Note
Standalone refers to the standalone topology of CIS deployment. See Standalone CIS
Following is the example for a primary CIS deployment:
spec:
containers:
- args:
- --bigip-partition
- <partition>
- --bigip-url
- <ip-address>
- --bigip-username
- <user-name>
- --bigip-password
- <password>
- --log-level
- DEBUG
- --insecure
- --controller-mode=openshift
- --extended-spec-configmap=kube-system/extended-spec-config
- --route-label=systest
- --pool-member-type
- nodeport
- --multi-cluster-mode=primary
command:
- /app/bin/k8s-bigip-ctlr
image: <image-name>
Note
- Update the
multi-cluster-mode
to secondary for secondary CIS deployments in high availablility topology. See High Availability CIS - Update the
multi-cluster-mode
to standalone for standalone topology. See Standalone CIS
Note
- For HA mode [Active-Standby, Active-Active, Ratio], CIS monitored resource manifests(such as routes, CRDs, extendedConfigmaps) must be available in both the clusters.
- The CIS monitored resource manifests must be identical on both primary and Secondary Clusters
- So, In case of CIS failover, CIS on Secondary Cluster will take control and will start processing the CIS monitored resource manifests.
- CIS on Secondary Cluster will not process the CIS monitored resource manifests if they are not available in Secondary Cluster.
- MakeSure to have identical resource manifests in both the clusters to avoid any issues during CIS failover.
Extended ConfigMap Parameters¶
Parameter | Type | Required | Description | Default | Examples |
---|---|---|---|---|---|
clusterName | String | Mandatory | Name of the cluster | cluster1 | |
secret | String | Mandatory | Name of the secret created for kubeconfig. Format: namespace/secretname | test/secret- kubeconfig1 | |
ratio | Integer | Optional | Ratio at which the traffic has to be distributed over clusters. | 1 | 3 |
adminState | String | Optional | adminState can be used to disable/enable/offline /no-pool clusters | enable | disable |
Note
Avoid specifying HA cluster (primary/secondary cluster) configs in externalClustersConfig.
Parameter | Type | Required | Description | Default | Examples |
---|---|---|---|---|---|
mode | Object | Optional | Type of high availability mode (active-active/active-standby/ratio) | active-standby | active-active |
Specifies whether the CIS HA cluster is configured with active-active mode, active-standby mode or ratio mode.
- If mode Type: active-active, then CIS fetches services from both the HA clusters whenever it’s referenced in Route Spec.
- If mode Type: active-standby, the default, CIS fetches service from only the local cluster whenever it’s referenced in a Route Spec.
- If mode Type: ratio, then CIS works in active-active mode and, splits the traffic according to the ratio specified for each cluster.
Parameter | Type | Required | Description | Default | Example |
---|---|---|---|---|---|
localClusterRatio | Int | Optional | Ratio for the local cluster where CIS is running. Specify only when using ratio in a CIS non-HA environment. | 1 | 3 |
Note: It is not needed in the case of using ratio in a CIS HA environment as ratio of the primary cluster does the same thing. If specified in this scenario, then it will be ignored. |
Parameter | Type | Required | Description | Default | Examples |
---|---|---|---|---|---|
primaryClusterEndPoint | String | Mandatory | Endpoint to check the health of the primary cluster. | http://10.145.72.114:8001 | |
probeInterval | Integer | Optional | Time interval between health checks, in seconds. | 60 | 30 |
retryInterval | Integer | Optional | Time interval between recheck when primary cluster is down, in seconds. | 15 | 3 |
primaryCluster | Object | Mandatory | Primary cluster config. | ||
secondaryCluster | Object | Mandatory | Secondary cluster config. |
Parameter | Type | Required | Description | Default | Examples |
---|---|---|---|---|---|
clusterName | String | Mandatory | Name of the cluster. | cluster1 | |
secret | String | Mandatory | Name of the secret created for kubeconfig. Format: namespace/secret-name. | test/secret-kubeconfig1 | |
ratio | Integer | Optional | Ratio at which the traffic has to be distributed over clusters. | 1 | 3 |
adminState | String | Optional | adminState can be used to disable/enable/offline /no-pool clusters. | enable | disable |
Note
In order to run CIS in high availability mode, multi-cluster-mode parameter (primary/secondary) needs to be set in the CIS deployment arguments.
- It is recommended to provide both primaryCluster and secondaryCluster configs in the extended ConfigMap.
- If no traffic has to be forwarded to a specific cluster, then set the ratio field to 0.
CIS Primary Cluster Endpoint
Health probe parameters are provided in highAvailabilityCIS in extended ConfigMap, helping to ensure high availability of CIS. CIS running in the secondary cluster continuously monitors the health of the primary cluster. If it is down, then the secondary CIS takes the responsibility of posting declarations to BIG-IP.
Note
primaryEndPoint
is a mandatory parameter if CIS is intended to run in multi-cluster HA mode. If this is not specified, then the secondary CIS will not run.
Route Annotation for Multi-ClusterServices¶
Services running in any other OpenShift clusters, apart from the HA cluster pair, can be referenced in the route annotations as mentioned below:
Note
Starting with CIS 2.16.0, in multicluster, serviceName is replaced with service and port is replaced with servicePort in the multiClusterServices
.
virtual-server.f5.com/multiClusterServices:
'[
{
"clusterName": "cluster2",
"service": "svc-pytest-foo-1-com",
"namespace": "foo",
"servicePort": 80,
"weight": 30,
}
]'
CRD Resources with Multi-Cluster¶
CIS Deployment Parameters for CRDs¶
Note
In this example, standalone refers to standalone topology of CIS deployment, See Standalone CIS
Sample deployment for primary CIS deployment:
spec:
containers:
- args:
- --bigip-partition
- <partition>
- --bigip-url
- <ip-address>
- --bigip-username
- <user-name>
- --bigip-password
- <password>
- --log-level
- DEBUG
- --insecure
- --custom-resource-mode=true
- --extended-spec-configmap=kube-system/extended-spec-config
- --pool-member-type
- nodeport
- --multi-cluster-mode=primary
command:
- /app/bin/k8s-bigip-ctlr
image: <image-name>
Note
- Update the
multi-cluster-mode
to secondary for secondary CIS deployments in high availability topology, See High Availability CIS - Update the
multi-cluster-mode
to standalone for standalone topology, See Standalone CIS - weight needs to be specified only in A/B scenario.
Note
- For HA mode [Active-Standby, Active-Active, Ratio], CIS monitored resource manifests(such as routes, CRDs, extendedConfigmaps) must be available in both the clusters.
- The CIS monitored resource manifests must be identical on both primary and Secondary Clusters
- So, In case of CIS failover, CIS on Secondary Cluster will take control and will start processing the CIS monitored resource manifests.
- CIS on Secondary Cluster will not process the CIS monitored resource manifests if they are not available in Secondary Cluster.
- MakeSure to have identical resource manifests in both the clusters to avoid any issues during CIS failover.
Virtual Server Pool with Multi-ClusterServices¶
Services running in any other OpenShift/Kubernetes clusters, apart from the HA cluster pair, can be referenced in the VS Pool as mentioned below:
Note
Starting with CIS 2.16.0, in multicluster, serviceName is replaced with service and port is replaced with servicePort in the extendedServiceReferences.
pools:
- path: /tea
serviceNamespace: tea
service: svc-2
servicePort: 80
extendedServiceReferences:
- clusterName: cluster2
namespace: ns1
servicePort: 8080
service: svc-1
- clusterName: cluster3
namespace: ns2
servicePort: 80
service: svc-ext-1
Transport Server Pool with Multi-ClusterServices¶
Services running in any other OpenShift/Kubernetes clusters, apart from the HA cluster pair, can be referenced in the TS Pool as mentioned below:
pool:
service: svc-1
servicePort: 8181
extendedServiceReferences:
- clusterName: cluster2
service: svc-1
namespace: ns1
servicePort: 8181
- clusterName: cluster3
service: svc-ext-1
namespace: ns2
servicePort: 8282
Static Routing Mode Support for Multi-Cluster¶
CIS supports configuring static routes in BIG-IP with node subnets assigned for the nodes in the OpenShift/Kubernetes cluster. This enables direct routing from BIG-IP to k8s pods in cluster mode without VxLAN tunnel configuration on BIG-IP.
Configuration¶
To enable the static route configuration, set --static-routing-mode
to true and --orchestration-cni
to CNI configured in the cluster.
Parameter | Type | Required | Description | Allowed Values | Default | Agent |
---|---|---|---|---|---|---|
static-routing-mode | Boolean | Optional | Adds static routes on the BIG-IP so traffic can be directly routed to the pods, without tunnels. | true/false | false | AS3 |
orchestration-cni | String | Optional | Kubernetes cluster CNI name. | cilium-k8s, flannel, ovn-k8s, antrea | flannel | AS3 |
shared-static-routes | Boolean | Optional | Flag to enable configuration of static routes on BIG-IP in common partition. | true/false | false | AS3 |
CIS Deployment with static-routing-mode enabled¶
Note
In this section, standalone refers to the standalone topology of CIS deployment, See Standalone CIS
Sample deployment for primary CIS deployment:
spec:
containers:
- args:
- --bigip-partition
- <partition>
- --bigip-url
- <ip-address>
- --bigip-username
- <user-name>
- --bigip-password
- <password>
- --log-level
- DEBUG
- --insecure
- --custom-resource-mode=true
- --static-routing-mode=true
- --orchestration-cni=ovn-k8s
- --shared-static-routes=true
- --extended-spec-configmap=kube-system/extended-spec-config
- --pool-member-type
- cluster
- --multi-cluster-mode=primary
command:
- /app/bin/k8s-bigip-ctlr
image: <image-name>
Parameter | Type | Required | Description | Default | Examples |
---|---|---|---|---|---|
clusterName | String | Mandatory | Name of the cluster. | cluster1 | |
service | String | Mandatory | Name of the service. | svc-1 | |
namespace | String | Mandatory | Namespace where the service is created. | test | |
servicePort | String/Int | Mandatory | Port of the service. For named port, use the string value. | 80 | |
weight | Integer | Optional | Weight used for traffic splitting. | 0 | 20 |
Cluster Wise Ratio for Traffic Distribution¶
CIS supports distribution of traffic across clusters as per the ratio configured for each cluster in the extended ConfigMap. It works even along with A/B where different weights are defined for each service. In such a case the ratio of traffic distribution is computed taking into consideration both the service weights and cluster ratio. However, the ratio of the clusters, those haven’t hosted any services linked to the concerned route are not taken into consideration while computing the final ratio. For more information on cluster ratio based traffic distribution, see Introduction to ratio in MultiCluster.
Note
- Cluster wise ratio for traffic distribution is supported in HA as well as non-HA CIS environments.
- Ratio is only supported for NextGen Routes and Virtual Server CR.
- Setting cluster adminState, in conjunction with cluster ratio, will affect the overall traffic distribution across clusters. As the clusters marked as disable or offline, they will not receive traffic. Any ratio defined for these clusters will be rendered ineffective. Thus, in such a scenario it’s recommended to set the cluster ratio to 0 for all clusters marked with disable/offline.
Set Cluster adminState to enable/disable/offline/no-pool a cluster¶
adminState
can be assigned to a cluster to dictate the state of the particular cluster.
Supported values for adminState are enable, disable, offline, and no-pool. By default the clusters are in an enabled state.
adminState enable: all new connections are allowed to the pool members from the cluster.
adminState disable: all new connections, except those which match an existing persistence session, are not allowed for the pool members from the cluster.
adminState offline: no new connections are allowed to the pool members from the cluster, even if they match an existing persistence session.
adminState: no-pool: in ratio
mode, service pool is not created for the affected cluster. For all other modes, pool members from the cluster are not added to the service pool. This configuration is helpful when we do not want to add pool or pool members from a particular cluster due to any reason such as cluster is under maintenance.
Note
- For HA mode [Active-Standby, Active-Active, Ratio], CIS monitored resource manifests(such as routes, CRDs, extendedConfigmaps) must be available in both the clusters.
- The CIS monitored resource manifests must be identical on both primary and Secondary Clusters
- So, In case of CIS failover, CIS on Secondary Cluster will take control and will start processing the CIS monitored resource manifests.
- CIS on Secondary Cluster will not process the CIS monitored resource manifests if they are not available in Secondary Cluster.
- MakeSure to have identical resource manifests in both the clusters to avoid any issues during CIS failover.
Examples¶
For more examples, see Config Examples
UserGuides¶
For userguides on multi-cluster, see UserGuides
Known issues¶
- Pool members are not getting populated for extended service in ratio mode.
- CIS doesn’t update pool members if the service doesn’t exist in the primary cluster, but exists in the secondary cluster for route.
- CIS, on start up in multiCluster mode, if any external cluster kube-api server is down/not reachable, CIS is stuck and not processing any valid clusters config. Workaround is to remove the unreachable cluster config from ConfigMap and restart CIS.
- CIS fails to post a VS declaration with health monitors in ratio mode. Issue is observed intermittently.
- Route status is not updated in other HA clusters. For example: Active Primary CIS cluster doesn’t update the route status in the Secondary HA cluster and vice-versa.
- CIS, on switch over from ratio to active-standby mode, doesn’t add the external cluster services. For any change to the mode, it is always recommended to restart CIS.
FAQ¶
Is –multi-cluster-mode is a required parameter for Multi-Cluster support?
Yes. Multi-Cluster support only works if --multi-cluster-mode
is defined in the CIS deployment.
Is extended ConfigMap mandatory for Multi-Cluster support?
Yes. Multi-Cluster support only works with extended ConfigMap.
How do you add a new cluster?
To add a new cluster, create a kube-config file with read-only permissions. Next, create a Kubernetes secret using the kube-config file. Refer this in secret in the extended ConfigMap to add the new cluster. CIS dynamically reads the new kube-config of the new cluster and starts listening to the services and endpoints in the new cluster when a route refers this new cluster.
Where do you manage the manifest or configuration objects such as routes, VS/TS CRs, extendedConfigmaps, etc.?
Manifests or configuration objects are managed centralized in the primary cluster and, if HA is desired, the same manifests are expected to be in the secondary cluster.
What are the supported CNIs?
Currently, NodePort mode is supported. For cluster mode, static routing mode is supported to enable configuration of static routes on BIG-IP for pod network subnets for direct routing from BIG-IP to k8s pods
What kind of providers are supported?
CIS supports hybrid cloud, and any public cloud providers such as; AWS, Azure, GCP, On-Prem, VmWare, Tanzu etc., which are in the same network/datacenter and can communicate with each other.
What kind of clusters are supported?
CIS multicluster solution is currently validated with OpenShift clusters and k8s clusters
How does CIS start as a secondary cluster?
CIS recognizes the cluster as secondary when it starts with a deployment parameter i.e. --multi-cluster-mode=secondary
How does the secondary CIS learn about the primary cluster endpoint state in HA mode?
Both of the CIS’ will communicate with both k8s API servers and prepares the AS3 declaration. The secondary CIS only sends the declaration when the primary cluster’s health probe fails. As soon as primary cluster comes up, secondary CIS stops sending the declaration.
What kind of permission is required for HA or standalone deployment of CIS?
No RBAC change for CIS deployment with multiCluster support. Only an additional kube-config configuration with read only permission is required to access the endpoints from an external cluster.
What kind of permission is required to access external clusters, apart from HA and standalone?
CIS requires read-only permissions in kube-config of external clusters to access resources like pods, services, endpoints and nodes.
Can CIS manage multiple BIG-IPs?
No. CIS can manage only standalone BIG-IP or an HA BIG-IP. In other words, CIS acts as a single point of BIG-IP orchestrator and supports multi-cluster.
Is traffic splitting with cluster ratio supported?
Yes. CIS supports traffic splitting as per the ratio specified for each cluster and also works with A/B as well.
Is A/B supported in multiCluster mode?
Yes. CIS supports A/B with multiCluster.
Is A/B custom persistence supported in all the modes?
No. A/B persistence is supported in ratio mode and pool member type as cluster.
Does extended ConfigMap update require CIS restart?
No. It’s recommended to restart CIS if any HA configuration, or external cluster configurations, are updated in extended ConfigMap. However, CIS restart is not required when updating ratio in the extended ConfigMap.
Does mode update require CIS restart?
Yes. CIS has to be restarted when there is a change in the mode.
Does Secondary CIS require resource manifests existing in Primary Cluster?
Yes. CIS on Secondary Cluster will not process the CIS monitored resource manifests[NextGen Routes, CRDs, extendedConfigmap] if they are not available in Primary Cluster. This is required in case of CIS failover, CIS on Secondary Cluster will take control and will start processing the CIS monitored resource manifests. It is suggested to maintain identical CIS monitored resource manifests in both the clusters to avoid any issues during CIS failover. This requirement is applicable in case of CIS HA mode [Active-Standby, Active-Active, Ratio].
How to configure primaryEndPoint in HA mode?
primaryEndPoint is a mandatory parameter if CIS is intended to run in Multi-Cluster HA mode[Active-Standby, Active-Active, Ratio]. If this is not specified the secondary CIS will not run. Secondary CIS will continuously monitor the health of the primary cluster based on the primaryEndPoint value. If it’s down, then the secondary CIS takes the responsibility of posting declarations to BIG-IP. Supported Protocols for primaryEndPoint are HTTP and TCP. Generally, it’s suggested to use the primaryEndPoint as
a) any available endpoint to check the health of the primary cluster.
b) Primary CIS cluster's health check endpoint (/ready) if accessible.
c) Primary CIS cluster's kube-api server endpoint if accessible.
Response code 200 OK is expected from the primaryEndPoint in case of HTTP Protocol. Successful TCP connection is expected from the primaryEndPoint in case of TCP Protocol. Secondary CIS become active if the primaryEndPoint is not accessible. PrimaryEndPoint is optional to configure for Primary CIS. Note: Primary CIS will not monitor the health of the Secondary CIS cluster.
How to configure primaryEndPoint in Standalone mode?
primaryEndPoint is not applicable in Standalone mode. It’s only applicable in HA mode.
How to use CIS /ready endpoint to check the health of CIS?
Fetch the CIS PodIP and use it in the curl command as shown below from any of the cluster nodes:
curl http://<CIS-PodIP>:8080/ready
Response code 200 OK is expected from the CIS /ready endpoint if kube-api server is accessible. Example:
[root@cluster-1-worker0 ~]# curl http://10.244.1.213:8080/ready
Ok[root@cluster-1-worker0 ~]# curl http://10.244.1.213:8080/ready -v
* About to connect() to 10.244.1.213 port 8080 (#0)
* Trying 10.244.1.213...
* Connected to 10.244.1.213 (10.244.1.213) port 8080 (#0)
> GET /ready HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.244.1.213:8080
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 07 Dec 2023 08:28:57 GMT
< Content-Length: 2
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host 10.244.1.213 left intact
Ok[root@cluster-1-worker0 ~]#
where 10.244.1.213 is the CIS PodIP.
How extendedServiceReferences is different from multiClusterServices?
extendedServiceReferences is applicable for Virtual Server CR or Transport Server CR and multiClusterServices is applicable for NextGen Routes. extendedServiceReferences is used to refer the services running in any other OpenShift/Kubernetes clusters, apart from the HA cluster pair, in the VS Pool or TS Pool. multiClusterServices is used to refer the services running in any other OpenShift/Kubernetes clusters, apart from the HA cluster pair, in the Route annotation.
How multiClusterServices and extendedServiceReferences are similar?
multiClusterServices and extendedServiceReferences are similar in terms of referring the services running in any other OpenShift/Kubernetes clusters, apart from the HA cluster pair. Both are applicable to refer services running in external Clusters, apart from the HA cluster pair
How to configure multiClusterServices in Route annotation?
multiClusterServices is a Route annotation. Below is the sample Route annotation with multiClusterServices:
virtual-server.f5.com/multiClusterServices:
'[
{
"clusterName": "cluster2",
"service": "svc-pytest-foo-1-com",
"namespace": "foo",
"servicePort": 80,
"weight": 30,
}
]'
where clusterName is the name of the cluster where the service is running, namespace is the namespace where the service is running, servicePort is the port of the service and service is the name of the service.
where cluster2 is the external cluster apart from the HA cluster pair.
Note: External Clusters do not need to install CIS
How to configure extendedServiceReferences in Virtual Server CR or Transport Server CR?
extendedServiceReferences is a field in Virtual Server CR or Transport Server CR. Below is the sample Virtual Server CR with extendedServiceReferences:
Note
Starting with CIS 2.16.0, in multicluster, serviceName is replaced with service and port is replaced with servicePort in the extendedServiceReferences.
pools:
- path: /tea
serviceNamespace: tea
service: svc-2
servicePort: 80
extendedServiceReferences:
- clusterName: cluster3
namespace: ns1
servicePort: 8080
service: svc-1
- clusterName: cluster4
namespace: ns2
servicePort: 80
service: svc-ext-1
where clusterName is the name of the cluster where the service is running, namespace is the namespace where the service is running, servicePort is the port of the service and service is the name of the service.
where cluster3 and cluster4 are the external clusters apart from the HA cluster pair.
Note: External Clusters do not need to install CIS