Last updated on: 2024-04-23 04:45:25.

OpenShift/Kubernetes Multi-Cluster¶

This page documents a new feature in CIS for Multi-Cluster.

Note

To provide feedback on Container Ingress Services or this documentation, please file a GitHub Issue.

Contents¶

Overview

Topologies

Configuration

Extended ConfigMap Parameters

Overview¶

Multi-Cluster Support in CIS allows users to expose multiple apps spread across OpenShift/Kubernetes clusters using a single BIG-IP Virtual Server. An app can be deployed in different OpenShift/Kubernetes clusters exposing them using a Route resource and VS/TS CR. Using a Multi-Cluster implementation, CIS can be deployed in an HA topology or a standalone CIS to expose the apps spread across OpenShift/Kubernetes clusters.

Note

CIS supports processing of routes in the traditional way as well as with NextGen Controller and multi-cluster support.
At present, nodePort mode is supported and Cluster mode is available with static route configuration on BIG-IP (No tunnels).

Prerequisites¶

Cluster node, where CIS is deployed, should be able to reach the API server of all OpenShift/Kubernetes clusters.
extended ConfigMap needs to be created to run CIS in multi-cluster mode.
kube-config files for each cluster should be available for CIS to access resources such as pods, services, endpoints and nodes.

Topologies¶

Standalone CIS¶

In a Standalone deployment of CIS, CIS is only deployed in one cluster. Next, create a route resource with a multi-cluster annotation, or CRD resource, with extendedServiceReferences to expose the apps in different OpenShift/Kubernetes clusters.

../../_images/multi-clusterstandalone.png

Below is the sample multi-cluster config in an extended ConfigMap.

extendedSpec: |
  externalClustersConfig:    -------------------------------------|----------------------------|                            |
  - clusterName: cluster3                                      |                            |
    secret: default/kubeconfig3                                |---> Cluster configs for    |
  - clusterName: cluster4                                      |     all other clusters     |
    secret: default/kubeconfig4                                |     except HA clusters     |
  - clusterName: cluster5                                      |                            |
    secret: default/kubeconfig5  ------------------------------|----------------------------|
  extendedRouteSpec:
  - namespace: foo   -------------------------------------|
    vserverAddr: 10.8.0.4                                 |
    vserverName: nextgenroutes                            |----------------> RouteGroup with namespace
    allowOverride: true                                   |
    bigIpPartition: MultiTenant                           |
    policyCR: default/sample-policy  _____________________|
  - namespace: bar -------------------------------------|
    vserverAddr: 10.8.0.5                               |----------------> RouteGroup with namespace
    allowOverride: false           _____________________|

Note

extendedRouteSpec is only applicable in case of OpenShift route resources, not for CRD resources.

High Availability CIS¶

Prerequisites¶

A pair of high availability OpenShift/Kubernetes clusters should be available, which have the same applications running in both clusters.
HealthCheck endpoint should be available to check the health of the primary cluster. Currently, TCP/HTTP Health endpoints are supported.

In an HA deployment of CIS, CIS needs to be deployed in both the primary and secondary cluster. Also, the same extended ConfigMap needs to be deployed in both the primary and secondary cluster. CIS will look for the same service name in both primary and secondary clusters to expose the application via routes. Additionally, a multi-cluster annotation is created in the route definition exposing the applications in the other clusters.

Deploying CIS HA in Two Modes:

active-active and active-standby modes are used if CIS is running in high availability mode.
Based on the provided value the CIS running on Primary cluster decides whether to monitor the workloads running on Secondary Cluster or not.
If CIS is running in active-active mode on the primary cluster, the pool members from both virtual servers’ clusters (primary and secondary) are part of the HA cluster, as well as members from other remotely monitored clusters.
If CIS is running in active-standby mode on the primary cluster, the pool members are updated for virtual servers only from the primary cluster, and from all the other remotely monitored clusters except the secondary cluster.
If the primary cluster is down, and CIS on the secondary cluster has taken control, then pool members from the secondary cluster, as well as all other remotely monitored clusters, are populated for the virtual servers irrespective of the value of HA mode.

Note

Additionally, there is another supported mode called ratio, in which pool members from all the Kubernetes/Openshift clusters are updated for the virtual server. However, the traffic distribution is done based on the ratio values defined for each cluster.
Ratio does not require CIS to be running in an HA environment. It is supported in both CIS HA and non-HA environments.

Note

For HA mode [Active-Standby, Active-Active, Ratio], CIS monitored resource manifests(such as routes, CRDs, extendedConfigmaps) must be available in both the clusters.
The CIS monitored resource manifests must be identical on both primary and Secondary Clusters
So, In case of CIS failover, CIS on Secondary Cluster will take control and will start processing the CIS monitored resource manifests.
CIS on Secondary Cluster will not process the CIS monitored resource manifests if they are not available in Secondary Cluster.
MakeSure to have identical resource manifests in both the clusters to avoid any issues during CIS failover.

The tables below represent the scenarios mentioned above.

When the primary cluster is healthy and CIS is running on it.¶
HA Mode Values	Primary Cluster Pool Members	Secondary Cluster Pool Members	Other Remotely Monitored Cluster Pool Members	Traffic Distribution Based on Ratio
active-active	Yes	Yes	Yes	No
active-standby	Yes	No	Yes	No
ratio	Yes	Yes	Yes	Yes

When the primary cluster is down, which means the CIS secondary cluster in control.¶
HA Mode Values	Primary Cluster Pool Members	Secondary Cluster Pool Members	Other Remotely Monitored Cluster Pool Members	Traffic Distribution Based on Ratio
active-active	No	Yes	Yes	No
active-standby	No	Yes	Yes	No
ratio	Yes	Yes	Yes	Yes

Below is an example of multi-cluster configs, with HA, in an extended ConfigMap.

extendedSpec: |
mode: active-active       -----------------------------------|---->  HA Mode              |
highAvailabilityCIS:   --------------------------------------|----------------------------|
  primaryEndPoint: http://10.145.72.114:8001                 |                            |
  probeInterval: 30                                          |                            |
  retryInterval: 3                                           |                            |
  primaryCluster:                                            |---> Cluster configs for    |
    clusterName: cluster1                                    |     High availability      |
    secret: default/kubeconfig1                              |     clusters               |---> Multi-Cluster configs
  secondaryCluster:                                          |                            |
    clusterName: cluster2                                    |                            |
    secret: default/kubeconfig2                              |                            |
externalClustersConfig:    -------------------------------------|                            |
- clusterName: cluster3                                      |                            |
  secret: default/kubeconfig3                                |---> Cluster configs for    |
- clusterName: cluster4                                      |     all other clusters     |
  secret: default/kubeconfig4                                |     except HA clusters     |
- clusterName: cluster5                                      |                            |
  secret: default/kubeconfig5  ------------------------------|----------------------------|
extendedRouteSpec:
- namespace: foo   -------------------------------------|
  vserverAddr: 10.8.0.4                                 |
  vserverName: nextgenroutes                            |----------------> RouteGroup with namespace
  allowOverride: true                                   |
  bigIpPartition: MultiTenant                           |
  policyCR: default/sample-policy  _____________________|
- namespace: bar -------------------------------------|
  vserverAddr: 10.8.0.5                               |----------------> RouteGroup with namespace
  allowOverride: false           _____________________|

Below is the sample multi-cluster configs, with HA and ratio, in an extended ConfigMap.

extendedSpec: |
mode: ratio        ------------------------------------------|----------------------------|
localClusterRatio: 4                                         |                            |
highAvailabilityCIS:   --------------------------------------|                            |
  primaryEndPoint: http://10.145.72.114:8001                 |                            |
  probeInterval: 30                                          |                            |
  retryInterval: 3                                           |                            |
  primaryCluster:                                            |---> Cluster configs for    |
    clusterName: cluster1                                    |     High availability      |
    secret: default/kubeconfig1                              |     clusters               |---> Multi-Cluster configs
    ratio: 3                                                 |                            |
  secondaryCluster:                                          |                            |
    clusterName: cluster2                                    |                            |
    secret: default/kubeconfig2                              |                            |
    ratio: 2                                                 |                            |
externalClustersConfig:    -------------------------------------|                            |
- clusterName: cluster3                                      |                            |
  secret: default/kubeconfig3                                |---> Cluster configs for    |
  ratio: 2                                                   |     all other clusters     |
- clusterName: cluster4                                      |     except HA clusters     |
  secret: default/kubeconfig4                                |                            |
- clusterName: cluster5                                      |                            |
  secret: default/kubeconfig5                                |                            |
  ratio: 1                     ------------------------------|----------------------------|
extendedRouteSpec:
- namespace: foo   -------------------------------------|
  vserverAddr: 10.8.0.4                                 |
  vserverName: nextgenroutes                            |----------------> RouteGroup with namespace
  allowOverride: true                                   |
  bigIpPartition: MultiTenant                           |
  policyCR: default/sample-policy  _____________________|
- namespace: bar -------------------------------------|
  vserverAddr: 10.8.0.5                               |----------------> RouteGroup with namespace
  allowOverride: false           _____________________|

Note

extendedRouteSpec is only applicable in the case of openshift route resources, not for CRD resources.

Below is the sample Multi-Cluster Configs with HA and cluster AdminState in extended ConfigMap.

extendedSpec: |
  mode: active-active    --------------------------------------|----------------------------|
  highAvailabilityCIS:   --------------------------------------|                            |
    primaryEndPoint: http://10.145.72.114:8001                 |                            |
    probeInterval: 30                                          |                            |
    retryInterval: 3                                           |                            |
    primaryCluster:                                            |---> Cluster configs for    |
      clusterName: cluster1                                    |     High availability      |
      secret: default/kubeconfig1                              |     clusters               |---> Multi-Cluster configs
    secondaryCluster:                                          |                            |
      clusterName: cluster2                                    |                            |
      secret: default/kubeconfig2                              |                            |
      adminState: enable                                       |                            |
  externalClustersConfig:    ----------------------------------|                            |
  - clusterName: cluster3                                      |                            |
    secret: default/kubeconfig3                                |---> Cluster configs for    |
    adminState: disable                                        |     all other clusters     |
  - clusterName: cluster4                                      |     except HA clusters     |
    secret: default/kubeconfig4                                |                            |
  - clusterName: cluster5                                      |                            |
    secret: default/kubeconfig5                                |                            |
    adminState: offline          ------------------------------|----------------------------|
  extendedRouteSpec:
  - namespace: foo   -------------------------------------|
    vserverAddr: 10.8.0.4                                 |
    vserverName: nextgenroutes                            |----------------> RouteGroup with namespace
    allowOverride: true                                   |
    bigIpPartition: MultiTenant                           |
    policyCR: default/sample-policy  _____________________|
  - namespace: bar -------------------------------------|
    vserverAddr: 10.8.0.5                               |----------------> RouteGroup with namespace
    allowOverride: false           _____________________|

Note

extendedRouteSpec is only applicable for OpenShift route resources, not for CRD resources.

Below is the sample Multi-Cluster Configs with standalone CIS and cluster AdminState in extended ConfigMap.

extendedSpec: |
  localClusterAdminState: disable  ----------------------------|AdminState for local cluster|
  externalClustersConfig:    ----------------------------------|----------------------------|
  - clusterName: cluster3                                      |                            |
    secret: default/kubeconfig3                                |---> Cluster configs for    |
    adminState: enable                                         |     all other clusters     |
  - clusterName: cluster4                                      |     except HA clusters     |
    secret: default/kubeconfig4                                |                            |
  - clusterName: cluster5                                      |                            |
    secret: default/kubeconfig5                                |                            |
    adminState: offline          ------------------------------|----------------------------|
  extendedRouteSpec:
  - namespace: foo   -------------------------------------|
    vserverAddr: 10.8.0.4                                 |
    vserverName: nextgenroutes                            |----------------> RouteGroup with namespace
    allowOverride: true                                   |
    bigIpPartition: MultiTenant                           |
    policyCR: default/sample-policy  _____________________|
  - namespace: bar -------------------------------------|
    vserverAddr: 10.8.0.5                               |----------------> RouteGroup with namespace
    allowOverride: false           _____________________|

Note

localClusterAdminState is only applicable in case of standalone CIS. It’s ignored if specified in HA CIS mode. extendedRouteSpec is only applicable for OpenShift route resources, not for CRD resources.

Configuration¶

OpenShift Routes with multi-cluster¶

CIS Deployment Parameter for NextGenRoutes¶

If you are using multi-cluster mode, the --multi-cluster-mode parameter is required.

Parameter	Type	Required	Description	Allowed Values
multi-cluster-mode	String	Required	Specify whether CIS is running standalone or as primary/secondary in the case of a high availability topology	standalone, primary, or secondary

Note

Standalone refers to the standalone topology of CIS deployment. See Standalone CIS

Following is the example for a primary CIS deployment:

spec:
  containers:
  - args:
    - --bigip-partition
    - <partition>
    - --bigip-url
    - <ip-address>
    - --bigip-username
    - <user-name>
    - --bigip-password
    - <password>
    - --log-level
    - DEBUG
    - --insecure
    - --controller-mode=openshift
    - --extended-spec-configmap=kube-system/extended-spec-config
    - --route-label=systest
    - --pool-member-type
    - nodeport
    - --multi-cluster-mode=primary
    command:
    - /app/bin/k8s-bigip-ctlr
    image: <image-name>

Note

Update the multi-cluster-mode to secondary for secondary CIS deployments in high availablility topology. See High Availability CIS
Update the multi-cluster-mode to standalone for standalone topology. See Standalone CIS

Note

For HA mode [Active-Standby, Active-Active, Ratio], CIS monitored resource manifests(such as routes, CRDs, extendedConfigmaps) must be available in both the clusters.
The CIS monitored resource manifests must be identical on both primary and Secondary Clusters
So, In case of CIS failover, CIS on Secondary Cluster will take control and will start processing the CIS monitored resource manifests.
CIS on Secondary Cluster will not process the CIS monitored resource manifests if they are not available in Secondary Cluster.
MakeSure to have identical resource manifests in both the clusters to avoid any issues during CIS failover.

Extended ConfigMap Parameters¶

externalClustersConfig Parameters¶
Parameter	Type	Required	Description	Default	Examples
clusterName	String	Mandatory	Name of the cluster		cluster1
secret	String	Mandatory	Name of the secret created for kubeconfig. Format: namespace/secretname		test/secret- kubeconfig1
ratio	Integer	Optional	Ratio at which the traffic has to be distributed over clusters.	1	3
adminState	String	Optional	adminState can be used to disable/enable/offline /no-pool clusters	enable	disable

Note

Avoid specifying HA cluster (primary/secondary cluster) configs in externalClustersConfig.

High Availability Mode (optional parameter)¶
Parameter	Type	Required	Description	Default	Examples
mode	Object	Optional	Type of high availability mode (active-active/active-standby/ratio)	active-standby	active-active

Specifies whether the CIS HA cluster is configured with active-active mode, active-standby mode or ratio mode.

If mode Type: active-active, then CIS fetches services from both the HA clusters whenever it’s referenced in Route Spec.
If mode Type: active-standby, the default, CIS fetches service from only the local cluster whenever it’s referenced in a Route Spec.
If mode Type: ratio, then CIS works in active-active mode and, splits the traffic according to the ratio specified for each cluster.

Local cluster ratio (Optional parameter)¶
Parameter	Type	Required	Description	Default	Example
localClusterRatio	Int	Optional	Ratio for the local cluster where CIS is running. Specify only when using ratio in a CIS non-HA environment.	1	3
Note: It is not needed in the case of using ratio in a CIS HA environment as ratio of the primary cluster does the same thing. If specified in this scenario, then it will be ignored.

highAvailabilityCIS Parameters¶
Parameter	Type	Required	Description	Default	Examples
primaryClusterEndPoint	String	Mandatory	Endpoint to check the health of the primary cluster.		http://10.145.72.114:8001
probeInterval	Integer	Optional	Time interval between health checks, in seconds.	60	30
retryInterval	Integer	Optional	Time interval between recheck when primary cluster is down, in seconds.	15	3
primaryCluster	Object	Mandatory	Primary cluster config.
secondaryCluster	Object	Mandatory	Secondary cluster config.

primaryCluster/secondaryCluster Parameters¶
Parameter	Type	Required	Description	Default	Examples
clusterName	String	Mandatory	Name of the cluster.		cluster1
secret	String	Mandatory	Name of the secret created for kubeconfig. Format: namespace/secret-name.		test/secret-kubeconfig1
ratio	Integer	Optional	Ratio at which the traffic has to be distributed over clusters.	1	3
adminState	String	Optional	adminState can be used to disable/enable/offline /no-pool clusters.	enable	disable

Note

In order to run CIS in high availability mode, multi-cluster-mode parameter (primary/secondary) needs to be set in the CIS deployment arguments.

It is recommended to provide both primaryCluster and secondaryCluster configs in the extended ConfigMap.
If no traffic has to be forwarded to a specific cluster, then set the ratio field to 0.

CIS Primary Cluster Endpoint

Health probe parameters are provided in highAvailabilityCIS in extended ConfigMap, helping to ensure high availability of CIS. CIS running in the secondary cluster continuously monitors the health of the primary cluster. If it is down, then the secondary CIS takes the responsibility of posting declarations to BIG-IP.

Note

primaryEndPoint is a mandatory parameter if CIS is intended to run in multi-cluster HA mode. If this is not specified, then the secondary CIS will not run.

Route Annotation for Multi-ClusterServices¶

Services running in any other OpenShift clusters, apart from the HA cluster pair, can be referenced in the route annotations as mentioned below:

Note

Starting with CIS 2.16.0, in multicluster, serviceName is replaced with service and port is replaced with servicePort in the multiClusterServices.

virtual-server.f5.com/multiClusterServices:
'[
     {
        "clusterName": "cluster2",
        "service": "svc-pytest-foo-1-com",
        "namespace": "foo",
        "servicePort": 80,
        "weight": 30,
     }
]'

CRD Resources with Multi-Cluster¶

CIS Deployment Parameters for CRDs¶

Note

In this example, standalone refers to standalone topology of CIS deployment, See Standalone CIS

Sample deployment for primary CIS deployment:

spec:
  containers:
  - args:
    - --bigip-partition
    - <partition>
    - --bigip-url
    - <ip-address>
    - --bigip-username
    - <user-name>
    - --bigip-password
    - <password>
    - --log-level
    - DEBUG
    - --insecure
    - --custom-resource-mode=true
    - --extended-spec-configmap=kube-system/extended-spec-config
    - --pool-member-type
    - nodeport
    - --multi-cluster-mode=primary
    command:
    - /app/bin/k8s-bigip-ctlr
    image: <image-name>

Note

Update the multi-cluster-mode to secondary for secondary CIS deployments in high availability topology, See High Availability CIS
Update the multi-cluster-mode to standalone for standalone topology, See Standalone CIS
weight needs to be specified only in A/B scenario.

Note

For HA mode [Active-Standby, Active-Active, Ratio], CIS monitored resource manifests(such as routes, CRDs, extendedConfigmaps) must be available in both the clusters.
The CIS monitored resource manifests must be identical on both primary and Secondary Clusters
So, In case of CIS failover, CIS on Secondary Cluster will take control and will start processing the CIS monitored resource manifests.
CIS on Secondary Cluster will not process the CIS monitored resource manifests if they are not available in Secondary Cluster.
MakeSure to have identical resource manifests in both the clusters to avoid any issues during CIS failover.

Virtual Server Pool with Multi-ClusterServices¶

Services running in any other OpenShift/Kubernetes clusters, apart from the HA cluster pair, can be referenced in the VS Pool as mentioned below:

Note

Starting with CIS 2.16.0, in multicluster, serviceName is replaced with service and port is replaced with servicePort in the extendedServiceReferences.

pools:
- path: /tea
  serviceNamespace: tea
  service: svc-2
  servicePort: 80
  extendedServiceReferences:
  - clusterName: cluster2
    namespace: ns1
    servicePort: 8080
    service: svc-1
  - clusterName: cluster3
    namespace: ns2
    servicePort: 80
    service: svc-ext-1

Transport Server Pool with Multi-ClusterServices¶

Services running in any other OpenShift/Kubernetes clusters, apart from the HA cluster pair, can be referenced in the TS Pool as mentioned below:

pool:
  service: svc-1
  servicePort: 8181
  extendedServiceReferences:
  - clusterName: cluster2
    service: svc-1
    namespace: ns1
    servicePort: 8181
  - clusterName: cluster3
    service: svc-ext-1
    namespace: ns2
    servicePort: 8282

Static Routing Mode Support for Multi-Cluster¶

CIS supports configuring static routes in BIG-IP with node subnets assigned for the nodes in the OpenShift/Kubernetes cluster. This enables direct routing from BIG-IP to k8s pods in cluster mode without VxLAN tunnel configuration on BIG-IP.

Configuration¶

To enable the static route configuration, set --static-routing-mode to true and --orchestration-cni to CNI configured in the cluster.

Parameter	Type	Required	Description	Allowed Values	Default	Agent
static-routing-mode	Boolean	Optional	Adds static routes on the BIG-IP so traffic can be directly routed to the pods, without tunnels.	true/false	false	AS3
orchestration-cni	String	Optional	Kubernetes cluster CNI name.	cilium-k8s, flannel, ovn-k8s, antrea	flannel	AS3
shared-static-routes	Boolean	Optional	Flag to enable configuration of static routes on BIG-IP in common partition.	true/false	false	AS3

CIS Deployment with static-routing-mode enabled¶

Note

In this section, standalone refers to the standalone topology of CIS deployment, See Standalone CIS

Sample deployment for primary CIS deployment:

spec:
  containers:
  - args:
    - --bigip-partition
    - <partition>
    - --bigip-url
    - <ip-address>
    - --bigip-username
    - <user-name>
    - --bigip-password
    - <password>
    - --log-level
    - DEBUG
    - --insecure
    - --custom-resource-mode=true
    - --static-routing-mode=true
    - --orchestration-cni=ovn-k8s
    - --shared-static-routes=true
    - --extended-spec-configmap=kube-system/extended-spec-config
    - --pool-member-type
    - cluster
    - --multi-cluster-mode=primary
    command:
    - /app/bin/k8s-bigip-ctlr
    image: <image-name>

Route Annotation / VS or TS MultiClusterServices Parameters¶
Parameter	Type	Required	Description	Default	Examples
clusterName	String	Mandatory	Name of the cluster.		cluster1
service	String	Mandatory	Name of the service.		svc-1
namespace	String	Mandatory	Namespace where the service is created.		test
servicePort	String/Int	Mandatory	Port of the service. For named port, use the string value.		80
weight	Integer	Optional	Weight used for traffic splitting.	0	20

Cluster Wise Ratio for Traffic Distribution¶

CIS supports distribution of traffic across clusters as per the ratio configured for each cluster in the extended ConfigMap. It works even along with A/B where different weights are defined for each service. In such a case the ratio of traffic distribution is computed taking into consideration both the service weights and cluster ratio. However, the ratio of the clusters, those haven’t hosted any services linked to the concerned route are not taken into consideration while computing the final ratio. For more information on cluster ratio based traffic distribution, see Introduction to ratio in MultiCluster.

Note

Cluster wise ratio for traffic distribution is supported in HA as well as non-HA CIS environments.
Ratio is only supported for NextGen Routes and Virtual Server CR.
Setting cluster adminState, in conjunction with cluster ratio, will affect the overall traffic distribution across clusters. As the clusters marked as disable or offline, they will not receive traffic. Any ratio defined for these clusters will be rendered ineffective. Thus, in such a scenario it’s recommended to set the cluster ratio to 0 for all clusters marked with disable/offline.

Set Cluster adminState to enable/disable/offline/no-pool a cluster¶

adminState can be assigned to a cluster to dictate the state of the particular cluster. Supported values for adminState are enable, disable, offline, and no-pool. By default the clusters are in an enabled state.

adminState enable: all new connections are allowed to the pool members from the cluster.

adminState disable: all new connections, except those which match an existing persistence session, are not allowed for the pool members from the cluster.

adminState offline: no new connections are allowed to the pool members from the cluster, even if they match an existing persistence session.

adminState: no-pool: in ratio mode, service pool is not created for the affected cluster. For all other modes, pool members from the cluster are not added to the service pool. This configuration is helpful when we do not want to add pool or pool members from a particular cluster due to any reason such as cluster is under maintenance.

Note

For HA mode [Active-Standby, Active-Active, Ratio], CIS monitored resource manifests(such as routes, CRDs, extendedConfigmaps) must be available in both the clusters.
The CIS monitored resource manifests must be identical on both primary and Secondary Clusters
So, In case of CIS failover, CIS on Secondary Cluster will take control and will start processing the CIS monitored resource manifests.
CIS on Secondary Cluster will not process the CIS monitored resource manifests if they are not available in Secondary Cluster.
MakeSure to have identical resource manifests in both the clusters to avoid any issues during CIS failover.

Examples¶

For more examples, see Config Examples

UserGuides¶

For userguides on multi-cluster, see UserGuides

Known issues¶

Pool members are not getting populated for extended service in ratio mode.
CIS doesn’t update pool members if the service doesn’t exist in the primary cluster, but exists in the secondary cluster for route.
CIS, on start up in multiCluster mode, if any external cluster kube-api server is down/not reachable, CIS is stuck and not processing any valid clusters config. Workaround is to remove the unreachable cluster config from ConfigMap and restart CIS.
CIS fails to post a VS declaration with health monitors in ratio mode. Issue is observed intermittently.
Route status is not updated in other HA clusters. For example: Active Primary CIS cluster doesn’t update the route status in the Secondary HA cluster and vice-versa.
CIS, on switch over from ratio to active-standby mode, doesn’t add the external cluster services. For any change to the mode, it is always recommended to restart CIS.

FAQ¶

Is –multi-cluster-mode is a required parameter for Multi-Cluster support?

Yes. Multi-Cluster support only works if --multi-cluster-mode is defined in the CIS deployment.

Is extended ConfigMap mandatory for Multi-Cluster support?

Yes. Multi-Cluster support only works with extended ConfigMap.

How do you add a new cluster?

To add a new cluster, create a kube-config file with read-only permissions. Next, create a Kubernetes secret using the kube-config file. Refer this in secret in the extended ConfigMap to add the new cluster. CIS dynamically reads the new kube-config of the new cluster and starts listening to the services and endpoints in the new cluster when a route refers this new cluster.

Where do you manage the manifest or configuration objects such as routes, VS/TS CRs, extendedConfigmaps, etc.?

Manifests or configuration objects are managed centralized in the primary cluster and, if HA is desired, the same manifests are expected to be in the secondary cluster.

What are the supported CNIs?

Currently, NodePort mode is supported. For cluster mode, static routing mode is supported to enable configuration of static routes on BIG-IP for pod network subnets for direct routing from BIG-IP to k8s pods

What kind of providers are supported?

CIS supports hybrid cloud, and any public cloud providers such as; AWS, Azure, GCP, On-Prem, VmWare, Tanzu etc., which are in the same network/datacenter and can communicate with each other.

What kind of clusters are supported?

CIS multicluster solution is currently validated with OpenShift clusters and k8s clusters

How does CIS start as a secondary cluster?

CIS recognizes the cluster as secondary when it starts with a deployment parameter i.e. --multi-cluster-mode=secondary

How does the secondary CIS learn about the primary cluster endpoint state in HA mode?

Both of the CIS’ will communicate with both k8s API servers and prepares the AS3 declaration. The secondary CIS only sends the declaration when the primary cluster’s health probe fails. As soon as primary cluster comes up, secondary CIS stops sending the declaration.

What kind of permission is required for HA or standalone deployment of CIS?

No RBAC change for CIS deployment with multiCluster support. Only an additional kube-config configuration with read only permission is required to access the endpoints from an external cluster.

What kind of permission is required to access external clusters, apart from HA and standalone?

CIS requires read-only permissions in kube-config of external clusters to access resources like pods, services, endpoints and nodes.

Can CIS manage multiple BIG-IPs?

No. CIS can manage only standalone BIG-IP or an HA BIG-IP. In other words, CIS acts as a single point of BIG-IP orchestrator and supports multi-cluster.

Is traffic splitting with cluster ratio supported?

Yes. CIS supports traffic splitting as per the ratio specified for each cluster and also works with A/B as well.

Is A/B supported in multiCluster mode?

Yes. CIS supports A/B with multiCluster.

Is A/B custom persistence supported in all the modes?

No. A/B persistence is supported in ratio mode and pool member type as cluster.

Does extended ConfigMap update require CIS restart?

No. It’s recommended to restart CIS if any HA configuration, or external cluster configurations, are updated in extended ConfigMap. However, CIS restart is not required when updating ratio in the extended ConfigMap.

Does mode update require CIS restart?

Yes. CIS has to be restarted when there is a change in the mode.

Does Secondary CIS require resource manifests existing in Primary Cluster?

Yes. CIS on Secondary Cluster will not process the CIS monitored resource manifests[NextGen Routes, CRDs, extendedConfigmap] if they are not available in Primary Cluster. This is required in case of CIS failover, CIS on Secondary Cluster will take control and will start processing the CIS monitored resource manifests. It is suggested to maintain identical CIS monitored resource manifests in both the clusters to avoid any issues during CIS failover. This requirement is applicable in case of CIS HA mode [Active-Standby, Active-Active, Ratio].

How to configure primaryEndPoint in HA mode?

primaryEndPoint is a mandatory parameter if CIS is intended to run in Multi-Cluster HA mode[Active-Standby, Active-Active, Ratio]. If this is not specified the secondary CIS will not run. Secondary CIS will continuously monitor the health of the primary cluster based on the primaryEndPoint value. If it’s down, then the secondary CIS takes the responsibility of posting declarations to BIG-IP. Supported Protocols for primaryEndPoint are HTTP and TCP. Generally, it’s suggested to use the primaryEndPoint as

a) any available endpoint to check the health of the primary cluster.
b) Primary CIS cluster's health check endpoint (/ready) if accessible.
c) Primary CIS cluster's kube-api server endpoint if accessible.

Response code 200 OK is expected from the primaryEndPoint in case of HTTP Protocol. Successful TCP connection is expected from the primaryEndPoint in case of TCP Protocol. Secondary CIS become active if the primaryEndPoint is not accessible. PrimaryEndPoint is optional to configure for Primary CIS. Note: Primary CIS will not monitor the health of the Secondary CIS cluster.

How to configure primaryEndPoint in Standalone mode?

primaryEndPoint is not applicable in Standalone mode. It’s only applicable in HA mode.

How to use CIS /ready endpoint to check the health of CIS?

Fetch the CIS PodIP and use it in the curl command as shown below from any of the cluster nodes:

curl  http://<CIS-PodIP>:8080/ready

Response code 200 OK is expected from the CIS /ready endpoint if kube-api server is accessible. Example:

[root@cluster-1-worker0 ~]# curl http://10.244.1.213:8080/ready
Ok[root@cluster-1-worker0 ~]# curl http://10.244.1.213:8080/ready -v
* About to connect() to 10.244.1.213 port 8080 (#0)
*   Trying 10.244.1.213...
* Connected to 10.244.1.213 (10.244.1.213) port 8080 (#0)
> GET /ready HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.244.1.213:8080
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 07 Dec 2023 08:28:57 GMT
< Content-Length: 2
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host 10.244.1.213 left intact
Ok[root@cluster-1-worker0 ~]#

where 10.244.1.213 is the CIS PodIP.

How extendedServiceReferences is different from multiClusterServices?

extendedServiceReferences is applicable for Virtual Server CR or Transport Server CR and multiClusterServices is applicable for NextGen Routes. extendedServiceReferences is used to refer the services running in any other OpenShift/Kubernetes clusters, apart from the HA cluster pair, in the VS Pool or TS Pool. multiClusterServices is used to refer the services running in any other OpenShift/Kubernetes clusters, apart from the HA cluster pair, in the Route annotation.

How multiClusterServices and extendedServiceReferences are similar?

multiClusterServices and extendedServiceReferences are similar in terms of referring the services running in any other OpenShift/Kubernetes clusters, apart from the HA cluster pair. Both are applicable to refer services running in external Clusters, apart from the HA cluster pair

How to configure multiClusterServices in Route annotation?

multiClusterServices is a Route annotation. Below is the sample Route annotation with multiClusterServices:

virtual-server.f5.com/multiClusterServices:
'[
     {
         "clusterName": "cluster2",
         "service": "svc-pytest-foo-1-com",
         "namespace": "foo",
         "servicePort": 80,
         "weight": 30,
     }
]'

where clusterName is the name of the cluster where the service is running, namespace is the namespace where the service is running, servicePort is the port of the service and service is the name of the service.
where cluster2 is the external cluster apart from the HA cluster pair.
Note: External Clusters do not need to install CIS

How to configure extendedServiceReferences in Virtual Server CR or Transport Server CR?

extendedServiceReferences is a field in Virtual Server CR or Transport Server CR. Below is the sample Virtual Server CR with extendedServiceReferences:

Note

Starting with CIS 2.16.0, in multicluster, serviceName is replaced with service and port is replaced with servicePort in the extendedServiceReferences.

  pools:
  - path: /tea
    serviceNamespace: tea
    service: svc-2
    servicePort: 80
    extendedServiceReferences:
    - clusterName: cluster3
      namespace: ns1
      servicePort: 8080
      service: svc-1
    - clusterName: cluster4
      namespace: ns2
      servicePort: 80
      service: svc-ext-1

where clusterName is the name of the cluster where the service is running, namespace is the namespace where the service is running, servicePort is the port of the service and service is the name of the service.
where cluster3 and cluster4 are the external clusters apart from the HA cluster pair.
Note: External Clusters do not need to install CIS

Previous Next