Last updated on: 2023-09-12 04:11:59.

OpenShift/Kubernetes Multi-Cluster¶

This page documents a new feature in CIS for Multi-Cluster.

Note

To provide feedback on Container Ingress Services or this documentation, please file a GitHub Issue.

Contents¶

Overview

Topologies

Configuration

Extended ConfigMap Parameters

Examples

Known issues

FAQ

Overview¶

Multi-Cluster Support in CIS allows users to expose multiple apps spread across OpenShift/Kubernetes clusters using a single BIG-IP Virtual Server. An app can be deployed in different OpenShift clusters exposing them using a route resource. Using a Multi-Cluster implementation, CIS can be deployed in an HA topology or a standalone CIS to expose the apps spread across OpenShift clusters.

Note

CIS supports processing of routes in the traditional way as well as with NextGen Controller and multi-cluster support.
Currently, nodePort is supported. Cluster mode is supported with static route configuration on BIG-IP (no tunnels).

Prerequisites¶

Cluster node, where CIS is deployed, should be able to reach the API server of all OpenShift/Kubernetes clusters.
extendedConfigMap needs to be created to run CIS in multi-cluster mode.
kube-config files for each cluster should be available for CIS to access resources such as pods, services, endpoints and nodes.

Topologies¶

Standalone CIS¶

In a Standalone deployment of CIS, CIS is only deployed in one cluster. Next, create a route resource with a multi-cluster annotation, or CRD resource, with extendedServiceReferences to expose the apps in different OpenShift/Kubernetes clusters.

../../_images/multi-clusterstandalone.png

Below is the sample multi-cluster config in an extended global ConfigMap.

extendedSpec: |
  externalClustersConfig:    -------------------------------------|----------------------------|                            |
  - clusterName: cluster3                                      |                            |
    secret: default/kubeconfig3                                |---> Cluster configs for    |
  - clusterName: cluster4                                      |     all other clusters     |
    secret: default/kubeconfig4                                |     except HA clusters     |
  - clusterName: cluster5                                      |                            |
    secret: default/kubeconfig5  ------------------------------|----------------------------|
  extendedRouteSpec:
  - namespace: foo   -------------------------------------|
    vserverAddr: 10.8.0.4                                 |
    vserverName: nextgenroutes                            |----------------> RouteGroup with namespace
    allowOverride: true                                   |
    bigIpPartition: MultiTenant                           |
    policyCR: default/sample-policy  _____________________|
  - namespace: bar -------------------------------------|
    vserverAddr: 10.8.0.5                               |----------------> RouteGroup with namespace
    allowOverride: false           _____________________|

Note

extendedRouteSpec is only applicable in case of OpenShift route resources, not for CRD resources.

High Availability CIS¶

Prerequisites¶

A pair of high availability OpenShift/Kubernetes clusters should be available, which have the same applications running in both clusters.
HealthCheck endpoint should be available to check the health of the primary cluster. Currently, TCP/HTTP Health endpoints are supported.

In an HA deployment of CIS, CIS needs to be deployed in both the primary and secondary cluster. Also, the same extendedConfigMap needs to be deployed in both the primary and secondary cluster. CIS will look for the same service name in both primary and secondary clusters to expose the application via routes. Additionally, a multi-cluster annotation is created in the route definition exposing the applications in the other clusters.

Deploying CIS HA in Two Modes:

active-active and active-standby modes are used if CIS is running in high availability mode.
Based on the provided value the CIS running on Primary cluster decides whether to monitor the workloads running on Secondary Cluster or not.
If CIS is running in active-active mode on the primary cluster, the pool members from both virtual servers’ clusters (primary and secondary) are part of the HA cluster, as well as members from other remotely monitored clusters.
If CIS is running in active-standby mode on the primary cluster, the pool members are updated for virtual servers only from the primary cluster, and from all the other remotely monitored clusters except the secondary cluster.
If the primary cluster is down, and CIS on the secondary cluster has taken control, then pool members from the secondary cluster, as well as all other remotely monitored clusters, are populated for the virtual servers irrespective of the value of HA mode.

Note

Additionally, there is another supported mode called ratio, in which pool members from all the Kubernetes/Openshift clusters are updated for the virtual server. However, the traffic distribution is done based on the ratio values defined for each cluster.
Ratio doesn’t require CIS to be running in an HA environment. It is supported in both CIS HA and non-HA environments.

The tables below represent the scenarios mentioned above.

When the primary cluster is healthy and CIS is running on it.¶
HA Mode Values	Primary Cluster Pool Members	Secondary Cluster Pool Members	Other Remotely Monitored Cluster Pool Members	Traffic Distribution Based on Ratio
active-active	Yes	Yes	Yes	No
active-standby	Yes	No	Yes	No
ratio	Yes	Yes	Yes	Yes

When the primary cluster is down, which means the CIS secondary cluster in control.¶
HA Mode Values	Primary Cluster Pool Members	Secondary Cluster Pool Members	Other Remotely Monitored Cluster Pool Members	Traffic Distribution Based on Ratio
active-active	No	Yes	Yes	No
active-standby	No	Yes	Yes	No
ratio	Yes	Yes	Yes	Yes

Below is an example of multi-cluster configs, with HA, in an extended global ConfigMap.

extendedSpec: |
mode: active-active       -----------------------------------|---->  HA Mode              |
highAvailabilityCIS:   --------------------------------------|----------------------------|
  primaryEndPoint: http://10.145.72.114:8001                 |                            |
  probeInterval: 30                                          |                            |
  retryInterval: 3                                           |                            |
  primaryCluster:                                            |---> Cluster configs for    |
    clusterName: cluster1                                    |     High availability      |
    secret: default/kubeconfig1                              |     clusters               |---> Multi-Cluster configs
  secondaryCluster:                                          |                            |
    clusterName: cluster2                                    |                            |
    secret: default/kubeconfig2                              |                            |
externalClustersConfig:    -------------------------------------|                            |
- clusterName: cluster3                                      |                            |
  secret: default/kubeconfig3                                |---> Cluster configs for    |
- clusterName: cluster4                                      |     all other clusters     |
  secret: default/kubeconfig4                                |     except HA clusters     |
- clusterName: cluster5                                      |                            |
  secret: default/kubeconfig5  ------------------------------|----------------------------|
extendedRouteSpec:
- namespace: foo   -------------------------------------|
  vserverAddr: 10.8.0.4                                 |
  vserverName: nextgenroutes                            |----------------> RouteGroup with namespace
  allowOverride: true                                   |
  bigIpPartition: MultiTenant                           |
  policyCR: default/sample-policy  _____________________|
- namespace: bar -------------------------------------|
  vserverAddr: 10.8.0.5                               |----------------> RouteGroup with namespace
  allowOverride: false           _____________________|

Below is the sample multi-cluster configs, with HA and ratio, in an extended global ConfigMap.

extendedSpec: |
mode: ratio        ------------------------------------------|----------------------------|
localClusterRatio: 4                                         |                            |
highAvailabilityCIS:   --------------------------------------|                            |
  primaryEndPoint: http://10.145.72.114:8001                 |                            |
  probeInterval: 30                                          |                            |
  retryInterval: 3                                           |                            |
  primaryCluster:                                            |---> Cluster configs for    |
    clusterName: cluster1                                    |     High availability      |
    secret: default/kubeconfig1                              |     clusters               |---> Multi-Cluster configs
    ratio: 3                                                 |                            |
  secondaryCluster:                                          |                            |
    clusterName: cluster2                                    |                            |
    secret: default/kubeconfig2                              |                            |
    ratio: 2                                                 |                            |
externalClustersConfig:    -------------------------------------|                            |
- clusterName: cluster3                                      |                            |
  secret: default/kubeconfig3                                |---> Cluster configs for    |
  ratio: 2                                                   |     all other clusters     |
- clusterName: cluster4                                      |     except HA clusters     |
  secret: default/kubeconfig4                                |                            |
- clusterName: cluster5                                      |                            |
  secret: default/kubeconfig5                                |                            |
  ratio: 1                     ------------------------------|----------------------------|
extendedRouteSpec:
- namespace: foo   -------------------------------------|
  vserverAddr: 10.8.0.4                                 |
  vserverName: nextgenroutes                            |----------------> RouteGroup with namespace
  allowOverride: true                                   |
  bigIpPartition: MultiTenant                           |
  policyCR: default/sample-policy  _____________________|
- namespace: bar -------------------------------------|
  vserverAddr: 10.8.0.5                               |----------------> RouteGroup with namespace
  allowOverride: false           _____________________|

Note

extendedRouteSpec is only applicable in the case of openshift route resources, not for CRD resources.

Configuration¶

OpenShift Routes with multi-cluster¶

CIS Deplpoyment Parameter¶

If you are using multi-cluster mode, the --multi-cluster-mode parameter is required.

Parameter	Type	Required	Description	Allowed Values
multi-cluster mode	String	Required	Specify whether CIS is running standalone or as primary/secondary in the case of a high availability topology	standalone, primary, or secondary

Note

Standalone refers to the standalone topology of CIS deployment. See Standalone CIS

Following is the example for a primary CIS deployment:

spec:
  containers:
  - args:
    - --bigip-partition
    - <partition>
    - --bigip-url
    - <ip-address>
    - --bigip-username
    - <user-name>
    - --bigip-password
    - <password>
    - --log-level
    - DEBUG
    - --insecure
    - --controller-mode=openshift
    - --extended-spec-configmap=kube-system/global-spec-config
    - --route-label=systest
    - --pool-member-type
    - nodeport
    - --multi-cluster-mode=primary
    command:
    - /app/bin/k8s-bigip-ctlr
    image: <image-name>

Note

Update the multi-cluster-mode to secondary for secondary CIS deployments in high availablility topology. See High Availability CIS
Update the multi-cluster-mode to standalone for standalone topology. See Standalone CIS

Extended ConfigMap Parameters¶

externalClustersConfig Parameters¶
Parameter	Type	Required	Description	Default	Examples
clusterName	String	Mandatory	Name of the cluster		cluster1
secret	String	Mandatory	Name of the secret created for kubeconfig. Format: namespace/secretname		test/secret- kubeconfig1

Note

Avoid specifying HA cluster (primary/secondary cluster) configs in externalClustersConfig.

High Availability Mode (optional parameter)¶
Parameter	Type	Required	Description	Default	Examples
mode	Object	Optional	Type of high availability mode (active-active/active-standby/ratio)	active-standby	active-active

Specifies whether the CIS HA cluster is configured with active-active mode, active-standby mode or ratio mode.

If mode Type: active-active, then CIS fetches services from both the HA clusters whenever it’s referenced in Route Spec.
If mode Type: active-standby, the default, CIS fetches service from only the local cluster whenever it’s referenced in a Route Spec.
If mode Type: ratio, then CIS works in active-active mode and, splits the traffic according to the ratio specified for each cluster.

Local cluster ratio (Optional parameter)¶
Parameter	Type	Required	Description	Default	Examples
localClusterRatio	Int	Optional	Ratio for the local cluster where CIS is running. Specify only when using ratio in a CIS non-HA environment.	active-standby	active-active
Note: It is not needed in the case of using ratio in a CIS HA environment as ratio of the primary cluster does the same thing. If specified in this scenario, then it will be ignored.

highAvailabilityCIS Parameters¶
Parameter	Type	Required	Description	Default	Examples
primaryClusterEndPoint	String	Mandatory	Endpoint to check the health of the primary cluster.		http://10.145.72.114:8001
probeInterval	Integer	Optional	Time interval between health checks, in seconds.	60	30
retryInterval	Integer	Optional	Time interval between recheck when primary cluster is down, in seconds.	15	3
primaryCluster	Object	Mandatory	Primary cluster config.
secondaryCluster	Object	Mandatory	Secondary cluster config.

primaryCluster/secondaryCluster Parameters¶
Parameter	Type	Required	Description	Default	Examples
clusterName	String	Mandatory	Name of the cluster.		cluster1
secret	String	Mandatory	Name of the secret created for kubeconfig. Format: namespace/secret-name.		test/secret-kubeconfig1
ratio	Integer	Optional	Ratio at which the traffic has to be distributed over clusters.	1	3

Note

In order to run CIS in high availability mode, multi-cluster-mode parameter (primary/secondary) needs to be set in the CIS deployment arguments.

It is recommended to provide both primaryCluster and secondaryCluster configs in the extendedConfigMap.
If no traffic has to be forwarded to a specific cluster, then set the ratio field to 0.

CIS Primary Cluster Endpoint

Health probe parameters are provided in highAvailabilityCIS in extended configmap, helping to ensure high availability of CIS. CIS running in the secondary cluster continuously monitors the health of the primary cluster. If it is down, then the secondary CIS takes the responsibility of posting declarations to BIG-IP.

Note

primaryEndPoint is a mandatory parameter if CIS is intended to run in multi-cluster HA mode. If this is not specified, then the secondary CIS will not run.

Route Annotation for Multi-ClusterServices¶

Services running in any other OpenShift clusters, apart from the HA cluster pair, can be referenced in the route annotations as mentioned below:

virtual-server.f5.com/multiClusterServices:
'[
     {
         "clusterName": "cluster2",
        "serviceName": "svc-pytest-foo-1-com",
        "namespace": "foo",
        "port": 80,
        "weight": 30,
     }
]'

CRD Resources with Multi-Cluster¶

CIS Deployment Parameter¶

Note

In this example, standalone refers to standalone topology of CIS deployment, See Standalone CIS

Sample deployment for primary CIS deployment:

spec:
  containers:
  - args:
    - --bigip-partition
    - <partition>
    - --bigip-url
    - <ip-address>
    - --bigip-username
    - <user-name>
    - --bigip-password
    - <password>
    - --log-level
    - DEBUG
    - --insecure
    - --custom-resource-mode=true
    - --extended-spec-configmap=kube-system/global-spec-config
    - --pool-member-type
    - nodeport
    - --multi-cluster-mode=primary
    command:
    - /app/bin/k8s-bigip-ctlr
    image: <image-name>

Note

Update the multi-cluster-mode to secondary for secondary CIS deployments in high availability topology, See High Availability CIS
Update the multi-cluster-mode to standalone for standalone topology, See Standalone CIS

weight needs to be specified only in A/B scenario.

Virutal Server Pool with Multi-ClusterServices¶

Services running in any other OpenShift/Kubernetes clusters, apart from the HA cluster pair, can be referenced in the VS Pool as mentioned below:

pools:
- path: /tea
  serviceNamespace: tea
  service: svc-2
  servicePort: 80
  extendedServiceReferences:
  - clusterName: cluster2
    namespace: ns1
    port: 8080
    serviceName: svc-1
  - clusterName: cluster3
    namespace: ns2
    port: 80
    serviceName: svc-ext-1

Transport Server Pool with Multi-ClusterServices¶

Services running in any other OpenShift/Kubernetes clusters, apart from the HA cluster pair, can be referenced in the TS Pool as mentioned below:

pool:
  service: svc-1
  servicePort: 8181
  extendedServiceReferences:
  - clusterName: cluster2
    serviceName: svc-1
    namespace: ns1
    port: 8181
  - clusterName: cluster3
    serviceName: svc-ext-1
    namespace: ns2
    port: 8282

Static Routing Mode¶

CIS supports configuring static routes in BIG-IP with node subnets assigned for the nodes in the OpenShift/Kubernetes cluster. This enables direct routing from BIG-IP to k8s pods in cluster mode without vxaln tunnel configuration on BIG-IP.

Configuration¶

To enable the static route configuration, set --static-routing-mode to true and --orchestration-cni to CNI configured in the cluster.

Parameter	Type	Required	Description	Allowed Values	Default	Agent
static-routing-mode	Boolean	Optional	Adds static routes on the BIG-IP so traffic can be directly routed to the pods, without tunnels.	true/false	false	AS3
orchestration-cni	String	Optional	Kubernetes cluster CNI name.	cilium-k8s, flannel, ovn-k8s, antrea	flannel	AS3
shared-static-routes	Boolean	Optional	Flag to enable configuration of static routes on BIG-IP in common partition.	true/false	false	AS3

CIS Deployment Parameter¶

Note

In this section, standalone refers to the standalone topology of CIS deployment, See Standalone CIS

Sample deployment for primary CIS deployment:

spec:
  containers:
  - args:
    - --bigip-partition
    - <partition>
    - --bigip-url
    - <ip-address>
    - --bigip-username
    - <user-name>
    - --bigip-password
    - <password>
    - --log-level
    - DEBUG
    - --insecure
    - --custom-resource-mode=true
    - --static-routing-mode=true
    - --orchestration-cni=ovn-k8s
    - --shared-static-routes=true
    - --extended-spec-configmap=kube-system/global-spec-config
    - --pool-member-type
    - cluster
    - --multi-cluster-mode=primary
    command:
    - /app/bin/k8s-bigip-ctlr
    image: <image-name>

Route Annotation / VS or TS MultiClusterServices Parameters¶
Parameter	Type	Required	Description	Default	Examples
clusterName	String	Mandatory	Name of the cluster.		cluster1
serviceName	String	Mandatory	Name of the service.		svc-1
namespace	String	Mandatory	Namespace where the service is created.		test
port	String/Int	Mandatory	Port of the service. For named port, use the string value.		80
weight	Integer	Optional	Weight used for traffic splitting.	0	20

Cluster Wise Ratio for Traffic Distribution¶

CIS supports distribution of traffic across clusters as per the ratio configured for each cluster in the extended ConfigMap. It works even along with A/B where different weights are defined for each service. In such a case the ratio of traffic distribution is computed taking into consideration both the service weights and cluster ratio. However, the ratio of the clusters, those haven’t hosted any services linked to the concerned route are not taken into consideration while computing the final ratio.

Note

Cluster wise ratio for traffic distribution is supported in HA as well as non-HA CIS environments.

Examples¶

For more examples, see Config Examples

Known issues¶

Pool members are not getting populated for extended service in ratio mode.
CIS doesn’t update pool members if the service doesn’t exist in the primary cluster, but exists in the secondary cluster for route.
CIS, on start up in multiCluster mode, if any external cluster kube-api server is down/not reachable, CIS is stuck and not processing any valid clusters config. Workaround is to remove the unreachable cluster config from configmap and restart CIS.
CIS fails to post a VS declaration with health monitors in ratio mode. Issue is observed intermittently.

FAQ¶

Is –multi-cluster-mode is a required parameter for Multi-Cluster support?

Yes. Multi-Cluster support only works if --multi-cluster-mode is defined in the CIS deployment.

Is extended configMap mandatory for Multi-Cluster support?

Yes. Multi-Cluster support only works with extended configmap.

Is extended configmap update require CIS restart?

Yes. It is recommended to restart CIS if any HA configuration or external cluster configurations are updated in extended configmap. However, CIS restart is not required when updating ratio in the extended configmap.

How do you add a new cluster?

To add a new cluster, create a kube-config file with read-only permissions. Next, create a Kubernetes secret using the kube-config file. Refer this in secret in the extended configmap to add the new cluster. CIS dynamically reads the new kube-config of the new cluster and starts listening to the services and endpoints in the new cluster when a route refers this new cluster.

Where do you manage the manifest or configuration objects such as routes, configmaps, etc.?

Manifests or configuration objects are managed centralized in the primary cluster and, if HA is desired, the same manifests are expected to be in the secondary cluster.

What are the supported CNIs?

Currently, NodePort mode is supported. For cluster mode, static routing mode is supported to enable configuration of static routes on BIG-IP for pod network subnets for direct routing from BIG-IP to k8s pods

What kind of providers are supported?

CIS supports hybrid cloud, and any public cloud providers such as; AWS, Azure, GCP, On-Prem, VmWare, Tanzu etc., which are in the same network/datacenter and can communicate with each other.

What kind of clusters are supported?

CIS multicluster solution is currently validated with OpenShift clusters and k8s clusters

How does CIS start as a secondary cluster?

CIS recognizes the cluster as secondary when it starts with a deployment parameter i.e. --multi-cluster-mode=secondary

How does the secondary CIS learn about the primary cluster endpoint state in HA mode?

Both of the CIS’ will communicate with both k8s API servers and prepares the AS3 declaration. The secondary CIS only sends the declaration when the primary cluster’s health probe fails. As soon as primary cluster comes up, secondary CIS stops sending the declaration.

What kind of permission is required for HA or standalone deployment of CIS?

No RBAC change for CIS deployment with multiCluster support. Only an additional kube-config configuration with read only permission is required to access the endpoints from an external cluster.

What kind of permission is required to access external clusters, apart from HA and standalone?

CIS requires read-only permissions in kube-config of external clusters to access resources like pods, services, endpoints and nodes.

Can CIS manage multiple BIG-IPs?

No. CIS can manage only standalone BIG-IP or an HA BIG-IP. In other words, CIS acts as a single point of BIG-IP orchestrator and supports multi-cluster.

Is traffic splitting with cluster ratio supported?

Yes. CIS supports traffic splitting as per the ratio specified for each cluster and also works with A/B as well.

Is A/B supported in multiCluster mode?

Yes. CIS supports A/B with multiCluster. However, it’s not supported with extendedServices/multiClusterServices.

Previous Next