OpenShift/Kubernetes Multi-Cluster

This page documents a new feature in CIS for Multi-Cluster.


Note

To provide feedback on Container Ingress Services or this documentation, please file a GitHub Issue.

Contents

Overview

Topologies

Configuration

multicluster_extended_configmap

multicluster_examples

multicluster_userguides

multicluster_known_issues

multicluster_faq

Overview

CIS Multi-Cluster support allow users to expose multiple applications spread across OpenShift/Kubernetes clusters using a single BIG-IP Virtual Server. An application can be deployed in different OpenShift/Kubernetes clusters exposing them using a ServiceTypeLB/Route/VS/TS CR resource. Using a Multi-Cluster implementation, the CIS can be deployed in a High Availability topology or Standalone CIS (see Topologies) to expose the application spread across OpenShift/Kubernetes clusters.

Note

At present, nodePort mode is supported and Cluster mode is available with static route configuration on BIG-IP (No tunnels).

Prerequisites

  • The API server of all OpenShift/Kubernetes clusters must be accessible by the cluster node where CIS is installed.
  • ExtendedConfigMap should be created to run the CIS in Multi-Cluster mode.
  • The CIS should have access to Kube-config files for each cluster in order to reach resources such as Pods, Services, Endpoints, and Nodes.

Topologies

Standalone CIS

When using a Standalone setup, CIS is solely installed on a single cluster. There are two modes supported in the Standalone CIS, namely default and ratio. - The default mode is supported with VS/TS CRs and ServiceType LB, see Default Mode. - The ratio mode is supported with VS/TS CRs and routes, see Ratio Mode.

../../_images/multi-clusterstandalone.png

High Availability CIS

The High Availability (HA) CIS offers three supported modes: default, active-active, and ratio.

  • The default mode is supported with VS/TS CRs and ServiceType LB, see Default Mode.
  • The Active-Active mode is supported with VS/TS CRs and routes, see Active-Active Mode.
  • The ratio mode is supported with VS/TS CRs and routes, see Ratio Mode.

Prerequisites

  • A pair of high availability OpenShift/Kubernetes clusters should be available, which have the same applications running in both clusters.
  • The primary cluster should have an available HealthCheck endpoint for checking its health. Currently, Health endpoints using TCP/HTTP are supported.
  • For a HA deployment of CIS, it is necessary to install CIS on both the primary and secondary cluster. Additionally, the extendedConfigMap must be deployed on both clusters as well.

Note

  • For HA mode [namely default, active-active, ratio], CIS monitored resource manifests such as routes, CRDs, extendedConfigmaps, and multiCluster serviceTypeLB must be available on both the clusters.
  • The CIS monitored resource manifests must be identical on both the primary and secondary clusters. So that, incase of CIS failover, CIS on secondary cluster will take control and start processing the CIS monitored resource manifests.
  • CIS on secondary cluster will not process the CIS monitored resource manifests if they are not available in secondary cluster.
../../_images/multi-clusterha.png

Configuration

If you are using multi-cluster mode, –multi-cluster-mode and –local-cluster-name are required parameters.

Parameter Type Required Description Allowed Values
Multi-cluster-mode String Required Specify whether CIS is running standalone or as primary/secondary in the case of a high availability topology. Standalone or primary or secondary
Local-cluster-name String Required Specify the cluster name where CIS is running. A valid cluster name defined in the extendedConfigMap
extended-spec-configmap String Required
extendedSpecConfigMap is required to configure the multi-cluster
configuration
 /

Note

Here standalone refers to standalone topology of CIS deployment, see Standalone CIS.

Following is the sample deployment for primary CIS deployment:

spec:
  containers:
  - args:
    - --bigip-partition
    - <partition>
    - --bigip-url
    - <ip-address>
    - --bigip-username
    - <user-name>
    - --bigip-password
    - <password>
    - --log-level
    - DEBUG
    - --insecure
    - --controller-mode=openshift
    - --extended-spec-configmap=kube-system/extended-spec-config
    - --route-label=systest
    - --pool-member-type
    - nodeport
    - --multi-cluster-mode=primary
    - --local-cluster-name=cluster1
    command:
    - /app/bin/k8s-bigip-ctlr
    image: <image-name>

Note

  • Update the multi-cluster-mode to secondary for secondary CIS deployment in high availablility topology, see :ref:`ha_cis.
  • Update the multi-cluster-mode to standalone for standalone topology, see Standalone CIS.
  • Update the local-cluster-name to the cluster name where CIS is running.

. - To configure the VS CR in active-active and ratio mode, see

FAQ

Is –multi-cluster-mode a required parameter for Multi-Cluster support?

Yes. Multi-Cluster support only works if –multi-cluster-mode is defined in the CIS deployment.

Is extended configMap mandatory for Multi-Cluster support?

Yes. Multi-Cluster support only works with extended configmap.

Is local-cluster-name mandatory for Multi-Cluster support?

Yes. Multi-Cluster support only works if –local-cluster-name is defined in the CIS deployment.

Does extended configmap update require CIS restart?

No. It is recommended to restart CIS if any HA configuration or external cluster configurations are updated in extended Configmap. However, CIS restart is not required when updating ratio in the extended Configmap.

Does mode update require CIS restart?

Yes. CIS has to be restarted when there is a change in the mode.

How do you add a new cluster?

To add a new cluster, create a kube-config file with read only permissions. Then create a Kubernetes secret using the kube-config file. Refer this in secret in the extended ConfigMap to add the new cluster. CIS dynamically reads the new kube-config of the new cluster and starts listening to the services and endpoints in the new cluster when a route refers this new cluster.

Where do you manage the manifest or Configuration Objects like Routes, CRDs, ExtendedConfigmaps etc.?

Manifests or Configuration objects are managed centrally in Primary Cluster. If CIS HA is desired, the same manifests are expected to be in Secondary Cluster.

What are the supported CNIs?

Currently, NodePort mode is supported. For cluster mode, static routing mode is supported to enable configuration of static routes on BIG-IP for pod network subnets for direct routing from BIG-IP to k8s Pods.

What kind of providers are supported?

CIS supports Hybrid Cloud, any public Cloud providers such as; AWS, Azure, GCP, On-Prem, VmWare, and Tanzu etc. which is in same network/datacenter and can communicate with each other.

What kind of clusters are supported?

CIS multicluster solution is currently validated with openshift clusters and Kubernetes clusters.

How does CIS start as a secondary Cluster?

CIS recognizes a Secondary cluster when it starts with a deployment parameter –multi-cluster-mode is set to secondary.

How does Secondary CIS learn about the Primary Cluster endpoint state in HA mode?

Both of the CIS will communicate with both K8s API servers and prepares the AS3 declaration. But, the secondary CIS only sends the declaration when the Primary cluster’s health probe fails. As soon as primary cluster comes up, the secondary CIS stops sending the declaration.

What kind of permission is required for HA or StandAlone deployment of CIS?

No RBAC change for CIS deployment with multiCluster support. Only additional kube-config configuration with read only permission is required to access the endpoints from external cluster.

What kind of permission is required to access external clusters (apart from HA and StandAlone)?

CIS requires read-only permission in kube-config of external clusters to access resources like Pods/Services/Endpoints/Nodes.

Can CIS manage multiple BIG-IPs?

No. CIS can manage only Standalone BIG-IP or HA BIG-IP. In other words, CIS acts as a single point of BIG-IP Orchestrator and supports Multi-Cluster.

Does Secondary CIS require resource manifests existing in Primary Cluster?

Yes. CIS on Secondary Cluster will not process the CIS monitored resource manifests[NextGen Routes, CRDs, ServiveTypeLB, extendedConfigmap] if they are not available in Primary Cluster. This is required in case of CIS failover, CIS on Secondary Cluster will take control and will start processing the CIS monitored resource manifests. It is suggested to maintain identical CIS monitored resource manifests in both the clusters to avoid any issues during CIS failover. This requirement is applicable in case of CIS HA mode [namely default, active-active, ratio].

How to configure the primaryEndPoint in HA mode?

The primaryEndPoint is a mandatory parameter if CIS is intended to run in Multi-Cluster HA mode [namely default, active-active, ratio]. If this is not specified, the secondary CIS will not run. Secondary CIS will continuously monitor the health of the primary cluster based on the primaryEndPoint value. If it is down, then the secondary CIS takes the responsibility of posting declarations to BIG-IP. Supported Protocols for the primaryEndPoint are HTTP and TCP. Generally, it is suggested to use the primaryEndPoint as:

  1. Any available endpoint to check the health of the primary cluster.
  2. Primary CIS cluster’s health check endpoint (/health) if accessible.

c. Primary CIS cluster’s kube-api server endpoint if accessible. Response code 200 OK is expected from the primaryEndPoint in case of HTTP Protocol. Successful TCP connection is expected from the primaryEndPoint in case of TCP Protocol. Secondary CIS become active if the primaryEndPoint is not accessible. PrimaryEndPoint is optional to configure for Primary CIS.

Note

Primary CIS will not monitor the health of the Secondary CIS cluster.

How to configure the primaryEndPoint in Standalone mode?

The primaryEndPoint is not applicable in Standalone mode. It is only applicable in HA mode.

How to use CIS / health endpoint to check the health of CIS?

Retrieve the CIS PodIP and incorporate it into the curl command displayed below from any of the cluster nodes.

curl  http://<CIS-PodIP>:8080/health

Response code 200 OK is expected from the CIS /health endpoint if kube-api server is accessible. Example:

[root@cluster-1-worker0 ~]# curl http://10.244.1.213:8080/health
Ok[root@cluster-1-worker0 ~]# curl http://10.244.1.213:8080/health -v
* About to connect() to 10.244.1.213 port 8080 (#0)
*   Trying 10.244.1.213...
* Connected to 10.244.1.213 (10.244.1.213) port 8080 (#0)
> GET /health HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.244.1.213:8080
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 07 Dec 2023 08:28:57 GMT
< Content-Length: 2
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host 10.244.1.213 left intact
Ok[root@cluster-1-worker0 ~]#

where 10.244.1.213 is the CIS PodIP.

Are the custom resources like TS and VS are same in all modes(default, active-active, ratio)?

No, Custom resources like TS and VS are different in default mode and non-default mode. See Default Mode for default mode and Overview for active-active and ratio mode.

Are the routes supported in all modes(default, active-active, ratio)?

No, Routes are only supported in active-active and ratio mode See Overview for active-active and ratio mode.

Are policy CR supported in all modes(default, active-active, ratio)?

Yes, Policy CR is supported in all modes same as non-multi-cluster mode.