How to deploy CRDs using Non Default RouteDomains in ClusterMode¶
Prerequisites¶
- All Self IPs and VLAN interfaces must be removed.
- Remove any existing VXLAN Tunnel, VXLAN FDBs, Static Routes, and Static ARPs associated with the CIS-managed BIG-IP partition.
Objective: CIS supports RouteDomain for both multipartition and default partition (single partition) using the following procedures.
Note
CIS can support only a single RD across CRDs. If it is intended to use different RDs, best practice is to use one CIS per RD.
Configuring non-default RouteDomains¶
Below are two approaches to configure non-default Route Domains, however there may be more as well.
- Create all NET Objects (i.e. VLANs, Self IPs interfaces, Tunnel Interfaces, Tunnels, Default Tunnel Route) in the CIS-managed partition except RouteDomain. Create the RouteDomain in the Common partition. First Approach Procedure
- Create all NET Objects in the CIS-managed partition except RouteDomain, VLAN, and respective VLAN Self IPs in the Common partition. Second Approach Procedure
Note: Both approaches in this example use Cilium CNI, however the same can be applied to other CNIs as well.
First Approach Procedure¶
- Remove any existing VXLAN Tunnels, VXLAN FDBs, VXLAN Self IPs, Static Routes, and Static ARPs (if any) associated with the CIS-managed BIG-IP partition or default RouteDomain 0.
Below is the configuration on the BIG-IP after removing:
Self IPs

Route List

Tunnel List

Static ARP List

Route Domain List

Self IPs

VLAN List









VXLAN Tunnel
Profile: fl-vxlan, Key: 2, Local Address: 10.4.0.28


VXLAN Self IP


VXLAN Static Route


- Start CIS with default-route-domain as 5.
spec:
containers:
- args:
- --bigip-partition
- test
- --bigip-url
- 10.x.x.x
- --bigip-username
- adxxx
- --bigip-password
- adxxx
- --insecure
- --custom-resource-mode=true
- --cilium-name
- /test/vxlan-tunnel-mp
- --pool-member-type
- cluster
- --default-route-domain=5
command:
- /app/bin/k8s-bigip-ctlr
image: f5networks/k8s-bigip-ctlr:2.13.0
- Create TransportServer CR.

- LTM and NET Objects are successfully pushed to the BIG-IP.





Second Approach Procedure¶
- Remove any existing VXLAN Tunnels, VXLAN FDBs, VXLAN Self IPs, Static Routes, and Static ARPs (if any) associated with the CIS-managed BIG-IP partition or default RouteDomain 0.
Below is the configuration on the BIG-IP after removing:
Self IPs

Route List

Tunnel List

Static ARP List

Route Domain List

Self IPs

VLANs









VXLAN Tunnel
Profile: fl-vxlan, Key: 2, Local Address: 10.4.0.28


VXLAN Self IP


VXLAN Static Route


- Start CIS with default-route-domain=5.
spec:
containers:
- args:
- --bigip-partition
- test
- --bigip-url
- 10.x.x.x
- --bigip-username
- adxxx
- --bigip-password
- adxxx
- --insecure
- --custom-resource-mode=true
- --cilium-name
- /test/vxlan-tunnel-mp
- --pool-member-type
- cluster
- --default-route-domain=5
command:
- /app/bin/k8s-bigip-ctlr
image: f5networks/k8s-bigip-ctlr:2.13.0
- Create a TransportServer CR.

- LTM and NET objects are successfully pushed to the BIG-IP.





CIS Logs
- Create Virtual Server CR.

- CIS successfully learns the new VS CR, populating the BIG-IP.




CIS Logs
Cluster Information
(systest-venv) [centos@firm-aphid-bastion ~]$ kubectl get nodes -owide
NAME | STATUS | ROLES | AGE | VERSION | INTERNAL-IP | EXTERNAL-IP | OS-IMAGE | KERNEL-VERSION | CONTAINER-RUNTIME | |
---|---|---|---|---|---|---|---|---|---|---|
k8s-cilium-centos8-1.pdsea.f5net.com | Ready | control-plane | 7d20h | v1.27.1 | 10.4.2.0 | <none> | CentOS Linux 8 (Core) | 5.4.242-1.el8.elrepo.x86_64 | containerd://1.6.20 | |
k8s-cilium-centos8-2.pdsea.f5net.com | Ready | <none> | 7d20h | v1.27.1 | 10.4.2.158 | <none> | CentOS Linux 8 (Core) | 5.4.242-1.el8.elrepo.x86_64 | containerd://1.6.20 | |
k8s-cilium-centos8-3.pdsea.f5net.com | Ready | <none> | 7d20h | v1.27.1 | 10.4.1.65 | <none> | CentOS Linux 8 (Core) | 5.4.242-1.el8.elrepo.x86_64 | containerd://1.6.20 |
(systest-venv) [centos@firm-aphid-bastion ~]$ kubectl get pods -owide
NAME | READY | STATUS | RESTARTS | AGE | IP | NODE | NOMINATED NODE | READINESS GATES |
---|---|---|---|---|---|---|---|---|
pytest-svc-1-5f6fd5b76c-rpg7b | 1/1 | Running | 0 | 45h | 10.0.2.110 | k8s-cilium-centos8-2.pdsea.f5net.com | <none> | <none> |
svc-1-6b4ff4bf47-rgxjp | 1/1 | Running | 0 | 11h | 10.0.2.117 | k8s-cilium-centos8-2.pdsea.f5net.com | <none> | <none> |
(systest-venv) [centos@firm-aphid-bastion ~]$ kubectl get svc
NAME | TYPE | CLUSTER-IP | EXTERNAL-IP | PORT(S) | AGE |
---|---|---|---|---|---|
kubernetes | ClusterIP | 10.96.0.1 | <none> | 443/TCP | 7d20h |
pytest-svc-1 | NodePort | 10.96.133.168 | <none> | 1344:30982/TCP,8080:31835/TCP | 45h |
svc-1 | NodePort | 10.104.85.86 | <none> | 80:32082/TCP | 12h |
(systest-venv) [centos@firm-aphid-bastion ~]$ kubectl get ep
NAME | ENDPOINTS | AGE |
---|---|---|
kubernetes | 10.4.2.0:6443 | 7d20h |
pytest-svc-1 | 10.0.2.110:1344,10.0.2.110:8080 | 45h |
svc-1 | 10.0.2.117:80 | 12h |
(systest-venv) [centos@firm-aphid-bastion ~]$ kubectl get ts,vs,tls
NAME | VIRTUALSERVERADDRESS | VIRTUALSERVERPORT | POOL | POOLPORT | IPAMLABEL | IPAMVSADDRESS | STATUS | AGE |
---|---|---|---|---|---|---|---|---|
transportserver.cis.f5.com/cr-transport-server | 1344 | pytest-svc-1 | 1344 | Dev | 10.8.128.1 | Ok | 13h |
NAME | HOST | TLSPROFILENAME | HTTPTRAFFIC | IPADDRESS | IPAMLABEL | IPAMVSADDRESS | STATUS | AGE |
---|---|---|---|---|---|---|---|---|
virtualserver.cis.f5.com/cr-vs-foo-svc-1 | foo.com | cr-tls-foo-svc-1 | none | Test | 10.9.0.1 | Ok | 12h |
NAME | AGE |
---|---|
tlsprofile.cis.f5.com/cr-tls-foo-svc-1 | 12h |
(systest-venv) [centos@firm-aphid-bastion ~]$ kubectl get ciliumnodes.cilium.io
NAME | AGE |
---|---|
k8s-cilium-centos8-1.pdsea.f5net.com | 7d20h |
k8s-cilium-centos8-2.pdsea.f5net.com | 7d20h |
k8s-cilium-centos8-3.pdsea.f5net.com | 7d20h |
(systest-venv) [centos@firm-aphid-bastion ~]$