Managing BIG-IP HA Clusters in OpenShift

× Warning! This solution applies to BIG-IP devices v13.x and later only.

You can use the BIG-IP Controller for OpenShift to manage a BIG-IP High Availability (HA) active-standby pair or device group [1]. You will need to deploy one BIG-IP Controller instance for each BIG-IP device. Each device will connect to a shared subnet that passes client traffic and to a management subnet.

A diagram showing the BIG-IP High Availability solution in an OpenShift cluster. The solution features 2 BIG-IP devices, each of which has its own BIG-IP Controller. Both BIG-IPs have 2 interfaces; one connects to the floating subnet used for client traffic, and the other connects to a subnet used for health monitoring.

Complete the steps below to set up the solution shown in the diagram. Be sure to use the correct IP addresses and subnet masks for your OpenShift Cluster.

Tasks
Step Task
Initial BIG-IP Device Setup

Add BIG-IP devices to OpenShift

Set up the VXLAN on the BIG-IP devices

Deploy the BIG-IP Controller

Initial BIG-IP Device Setup

Important

The steps in this section require either Administrator or Resource Administrator permissions on the BIG-IP system.

  1. If you want to use BIG-IP High Availability (HA), set up two or more F5 BIG-IPs in a Device Service Cluster (DSC).

  2. Create a new partition on your BIG-IP system.

    Note

    • The BIG-IP Controller can not manage objects in the /Common partition.
    • [Optional] The Controller can decorate the IP addresses it configures on the BIG-IP with a Route Domain identifier. You may want to use route domains if you have many applications using the same IP address space that need isolation from one another. After you create the partition on your BIG-IP system, you can 1) create a route domain and 2) assign the route domain as the partition’s default. See create and set a non-zero default Route Domain for a partition for setup instructions.
    • [Optional] If you’re using a BIG-IP HA pair or cluster, sync your changes across the group.
  3. Store your BIG-IP login credentials in a Secret.

  4. If you need to pull the k8s-bigip-ctlr image from a private Docker registry, store your Docker login credentials as a Secret.

Add BIG-IP devices to OpenShift

Important

The examples below add two BIG-IP devices to the OpenShift cluster. If you have more than two BIG-IPs, be sure to repeat the steps here for each additional device.

Define HostSubnets

Tip

HostSubnets must use valid YAML or JSON. Using a linter prior to deploying your HostSubnet can cut down on troubleshooting later!

  1. Create one HostSubnet for each BIG-IP device. These will handle health monitor traffic.

    HostSubnet for BIG-IP 1
    apiVersion: v1
    kind: HostSubnet
    metadata:
      name: f5-bigip-node01
      annotations:
        pod.network.openshift.io/fixed-vnid-host: "0"
        pod.network.openshift.io/assign-subnet: "true"
    # provide a name for the node that will serve as BIG-IP's entry into the cluster
    host: f5-bigip-node01
    # The hostIP address will be the BIG-IP interface address routable to the
    # OpenShift Origin nodes.
    # This address is the BIG-IP VTEP in the SDN's VXLAN.
    hostIP: 172.16.1.28
    

    f5-kctlr-openshift-hostsubnet-node01.yaml

    HostSubnet for BIG-IP 2
    apiVersion: v1
    kind: HostSubnet
    metadata:
      name: f5-bigip-node02
      annotations:
        pod.network.openshift.io/fixed-vnid-host: "0"
        pod.network.openshift.io/assign-subnet: "true"
    # provide a name for the BIG-IP device's host Node
    host: f5-bigip-node02
    # Provide an IP address to serve as the BIG-IP VTEP in the OpenShift SDN
    hostIP: 172.16.1.29
    

    f5-kctlr-openshift-hostsubnet-node02.yaml

  2. Create one HostSubnet to pass client traffic. You will create the floating IP address for the active device in this subnet.

    HostSubnet for client traffic
    apiVersion: v1
    kind: HostSubnet
    metadata:
      name: f5-bigip-float
      annotations:
        pod.network.openshift.io/fixed-vnid-host: "0"
        pod.network.openshift.io/assign-subnet: "true"
    # provide a name for the node that will serve as BIG-IP's entry into the cluster
    host: f5-bigip-float
    # The hostIP address will be the BIG-IP interface address routable to the
    # OpenShift Origin nodes.
    # This address is the BIG-IP VTEP in the SDN's VXLAN.
    hostIP: 172.16.1.30
    

    f5-kctlr-openshift-hostsubnet-float.yaml

Upload the HostSubnet files to the OpenShift API server

You can upload the files individually using separate oc create commands or upload them all at once, as shown below.

oc create -f f5-kctlr-openshift-hostsubnet-node01.yaml -f f5-kctlr-openshift-hostsubnet-node02.yaml -f f5-kctlr-openshift-hostsubnet-float.yaml
hostsubnet f5-bigip-node01 created
hostsubnet f5-bigip-node02 created
hostsubnet f5-bigip-float created

Verify creation of the HostSubnets

Use oc get to retrieve information about your newly-created HostSubnets. Record the hostIP and subnet values for each; you will use these when setting up the VXLAN on your BIG-IP devices.

oc get hostsubnet
NAME                HOST                    HOST IP       SUBNET

f5-bigip-float       f5-bigip-float       172.16.1.30   10.129.6.0/14
f5-bigip-node01      f5-bigip-node01      172.16.1.28   10.129.2.0/14
f5-bigip-node02      f5-bigip-node02      172.16.1.29   10.129.4.0/14
...

Set up the VXLAN on the BIG-IP devices

Important

The steps in this section require either Administrator or Resource Administrator permissions on the BIG-IP system.

Take the steps below on each BIG-IP device in the pair or cluster.

Create a VXLAN profile

In a TMOS shell, create a VXLAN profile that uses multi-cast flooding.

create /net tunnels vxlan ose-vxlan flooding-type multipoint

Create a VXLAN tunnel

  • Use the hostIP IP address provided for the “float” HostSubnet as the VXLAN’s local-address.
  • Use the hostIP IP address provided for the BIG-IP node’s HostSubnet as the VXLAN’s secondary-address.
  • Set the key to 0 to grant the BIG-IP device access to all OpenShift projects and subnets.

BIG-IP Node 01

create /net tunnels tunnel openshift_vxlan key 0 profile ose-vxlan local-address 172.16.1.30 secondary-address 172.16.1.28 traffic-group traffic-group-1

BIG-IP Node 02

create /net tunnels tunnel openshift_vxlan key 0 profile ose-vxlan local-address 172.16.1.30 secondary-address 172.16.1.29 traffic-group traffic-group-1

Note

When using a non-zero Route Domain, add remote-address any to the command, as shown below.

create /net tunnels tunnel openshift_vxlan key 0 profile ose-vxlan local-address 172.16.1.30 secondary-address 172.16.1.28 traffic-group traffic-group-1 remote-address any

BIG-IP Node 02

create /net tunnels tunnel openshift_vxlan key 0 profile ose-vxlan local-address 172.16.1.30 secondary-address 172.16.1.29 traffic-group traffic-group-1 remote-address any

Create a self IP in the VXLAN

Create a self IP address in the VXLAN on each device.

  • The self IP range must fall within the cluster subnet mask. Use the command oc get clusternetwork to find the correct subnet mask for your cluster.
  • If you use the BIG-IP configuration utility to create a self IP, you may need to provide the full netmask instead of the CIDR notation.
  • Be sure to specify a floating traffic group (for example, traffic-group-1). Otherwise, the self IP will use the BIG-IP system’s default.

BIG-IP Node 01

create /net self 10.129.2.3/14 allow-service none vlan openshift_vxlan

BIG-IP Node 02

create /net self 10.129.4.3/14 allow-service none vlan openshift_vxlan

Create a floating IP in the VXLAN

  1. On the active device, create a floating IP address in the subnet assigned by the OpenShift SDN.

    create /net self 10.129.6.4/14 allow-service none traffic-group traffic-group-1 vlan openshift_vxlan
    
  2. In a TMOS shell, run the config-sync command to sync your changes to the device group.

    run /cm config-sync to-group <sync_group>
    

Deploy the BIG-IP Controller

Take the steps below to deploy a BIG-IP Controller for each BIG-IP device in the cluster.

Set up RBAC

You can create RBAC resources in the project in which you will run your BIG-IP Controller. Each Controller that manages a device in a cluster or active-standby pair can use the same Service Account, Cluster Role, and Cluster Role Binding.

Required RBAC Permissions
API groups Resources Actions
“” endpoints, namespaces, nodes, routes, services, secrets get, list, watch
“extensions” ingresses get, list, watch
“” configmaps, events get, list, watch, update, create, patch
“extensions” ingresses/status get, list, watch, update, create, patch

Tip

Create the RBAC resources in the same Project (or namespace) as the BIG-IP Controller, or in a Project the BIG-IP Controller can access.

If you need to be able to access the RBAC resources from all Projects, an OpenShift administrator should create them in the kube-system namespace (-n kube-system).

In these cases, you can either:

  • use the Controller’s default “watch all namespaces” setting (requires no additional configuration); or
  • set the Controller to watch both the kube-system namespace and the Project’s namespace.
  1. Create a Service Account for the BIG-IP Controller.

    oc create serviceaccount bigip-ctlr [-n kube-system]
    serviceaccount "bigip-ctlr" created
    
  2. Create a Cluster Role and Cluster Role Binding with the required permissions.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    # For use in OpenShift clusters
    apiVersion: v1
    kind: ClusterRole
    metadata:
      annotations:
        authorization.openshift.io/system-only: "true"
      name: system:bigip-ctlr
    rules:
    - apiGroups: ["", "extensions"]
      resources: ["nodes", "services", "endpoints", "namespaces", "ingresses", "routes" ]
      verbs: ["get", "list", "watch"]
    - apiGroups: ["", "extensions"]
      resources: ["configmaps", "events", "ingresses/status"]
      verbs: ["get", "list", "watch", "update", "create", "patch" ]
    - apiGroups: ["", "extensions"]
      resources: ["secrets"]
      resourceNames: ["<secret-containing-bigip-login>"]
      verbs: ["get", "list", "watch"]
    
    ---
    
    apiVersion: v1
    kind: ClusterRoleBinding
    metadata:
        name: bigip-ctlr-role
    userNames:
    - system:serviceaccount:kube-system:bigip-ctlr
    subjects:
    - kind: ServiceAccount
      name: bigip-ctlr
    roleRef:
      name: system:bigip-ctlr
    

    f5-kctlr-openshift-clusterrole.yaml

  3. Upload the Cluster Role and Cluster Role Binding to the API server.

    oc create -f f5-kctlr-openshift-clusterrole.yaml [-n kube-system]
    clusterrole "system:bigip-ctlr" created
    clusterrolebinding "bigip-ctlr-role" created
    

Create Deployments

Create an OpenShift Deployment for each Controller (one per BIG-IP device):

  • Provide a unique metadata.name for each Controller.
  • Provide a unique --bigip-url in each Deployment (each Controller manages a separate BIG-IP device).
  • Use the same --bigip-partition in all Deployments.

Important

Do not define multiple Deployment configs in a single manifest.

If you launch multiple BIG-IP Controller instances using a single manifest, they will run on the same Pod. This means that if the Pod goes down, you lose all of your Controllers.

The example Deployments below include the settings that the BIG-IP Controller needs to manage OpenShift Routes. If you don’t need/want to manage Routes, exclude the following settings:

  • "--manage-routes=true"
  • "--route-vserver-addr=1.2.3.4"
  • "--route-label=App1"

See Attach Virtual Servers to OpenShift Routes for additional information.

BIG-IP Controller 1
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: f5-bigip-ctlr-01
spec:
  replicas: 1
  template:
    metadata:
      name: k8s-bigip-ctlr
      labels:
        app: k8s-bigip-ctlr
    spec:
      # Name of the Service Account bound to a Cluster Role with the required
      # permissions
      serviceAccountName: bigip-ctlr
      containers:
        - name: k8s-bigip-ctlr
          # replace the version as needed
          image: "f5networks/k8s-bigip-ctlr:1.5"
          env:
            - name: BIGIP_USERNAME
              valueFrom:
                secretKeyRef:
                  # Replace with the name of the Secret containing your login
                  # credentials
                  name: bigip-login
                  key: username
            - name: BIGIP_PASSWORD
              valueFrom:
                secretKeyRef:
                  # Replace with the name of the Secret containing your login
                  # credentials
                  name: bigip-login
                  key: password
          command: ["/app/bin/k8s-bigip-ctlr"]
          args: [
            # See the k8s-bigip-ctlr documentation for information about
            # all config options
            # http://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/latest
            "--bigip-username=$(BIGIP_USERNAME)",
            "--bigip-password=$(BIGIP_PASSWORD)",
            "--bigip-url=<ip_address_of_bigip_1>",
            "--bigip-partition=my_bigip_partition",
            "--pool-member-type=cluster",
            "--openshift-sdn-name=/Common/openshift_vxlan",
            "--manage-routes=true",
            "--route-vserver-addr=1.2.3.4"
            "--route-label=App1"
            "--vs-snat-pool-name=<snat-pool>",
            ]
      imagePullSecrets:
        # Secret that gives access to a private Docker registry
        - name: f5-docker-images
        # Secret containing the BIG-IP system login credentials
        - name: bigip-login

Download f5-k8s-bigip-ctlr_openshift_ha-node01.yaml

BIG-IP Controller 2
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
apiVersion: extensions/v1beta1
kind: Deployment
metadata
  name: f5-bigip-ctlr-02
spec:
  replicas: 1
  template:
    metadata:
      name: k8s-bigip-ctlr
      labels:
        app: k8s-bigip-ctlr
    spec:
      # Name of the Service Account bound to a Cluster Role with the required
      # permissions
      serviceAccountName: bigip-ctlr
      containers:
        - name: k8s-bigip-ctlr
          # replace the version as needed
          image: "f5networks/k8s-bigip-ctlr:1.5"
          env:
            - name: BIGIP_USERNAME
              valueFrom:
                secretKeyRef:
                  # Replace with the name of the Secret containing your login
                  # credentials
                  name: bigip-login
                  key: username
            - name: BIGIP_PASSWORD
              valueFrom:
                secretKeyRef:
                  # Replace with the name of the Secret containing your login
                  # credentials
                  name: bigip-login
                  key: password
          command: ["/app/bin/k8s-bigip-ctlr"]
          args: [
            # See the k8s-bigip-ctlr documentation for information about
            # all config options
            # http://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/latest
            "--bigip-username=$(BIGIP_USERNAME)",
            "--bigip-password=$(BIGIP_PASSWORD)",
            "--bigip-url=<ip_address_of_bigip_2>",
            "--bigip-partition=my_bigip_partition",
            "--pool-member-type=cluster",
            "--openshift-sdn-name=/Common/openshift_vxlan",
            "--manage-routes=true",
            "--route-vserver-addr=1.2.3.5"
            "--route-label=App1",
            "--vs-snat-pool-name=<snat-pool>",
            ]
      imagePullSecrets:
        - name: f5-docker-images
        - name: bigip-login

Download f5-k8s-bigip-ctlr_openshift_ha-node02.yaml

Upload the Deployments

  1. Upload the Deployments to the OpenShift API server.

    oc create -f f5-k8s-bigip-ctlr_openshift-node01-routes.yaml -f f5-k8s-bigip-ctlr_openshift-node02-routes.yaml
    deployment "f5-bigip-ctlr-01" created
    deployment "f5-bigip-ctlr-02" created
    
  2. Verify Pod creation.

    oc get pods
    NAME                                    READY     STATUS    RESTARTS   AGE
    f5-bigip-ctlr-01-1530682540-7rs5s         1         1         1         5m
    f5-bigip-ctlr-02-5973567192-trh2W         1         1         1         5m
    

Footnotes

[1]Does not apply to BIG-IP devices v12.x and earlier.