Kubernetes

How to deploy Kubernetes

Overview

CIS can be configured in multiple ways depending on the customer scenario. CIS can be deployed on Kubernetes and OpenShift platform. CIS installation may differ based on the resources (for example: ConfigMap, Ingress, Routes, and CRD) used by the customer to expose the Kubernetes services. CIS installation also depends on BIG-IP deployment (stand alone and HAproxy configuration) and Kubernetes cluster networking (Flannel/Calico).

Prerequisites

These are the mandatory requirements for deploying CIS:

  • Kubernetes Cluster must be up and running

  • AS3: 3.18+

  • BIG-IP partition to create Kubernetes cluster objects which can be created on the BIG-IP using the following tmos command:

    create auth partition <cis_managed_partition>
    
  • You need a user with administrative access to this partition.

  • If you need to pull the k8s-bigip-ctlr image from a private Docker registry, store your Docker login credentials as a Secret.

Additionally, if you are deploying the CIS in Cluster Mode you need to have following prerequisites. For more information, see Deployment Options.

Installing CIS Manually

  1. Add BIG-IP credentials as K8S secrets.

    kubectl create secret generic bigip-login -n kube-system --from-literal=username=admin --from-literal=password=<password>
    
  2. Create a service account for deploying CIS.

    kubectl create serviceaccount k8s-bigip-ctlr -n kube-system
    
  3. Create a Cluster Role and Cluster Role Binding on the Kubernetes Cluster using the examples below.

    kubectl apply -f  k8s_rbac.yaml
    

    The example below shows the broadest supported permission set. You can narrow the permissions down to specific resources, namespaces, etc. to suit your needs. See the Kubernetes RBAC documentation for more information.

    k8s_rbac.yaml
     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
     # for use in k8s clusters only
     kind: ClusterRole
     apiVersion: rbac.authorization.k8s.io/v1
     metadata:
       name: bigip-ctlr-clusterrole
     rules:
     - apiGroups: ["", "extensions"]
       resources: ["nodes", "services", "endpoints", "namespaces", "ingresses", "pods"]
       verbs: ["get", "list", "watch"]
     - apiGroups: ["", "extensions"]
       resources: ["configmaps", "events", "ingresses/status"]
       verbs: ["get", "list", "watch", "update", "create", "patch"]
     - apiGroups: ["", "extensions"]
       resources: ["secrets"]
       resourceNames: ["<secret-containing-bigip-login>"]
       verbs: ["get", "list", "watch"]
     - apiGroups: ["cis.f5.com"]
       resources: ["virtualservers", "tlsprofiles", "transportservers"]
       verbs: ["get", "list", "watch", "update"]
    
     ---
    
     kind: ClusterRoleBinding
     apiVersion: rbac.authorization.k8s.io/v1
     metadata:
       name: bigip-ctlr-clusterrole-binding
       namespace: kube-system
     roleRef:
       apiGroup: rbac.authorization.k8s.io
       kind: ClusterRole
       name: bigip-ctlr-clusterrole
     subjects:
     - apiGroup: ""
       kind: ServiceAccount
       name: bigip-ctlr
       namespace: kube-system
    
  4. Create a CIS deployment using cis_deploy.yaml as shown below:

    kubectl apply -f  cis_deploy.yaml
    
    cis_deploy.yaml
     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: k8s-bigip-ctlr-deployment
      namespace: kube-system
    spec:
    # DO NOT INCREASE REPLICA COUNT
      replicas: 1
      selector:
        matchLabels:
          app: k8s-bigip-ctlr-deployment
      template:
        metadata:
          labels:
            app: k8s-bigip-ctlr-deployment
        spec:
          # Name of the Service Account bound to a Cluster Role with the required
          # permissions
          containers:
            - name: k8s-bigip-ctlr
              image: "f5networks/k8s-bigip-ctlr"
              env:
                - name: BIGIP_USERNAME
                  valueFrom:
                    secretKeyRef:
                    # Replace with the name of the Secret containing your login
                    # credentials
                      name: bigip-login
                      key: username
                - name: BIGIP_PASSWORD
                  valueFrom:
                    secretKeyRef:
                    # Replace with the name of the Secret containing your login
                    # credentials
                      name: bigip-login
                      key: password
              command: ["/app/bin/k8s-bigip-ctlr"]
              args: [
                # See the k8s-bigip-ctlr documentation for information about
                # all config options
                # https://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/latest
                "--bigip-username=$(BIGIP_USERNAME)",
                "--bigip-password=$(BIGIP_PASSWORD)",
                "--bigip-url=<ip_address-or-hostname>",
                "--bigip-partition=<name_of_partition>",
                "--pool-member-type=nodeport",
                "--insecure",
                # for secure communication provide the internal ca certificates using config-map with below option and remove insecure parameter
                #"--trusted-certs-cfgmap=<namespace/configmap>",
                ]
          serviceAccount: bigip-controller
          serviceAccountName: bigip-controller
        imagePullSecrets:
            # Secret that gives access to a private docker registry
            #- name: f5-docker-images
            # Secret containing the BIG-IP system login credentials
            - name: bigip-login
    

Creating VXLAN Tunnels on Kubernetes Cluster

  1. Login to BIG-IP and create a VXLAN tunnel profile.

    tmsh create auth partition kubernetes
    
  2. Create a VXLAN tunnel.

    tmsh create net tunnels tunnel k8s-tunnel { app-service none key 1 local-address 10.1.1.4 profile k8s-vxlan }
    
  3. Create the VXLAN tunnel self-IP.

    tmsh create net self k8s-vxlan-selfip { address 10.244.20.1/16 vlan k8s-tunnel allow-service all }
    
  4. Save the configuration.

    tmsh save sys config
    
  5. Before deploying CIS in ClusterIP mode, you need to configure BIG-IP as a node in the Kubernetes cluster. To do so you will need to modify bigip-node.yaml with the MAC address auto-created from the previous steps. From the jumpbox terminal, run the following command at bigip1. Copy the displayed MAC Address.

    tmsh show net tunnels tunnel k8s-tunnel all-properties
    
  6. Update the MAC address obtained in the previous step to following yaml file:

    bigip-node.yaml
     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    apiVersion: v1
    kind: Node
    metadata:
      name: bigip1
      annotations:
        #Replace IP with Self-IP for your deployment
        flannel.alpha.coreos.com/public-ip: "10.1.1.4"
        #Replace MAC with your BIG-IP Flannel VXLAN Tunnel MAC
        flannel.alpha.coreos.com/backend-data: '{"VtepMAC":"2c:c2:60:23:0c:58"}'
        flannel.alpha.coreos.com/backend-type: "vxlan"
        flannel.alpha.coreos.com/kube-subnet-manager: "true"
    spec:
      #Replace Subnet with your BIG-IP Flannel Subnet
      podCIDR: "10.244.20.0/24"
    
  7. Create the BIG-IP node:

    kubectl create -f bigip-node.yaml
    
  8. Verify “bigip1” node is created:

    kubectl get nodes
    

Examples Repository

View more examples on GitHub.


Note

To provide feedback on Container Ingress Services or this documentation, you can file a GitHub Issue.