Lab 2.1 - Install & Configure CIS in ClusterIP Mode

In the previous moudule we learned about Nodeport Mode. Here we’ll learn about ClusterIP Mode.

See also

For more information see BIG-IP Deployment Options

BIG-IP Setup

With ClusterIP we’re utilizing VXLAN to communicate with the application pods. To do so we’ll need to configure BIG-IP first.

If not already connected, RDP to the UDF lab “jumpbox” host. Otherwise resume previous session.

  1. Open firefox and connect to bigip1. For your convenience there’s a shortcut on the toolbar. Username and password are: admin/admin

    Attention

    Be sure to be in the Common partition before creating the following objects.

    ../../_images/f5-check-partition3.png
  2. First we need to setup a partition that will be used by F5 Container Ingress Service.

    Note

    This step was performed in the previous module. Verify the “okd” partion exists and if not follow the instructions below.

    • GoTo: System ‣ Users ‣ Partition List
    • Create a new partition called “okd” (use default settings)
    • Click Finished
    ../../_images/f5-container-connector-bigip-partition-setup4.png
    # From the CLI:
    ssh admin@10.1.1.4 tmsh create auth partition okd
    
  3. Install AS3 via the management console

    Attention

    This has been done to save time. If needed see Module1 / Lab 1.1 / Install AS3 Steps

  4. Create a vxlan tunnel profile

    • GoTo: Network ‣ Tunnels ‣ Profiles ‣ VXLAN
    • Create a new profile called “okd-vxlan”
    • set Port = 4789
    • Set the Flooding Type = Multipoint
    • Click Finished
    ../../_images/create-okd-vxlan-profile.png
    # From the CLI:
    ssh admin@10.1.1.4 tmsh create net tunnel vxlan okd-vxlan { app-service none port 4789 flooding-type multipoint }
    
  5. Create a vxlan tunnel

    • GoTo: Network ‣ Tunnels ‣ Tunnel List
    • Create a new tunnel called “okd-tunnel”
    • Set the Profile to the one previously created called “okd-vxlan”
    • set the key = 0
    • Set the Local Address to 10.1.1.4
    • Click Finished
    ../../_images/create-okd-vxlan-tunnel.png
    # From the CLI:
    ssh admin@10.1.1.4 tmsh create net tunnel tunnel okd-tunnel { app-service none key 0 local-address 10.1.1.4 profile okd-vxlan }
    
  6. Create the vxlan tunnel self-ip

    Tip

    For your SELF-IP subnet, remember it is a /14 and not a /23.

    Why? The Self-IP has to know all other /23 subnets are local to this namespace, which includes Master1, Node1, Node2, etc. Each of which have their own /23.

    Many students accidently use /23, doing so would limit the self-ip to only communicate with that subnet. When trying to ping services on other /23 subnets from the BIG-IP for instance, communication will fail as your self-ip doesn’t have the proper subnet mask to know the other subnets are local.

    • GoTo: Network ‣ Self IPs
    • Create a new Self-IP called “okd-vxlan-selfip”
    • Set the IP Address to “10.131.0.1”.
    • Set the Netmask to “255.252.0.0”
    • Set the VLAN / Tunnel to “okd-tunnel” (Created earlier)
    • Set Port Lockdown to “Allow All”
    • Click Finished
    ../../_images/create-okd-vxlan-selfip.png
    # From the CLI:
    ssh admin@10.1.1.4 tmsh create net self okd-vxlan-selfip { app-service none address 10.131.0.1/14 vlan okd-tunnel allow-service all }
    

CIS Deployment

Note

  • For your convenience the file can be found in /home/ubuntu/agilitydocs/docs/class2/openshift (downloaded earlier in the clone git repo step).
  • Or you can cut and paste the file below and create your own file.
  • If you have issues with your yaml and syntax (indentation MATTERS), you can try to use an online parser to help you : Yaml parser
  1. On the jumphost open a terminal and start an SSH session with okd-master1.

    Note

    This session should be up and running from the previous module.

    # If directed to, accept the authenticity of the host by typing "yes" and hitting Enter to continue.
    
    ssh centos@okd-master1
    
  2. Just like the previous module where we deployed CIS in NodePort mode we need to create a “secret”, “serviceaccount”, and “clusterrolebinding”.

    Important

    This step can be skipped if previously done in module1(NodePort). Some classes may choose to skip module1.

    oc create secret generic bigip-login -n kube-system --from-literal=username=admin --from-literal=password=admin
    oc create serviceaccount k8s-bigip-ctlr -n kube-system
    oc create clusterrolebinding k8s-bigip-ctlr-clusteradmin --clusterrole=cluster-admin --serviceaccount=kube-system:k8s-bigip-ctlr
    
  3. Next let’s explore the f5-hostsubnet.yaml file

    cd ~/agilitydocs/docs/class2/openshift
    
    cat bigip-hostsubnet.yaml
    

    You’ll see a config file similar to this:

    bigip-hostsubnet.yaml
    1
    2
    3
    4
    5
    6
    7
    8
    9
    apiVersion: v1
    kind: HostSubnet
    metadata:
      name: openshift-f5-node
      annotations:
        pod.network.openshift.io/fixed-vnid-host: "0"
    host: openshift-f5-node
    hostIP: 10.1.1.4
    subnet: "10.131.0.0/23"
    

    Attention

    This YAML file creates an OpenShift Node and the Host is the BIG-IP with an assigned /23 subnet of IP 10.131.0.0 (3 images down).

  4. Next let’s look at the current cluster, you should see 3 members (1 master, 2 nodes)

    oc get hostsubnet
    
    ../../_images/F5-OC-HOSTSUBNET1.png
  5. Now create the connector to the BIG-IP device, then look before and after at the attached devices

    oc create -f bigip-hostsubnet.yaml
    

    You should see a successful creation of a new OpenShift Node.

    ../../_images/F5-OS-NODE.png
  6. At this point nothing has been done to the BIG-IP, this only was done in the OpenShift environment.

    oc get hostsubnet
    

    You should now see OpenShift configured to communicate with the BIG-IP

    ../../_images/F5-OC-HOSTSUBNET2.png

    Important

    The Subnet assignment, in this case is 10.131.0.0/23, was assigned by the subnet: “10.131.0.0/23” line in “HostSubnet” yaml file.

    Note

    In this lab we’re manually assigning a subnet. We have the option to let openshift auto assign ths by removing subnet: “10.131.0.0/23” line at the end of the “hostsubnet” yaml file and setting the assign-subnet: “true”. It would look like this:

    apiVersion: v1
    kind: HostSubnet
    metadata:
       name: openshift-f5-node
       annotations:
          pod.network.openshift.io/fixed-vnid-host: "0"
          pod.network.openshift.io/assign-subnet: "true"
    host: openshift-f5-node
    hostIP: 10.1.1.4
    
  7. Now that we have added a HostSubnet for bigip1 we can launch the CIS deployment. It will start the f5-k8s-controller container on one of the worker nodes.

    Attention

    This may take around 30s to get to a running state.

    cd ~/agilitydocs/docs/class2/openshift
    
    cat cluster-deployment.yaml
    

    You’ll see a config file similar to this:

    cluster-deployment.yaml
     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: k8s-bigip-ctlr
      namespace: kube-system
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: k8s-bigip-ctlr
      template:
        metadata:
          name: k8s-bigip-ctlr
          labels:
            app: k8s-bigip-ctlr
        spec:
          serviceAccountName: k8s-bigip-ctlr
          containers:
            - name: k8s-bigip-ctlr
              image: "f5networks/k8s-bigip-ctlr:2.4.1"
              imagePullPolicy: IfNotPresent
              env:
                - name: BIGIP_USERNAME
                  valueFrom:
                    secretKeyRef:
                      name: bigip-login
                      key: username
                - name: BIGIP_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: bigip-login
                      key: password
              command: ["/app/bin/k8s-bigip-ctlr"]
              args: [
                "--bigip-username=$(BIGIP_USERNAME)",
                "--bigip-password=$(BIGIP_PASSWORD)",
                "--bigip-url=https://10.1.1.4:8443",
                "--insecure=true",
                "--bigip-partition=okd",
                "--namespace=default",
                "--manage-routes=true",
                "--route-vserver-addr=10.1.1.4",
                "--route-http-vserver=okd_http_vs",
                "--route-https-vserver=okd_https_vs",
                "--route-label=hello-world",
                "--pool-member-type=cluster",
                "--openshift-sdn-name=/Common/okd-tunnel"
              ]
    
  8. Create the CIS deployment with the following command

    oc create -f cluster-deployment.yaml
    
  9. Verify the deployment “deployed”

    oc get deployment k8s-bigip-ctlr --namespace kube-system
    
    ../../_images/f5-container-connector-launch-deployment-controller1.png
  10. To locate on which node CIS is running, you can use the following command:

    oc get pods -o wide -n kube-system
    

    We can see that our container, in this example, is running on okd-node1 below.

    ../../_images/F5-CTRL-RUNNING.png

Troubleshooting

Check the container/pod logs via oc command. You also have the option of checking the Docker container as described in the previos module.

  1. Using the full name of your pod as showed in the previous image run the following command:

    # For example:
    oc logs k8s-bigip-ctlr-79b8f9cbd8-smsqs -n kube-system
    
    ../../_images/f5-container-connector-check-logs-kubectl21.png

    Attention

    You will see ERROR in this log output. These errors can be ignored. The lab will work as expected.