Attach Virtual Servers to OpenShift Routes

Overview

As described in the OpenShift documentation, the IP address assigned to an OpenShift Pod is only accessible from within the cluster network. You can use the BIG-IP Controller for OpenShift as a router to expose Services to external traffic.

When you use the BIG-IP Controller as a Router, you can

Attention

All Route resources share two virtual servers:

  • “ose-vserver” for HTTP traffic, and
  • “https-ose-vserver” for HTTPS traffic.

The Controller assigns the names shown above by default. To set set custom names, define route-http-vserver and route-https-vserver in the BIG-IP Controller Deployment.

Task Summary
Step Task

Deploy the BIG-IP Controller

Create an OpenShift Route Resource
Upload the Route to the OpenShift API server
Verify creation of BIG-IP objects

Manage BIG-IP objects for Routes

Attach Routes to Existing BIG-IP Virtual Servers

If you need to use BIG-IP system functionality that isn’t natively supported by the BIG-IP Controller, you can attach a Route to an existing BIG-IP virtual server. Take the steps below before you deploy the BIG-IP Controller.

If you want the BIG-IP Controller to create a new virtual server for your Route, skip to the Basic Deployment section.

  1. Create a virtual server in a BIG-IP partition that isn’t already managed by a BIG-IP Controller instance.

  2. Customize the virtual server as needed. Be sure the settings applied don’t conflict with those you want the Controller to apply for the Route.

  3. In a TMOS shell, run the commands shown below to set the cccl-whitelist metadata field. This field tells the Controller it should merge its configuration into the existing virtual instead of overwriting it.

    • Make sure you’re in the correct partition (for example, user@(BIG-IP)(cfg-sync Standalone)(Active)(/myPartition)(tmos)).
    • Replace “myVirtual” with the name of the virtual server on your BIG-IP device.
    modify ltm virtual myVirtual metadata add { cccl-whitelist { value 1 }}
    

Deploy the BIG-IP Controller

Note

OpenShift supports two types of Deployments: Deployment Configurations and Kubernetes Deployments. See Kubernetes Deployment Support in the OpenShift documentation for more information.

Basic Deployment

Create a Kubernetes Deployment using valid YAML or JSON. Define the BIG-IP Controller Route configuration parameters as appropriate to suit your needs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: k8s-bigip-ctlr
spec:
  replicas: 1
  template:
    metadata:
      name: k8s-bigip-ctlr
      labels:
        app: k8s-bigip-ctlr
    spec:
      # Name of the Service Account bound to a Cluster Role with the required
      # permissions
      serviceAccountName: bigip-ctlr
      containers:
        - name: k8s-bigip-ctlr
          image: "f5networks/k8s-bigip-ctlr:1.4"
          env:
            - name: BIGIP_USERNAME
              valueFrom:
                secretKeyRef:
                  # Replace with the name of the Secret containing your login
                  # credentials
                  name: bigip-login
                  key: username
            - name: BIGIP_PASSWORD
              valueFrom:
                secretKeyRef:
                  # Replace with the name of the Secret containing your login
                  # credentials
                  name: bigip-login
                  key: password
          command: ["/app/bin/k8s-bigip-ctlr"]
          args: [
            # See the k8s-bigip-ctlr documentation for information about
            # all config options
            # http://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/latest
            "--bigip-username=$(BIGIP_USERNAME)",
            "--bigip-password=$(BIGIP_PASSWORD)",
            "--bigip-url=10.10.10.10",
            "--bigip-partition=openshift",
            "--pool-member-type=cluster",
            "--openshift-sdn-name=/Common/openshift_vxlan",
            "--manage-routes=true",
            "--route-vserver-addr=1.2.3.4",
            "--route-label="App1"
            ]
      imagePullSecrets:
        - name: f5-docker-images
        - name: bigip-login

f5-k8s-bigip-ctlr_openshift_routes.yaml

Warning

Use caution when setting the --route-vserver-addr and specifying a BIG-IP SNAT pool.

If you choose to set both options, make sure the IP address defined for the virtual server falls within the range of the selected SNAT pool.

Manage a Pre-Existing Virtual Server

Create a Kubernetes Deployment using valid YAML or JSON.

  • Define the BIG-IP Controller Route configuration parameters as appropriate to suit your needs.
  • Provide the name of the BIG-IP virtual server to which you want to attach the Route to the BIG-IP Controller Deployment. The config parameter to use depends on the type of virtual server (HTTP or HTTPS)
    • route-http-vserver – HTTP virtual server.
    • route-https-vserver – HTTPS virtual server.

Example k8s-bigip-ctlr args:

args: [
      "--bigip-username=$(BIGIP_USERNAME)",
      "--bigip-password=$(BIGIP_PASSWORD)",
      "--bigip-url=10.10.10.10",
      "--bigip-partition=myPartition",
      "--pool-member-type=cluster",
      "--openshift-sdn-name=/Common/openshift_vxlan",
      "--manage-routes=true",
      "--route-http-vserver=myVirtual"
      ]

Warning

When you attach an OpenShift Route to an existing BIG-IP virtual server, the BIG-IP Controller attempts to merge its settings with the existing object configurations on the BIG-IP device. If conflicts occur, the Controller will attempt to replace the existing setting on the BIG-IP system with its own configuration. If the BIG-IP Controller cannot create the requested objects, you can find the resulting error message in the BIG-IP Controller logs.

See OpenShift troubleshooting for more information about viewing the Controller logs.

Upload the Deployment to the OpenShift API Server

Use the oc create command to upload the Deployment to the OpenShift API server.

oc create -f f5-k8s-bigip-ctlr_openshift-sdn.yaml [-n kube-system]
deployment "k8s-bigip-ctlr" created

See also

See Upload the Deployment to the OpenShift API Server for additional information.

Create an OpenShift Route Resource

To use the BIG-IP device as an OpenShift Router, add the BIG-IP Controller OpenShift Route Annotations to a Route Resource. The BIG-IP Controller supports the following types of Route Resource:

Rewrite URLs for Routes

The BIG-IP Controller can rewrite URLs for Routes. See Rewrite URLs for more information.

Unsecured

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
apiVersion: v1
kind: Route
metadata:
  labels:
    name: myService
  name: myService-route-unsecured
  annotations:
    # See the k8s-bigip-ctlr documentation for information about
    # all Route Annotations
    # http://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/latest/#supported-route-annotations
    virtual-server.f5.com/balance: least-connections-node
spec:
  host: mysite.example.com
  path: "/myApp"
  port:
    targetPort: 80
  to:
    kind: Service
    name: myService

f5-openshift-unsecured-route.yaml

Edge Termination

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
apiVersion: v1
kind: Route
metadata:
  labels:
    name: myService
    # Tells the BIG-IP Controller to watch Routes with a specific label.
    # Include the label in the k8s-bigip-ctlr Deployment under "args"
    # (for example, "--route-label="App1")
    f5type: App1
  name: myService-route-edge
  annotations:
    # See the k8s-bigip-ctlr documentation for information about
    # all Route Annotations
    # http://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/latest/#supported-route-annotations
    virtual-server.f5.com/balance: least-connections-member
    virtual-server.f5.com/clientssl: /Common/client-ssl
    virtual-server.f5.com/serverssl: /Common/server-ssl
spec:
  host: mysite.example.com
  path: "/myApp"
  port:
    targetPort: 443
  tls:
    certificate: |
      -----BEGIN CERTIFICATE-----
      [...]
      -----END CERTIFICATE-----
    key: |
      -----BEGIN PRIVATE KEY-----
      [...]
      -----END PRIVATE KEY-----
    caCertificate: |
      -----BEGIN CERTIFICATE-----
      [...]
      -----END CERTIFICATE-----
    termination: edge
    insecureEdgeTerminationPolicy: Allow
  to:
    kind: Service
    name: myService

f5-openshift-edge-route.yaml

Passthrough Termination

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
apiVersion: v1
kind: Route
metadata:
  labels:
    name: myService
  name: myService-route-passthrough
  annotations:
    # See the k8s-bigip-ctlr documentation for information about
    # all Route Annotations
    # http://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/latest/#supported-route-annotations
    virtual-server.f5.com/balance: least-connections-member
    virtual-server.f5.com/clientssl: /Common/client-ssl
    virtual-server.f5.com/serverssl: /Common/server-ssl
spec:
  host: mysite.example.com
  path: "/myApp"
  port:
    targetPort: 443
  tls:
    termination: passthrough
  to:
    kind: Service
    name: myService

f5-openshift-passthrough-route.yaml

Re-encryption Termination

Important

The BIG-IP Controller does not support path-based Routes for TLS re-encryption.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
apiVersion: v1
kind: Route
metadata:
  labels:
    name: myService
  name: myService-route-reencrypt
  annotations:
    # See the k8s-bigip-ctlr documentation for information about
    # all Route Annotations
    # http://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/latest/#supported-route-annotations
    virtual-server.f5.com/balance: round-robin
    virtual-server.f5.com/clientssl: /Common/client-ssl
    virtual-server.f5.com/serverssl: /Common/server-ssl
    virtual-server.f5.com/secure-serverssl: True
spec:
  host: mysite.example.com
  path: "/myApp"
  port:
    targetPort: https
  tls:
    certificate:  |
      -----BEGIN CERTIFICATE-----
      [...]
      -----END CERTIFICATE-----
    key:  |
      -----BEGIN CERTIFICATE-----
      [...]
      -----END CERTIFICATE-----
    destinationCACertificate: |
      -----BEGIN CERTIFICATE-----
      [...]
      -----END CERTIFICATE-----
    termination: reencrypt
  to:
    kind: Service
    name: myService
    weight: 100

f5-openshift-reencrypt-route.yaml

Upload the Route to the OpenShift API server

Use the oc apply command to upload your Route resource to the OpenShift API server.

Tip

When uploading resources that don’t reside in the default or current Project, specify the correct Project using the --namespace (or -n) flag.

openshift cli
oc apply -f <filename.yaml> [--namespace=<resource-project>]

Verify creation of BIG-IP objects

You can use the BIG-IP configuration utility or a TMOS shell to verify creation/modification/deletion of BIG-IP objects.

Configuration Utility

  • Go to Local Traffic ‣ Virtual Servers.
  • Select the correct partition from the Partition drop-down menu.

TMOS Management Console

admin@(bigip)(cfg-sync Standalone)(Active)(/Common) cd my-partition
admin@(bigip)(cfg-sync Standalone)(Active)(/my-partition) tmsh
admin@(bigip)(cfg-sync Standalone)(Active)(/my-partition)(tmos)$ show ltm virtual
------------------------------------------------------------------
Ltm::Virtual Server: default_myApp.vs_173.16.2.2_80
------------------------------------------------------------------
Status
  Availability     : available
  State            : enabled
  Reason           : The virtual server is available
  CMP              : enabled
  CMP Mode         : all-cpus
  Destination      : 173.16.2.2:80
...
Ltm::Virtual Server: default_myApp.vs_173.16.2.2_443
------------------------------------------------------------------
Status
  Availability     : available
  State            : enabled
  Reason           : The virtual server is available
  CMP              : enabled
  CMP Mode         : all-cpus
  Destination      : 173.16.2.2:443
...

Manage BIG-IP objects for Routes

Use the BIG-IP Controller Route annotations to attach various types of BIG-IP objects to the virtual servers corresponding to OpenShift Routes.

Health monitors

You can use the k8s-bigip-ctlr Route annotations to update/add health monitors to OpenShift Routes.

  1. Define the virtual-server.f5.com/health annotation JSON blob.

  2. Add the health monitor annotation to the Route Resource.

    Annotate an OpenShift Route using the cli
    oc annotate route myRoute virtual-server.f5.com/health='[{"path": "svc1.example.com/app1", "send": "HTTP GET /health/svc1", "interval": 5, "timeout": 10}]'
    

In the Route resource YAML file, the health monitor should look like this:

Example Health Monitor in a Route resource
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
apiVersion: v1
kind: Route
metadata:
  name: route-unsecured
  annotations:
    # See the k8s-bigip-ctlr documentation for information about
    # all Route Annotations
    # http://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/latest/#supported-route-annotations
    virtual-server.f5.com/balance: fastest-node
    virtual-server.f5.com/clientssl: /Common/client-ssl
    virtual-server.f5.com/serverssl: /Common/server-ssl
    virtual-server.f5.com/health: |
      [
        {
          "path":     "mysite.example.com/app1",
          "send":     "HTTP GET /health/app1",
          "interval": 5,
          "timeout":  10
        }
      ]
spec:
  host: mysite.example.com
  path: "/app1"
  to:
    kind: Service
    name: myService1

SSL Profiles

By default, the BIG-IP Controller creates custom BIG-IP SSL Profiles using the certificates and keys defined in the Route resource. You can also use an existing BIG-IP SSL profile to secure traffic for a Route.

  • For a Client SSL profile, annotate the Route resource as shown below:

    oc annotate route <route_name> virtual-server.f5.com/clientssl=</BIG-IP-partition/SSL-profile-name>
    
  • For a Server SSL profile, annotate the Route resource as shown below:

    oc annotate route <route_name> virtual-server.f5.com/serverssl=</BIG-IP-partition/SSL-profile-name>
    

Note

  • Each SSL profile applies to one Route.
  • The BIG-IP Controller creates one client ssl and one server ssl profile for the HTTPS virtual server. These profiles – “default-client-ssl” and “default-server-ssl” – are the default profiles used for SNI.

Delete a Route’s virtual server

If you want to remove the virtual server associated with a Route from the BIG-IP system, but keep the Route:

  1. Remove the BIG-IP Controller Annotations from the Route definition.

  2. Update the OpenShift API server.

    Tip

    When uploading resources that don’t reside in the default or current Project, specify the correct Project using the --namespace (or -n) flag.

    openshift cli
    oc apply -f <filename.yaml> [--namespace=<resource-project>]
    

See also

See Manage Your BIG-IP Virtual Servers for more information.