F5 BIG-IP Controller for Kubernetes¶

The F5 BIG-IP Controller for Kubernetes lets you manage your F5 BIG-IP device from Kubernetes or OpenShift using either environment’s native CLI/API.

Features¶

Dynamically creates, manages, and destroys BIG-IP objects.
Forwards traffic from the BIG-IP device to Kubernetes clusters via NodePort or ClusterIP.
Support for F5 AS3 Extension declarations.
Support for F5 iApps.
Handles F5-specific VirtualServer objects created in Kubernetes.
Handles standard Kubernetes Ingress objects using F5-specific extensions.
Handles OpenShift Route objects using F5-specific extensions.

Guides¶

See the F5 BIG-IP Controller for Kubernetes user documentation.

Installation¶

Kubernetes Installation
F5 AS3 Installation
OpenShift Installation
If you use helm you can install the k8s-bigip-ctlr using the f5-bigip-ctlr chart.

Overview¶

The F5 BIG-IP Controller for Kubernetes is a Docker container that runs in a Kubernetes Pod. It uses F5 Resource s to determine:

what objects to configure on your BIG-IP system, and
to which Kubernetes Service those objects belong.

The k8s-bigip-ctlr watches the Kubernetes API for the creation, modification, or deletion of Kubernetes objects. For some Kubernetes objects, the Controller responds by creating, modifying, or deleting objects in the BIG-IP system. The k8s-bigip-ctlr handles the following Kubernetes objects:

F5 Resource ConfigMap – creates Service-specific frontend virtual servers and/or pools on the BIG-IP system.
Kubernetes Ingress – creates a single front-end virtual server on the BIG-IP system that uses L7 policies to route to individual Services.
OpenShift Route – enables route-handling (specific to OpenShift).

One Controller can handle a mix of these objects simultaneously. See below for specifics regarding the handling of these objects.

For example, when run in NodePort mode, the k8s-bigip-ctlr does the following:

Discovers a new F5 virtualServer resource.
Creates a new virtual server object in the specified partition on the BIG-IP system. [1]
Creates a pool member on the virtual server for each node in the cluster. [2]
Monitors F5 resources, and linked Kubernetes resources, for changes.
Reconfigures the BIG-IP system when it discovers changes.

The BIG-IP system handles traffic for the Service at the specified virtual address and load balances to all nodes in the cluster. Within the cluster, the allocated NodePort load balances traffic to all pods.

Danger

F5 does not recommend making configuration changes to objects in any partition managed by the k8s-bigip-ctlr via any other means (for example, the configuration utility, TMOS, or by syncing configuration with another device or service group). Doing so may result in disruption of service or unexpected behavior.

The Controller allows one exception to this recommendation. When using named virtual servers for Openshift routes, you can set the Controller to merge its desired settings with a pre-existing virtual server(s). See Manage OpenShift Routes for more information.

Controller Configuration Parameters¶

All of the configuration parameters below are global.

General¶

Parameter	Type	Required	Default	Description	Allowed Values
http-listen-address	string	Optional	“0.0.0.0:8080”	Address at which to serve HTTP-based information (for example, `/metrics`, `health`) to Prometheus Beta feature
log-level	string	Optional	INFO	Log level	INFO, DEBUG, CRITICAL, WARNING, ERROR
node-poll-interval	integer	Optional	30	In seconds, the interval at which the `k8s-bigip-ctlr` polls the cluster to find all node members.
python-basedir	string	Optional	/app/python	Path to the python utilities directory
schema-db-base-dir	string	Optional	file:///app/vendor/src/f5/schemas	Path to the directory containing the F5 schema db
verify-interval	integer	n/a	30	In seconds, the interval at which the `k8s-bigip-ctlr` verifies that the BIG-IP configuration matches the state of the orchestration system. This value is not currently configurable
vs-snat-pool-name	string	Optional	n/a	Name of the SNAT pool that all virtual servers will reference. If it is not set, virtual servers use automap SNAT.

Note

The python-basedir setting lets you specify the path to an alternate python agent that can bridge between the k8s-bigip-ctlr and F5 CCCL.
The time it takes for the k8s-bigip-ctlr to reapply the system configurations to the BIG-IP device is normally low (a few ms) and won’t cause service disruption.
Use vs-snat-pool-name if you want virtual servers to reference a SNAT pool that already exists in the /Common partition on the BIG-IP device. See Overview of SNAT features on AskF5 for more information.

BIG-IP system¶

Parameter	Type	Required	Default	Description
bigip-partition	string	Required	n/a	The BIG-IP partition in which to configure objects.
bigip-password	string	Required	n/a	BIG-IP iControl REST password You can secure your BIG-IP credentials using a Kubernetes Secret.
bigip-url	string	Required	n/a	BIG-IP admin IP address
bigip-username	string	Required	n/a	BIG-IP iControl REST username The BIG-IP user account must have the appropriate role defined: For `nodeport` type pool members, the role must be `Administrator`. For `cluster` type pool members, the role must be `Administrator`.
credentials-directory	string	Optional	n/a	Directory that contains the BIG-IP username, password, or url files

Important

The credentials-directory option is an alternative to using the bigip-username, bigip-password, or bigip-url arguments.

When you use this argument, the controller looks for three files in the specified directory:

“username”, “password”, and “url”

If any of these files do not exist, the controller falls back to using the CLI arguments as parameters.

Each file should contain only the username, password, and url, respectively. You can create and mount the files as Kubernetes Secrets.

It is important to not project the Secret keys to specific paths, as the controller looks for the “username”, “password”, and “url” files directly within the credentials directory.

See example-bigip-credentials-directory.yaml for a deployment example.

VXLAN¶

Parameter	Type	Required	Default	Description	Allowed Values
openshift-sdn-name	string	Optional	n/a	Name of the VXLAN tunnel on the BIG-IP system that corresponds to an Openshift SDN HostSubnet. Only applicable in OpenShift.
flannel-name	string	Optional	n/a	Name of the VXLAN tunnel on the BIG-IP system that corresponds to a Flannel subnet.

Parameter

Type

Required

Default

Description

Allowed Values

openshift-sdn-name

string

Optional

n/a

Name of the VXLAN tunnel on the BIG-IP system that corresponds to an Openshift SDN HostSubnet.

Only applicable in OpenShift.

flannel-name

string

Optional

n/a

Name of the VXLAN tunnel on the BIG-IP system that corresponds to a Flannel subnet.

Kubernetes¶

Parameter	Type	Required	Default	Description	Allowed Values
default-ingress-ip	string	Optional	n/a	The controller configures a virtual server at this IP address for all Ingresses with the annotation: `virtual-server.f5.com/ip: 'controller-default'`
kubeconfig	string	Optional	./config	Path to the kubeconfig file
manage-configmaps	boolean	Optional	true	Tells the controller whether or not to watch Kubernetes ConfigMaps and apply their configuration. If false, the controller will ignore ConfigMap events.	true, false
namespace	string	Optional	All	Kubernetes namespace(s) to watch may be a comma-separated list watches all namespaces by default
namespace-label	string	Optional	n/a	Tells the `k8s-bigip-ctlr` to watch any namespace with this label
node-label-selector	string	Optional	n/a	Tells the `k8s-bigip-ctlr` to watch only nodes with this label
pool-member-type	string	Optional	nodeport	The type of BIG-IP pool members you want to create. Use `cluster` to create pool members for each of the endpoints for the Service (the pod’s InternalIP) Use `nodeport` to create pool members for each schedulable node using the Service’s NodePort.	cluster, nodeport
resolve-ingress-names	string	Optional	n/a	Tells the controller to resolve the first Host in an Ingress resource to an IP address. This IP address will be used as the virtual server address for the Ingress resource. A value of “LOOKUP” will use local DNS to resolve the Host. Any other value is a custom DNS server and the controller sends resolution queries through that server instead. Specifying the flag with no argument will default to LOOKUP.
running-in-cluster	boolean	Optional	true	Indicates whether or not a kubernetes cluster started `k8s-bigip-ctlr`	true, false
use-node-internal	boolean	Optional	true	filter Kubernetes InternalIP addresses for pool members	true, false
use-secrets	boolean	Optional	true	Tells the controller whether or not to load SSL profiles from Kubernetes Secrets for Ingresses and ConfigMaps. If false, the controller will only use profiles from the BIG-IP system.	true, false

Note

Use the node-label-selector parameter if you only want the controller to manage specific nodes from the cluster. For example, the BIG-IP device may not be able to reach certain nodes, or the BIG-IP device already manages certain nodes. Therefore, the controller should only watch the nodes that match the environmental constraints (by using a label).

OpenShift Routes¶

The following configuration parameters only apply to OpenShift.

Parameter	Type	Required	Default	Description	Allowed Values
custom-client-ssl	string	Optional	n/a	Specifies the name of a custom client SSL profile attached to the route HTTPS virtual server and used as default for SNI. This profile must have the Default for SNI field enabled.
custom-server-ssl	string	Optional	n/a	Specifies the name of a custom server SSL profile attached to the route HTTPS virtual server and used as default for SNI. This profile must have the Default for SNI field enabled.
manage-routes	boolean	Optional	false	Indicates if `k8s-bigip-ctlr` should handle OpenShift Route objects.	true, false
route-http-vserver	string	Optional	ose-vserver	The name of the http virtual server for OpenShift Routes.
route-https-vserver	string	Optional	https-ose-vserver	The name of the https virtual server for OpenShift Routes.
route-label	string	Optional	n/a	Tells the `k8s-bigip-ctlr` to only watch for OpenShift Route objects with the `f5type` label set to this value.
route-vserver-addr	string	Optional	n/a	Bind address for virtual server for OpenShift Route objects.

Note

If the custom-client-ssl or custom-server-ssl parameters are not provided, then the controller creates default clientssl and serverssl profiles for the OpenShift Route HTTPS virtual server. The controller sets these profiles as Default for SNI.

F5 Resource ConfigMap Properties¶

F5 Resource ConfigMap objects tell the k8s-bigip-ctlr how to configure the BIG-IP system. See the Integration Overview for more information about F5 resources.

Note

The Controller uses the following naming structure when creating BIG-IP objects:

<service-namespace>_<configmap-name>

For a Service named “myService” running in the “default” namespace, the Controller would create a BIG-IP pool with the following name:

default_myService

Property	Description	Allowed Values
f5type	Tells `k8s-bigip-ctlr` about resources it should watch
schema	Verifies the `data` blob	See the F5 schema versions table
data	Defines the F5 resource
frontend	Defines BIG-IP objects	See Frontend
backend	Identifies the Kubernetes Service acting as the server pool	See Backend

F5 schema¶

The F5 schema allows the k8s-bigip-ctlr to communicate with BIG-IP systems.

Note

While all versions of the BIG-IP Controller and F5 schema are backwards-compatible, using an older schema may limit Controller functionality. Be sure to use the schema version that corresponds with your Controller version to ensure access to the full feature set.

See the F5 schema versions table for schema and Controller version compatibility.

Frontend¶

Virtual Servers¶

Use the options shown in the table below in the frontend section of an F5 resource ConfigMap to define BIG-IP virtual server(s), pool(s), and pool member(s).

Property	Type	Required	Default	Description	Allowed Values
partition	string	Required		The BIG-IP partition you want to manage
virtualAddress	JSON object	Optional		Assigns a BIG-IP self IP to the virtual server
bindAddr [4]	string	Required		Virtual IP address
port	integer	Required		Port number
mode	string	Optional	tcp	Sets the proxy mode	http, tcp, udp

balance	string	Optional	round-robin	Sets the load balancing mode	Any supported load balancing algorithm [3]
sslProfile [5]	JSON object	Optional		BIG-IP SSL profile to apply to the virtual server.
f5ProfileName	string	Optional		Name of the BIG-IP SSL profile you want to use. Uses format `partition_name/cert_name` Example: `Common/testcert`
f5ProfileNames	array of strings	Optional		Array of BIG-IP SSL profile names. Each SSL profile name uses the format `partition_name/cert_name`. Example: [ 'Common/testcert1', 'Common/testcert2' ]

Note

If you include virtualAddress in your Resource definition, you can specify the bindAddr and port you want the virtual server to use. Omit the virtualAddress section if you want to create pools without virtual servers.

If you’re creating pools without virtual servers, you should already have a BIG-IP virtual server that handles client connections configured with an iRule or local traffic policy that can forward requests to the correct pool for the Service.

You can also assign IP addresses to BIG-IP virtual servers using IPAM.

iApps¶

Use the options shown in the table below in the frontend section of an F5 resource ConfigMap to deploy an iApp on the BIG-IP system.

Tip

The iappOptions parameter should contain the information that you would provide if you deployed the iApp using the BIG-IP configuration utility.

Property	Type	Required	Description	Allowed Values
partition	string	Required	The BIG-IP partition you want the `k8s-bigip-ctlr` to manage.
iapp	string	Required	BIG-IP iApp template to use to create the application Service.	Any iApp template that already exists on the BIG-IP system.
iappPoolMemberTable	JSON object	Required	Define the name and layout of the pool member table in the iApp. See iApp Pool Member Table.
iappTables	JSON object array	Optional	Define iApp tables to apply to the Application Service Example: "iappTables": { "monitor__Monitors": {"columns": ["Index", "Name", "Type", "Options"], "rows": [[0, "mon1", "tcp", "" ], [1, "mon2", "http", ""]]}}"
iappOptions	key-value object	Required	Define the App configurations	Varies
iappVariables	key-value object	Required	Define the iApp variables needed for Service creation.

iApp Pool Member Table¶

You can use the iappPoolMemberTable option to describe the layout of the pool-member table that the Controller should configure. It is a JSON object with these properties:

name (required): A string that specifies the name of the table that contains the pool members.
columns (required): An array that specifies the columns that the Controller will configure in the pool-member table, in order.

Each entry in columns is an object that has a name property and either a kind or value property:

name (required): A string that specifies the column name.
kind: A string that tells the Controller what property from the node to substitute. The Controller supports "IPAddress" and "Port".
value: A string that specifies a value. The Controller will not perform any substitution, it uses the value as specified.

For example: If you configure an application with two pods at 1.2.3.4:20123 and 1.2.3.5:20321 and you specify the following JSON:

"iappPoolMemberTable" = {
  "name": "pool__members",
  "columns": [
    {"name": "Port", "kind": "Port"},
    {"name": "IPAddress", "kind": "IPAddress"},
    {"name": "ConnectionLimit", "value": "0"}
  ]
}

      
    

the k8s-bigip-ctlr creates the table below on the BIG-IP system.

{
  "name": "pool__members",
  "columnNames": [
    "Port",
    "IPAddress",
    "ConnectionLimit",
  ],
  "rows": [
    {
      "row": [
        "20123",
        "1.2.3.4",
        "0",
      ]
    },
    {
      "row": [
        "20321",
        "1.2.3.5",
        "0",
      ]
    },
  ]
}

      
    

You will need to adjust this for the particular iApp template that you are using. One way to discover the format is to configure an iApp manually from a template, then check its configuration using tmsh list sys app Service <appname>.

Backend¶

The backend section tells the k8s-bigip-ctlr about the Service you want to manage.

Property		Type	Required	Default	Description	Allowed Values
ServiceName		string	Required	none	The Kubernetes Service representing the server pool.
ServicePort		integer	Required	none	Kubernetes Service port number
healthMonitors		JSON object array	Optional	none	Array of TCP, UDP or HTTP Health Monitors.
	protocol	string	Required	N/A	Protocol used to confim health.	http, tcp, udp
	interval	integer	Optional	5	Seconds between health queries.	1 to 86,400.
	timeout	integer	Optional	16	Seconds before query fails.	Integer from 1 to 86,400.
	send	string	Optional	“GET /rn”	HTTP request string to send.	String values.
	recv	string	Optional	none	String or RegEx pattern to match in first 5,120 bytes of backend response.	String or valid RegEx.

F5 AS3 Integration Resources¶

To expose services to external traffic using As3 Extension declarations, refer to Container Ingress Services and AS3 Extension integration.

Kubernetes Ingress Resources¶

You can use the k8s-bigip-ctlr to Expose Services to External Traffic using Ingresses.

Supported Ingress Annotations¶

Annotation		Type	Required	Description	Default	Allowed Values
virtual-server.f5.com/ip		string	Required	The IP address you want to assign to the virtual server. Set to “controller-default” if you want to use the `default-ingress-ip` specified in the Configuration Parameters above.	N/A	numerical IP address “controller-default”
virtual-server.f5.com/partition		string	Optional	The BIG-IP partition in which the Controller should create/update/delete objects for this Ingress.	N/A
kubernetes.io/ingress.class		string	Optional	Tells the Controller it should only manage Ingress resources in the `f5` class. If defined, the value must be `f5`.	f5	“f5”
virtual-server.f5.com/balance		string	Optional	Sets the load balancing mode.	round-robin	Any supported load balancing algorithm [3]
virtual-server.f5.com/http-port		integer	Optional	Specifies the HTTP port.	80
virtual-server.f5.com/https-port		integer	Optional	Specifies the HTTPS port.	443
virtual-server.f5.com/health		JSON object	Optional	Defines a health monitor for the Ingress resource.	N/A
	path	string	Required [6]	The path for the Service specified in the Ingress resource.	N/A
	interval	integer	Required [6]	The interval at which to check the health of the virtual server.	N/A
	timeout	integer	Required [6]	Number of seconds before the check times out.	N/A
	send	string	Required [6]	The send string to set in the health monitor. [7]	N/A
	recv	string	Optional	String or RegEx pattern to match in first 5,120 bytes of backend response.	N/A
	type	string	Optional	Health monitor type. Typically http or https.	http
ingress.kubernetes.io/allow-http		boolean	Optional	Tells the Controller to allow HTTP traffic for HTTPS Ingress resources.	false	“true”, “false”
ingress.kubernetes.io/ssl-redirect		boolean	Optional	Tells the Controller to redirect HTTP traffic to the HTTPS port for HTTPS Ingress resources (see TLS Ingress resources, below).	true	“true”, “false”
virtual-server.f5.com/serverssl		string	Optional	The name of a pre-configured server ssl profile on the BIG-IP system.	N/A
virtual-server.f5.com/rewrite-app-root		string	Optional	Root path redirection for the application.	N/A
virtual-server.f5.com/rewrite-target-url		string	Optional	URL host, path, or host and path to be rewritten.	N/A
virtual-server.f5.com/whitelist-source-range		string	Optional	Comma separated list of CIDR addresses to allow inbound to Ingress services.	N/A	Comma separated, CIDR formatted, IP addresses. ex. 1.2.3.4/32,2.2.2.0/24

Ingress Health Monitors¶

To configure health monitors on your Ingress resource, define the virtual-server.f5.com/health annotation with a JSON object. Provide an array for each path specified in the Ingress resource.

For example

{
"path": "ServiceName/path",
"send": "<send string to set in the health monitor>",
"interval": <health check interval>,
"timeout": <number of seconds before the check has timed out>
}

      
    

TLS Ingress Resources¶

If the Ingress resource contains a tls section, the allow-http and ssl-redirect annotations provide a method of controlling HTTP traffic. In this case, the Controller uses the value set in the allow-http annotation to enable or disable HTTP traffic. Use the ssl-redirect annotation to redirect all HTTP traffic to the HTTPS Virtual Server.

You can specify one (1) or more SSL profiles in the Ingress resource.

Profiles must already exist either in Kubernetes/OpenShift –OR– on the BIG-IP system;
- If the controller looks for a Kubernetes Secret with the name(s) provided first;
- if it doesn’t find a matching Secret, the Controller assumes that the name(s) matches a profile that already exists on the BIG-IP system.
- If naming an existing BIG-IP profile, provide the full path to the profile (for example, /Common/clientssl).

OpenShift Route Resources¶

Note

You can use OpenShift Route resources in an existing deployment once you replace the OpenShift F5 Router with the BIG-IP Controller.

Supported Route Configurations¶

Important

The k8s-bigip-ctlr supports a single path-based route for TLS re-encryption. Multiple path-based routes are not currently supported.

Type	Client Connection Encrypted	Server Connection Encrypted	Path Support	SSL Termination on BIG-IP	Description
Unsecured	No	No	Yes	No	The BIG-IP system forwards unsecured traffic from the client to the endpoint.
Edge Terminated	Yes	No	Yes	Yes	The Controller maintains a new client SSL profile on the BIG-IP system based on the client certificate and key from the Route resource. Set insecureEdgeTerminationPolicy in the Route resource to Allow to enable support for insecure client connections. Set insecureEdgeTerminationPolicy in the Route resource to Redirect to redirect HTTP client connections to the HTTPS endpoint.
Passthrough Terminated	Yes	Yes	No	No	The BIG-IP system uses an iRule to select the destination pool based on SNI and forward the re-encrypted traffic.
Re-encrypt Terminated	Yes	Yes	Yes	Yes	The Controller maintains a new BIG-IP client SSL profile based on the client certificate and key from the Route resource. The Controller maintains a new BIG-IP server SSL profile based on the server CA certificate from the Route resource for re-encrypting the traffic. The BIG-IP system uses an iRule to select the destination pool based on SNI and forward the re-encrypted traffic.

Supported Route Annotations¶

Annotation		Type	Required	Description	Default	Allowed Values
virtual-server.f5.com/balance		string	Optional	Sets the load balancing mode.	round-robin	Any supported load balancing algorithm [3]
virtual-server.f5.com/clientssl		string	Optional	The name of a pre-configured client ssl profile on the BIG-IP system. The controller uses this profile instead of the certificate and key within the Route’s configuration.	N/A
virtual-server.f5.com/serverssl		string	Optional	The name of a pre-configured server ssl profile on the BIG-IP system. The controller uses this profile instead of the certificate within the Route’s configuration.	N/A
virtual-server.f5.com/health		JSON object	Optional	Defines a health monitor for the Route resource.	N/A
	path	string	Required [6]	The path for the Service specified in the Route resource.	N/A
	interval	integer	Required [6]	The interval at which to check the health of the virtual server.	N/A
	timeout	integer	Required [6]	Number of seconds before the check times out.	N/A
	send	string	Required [6]	The send string to set in the health monitor. [7]	N/A
	recv	string	Optional	String or RegEx pattern to match in first 5,120 bytes of backend response.	N/A
virtual-server.f5.com/secure-serverssl		boolean	Optional	Specify to validate the server-side SSL certificate of re-encrypt terminated routes.	false	“true”, “false”
virtual-server.f5.com/rewrite-app-root		string	Optional	Root path redirection for the application.	N/A
virtual-server.f5.com/rewrite-target-url		string	Optional	URL host, path, or host and path to be rewritten.	N/A
virtual-server.f5.com/whitelist-source-range		string	Optional	Comma separated list of CIDR addresses to allow inbound to Route services.	N/A	Comma separated, CIDR formatted, IP addresses. ex. 1.2.3.4/32,2.2.2.0/24
virtual-server.f5.com/waf		string	Optional	The name of a pre-configured WAF Policy on the BIG-IP system.	N/A

Important

For edge (client) termination, a Route must include either the certificate/key literal information in the Route Spec, or the clientssl annotation. For re-encrypt (server) termination, a Route must include either the destinationCaCertificate literal information in the Route Spec, or the serverssl annotation, in addition to the edge rules listed previously. If you want to use the configuration parameters default-clientssl or default-serverssl profiles for a Route, then specify those profile names in the Route annotations in addition to the controller configuration.

Please see the example configuration files for more details.

Example Configuration Files¶

OpenShift¶

Footnotes

[1]	The `k8s-bigip-ctlr` creates and manages objects in the BIG-IP partition defined in the F5 resource ConfigMap. It cannot manage objects in the `/Common` partition.

[2]	The `k8s-bigip-ctlr` forwards traffic to the NodePort assigned to the Service by Kubernetes. See the Kubernetes Service documentation for more information.

[3] (1, 2, 3) The k8s-bigip-ctlr supports BIG-IP load balancing algorithms that do not require additional configuration parameters. You can view the full list of supported algorithms in the f5-cccl schema. See the BIG-IP Local Traffic Management Basics user guide for information about each load balancing mode.

[4]	The Controller supports BIG-IP route domain specific addresses.

[5]	If you want to configure multiple SSL profiles, use `f5ProfileNames` instead of `f5ProfileName`. The two parameters are mutually exclusive.

[6]	(1, 2, 3, 4, 5, 6, 7, 8) Required if defining the `virtual-server.f5.com/health` Ingress/Route annotation.

[7]	(1, 2) See the HTTP monitor settings section of the BIG-IP LTM Monitors Reference Guide for more information about defining send strings.