BIG-IP and flannel VXLAN Integration

This document provides a general overview of the BIG-IP device integration with flannel VXLAN in Kubernetes. For set-up instructions, see Add BIG-IP device to flannel VXLAN.

Overview of Cluster Networking with flannel in Kubernetes

Flannel is a layer 3 network fabric (or, in their words, “a virtual network that attaches IP addresses to containers”). In Kubernetes, flannel runs as a Pod on each Node in the Cluster. The Pod contains the flannel daemon – flanneld – that provides network information to Nodes and reads information about Nodes from the Kubernetes API server.

Flannel assigns a subnet to each Kubernetes Node. It allocates an IP address within that subnet to each Pod running on the Node. Because flanneld runs on every Node, all of the Pods across the Cluster can talk to each other directly.

Note

In older versions of Kubernetes (pre-1.6), flannel used an etcd key-value store to read and write information about Kubernetes Nodes. Though etcd is still viable in later versions, it’s not commonly used.

BIG-IP Devices and the Kubernetes Cluster Network

As discussed in BIG-IP Controller Modes, when a BIG-IP device is part of the Kubernetes Cluster Network, it can load balance directly to any Pod in the Cluster. This is the case because, via flannel and the BIG-IP Controller, the BIG-IP can find each Pod’s public-ip address. Read on for an overview of how this works.

The BIG-IP device connects to the flannel network via a VXLAN tunnel. The BIG-IP Controller populates this tunnel with the following information about the flannel network:

  • forwarding database (FDB) records that map the MAC address of each Kubernetes Node’s flannel VXLAN interface to the Node IP address;
  • static ARP entries that map the flannel VXLAN interface’s MAC address to the Pod’s flannel public-ip.

The BIG-IP Controller also assigns each Pod’s flannel public-ip address to a node on the BIG-IP.

Example:

Node1 has the NodeIP address, MAC address, and Pod public-ip address shown in the table below.

Kubernetes Node1
Node IP address 172.16.2.10
MAC address of Node’s flannel VXLAN interface 98:ba:76:dc:54:fe
Pod public-ip address assigned by flannel 10.244.1.2

The BIG-IP Controller uses this information to create an FDB record and a static ARP entry for the Node on the BIG-IP system:

FDB record
flannel_vxlan {
 records [
    98:ba:76:dc:54:fe {
      endpoint: 172.16.2.10
    }
 ]
}
static ARP entry
{
   name: k8s-10.244.1.2
   ipaddress: 10.244.1.2
   macaddress: 98:ba:76:dc:54:fe
}

Together, these records tell the BIG-IP device that a Pod on Node1 should receive traffic from the BIG-IP node with the IP address “10.244.1.2”.

Use BIG-IP SNAT Pools and SNAT automap

Note

By default, the BIG-IP Controller uses BIG-IP Automap SNAT for all of the virtual servers it creates. From k8s-bigip-ctlr v1.5.0 forward, you can designate a specific SNAT pool in the Controller Deployment instead of using SNAT automap.

In environments where the BIG-IP connects to the Cluster network, the self IP used as the BIG-IP VTEP serves as the SNAT pool for all origin addresses within the Cluster. The subnet mask you provide when you create the self IP defines the addresses available to the SNAT pool.

See BIG-IP SNATs and SNAT automap for more information.

How flannel knows about the BIG-IP device

At this point, your BIG-IP device knows how to route to the Kubernetes network, but flannel doesn’t know about the BIG-IP device. Flannel’s kube-subnet-manager uses the Kubernetes API to discover information about Kubernetes Nodes. This means that, to add the BIG-IP device to the flannel network, we need to add the BIG-IP device as a Node in Kubernetes.

When you add a new Node to Kubernetes to represent the BIG-IP device, add the flannel Annotations and podCIDR to the Node resource. Once the Node is up and running, flannel will discover its Annotations and add the BIG-IP device to the VXLAN.

With all of these pieces in place, you can successfully send traffic from (or through) a BIG-IP virtual server to a specific Kubernetes Pod!