F5 BIG-IP Controller for Cloud Foundry

Slack

The F5 BIG-IP Controller for Cloud Foundry (cf-bigip-ctlr) lets you manage your F5 BIG-IP device from Cloud Foundry using the environment’s native CLI/API.

Release Notes

Attributions

Attributions.md

Features

  • Dynamically creates, manages, and destroys BIG-IP objects.
  • Forwards traffic from the BIG-IP device to Cloud Foundry clouds via Diego cell virtual machine addressing.
  • Supports Cloud Foundry HTTP and TCP routing.
  • Supports use of BIG-IP profiles and policies with Cloud Foundry routes.

Guides

See the F5 BIG-IP Controller for Cloud Foundry user documentation.

Overview

The cf-bigip-ctlr is a Docker container that runs in a Cloud Foundry cell. It emulates the behavior of the Cloud Foundry Gorouter as follows:

  • subscribes to the Cloud Foundry NATS message bus and routing API;
  • gathers application route information (HTTP and TCP); and
  • configures routing policy rules on the BIG-IP system.

The cf-bigip-ctlr receives Route updates and transforms them into BIG-IP policy rules when the following events occur:

  • push, delete, and scale applications (with mapped routes to the application);
  • map and unmap routes.

For example:

  1. Developer pushes “myApp” to Cloud Foundry (without the Cloud Foundry --no-route option).
  2. cf-bigip-ctlr discovers new route information for myApp.
  3. Controller creates a pool and pool member(s) for each myApp instance.
  4. Controller updates BIG-IP routing policy directing App requests to the myApp pool.
  5. Controller monitors Cloud Foundry routing table and, when it discovers changes, reconfigures the BIG-IP device.

The BIG-IP device handles traffic for every Cloud Foundry-mapped Route and load balances traffic to each Application instance. When run in the “global” default mode, the cf-bigip-ctlr can create a total of two BIG-IP virtual servers for HTTP Routes: one for HTTP and one for HTTPS. When run in broker_mode, the cf-bigip-ctlr can create per-route virtual servers.

The Controller creates a virtual server for each TCP route.

All HTTP traffic goes through either the port 80 or port 443 virtual server. The Controller routes TCP traffic through dedicated virtual servers configured to listen on non-HTTP ports.

You can attach the BIG-IP objects listed below to the HTTP virtual servers the cf-bigip-ctlr creates for Cloud Foundry. To do so, create the desired objects on the BIG-IP manually before adding them to the Controller configuration in Cloud Foundry.

  • policies
  • profiles
  • SSL profiles
  • health monitors

Danger

The cf-bigip-ctlr monitors the BIG-IP partition it manages for configuration changes. If it discovers changes, the Controller reapplies its own configuration to the BIG-IP system.

F5 does not recommend making configuration changes to objects in any partition managed by the cf-bigip-ctlr via any other means (for example, the configuration utility, TMOS, or by syncing configuration with another device or service group). Doing so may result in disruption of service or unexpected behavior.

Configuration Parameters

The configuration parameters below customize the cf-bigip-ctlr behavior. Define the parameters in the env section of the Controller application manifest using the environment variable BIGIP_CTLR_CFG.

Tip

It’s possible to confuse the profiles configuration parameter with the ssl_profiles parameter.

  • profiles tells the cf-bigip-ctlr what BIG-IP profiles you want to attach to the virtual server(s) (for example, TCP acceleration or the X-Forwarded-For header).
  • ssl_profiles tells the cf-bigip-ctlr that it should create an HTTPS virtual server that uses the specified BIG-IP SSL profiles.

BIG-IP Controller for Cloud Foundry configuration parameters
Parameter Type Required Default Description Allowed Values

bigip

object Required n/a A YAML blob defining BIG-IP parameters.  
  url string Required n/a BIG-IP admin IP address  
  user string Required n/a BIG-IP iControl REST username [1]  
  pass string Required n/a BIG-IP iControl REST password  
  partition array Required n/a The BIG-IP partition in which to configure objects.  
  balance string Optional round-robin Set the load balancing mode Any supported load balancing algorithm [2]
  verify_interval integer Optional 30 In seconds; interval at which to verify the BIG-IP configuration.  
  external_addr [3] string Required n/a Virtual address on the BIG-IP to use for cloud ingress.  
  tier2_ip_range string Optional 172.0.0.0/24 IP range to assign to the tier2 vips (used in Service Broker mode only) Must use CIDR notation
  ssl_profiles array Optional n/a List of BIG-IP SSL policies to attach to the HTTPS routing virtual server. [4]  
  policies array Optional n/a Additional pre-configured BIG-IP policies to attach to routing virtual servers  
  profiles array Optional n/a Additional pre-configured BIG-IP profiles to attach to routing virtual servers  
  health_monitors array Optional n/a Health monitors attached to each configured routing pool  
status

object Optional n/a Basic authorization credentials; used to access debug information and the Service Broker API  
  user string Optional n/a Provide a username the Cloud Controller can use to access the cf-bigip-ctlr API  
  pass string Optional n/a Provide a secure password  

nats

array Required n/a NATS message bus  
  host string Required n/a NATS host  
  port integer Required n/a NATS port  
  user string Required n/a NATS username  
  pass string Required n/a NATS password  

logging

object Optional n/a Logging configuration  
  file string Optional n/a Logging file name  
  syslog string Optional n/a Syslog ID  
  level string Optional debug Logging level  
  loggregator_enabled boolean Optional false Is loggregator facility enabled  
  metron_address string Optional localhost:3457 Metron address  

oauth

object Optional n/a UAA token server configuration  
  token_endpoint string Optional n/a UAA token server  
  client_name string Optional n/a UAA username  
  client_secret string Optional n/a UAA password  
  port string Optional n/a UAA listen port  
  skip_ssl_validation boolean Optional false Should skip SSL verification  
  ca_certs string Optional n/a CA cert bundle  

routing_api

object Optional n/a Routing API configuratoin  
  uri string Optional n/a Routing API endpoint  
  port integer Optional n/a Routing API listen port  
  auth_disabled boolean Optional false Routing API authorization status  
go_max_procs integer Optional -1 Golang GOMAXPROCS limits  
prune_stale_droplets_interval integer Optional 30 In seconds, interval to check and prune stale routes  
droplet_stale_threshold integer Optional 120 In seconds, threshold to consider route stale  
suspend_prune_if_nats_unavailable boolean Optional false If NATS becomes unavailable should pruning suspend  
route_mode string Optional http Route type you want to watch; must be a single value http, tcp, all
session_persistence boolean Optional true Enable JSESSIONID cookie session persistence on the BIG-IP device true, false
broker_mode boolean Optional false Run the controller as a Service Broker true, false
start_response_delay_interval integer Optional 5 In seconds, wait time to achieve steady state from routing message bus  
token_fetcher_max_retries integer Optional 3 Number of retries to fetch auth token  
token_fetcher_retry_interval integer Optional 5 In seconds, time to wait between token fetch retries  
token_fetcher_expiration_buffer_time integer Optional 30 In seconds, time to re-fetch auth token  
tcp_router_group string Optional default-tcp Name of TCP router group  

JSESSIONID Session Persistence

The cf-bigip-ctlr enables session persistence on the BIG-IP device using cookies by default. To turn it off, set the session_persistence configuration parameter to “false”, as shown below.

session_persistence: false

To configure session persistence for the cf-bigip-ctlr, set cookies in your application’s HTTP response headers as follows:

Set-Cookie: JSESSIONID=<value>; Max-Age=<age>

Note

  • The cookie name of JSESSIONID is case insensitive; jsessionid and JSessionID are also valid.
  • The cookie value can be any distinct value (in other words, be sure it’s not the same as other cookies set by other responses).
  • Setting the cookie Max-Age to the following values results in the behavior noted in the table below.
Cookie Max-Age values
Age Behavior
< 0 Session persists until client connection closes or one hour elapses.
> 0 Session persists until the Max-Age of the cookie.
= 0 Delete session persistence record if one exists on the BIG-IP system.

Supported Route Types

The cf-bigip-ctlr accepts three (3) values for the route_type configuration parameter. Each of the supported route_mode options have different configuration requirements:

When managing HTTP routes, the cf-bigip-ctlr will create an HTTP virtual server (virtual address port 80) for routing into Cloud Foundry. If you define an SSL profile in the configuration (the ssl_profiles parameter), the Controller creates an additional HTTPS virtual server (virtual address port 443). You can attach multiple certificate/key pairs to the HTTPS virtual server.

The BIG-IP device uses TLS Server Name Indication (SNI) to choose the correct certificate to present to the client; SNI allows the Cloud Foundry instance to support multiple hostnames (foo.mycf.com and bar.mycf.com). Some of these cert/key pairs can be wildcard (*.mycf.com).

Configure per-Route Virtual Servers

You can use the cf-bigip-ctlr as a Cloud Foundry Service Broker to apply per-Route configurations to your BIG-IP device(s). See the user documentation for more information.

Define the configuration parameters below in the SERVICE_BROKER_CONFIG section of your Application Manifest. See below for configuration examples.

Parameter Type Required Description Allowed Values

plans

array Required A YAML array defining service broker plans.  
name
string Required The name of the plan.  
  description string Required A short description of the plan.  
  virtualServer object Optional A YAML blob defining a virtual server configuration.  
    policies array Optional An array of strings of BIG-IP device policy names.  
    profiles array Optional An array of strings of BIG-IP device profile names.  
    sslProfiles array Optional An array of strings of BIG-IP device server side SSL profile names.  
  pool object Optional A YAML blob defining a pool configuration.  
    balance string Optional The load balancing mode of the pool. Any BIG-IP-supported mode [2]
    healthMonitors array Optional An array of health monitor configuration objects See table and examples below

Per-Route Health Monitors

You can use an existing BIG-IP health monitor, define a new health monitor, or both. Use parameters shown in the table below to define a custom health monitor. See the configuration examples below for usage examples.

Parameter Type Required Default Description Allowed Values
name string Required   Name of the custom health monitor.  
type string Required   Type of the custom health monitor. http, tcp
interval integer Optional 5 Health monitor probe interval in seconds. 1 to 86400
timeout integer Optional 16 Probe timeout in seconds. 1 to 86400
send string Optional   Message sent by the health monitor.  
recv string Optional   Response expected by the health monitor.  

Cloud Foundry Health Checks

The cf-bigip-ctlr supports native Cloud Foundry health checks for itself and will manage BIG-IP health checks for applications.

To configure Cloud Foundry health checks for the cf-bigip-ctlr (this ensures the orchestration system will manage the Controller availability for you), define the health-check-type and health-check-http-endpoint settings as follows:

health-check-type: http
health-check-http-endpoint: /health

The cf-bigip-ctlr will also manage BIG-IP health checking of the managed applications. To use any health monitor(s) that already exists on the BIG-IP system, add the name to the application manifest under bigip.health_monitors. Because these monitors apply to all applications in the system, the cf-bigip-ctlr uses the /Common/tcp_half_open monitor by default.

Cookie Max-Age values
Age Behavior
< 0 Session persists until client connection closes or one hour elapses.
> 0 Session persists until the Max-Age of the cookie.
= 0 Delete session persistence record if one exists on the BIG-IP system.

Configuration Examples

The example cf-bigip-ctlr application manifest below defines the following:

  • Cloud Foundry health check;
  • BIG-IP URL;
  • BIG-IP login credentials;
  • BIG-IP partition the cf-bigip-ctlr should manage;
  • BIG-IP virtual IP address that should serve as the cloud ingress;
  • Cloud Foundry NATS message bus to subscribe to for routing information;
  • Cloud Foundry NATS message bus login credentials;
  • Cloud Foundry OAuth endpoint for API access;
  • Cloud Foundry OAuth API access credentials;
  • Cloud Foundry routing API endpoint;
  • Type of routes to watch in Cloud Foundry (all routes, in this case).
applications:
  - name: cf-bigip-ctlr
    health-check-type: http
    health-check-http-endpoint: /health
    env:
      BIGIP_CTLR_CFG: |
                      # Required for all modes
                      bigip:
                        url: https://bigip.example.com
                        user: admin
                        pass: password
                        partition:
                          - cf
                        balance: least-connections-node
                        external_addr: 192.168.1.1
                        tier2_ip_range: 255.255.255.0
                        policies:
                          - /Example/policy
                        profiles:
                          - /Example/profile
                        health_monitors:
                          - /Common/tcp_half_open

                      # Required to run the BIG-IP Controller as a Service
                      # Broker (introduced in v1.1.0)
                      broker_mode: true

                      logging:
                        level: info

                      route_mode: all

                      # Required for HTTP routing
                      nats:
                        - host: 192.168.10.1
                          port: 4222
                          user: nats
                          pass: nats-password

                      # Required for TCP routing
                      oauth:
                        token_endpoint: uaa.system.pcf.local
                        client_name: uaa-client
                        client_secret: uaa-secret
                        port: 443
                        skip_ssl_validation: true
                        ca_certs:

                      routing_api:
                        uri: http://api.system.pcf.local
                        port: 80
                        auth_disabled: false

                      # User account for authentication to the Service Broker API
                      status:
                        user: user
                        pass: pass

      # Include the section below to use the cf-bigip-ctlr as a Service Broker
      # THE SETTINGS IN THIS SECTION ARE ROUTE-SPECIFIC
      SERVICE_BROKER_CONFIG: |
                             plans:
                              - description: plan for policy A,
                                name: planA,
                                virtualServer:
                                  - policies:
                                    - policyA
                                  - profiles:
                                    - profileA
                                  - sslProfiles:
                                    - sslProfileA
                                pool:
                                  balance: ratio-member
                                  healthMonitors:
                                    - name: /Common/http.get
                                    - name: hm-test
                                      type: http
                                      interval: 12
                                      timeout: 5
                                      send: hello
                                      recv: healthy

Controller API Endpoints

You can access health and route information using the cf-bigip-ctlr API.

Important

Use of /routes requires HTTP basic authentication. You can define a username and password that can access the Controller API in the Application Manifest:

status:
  password: some_password
  user: some_user
  • /health: The Controller health endpoint.

    The Controller returns 200 OK to indicate health; any other response is unhealthy. You can set the health Controller endpoint to the status.port property in the application configuration for development purposes. The Diego PORT value provided to the container environment will override this setting in production environments.

    curl -v http://10.0.32.15/health
    *   Trying 10.0.32.15..
    * Connected to 10.0.32.15 (10.0.32.15) port 80 (#0)
    > GET /health HTTP/1.1
    > Host: 10.0.32.15
    > User-Agent: curl/7.43.0
    > Accept: */*
    >
    < HTTP/1.1 200 OK
    < Cache-Control: private, max-age=0
    < Expires: 0
    < Date: Thu, 22 Sep 2016 00:13:54 GMT
    < Content-Length: 3
    < Content-Type: text/plain; charset=utf-8
    <
    ok
    * Connection #0 to host 10.0.32.15 left intact
    
  • /routes: The routes endpoint returns the entire routing table as JSON. Each route has an associated array of host:port entries.

    Example
    curl "http://someuser:somepass@10.0.32.15/routes"
    {"api.catwoman.cf-app.com":[{"address":"10.244.0.138:9022","ttl":0,"tags":{"component":"CloudController"}}],"dora-dora.catwoman.cf-app.com":[{"address":"10.244.16.4:60035","ttl":0,"tags":{"component":"route-emitter"}},{"address":"10.244.16.4:60060","ttl":0,"tags":{"component":"route-emitter"}}]}
    

Instrumentation and Logging

You can define the desired logging level for the cf-bigip-ctlr in the application manifest. The Controller supports the following log levels:

  • info, debug - An expected event occurred.
  • error - An unexpected error occurred.
  • fatal - A fatal error occurred which makes the Controller unable to execute.
Sample log message
[2017-02-01 22:54:08+0000] {"log_level":0,"timestamp":1485989648.0895808,"message":"endpoint-registered","source":"vcap.cf-bigip-ctlr.registry","data":{"uri":"0-*.login.bosh-lite.com","backend":"10.123.0.134:8080","modification_tag":{"guid":"","index":0}}}
  • log_level: message log level
    • 0 = info
    • 1 = debug
    • 2 = error
    • 3 = fatal
  • timestamp: Epoch time of the log
  • message: Content of the log line
  • source: The Controller function that initiated the log message
  • data: Additional information, varies based on the message

Footnotes:

[1]The controller requires the BIG-IP user account to have a defined role of Administrator, Resource Administrator, or Manager. See BIG-IP User Roles for further details.
[2](1, 2) The cf-bigip-ctlr supports BIG-IP load balancing algorithms that do not require additional configuration parameters. You can view the full list of supported algorithms in the f5-cccl schema. See the BIG-IP Local Traffic Management Basics user guide for information about each load balancing mode.
[3]The controller supports BIG-IP route domain specific addresses.
[4]SSL profiles must already exist on the BIG-IP device in a partition accessible by the cf-bigip-ctlr (for example, /Common).