F5OS-A 1.3.0 - SYN Cookie Protection¶
Feature Overview¶
Global SYN cookie protection is a feature that allows for protecting against SYN flood attacks at the VLAN level. To achieve its goals, this feature leverages the FPGA hardware that is in the rSeries hardware platform similar to previous implementations on BIG-IP hardware.
Feature deeper overview¶
This document describes the F5OS implementation and troubleshooting of the F5OS portion of the feature. The feature has been a part of BIG-IP for a number of years, but the platform architecture has changed since previous iterations. The architecture leverages the ATSE FPGA’s and TCAM tables to achieve the same functionality as previous platforms and a set of containers to propogate config information to and from the FPGAs and Tenants.
In particular, the solution uses the following containers:
Tenant Platform Agent
API-SVC-Gateway
FPGA Manager
ConfD
Customer use case example¶
Admin user wants to enable SYN cookie protection at the VLAN level at wire speed.
CLI commands¶
The following commands can be used in the ConfD CLI:
appliance-1# show service-instances service-instance tenant_name
GUI Screen Shots¶
Not applicable, this is CLI and hardware functionality only.
API¶
Refer to the F5OS-A API for API endpoints.
New / Updated statistics¶
Stats for this feature are the same and are pulled via tmctl in the tenant:
E.g. tmctl epva_hwvipstat -s vlan_generated
New / Updated logs¶
The only logs in F5OS regarding VLAN Syncookie protection are the enabled and disabled messages when fpgamgr is set to debug mode:
2022-11-11T23:45:55.696051+00:00 appliance-1 fpgamgr[14]: priority="Debug" version=1.0 msgid=0x301000000000030 msg="" text="hdpMgr::synCookieVlan, unit 0 svc_grp 5".
2022-11-11T23:45:55.696786+00:00 appliance-1 fpgamgr[14]: priority="Debug" version=1.0 msgid=0x301000000000030 msg="" text="hdpMgr::synCookieVlan, unit 1 svc_grp 5".
Expected / Possible Problems¶
None Currently.
Known Issues¶
None Currently.
Licensing, Provisioning, and other Requirements¶
No special licensing is required for this feature, it is enabled at the tenant level.
Admin setup workflow¶
N/A - all configuration of this feature is done at the tenant level (e.g. in BIG-IP).
End user workflow¶
N/A - all configuration of this feature is done at the tenant level (e.g. in BIG-IP) and does not have an end user workflow.
Is it working or is it failing?¶
To see if the feature is available and configured at the F5OS level, the following will be available to see:
appliance-1# show service-instances service-instance test-tenant
service-instances service-instance test-tenant 1 1932593075
atse-mod-id 0
sub-mod-id 0
service-type ST_TENANT_SERVICE
tenant-id 6
service-ids [ 11 12 ]
num-seps 1
dm-offset 0
did 15
svc-grp 5
dos-grp 2
tcp-syn-cookie transform-info tco FALSE
tcp-syn-cookie transform-info wp FALSE
tcp-syn-cookie transform-info sp FALSE
tcp-syn-cookie transform-info tp FALSE
tcp-syn-cookie transform-info mss MSS_IDX_FIVE
tcp-syn-cookie transform-info tc 7
tcp-syn-cookie transform-info win 1
tcp-syn-cookie transform-info wscale 8
tcp-syn-cookie transform-info synackttl 255
tcp-syn-cookie transform-info synackhoplimit 64
tcp-syn-cookie secret-info index SECRET_IDX_ZERO
tcp-syn-cookie secret-info s1 1893190907
tcp-syn-cookie secret-info s2 4124231716
tcp-syn-cookie secret-info s3 2948136541
tcp-syn-cookie secret-info s4 1119058162
If an attack is in process and the feature is triggered, the state will appear and change for the VLAN like so:
appliance-1# show service-instances service-instance test-tenant
service-instances service-instance test-tenant 1 1932593075
atse-mod-id 0
sub-mod-id 0
service-type ST_TENANT_SERVICE
tenant-id 6
service-ids [ 11 12 ]
num-seps 1
dm-offset 0
did 15
svc-grp 5
dos-grp 2
tcp-syn-cookie transform-info tco FALSE
tcp-syn-cookie transform-info wp FALSE
tcp-syn-cookie transform-info sp FALSE
tcp-syn-cookie transform-info tp FALSE
tcp-syn-cookie transform-info mss MSS_IDX_FIVE
tcp-syn-cookie transform-info tc 7
tcp-syn-cookie transform-info win 1
tcp-syn-cookie transform-info wscale 8
tcp-syn-cookie transform-info synackttl 255
tcp-syn-cookie transform-info synackhoplimit 64
tcp-syn-cookie secret-info index SECRET_IDX_ZERO
tcp-syn-cookie secret-info s1 1893190907
tcp-syn-cookie secret-info s2 4124231716
tcp-syn-cookie secret-info s3 2948136541
tcp-syn-cookie secret-info s4 1119058162
VLAN
ID STATE
-------------
3863 TRUE
appliance-1# show service-instances service-instance test-tenant
service-instances service-instance test-tenant 1 1932593075
atse-mod-id 0
sub-mod-id 0
service-type ST_TENANT_SERVICE
tenant-id 6
service-ids [ 11 12 ]
num-seps 1
dm-offset 0
did 15
svc-grp 5
dos-grp 2
tcp-syn-cookie transform-info tco FALSE
tcp-syn-cookie transform-info wp FALSE
tcp-syn-cookie transform-info sp FALSE
tcp-syn-cookie transform-info tp FALSE
tcp-syn-cookie transform-info mss MSS_IDX_FIVE
tcp-syn-cookie transform-info tc 7
tcp-syn-cookie transform-info win 1
tcp-syn-cookie transform-info wscale 8
tcp-syn-cookie transform-info synackttl 255
tcp-syn-cookie transform-info synackhoplimit 64
tcp-syn-cookie secret-info index SECRET_IDX_ZERO
tcp-syn-cookie secret-info s1 1893190907
tcp-syn-cookie secret-info s2 4124231716
tcp-syn-cookie secret-info s3 2948136541
tcp-syn-cookie secret-info s4 1119058162
VLAN
ID STATE
-------------
3863 FALSE
Architecture¶
Below is a general architecture diagram of per TMM/VLAN based hardware SYN cookie feature in classic BIG-IP.
In classic BIG-IP, HSB register space is directly accessible from TMM either in bare metal or VCMP guest installs. To do this in the v15.0 code base or earlier the VLAN syncookie application layer called the HSB driver directly to configure VLAN syscookie related registers and tables as well as getting back SYN cookie related stats.
As a result of SmartNIC support in versions after v15.0, the VLAN syncookie application layer now calls on the NDAL (Network Device Abstraction Layer) which dispatches the calls into corresponding driver APIs. The NDAL is designed by the BIG-IP VE team to abstract away ifnet, and provides a set of generic APIs of the hardware device APIs to the application layer above.
Based on currentsystem architecture (Velocity Platform Interface and APIs),The velocity end-to-end VLAN Syncookie function will be implemented as the following functional blocks.
The HSB register space is no longer accessible for TMM inside Velocity VM tenants so the HSB driver is replaced by a MCP/Protobuf layer to send the VLAN SYN cookie commands out to the Velocity platform host.Then we have to route the SYN cookie commands from the tenant to the host through multiple hops using the proper IPC channels (mcp, grpc, etc): Tenant Platform Agent -> API-SVC-Gateway -> FPGA manager.
The VLAN SYN cookie operational data is also persisted through ConfD. Eventually FPGA Manager on the host will be responsible for configuring the VLAN SYN cookie related register tables. The reverse path will be used to transfer the stats back from the FPGA to the tenant TMM with the tmctl stats table being used to transfer the stats between FPGA manager and the API-SVC-Gateway.
There are some minor changes to VLAN SYN cookie application layer for the Velocity VM tenant. Since there is no PDE concept on Velocity FPGA, the per TMM/PDE based VLAN SYN cookie protection will be replaced by per Tenant based VLAN syncookie.
Interfaces and message type¶
Restrictions¶
On Velocity platforms, GRPC and protobuf are widely used for both internal and external APIs. Legacy TMMs are not able to run GRPC protocol due to following restrictions:
TMM is C code and there is no official GRPC implementation for C programming language at this time; there is no third-party implementation either.
GRPC is used as a system socket to transfer data over HTTP2, which is not available in TMM.
In order to communicate with the outside world, a tenant TMM’s existing mcp_socket is used.The Tenant Platform Agent was introduced and used as a proxy to communicate with TMM via mcp_socket.Rather than communicate with rawmcp_socket messages,an existing high level MCP messaging framework is actually used.
Interfaces¶
TMM <-> Tenant Platform Agent
MCP messages are used for this communication. In order to reuse definitions of message content with other GRPC based modules, protobuf is used as a data blob - which is the payload of MCP message and SYN cookie commands encapsulated into protobuf format. The Tenant Platform Agent forwards the protobuf payload from incoming MCP messages to API-SVC-Gateway via GRPC interface and sends protobuf responses back to TMM via MCP messages.
In order to achieve bidirectional MCP messages communication between TMM and the Tenant Platform Agent, a new hubproxy (syncookie-proxy) was added to handle MCP messages with protobuf-C payload between TMMs and the Tenant Platform Agent.
TBD
VLAN SYN cookie proto message definition:
Refer to:Global SYN Cookie Config Operations and APIs
What is the final proto definition? Is it pertinent to internal groups troubleshooting it?
Notes:
TMM must use the protobuf-C compiler and library in the TMM code base as the old protobuf-C doesn’t support Service definition in proto and may not support “proto 3” either.
Tenant Platform Agent<-> API-SVC-Gateway
The Tenant Platform Agent is a GRPC client of API-SVC-Gateway and VLAN SYN cookie requests will be issued by the Tenant Platform Agent and served by API-SVC-Gateway.
GRPC message definition:
Refer to:GRPC Service Definitions - API methods
What is the final proto definition? Is it pertinent to internal groups troubleshooting it?
Notes:
Need to figure it out how to share message definition in proto with TMM and define GRPC services separately in API-Svc-Gateway.
API-SVC-Gateway <-> FPGA Manager
There is no direct communication between API-SVC-Gateway and FPGA Managers. ConfD is used for persistence and syncing up configuration/stats information between the two.
API-SVC-Gateway will publish any requested VLAN SYN cookie configuration into the ConfDschema for incoming SYN Cookie commands it recieves in GRPC requests.
API-SVC-Gateway may subscribe to or directly read any VLAN SYN cookie stats information in ConfD.
FPGA Manager subscribes to the VLAN SYN cookie table/events and configures the FPGAs accordingly.
VLAN SYN cookie stats information coming from the FPGAs will be published back to ConfD via FPGA Manager.
VLAN syncookie Confd Schema
The following output table describes the ConfD scheme decided upon for VLAN SYN Cookie Protection:
appliance-1# show service-instances service-instance test-tenant service-instances service-instance test-tenant 1 1932593075 atse-mod-id 0 sub-mod-id 0 service-type ST_TENANT_SERVICE tenant-id 6 service-ids [ 11 12 ] num-seps 1 dm-offset 0 did 15 svc-grp 5 dos-grp 2 tcp-syn-cookie transform-info tco FALSE tcp-syn-cookie transform-info wp FALSE tcp-syn-cookie transform-info sp FALSE tcp-syn-cookie transform-info tp FALSE tcp-syn-cookie transform-info mss MSS_IDX_FIVE tcp-syn-cookie transform-info tc 7 tcp-syn-cookie transform-info win 1 tcp-syn-cookie transform-info wscale 8 tcp-syn-cookie transform-info synackttl 255 tcp-syn-cookie transform-info synackhoplimit 64 tcp-syn-cookie secret-info index SECRET_IDX_ZERO tcp-syn-cookie secret-info s1 1893190907 tcp-syn-cookie secret-info s2 4124231716 tcp-syn-cookie secret-info s3 2948136541 tcp-syn-cookie secret-info s4 1119058162
Configuring VLAN SYN Cookie Under the Hood¶
Tenant Platform Agent is the border between legacy TMM and the F5OS.API-SVC-Gateway is the single point of service access via GRPC APIs. At this moment, for simplicity, Unary RPCs are used for most services.Streaming RPCs will be considered at a later date.
In order not to block TMM, whenTenant Platform Agentreceives requests from TMM,it will send a response to TMM immediately to confirm the request is received before it makes any GRPC calls to API-SVC-Gateway (and GRPC responses are received).Tenant Platform Agent is responsible for resending GRPC requests for any negative GRPC responses from API-Svc-Gateway.
From API-SVC-Gateway’s perspective, a request is successfully served after data is updated in ConfD.The FPGA Manager will be responsible for writing the configuration into FPGAs based on latest configuration changes in ConfD.
Querying for and Publishing VLAN SYN Cookie Stats to the Tenant¶
After the VLAN SYN cookie feature is configured and enabledin the FPGA devices, FPGA Manager will constantly updateVLAN SYN cookie stats from the FPGAs into the tmstats tables via the tmctl interface. This is because TMM needs to know the current VLAN SYN cookie operational stats and take appropriate actions.The VLAN SYN cookie stats will be forwarded to TMM via MCP messages after the Tenant Platform Agent recieves them from API-SVC-Gateway.