Reporting¶
Overview¶
Policy Enforcement allows to create a rule within an enforcement policy that instructs the system to send usage data in high-speed logging (HSL) format to an external analytics server. The rule specifies what type of reporting data you are interested in. One of the actions it can take with the traffic is to send the information collected about it for processing, to a centralized analytics server. The system sends the information as a set of comma-separated values through Syslog transport.
You can choose to use session-based or flow-based reporting format, depending on the level of granularity you need. For example, a rule might collect session-based information about all audio and video traffic. You can specify how often to log the data and set the
destination as an HSL server or pool.
PE supports high-speed logging in flow/transaction and session granularity. The session is an aggregated log of all the flows of a subscriber.TMM container logging carries out the Logging for PE. Fluent-bit is used for processing the log files. Fluent-bit watches all files in /logs location and a few other locations, reads all of those log files, and adds a container-type ct field indicating which container logged the log message. All these messages are sent to Fluentd.
Procedure¶
Follow the below procedures to create a reporting profile:
Creating a log publisher¶
Create a log publisher to specify where the BIG-IP system has to send the alert messages.
apiVersion: "k8s.f5net.com/v2"
kind: F5BigLogHslpub
metadata:
name: "poolhslpublisher"
spec:
pool:
- name: "hslpublisher"
endpoint:
- "22.22.22.100:514"
syslog:
- name: "syslog1"
pool: "hslpublisher"
format: "rfc5424" # [rfc5424, rfc3164, legacy-bigip]
protocol: tcp
RFC formats examples¶
Following is an RFC format example:
#legacy-bigip #tcp
# Jan 6 12:34:06 f5-tmm-7f8b9d574f-9gxtj tmm[328038]: 27131905 "27131905","f5-tmm-7f8b9d574f-9gxtj","11.11.11.100","45734","22.22.22.100","900","TCP","default-virtual-SecureContext_vs","default-dpi-profile-profileclassification","","","allow","tcp","http","google","","","","","","0","Network_Management_and_Services","Network_Management_and_Services","Search_Engines","","","","","","Search_Engines","","","","302","60","3","","","","","","","",""
#rfc5424 #tcp
# Jan 6 12:37:24 f5-tmm-7f8b9d574f-9gxtj tmm[643455] "27131905","f5-tmm-7f8b9d574f-9gxtj","11.11.11.100","38124","22.22.22.100","900","TCP","default-virtual-SecureContext_vs","default-dpi-profile-profileclassification","","","allow","tcp","http","google","","","","","","0","Network_Management_and_Services","Network_Management_and_Services","Search_Engines","","","","","","","","","","302","60","3","","","","","","","",""
#rfc3164 #tcp
# Jan 6 12:40:29 f5-tmm-7f8b9d574f-9gxtj tmm[643455]: 27131905 "27131905","f5-tmm-7f8b9d574f-9gxtj","11.11.11.100","37668","22.22.22.100","900","TCP","default-virtual-SecureContext_vs","default-dpi-profile-profileclassification","","","allow","tcp","http","google","","","","","","0","Network_Management_and_Services","Network_Management_and_Services","Search_Engines","","","","","","","","","","302","60","3","","","","","","","",""
#legacy-bigip #udp
# Jan 6 12:41:54 f5-tmm-7f8b9d574f-9gxtj tmm[643455]: 27131905 "27131905","f5-tmm-7f8b9d574f-9gxtj","11.11.11.100","53602","22.22.22.100","900","TCP","default-virtual-SecureContext_vs","default-dpi-profile-profileclassification","","","allow","tcp","http","google","","","","","","0","Network_Management_and_Services","Network_Management_and_Services","Search_Engines","","","","","","","","","","302","60","3","","","","","","","",""
#rfc5424 #udp
# Jan 6 12:43:20 f5-tmm-7f8b9d574f-9gxtj tmm[643455] "27131905","f5-tmm-7f8b9d574f-9gxtj","11.11.11.100","40082","22.22.22.100","900","TCP","default-virtual-SecureContext_vs","default-dpi-profile-profileclassification","","","allow","tcp","http","google","","","","","","0","Network_Management_and_Services","Network_Management_and_Services","Search_Engines","","","","","","","","","","302","60","6","","","","","","","",""
#rfc3164 #udp
# Jan 6 12:43:55 f5-tmm-7f8b9d574f-9gxtj tmm[643455]: 27131905 "27131905","f5-tmm-7f8b9d574f-9gxtj","11.11.11.100","34766","22.22.22.100","900","TCP","default-virtual-SecureContext_vs","default-dpi-profile-profileclassification","","","allow","tcp","http","google","","","","","","0","Network_Management_and_Services","Network_Management_and_Services","Search_Engines","","","","","","","","","","302","60","4","","","","","","","",""
Creating a rule for high-speed logging for session reporting¶
Before you can create a high-speed logging (HSL) rule, you need to create a publisher that defines the destination server or pool where the HSL logs are sent.
apiVersion: "k8s.f5net.com/v1"
kind: F5BigLogProfile
metadata:
name: hsl-log-profile
spec:
publisher: "poolhslpublisher"
pe:
reportingType: "session-reporting"
reportingFields:
- "All"
intervalThreshold: 20
You have created a rule that sends data about the traffic to external high-speed logging servers. The CSV reporting format differs depending on whether the report granularity is flow-based or session-based.
Creating a rule for high-speed logging for flow reporting¶
Before you can create a high-speed logging (HSL) rule, you need to create a publisher that defines the destination server or pool where the HSL logs are sent. In an enforcement policy, a rule can specify that flow statistics about the traffic affected by the rule are sent to an external high-speed logging server.
apiVersion: "k8s.f5net.com/v1"
kind: F5BigLogProfile
metadata:
name: hsl-log-profile
spec:
publisher: "poolhslpublisher"
pe:
reportingType: "flow-reporting"
reportingFields:
- "All"
intervalThreshold: 20
You have created a rule that sends data about the traffic to external high-speed logging servers. The CSV reporting format differs depending on whether the report granularity is flow-based or session-based.
Session-based reporting format¶
In an enforcement policy, a rule can send session-based information about traffic that matches certain criteria to an external high-speed logging (HSL) server. The logs include the following comma-separated values in the order listed.
| Field | Description |
|---|---|
| PE ID | Identifies the reporting module (pe) and the field value is 23003143. |
| Version | Indicates the version of the format for backward compatibility. |
| Timestamp seconds | The time the information was logged (along with the timestamp in milliseconds), specifies seconds using Unix time format. |
| Timestamp msec | The time the information was logged (along with the timestamp in seconds), specifies milliseconds using Unix time format. |
| Report type | The type of report. Always set to 3 for session-based reporting. |
| Subscriber ID | A unique identifier (up to 64 characters) for the subscriber initiating the session, such as a phone number. The subscriber ID type determines the format. |
| Subscriber ID type | The format of the subscriber ID. It can be E.164, IMSI, NAI, or Private. |
| 3GPP parameters | The list of 3GPP parameters, which can be imsi, imeisv, tower_id, or username. |
| Policy ID | The identification of the policy. |
| Rule ID | The identification of the policy rule. |
| Application ID | A unique number that represents a particular application, and is used for classifying traffic. |
| Last Sent | The time, in seconds, since the last log entry was sent. |
| Bytes in | The number of bytes received during this session. |
| Bytes out | The number of bytes sent during this session. |
| Concurrent flows | Always 0 (unsupported). |
| Opened flows | Always 0 (unsupported). |
| Terminated flows | Always 0 (unsupported). |
| Total transactions | Always 0 (unsupported). |
| Successful transactions | Always 0 (unsupported). |
| Aggregated category duration | Summary of the duration of all flows for the session. |
| Reason | The reason for sending the record. It can be 0 - reserved, 1 - volume threshold reached, 2- interval time, 3 - subscriber logout, or 4 - inactivity. |
Example session-based reporting format¶
Following is an example format of session-based reporting:
Oct 10 17:19:45 172.31.63.64 23003143,1349914925,546879,3,404234567123456,IMSI,linux,f501, 404234567123456,35827001,16394,1349914913,5469633,308908379, 0,0,0,0,0,5052,1 Oct 10 17:19:57 172.31.63.64 23003143,1349914937,546661,3,404234567123456,IMSI,linux,f501, 404234567123456,35827001,16394,1349914925,5550857,313317479, 0,0,0,0,0,5063,1 Oct 10 17:20:09 172.31.63.64 23003143,1349914949,546676,3,404234567123456,IMSI,linux,f501, 404234567123456,35827001,16394,1349914937,5636605,318053179, 0,0,0,0,0,5074,1
Flow-based reporting format¶
In an enforcement policy, a rule can send flow-based information about traffic that matches certain criteria to an external high-speed logging (HSL) server. The logs include the following comma-separated values in the order listed.
| Field | Description |
|---|---|
| PE ID | Identifies the reporting module (pe) and the field value is 23003143. |
| Version | Indicates the version of the format for backward compatibility. |
| Timestamp seconds | The time the information was logged in Unix time format. |
| Timestamp msec | The msecs time value of the timestamp in Unix time format. |
| Report type | The type of report; 0 -- flow start, 1 -- flow interim, 2 -- flow end. |
| Subscriber ID | A unique identifier (up to 64 characters) for the subscriber initiating the session, such as a phone number. The subscriber ID type determines the format. |
| Subscriber ID type | The format of the subscriber ID. It can be E.164, IMSI, NAI, or Private. |
| Source IP | The IPv4 source address in the IP packet header. |
| Source port | The source port the subscriber. |
| Destination IP | The IPv4 destination address in the IP packet header. |
| Destination IP address | The destination IP of the traffic. |
| Destination port | The destination port for the traffic. |
| Protocol | The protocol of the traffic for this flow, TCP or UDP. |
| Route Domain | The route domain this flow belongs to. |
| VLAN | The VLAN this flow belongs to. |
| Application ID | A unique number that represents a particular application in this flow; it is used for classifying traffic. |
| Urlcat ID | The URL category ID that the flow belongs to. |
| Flow start time seconds | The time, in seconds, the flow started in Unix time format. |
| Flow start time msecs | The time in milliseconds of the flow start time. |
| Flow end time seconds | The time the flow ended in Unix time format. |
| Flow end time msecs | The time in milliseconds of the flow end time. |
| Transactions count | The count of full transactions seen in the flow. |
| Bytes in | The number of bytes received during this flow. |
| Bytes out | The number of bytes sent during this flow. |
| SNI | The Server Name Indication value in SSL traffic. |
| Video Content Provider | The name of the Video Content Provider. |
| Video Resolution | The number of pixels(vertical) of the video resolution. |
| Video Bitrate | The number of bits processed per unit time. |
| Handshake RTT | The Round Trip time of the handshake mechanism. |
Example flow-based reporting format¶
Following is an example format of flow-based reporting:
Sep 13 13:48:58 172.31.63.60 23003143,1347546777,654398,0,4086007577,E164,2001::10,52784,2001::2,80,6, 67,1347546774,628630,4278124286,4278124286,331,156, , youtube, 720p, 11234567, 123 Sep 13 13:48:58 172.31.63.60 23003143,1347546777,654398,2,4086007577,E164,2001::10,52784,2001::2,80,6, 67,1347546774,628630,1347546775,382473,547,864, , youtube, 720p, 11234567, 123
Example CRs of a Reporting use case¶
The following CRs help us to configure session-based reporting with all the necessary CRs required.
F5BigDpiPeOptions¶
For Reporting, enableHslFlowStartReport and enableHslFlowInterimReport has to be set to True in the F5BigDpiPeOptions CR. Also, the hslFlowReportVersion version needs to be REPORT_VER_16_1.
To view or edit the F5BigDpiPeOptions CR, see F5BigDpiPeOptions.
Example CR:
kind: F5BigDpiPeOptions
metadata:
name: "dpi-pe-default-options"
spec:
dpiGlobalOptions:
enableFlowBundling: True
sslSessionIdTimeout: 18000
gpaLogLevel: "Debug"
dpiLogLevel: "Debug"
urlcatLogLevel: "Debug"
dpiMaxPackets: 10
peGlobalOptions:
enablePeSrdb: True
peLogLevel: "Debug"
pePolicyReevaluationInterval: 5
peSessionInactivityTimeout: 2
peSpmMaxSessionLimit: 524288
reporting:
enableHslFlowStartReport: true
enableHslFlowInterimReport: true
hslFlowReportVersion: REPORT_VER_16_1
F5BigLogHslpub¶
Copy the following example in
poolhslpublisher.yamlfile. The following CR configures the endpoint which acts the server for the HSL logs.apiVersion: "k8s.f5net.com/v2" kind: F5BigLogHslpub metadata: name: "poolhslpublisher" spec: pool: - name: "hslpublisher" endpoint: - "22.22.22.100:514" syslog: - name: "syslog3" pool: "hslpublisher" format: "rfc5424" # [rfc5424, rfc3164, legacy-bigip] protocol: tcp
Run the following command to apply the
F5BigLogHslpubCR.kubectl apply -f poolhslpublisher.yaml -n <name_space>Verify that the
F5BigLogHslpubCR is applied by checking logs in f5ingress.I0221 08:02:55.145541 13 event.go:377] Event(v1.ObjectReference{Kind:"F5BigPeProfile", Namespace:"default", Name:"poolhslpublisher", UID:"1405a7e8-402a-4490-80b6-710ae6b3f79c", APIVersion:"", ResourceVersion:"12241", FieldPath:""}): type: 'Normal' reason: 'Added/Updated' peProfile default/poolhslpublisher was added/updated
For more information on Log Profile, see F5LogHslpub CRD page.
F5BigLogProfile¶
The following CR configures the reporting type and the interval threshold for the logs to be periodically sent to the endpoint.
Copy the following example in
hsl-log-profile.yamlfile.apiVersion: "k8s.f5net.com/v1" kind: F5BigLogProfile metadata: name: hsl-log-profile spec: publisher: "poolhslpublisher" pe: reportingType: "session-reporting" reportingFields: - "All" intervalThreshold: 20
Run the following command to apply the
F5BigLogProfileCR.kubectl apply -f poolhslpublisher.yaml -n <name_space>Verify that the
F5BigLogProfileCR is applied by checking logs in f5ingress.I0221 08:02:55.145541 13 event.go:377] Event(v1.ObjectReference{Kind:"F5BigPeProfile", Namespace:"default", Name:"hsl-log-profile", UID:"1405a7e8-402a-4490-80b6-710ae6b3f79c", APIVersion:"", ResourceVersion:"12241", FieldPath:""}): type: 'Normal' reason: 'Added/Updated' peProfile default/hsl-log-profile was added/updated
For more information on Log Profile, see F5BigLogProfile CRD page.
F5BigPePolicy¶
The following PE policy referes the F5BigLogProfile CR and the F5LogHslpub CR.
Copy the following example in
hsl-pe-policy.yamlfile.apiVersion: "k8s.f5net.com/v1" kind: F5BigPePolicy metadata: name: "hsl-pe-policy" spec: description: "my pem policy" rule: - name: "hsl-rule" precedence: 1 reportingProfile: hsl-log-profile publisher: poolhslpublisher filter: flow: - name: "test-flow" match: match sourceAddress: 11.11.11.100 action: enableGate: True
Run the following command to apply the
F5BigPePolicyCR.kubectl apply -f poolhslpublisher -n <name_space>Verify that the
F5BigPePolicyCR is applied by checking logs in f5ingress.I0221 08:02:55.145541 13 event.go:377] Event(v1.ObjectReference{Kind:"F5BigPeProfile", Namespace:"default", Name:"hsl-pe-policy", UID:"1405a7e8-402a-4490-80b6-710ae6b3f79c", APIVersion:"", ResourceVersion:"12241", FieldPath:""}): type: 'Normal' reason: 'Added/Updated' peProfile default/hsl-pe-policy was added/updated
For more information, see F5BigPePolicy CRD page.
F5BigPeProfile¶
Copy the following example in
pe-profile.yamlfile.apiVersion: "k8s.f5net.com/v1" kind: F5BigPeProfile metadata: name: "pe-profile" spec: description: 'pe-profile-hsl-logging' globalPolicy: {} unknownSubscriberpolicy: - "hsl-pe-policy"
Run the following command to apply the
F5BigPeProfileCR.kubectl apply -f pe-profile.yaml -n <name_space>Verify that the
F5BigPeProfileCR is applied by checking logs in f5ingress.I0317 01:51:31.017071 13 event.go:377] Event(v1.ObjectReference{Kind:"F5BigPePolicy", Namespace:"default", Name:"pem-profile", UID:"da1485a0-2812-4594-82f5-31ad9eddd38d", APIVersion:"", ResourceVersion:"15152", FieldPath:""}): type: 'Normal' reason: 'Added/Updated' PemPolicy default/pem-profile was added/updated
For more information, see F5BigPeProfile CRD page.
F5BigContextSecure¶
The following CR refers the DPI profile, PE profile and the HSL log profiles applied earlier.
Copy the following example in the
secure_context.yamlfile.apiVersion: "k8s.f5net.com/v1" kind: F5BigContextSecure metadata: name: "secure-context" spec: destinationAddress: 0.0.0.0/0 ipv6destinationAddress: ::/0 ipProtocol: "tcp" destinationPort: 0 snat: type: "automap" profile: "fastL4" logProfile: "hsl-log-profile" peProfile: "pe-profile"
Run the following command to apply the
F5BigContextSecureCR.kubectl apply -f secure_context.yaml -n <name_space>Verify that the
F5BigContextSecureCR is applied by checking logs in f5ingress.0221 08:06:28.013501 13 event.go:377] Event(v1.ObjectReference{Kind:"F5BigContextSecure", Namespace:"default", Name:"secure-context", UID:"ec203939-44ef-4c32-9bc6-8e84b1501869", APIVersion:"", ResourceVersion:"12572", FieldPath:""}): type: 'Normal' reason: 'Added/Updated' SecureContext default/secure-context was added/updated
For more information, see F5BigContextSecure CRD page.
Feedback
To provide feedback and help improve this document, please email us at cnfdocs@f5.com.