BIG-IP Next Access: Troubleshooting FAQ¶

This page of frequently asked questions is specific to BIG-IP Next Access. Use this section for general support and troubleshooting information.

Access Components¶

What are Access Components?¶

BIG-IP Next is composed of several components which are deployed as pods. In BIG-IP Next Access, the following components perform traffic processing:

f5-access-apmd: Runs the Access policy. An example of this Pod name is f5-access-apmd-5f4888bcfc-xjpr4.
f5-access-renderer: Renders the access related (login/logout) webpages. An example of this Pod name is f5-access-renderer-5f4666bcfc-xjpr6.
f5-access-session-manager: Handles all session management functions. An example of this Pod name is f5-access-session-manager-66cc784f88-8w26b.
f5-fsm-tmm: An example of f5-fsm-tmm pod name is f5-fsm-tmm-864f497c66-qhh5n. The f5-fsm-tmm pod has the following containers:

f5-fsm-tmm: Processes incoming traffic.

f5-access-websso: Processes the single-sign-on functionality.

How do I view information on the pods and containers?¶

Use one of the following kubectl command as per your requirement to enter a pod:

kubectl get pods | grep apmd
kubectl exec -it <podname> -- bash
kubectl exec -it <podname> -- sh
kubectl exec -it <podname> -c <container-name> -- sh
kubectl exec <podname> -- bash -c "ls /"
kubectl exec <podname> -c <container name> -- bash -c "ls /"

How do I list running pods and containers?¶

Use the following kubectl commands to list running pods on the BIG-IP Next:

kubectl get pods | grep f5-access
kubectl get pods | grep f5-access-apmd
kubectl get pods | grep f5-access-renderer
kubectl get pods | grep f5-access-session-manager

To view all pods:

kubectl get pods

To view all containers:

kubectl describe pod <podname>

To view all access pods:

kubectl get pods | grep "\(access\|fsm\)"

To view all access pods (except websso container):

kubectl get pods | grep access

Licenses¶

How many Concurrent Access sessions are supported?¶

The BIG-IP Next Access supports a maximum of 20000 Concurrent Access sessions.

Deployment¶

What configuration should my VE instance be for deployment?¶

All BIG-IP Next Access instances should use 8 core and 16 GB configuration for deployment.

Refer the following links for details on initial setup:

General Health¶

How do I check the general health of the modules?¶

To display a live stream of pod(s) resource usage statistics (such as CPU, memory, network, disk), use the following commands:

kubectl get pods | grep f5-access-apmd

This command returns the pod name for f5-access-apmd pod.
kubectl describe pod <podname>

This command describes pod status and health.
kubectl top pods

This command gives the CPU memory usage of a pod.

Refer to kubectl commands for more details.

How do I generate and download a QKView file?¶

Refer to the How to: Generate & download a QKView file and upload to F5 iHealth or Support link for more information.

Collecting and Viewing Logs¶

How do I collect and view logs?¶

Run the following commands on a shell to collect and view logs:

Use the GET /systems/{system-id}/logs API to get the list of available logs:

curl --location --insecure --request GET 'https://'${MBIP_MGMT_IP}':5443/api/v1/systems/{system-id}/logs' --header 'Authorization: Bearer '${TOKEN}

Response:

"/api/v1/logs/f5-access-apmd-stdout.log" 
...

"/api/v1/logs/f5-fsm-tmm-stdout.log"

Use the Use the GET /systems/{system-id}/logs/<log-file-name> API to get a specific log file such as f5-access-apmd-stdout.log:

curl --location --insecure --request GET 'https://'${MBIP_MGMT_IP}':5443/api/v1/systems/{system-id}/logs/f5-access-apmd-stdout.log' --header 'Authorization: Bearer '${TOKEN}

Response:

"{\"type\":\"apmd\",\"timestamp\":\"Timestamp here\",\"log\":\"Log statement here\",\"tier2tag\":\"stdout\"}"

Use the kubectl command to get real-time logs for apmd pod:

kubectl get pods | grep fluentdserver
kubectl exec -it <podname> – sh tail -f /TODA/log/f5-access-apmd-stdout.log

or

kubectl get pods | grep apmd
kubectl logs -f <podname> –-timestamps=true

Example:

kubectl logs -f f5-access-apmd-5f4888bcfc-xjpr4 –-timestamps=true

Use the kubectl command to get real-time logs for TMM container:

kubectl get pods | grep fluentdserver
kubectl exec -it <podname> – sh tail -f /TODA/log/f5-fsm-tmm-stdout.log

Example:

kubectl exec -it fluentdserver-7b65c47fd9-7d8gj – sh tail -f /TODA/log/f5-fsm-tmm-stdout.log

Another method:

kubectl get pods | grep tmm
kubectl logs -f <podname> -c f5-fsm-tmm

Example:

kubectl logs -f f5-fsm-tmm-864f497c66-qhh5n -c f5-fsm-tmm

How do I change log levels?¶

Setting log levels in Access

Using the service object, you can configure the system logs, analytics, cluster configuration, and WAF settings. The service object represents the name of the service or process that generated the log. To change Access log settings independently from other service properties, use the /services/{serviceId}/access endpoint. The serviceId is the service identifier. To retrieve a service ID, make a GET request to the endpoint /services. This returns a list of all the services.

The logging object defines an array of log settings and log levels of various Access components. Set log levels for the components using the following properties in the logging object.

component: Specifies the BIG-IP Next Access component.
policy: Specifies the name of the Access policy. This property is optional; if not specified, the log levels are used by Access for policies that do not have their defined log level.
level: Specifies the minimum log level for the component log messages. The possible values for the log level are DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT, and EMERGENCY. These values are not case-sensitive. The default log level for all the Access logs is Notice.

To get all services:

GET https://{{bigip_next_mgmt_floating_ip}}:5443/api/v1/services

To set the log level for the websso, tmm, and apmd components:

PUT /services/{serviceId}/access
{
  "logging": [
    {
      "component": "websso",
      "level": "WARNING"
    },
    {
      "component": "tmm",
      "level": "NOTICE"
    },
    {
      "component": "apmd",
      "policy": "access-policy",
      "level": "DEBUG"
    }
  ]
}

Response:

{
"id": "b56797e0-78fc-470f-aaa0-5e7ca3e10a32",
"verb": "PUT",
"uri": "/services/2addf4c6-1bda-44ec-a511-377af75c3a49/access",
"request": "{\"logging\":[{\"component\":\"websso\",\"level\":\"WARNING\"},{\"component\":\"tmm\",\"level\":\"NOTICE\"},{\"component\":\"apmd\",\"level\":\"DEBUG\",\"policy\":\"access-policy\"}]}",
"role": "AGENTCONFIG",
"status": "PENDING",
"message": {
"code": "",
"title": "",
"detail": "",
"links": {
"about": "",
"resourceLink": ""
}
},
"creationTime": "2022-08-10T22:44:43.347487Z",
"updateTime": "2022-08-10T22:44:43.347487Z",
"owner": "admin",
"authn": "local",
"authz": "global"
}

Configuration¶

What are the common configuration errors?¶

jobId is used to track the progress of the job and get useful information about it. Every PUT call returns a 202 with a Job ID that can be used to track the progress of the task and find out if it is successful, timed out, or failed.

Refer to the Support and Troubleshooting link for general support.

SessionDump Command¶

How do I use the sessiondump utility for BIG-IP Next?¶

SessionDump is a command-line utility that allows you to perform several operations within BIG-IP Next, such as listing, copying, or deleting Access session information. You can access this utility and run commands only from the f5-access-session-manager pod. To get into the f5-access-session-manager pod, you require access to the pod’s Kubernetes environment. Once you SSH into the f5-access-session-manager pod, you can retrieve and delete session information using the sessiondump command.

Use the kubectl exec command to get pod access:

kubectl get pods | grep f5-access-session-manager
kubectl exec -it <podname> -- bash
kubectl exec <session manager podname> – sessiondump --list

The sessiondump command syntax on CLI includes one operation and one or more arguments and flags. The general usage for sessiondump utility is as follows:

sessiondump <operation> <arguments> <flags>

Operations

Name	Description	Example
help	Show this help message	sessiondump --help
list	Show list of all sessions	sessiondump --list
allkeys	Show all session variables for all sessions	sessiondump --allkeys
locks	Show list of session locks	sessiondump --locks
ip	Show list of IP to session maps	sessiondump --ip
ntlm	Show list of NTLM credentials to session maps	sessiondump --ntlm

Arguments

Name	Description	Example
sid	Show all session variables for a session	sessiondump --sid 75d4d5c7 where 75d4d5c7 is the session key for a session
delete	Delete all or a specific session	sessiondump --delete all sessiondump --delete 75d4d5c7 where 75d4d5c7 is the session key for a session
lockdelete	Delete all or a specific session lock	sessiondump --lockdelete 75d4d5c8 where 75d4d5c8 is the session lock for a session

Flags

Name	Description	Example
savetofile	Save all results to a file	sessiondump --allkeys --savetofile
json	Show in json format	sessiondump --json

How do I send a sessiondump command from BIG-IP Next Central Manager?¶

To send a sessiondump command to BIG-IP Next Access, you can use an API call.

Authenticate with the BIG-IP Next Central Manager API. For details refer to How to: Authenticate with the BIG-IP Next Central Manager API.
Send the command by sending a Post to the /api/apm/v1/sessions/{{mbiq_instance_id}}/{{mbiq_session_id}}/variable endpoint.
```
POST https://{{big-ip_next_cm_mgmt_ip}}/api/apm/v1/sessions/{{mbiq_instance_id}}/{{mbiq_session_id}}/variable
```

SAML SP policy - Error Messages¶

Error message: SAML SSO: Error failed to base64 decode message¶

BIG-IP Next Access as SAML SP or IdP URL decodes POST binding SAML messages, and base64 decodes them. If Access as SAML SP or IdP receives a SAML message that is not URL encoded, then it will still URL decode the message and attempt to base64 decode, which may lead to failure in processing the SAML message and give the following error in Access logs.

SAML SSO: Error failed to base64 decode message:

Resolution

This is expected behavior.

Note: SAML messages can be found in Access logs and session variables. A good base64 SAML message contains a large block of ASCII characters. If the SAML message contains a space, then it indicates the SAML request message was not URL encoded.

Error message: SAML assertion is invalid, error: Recipient should match acs url of BIG-IP (as SP)¶

When attempting SP initiated SAML, where the BIG-IP Next is the Service Provider, you may get the following error message:

SAML assertion is invalid, error: Recipient should match acs url of BIG-IP (as SP)

Cause

There are multiple reasons this error can be generated:

EntityID in the SP configuration is set to use http:// instead of https:// to build out the ACS URL of the SAMLRequest that is sent to the Identity Provider.
The Host value also has a protocol, such as placing http:// or https:// in front of the hostname.

Resolution

Update SP configuration to use the correct https:// scheme.
Ensure there is no protocol included in the host field.

Error Message: SAML SSO: Error creating signed SAML message - failed to find certificate¶

BIG-IP Next as the SP fails to generate signed SAML requests/responses or SAML Single Logout (SLO) requests/responses after the signing certificate is modified or re-imported. SAML services are impacted when this happens, and the resources that need SAML authentication cannot be accessed. Subsequent SAML/SAML SLO requests/responses fail with the following error:

SAML SSO: Error creating signed SAML message - failed to find certificate

Cause

The certificate used for signing SAML requests/responses or SAML SLO requests/responses is modified or re-imported.

Resolution

Delete and re-create the Access configuration.

Kerberos SSO¶

What is Kerberos constrained delegation?¶

Kerberos delegation is a Microsoft feature that allows an application to reuse end-user credentials to access resources hosted on a different server. Kerberos constrained delegation is a form of specifying applications that are allowed to reuse the user credentials. A server is configured to pass along a client’s identity and credentials to a secondary resource server, accessing a resource or service on behalf of the client while limiting the scope of where application services can act on the client’s behalf.

For more information, refer to the Microsoft Kerberos Constrained Delegation Overview document.

Setting up a delegation account to support Kerberos SSO

Before you can configure Kerberos SSO in Access, you must create a delegation account in Active Directory. For every server realm, you must create a delegation account in that realm.

Note: For up-to-date instructions on configuring the Kerberos delegation for your account, refer to the Microsoft documentation.

Open the Active Directory Users and Computers administrative tool and create a new user account.

The user account should be dedicated for delegation. Ensure that you have enabled the account user with change password and set the password to never expires.
Set the service principal name (SPN) on the Windows server for the user account.

For the support tools that you can use, and for the commands, such as setspn and ktpass, refer to Microsoft documentation.

Note: If you use the ktpass command, it sets the SPN on the Windows server and creates a keytab file. Kerberos SSO for Access does not need or use a keytab file.
Verify the result of setting the SPN.

This example is purely for illustration. Refer to Microsoft documentation for up-to-date commands and correct usage.

C:\Users\Administrator>setspn-Lacc

Registered ServicePrincipalNames for CN=acc,OU=users,DC=loc,DC=lab,DC=dnet,DC=com: HTTP/acc.loc.lab.dnet.com where acc is the name of the user account that you created.
Return to the Active Directory Users and Computers screen to open your account again.

A Delegation tab should appear.
Select Trust this user for delegation to specified services only.
Select Use any authentication protocol, and add all your services to the list under Services to which this account can present delegated credentials.

Every service should have Service Type HTTP (or http) and the host name of the pool member or web application resource host that you will use in your configuration.
Click OK.

This creates the new delegation account.

DNS server

You must configure the BIG-IP Next with a DNS server that contains the proper Kerberos SRV records. SRV records are in the form of _kerberos._udp.<DOMAIN> or _kerberos._tcp.<DOMAIN>. These DNS entries are queried when the KDC field (Key Distribution Center) from the Kerberos SSO profile is empty and the dns_lookup_kdc value in the /etc/krb5.conf file is set to true. This allows DNS to know which KDC handles the authentication for a given realm. Without these entries, SSO might fail.

If multiple entries exist for a service, UDP is used by default. In the event that the response contains too many KDC records (more than 512 bytes), DNS will switch to TCP on port 53 to request the same entries.

Required ports

The following ports are used in Kerberos and must be allowed for proper communication and authentication.

Port	Protocol	IP(s)
53	UDP/TCP	DNS Server
88	UDP/TCP	KDC(s)
123	UDP	NTP server(s)
<Web Server port(s)>	TCP	Web server(s)

/etc/krb5.conf file

Example /etc/krb5.conf file:

[libdefaults]
 default_realm = EXAMPLE.COM
 dns_lookup_realm = false
 dns_lookup_kdc = true
 ticket_lifetime = 24h
 forwardable = yes
 
[realms]
 EXAMPLE = {
   kdc = 10.154.144.94
# kdc = add as many kdc entries a you like 
 }
 EMEA.EXAMPLE.COM = {
   kdc = 10.154.144.92
 }
 EXAMPLEDEMO.COM = {
   kdc = dc2016.exampledemo.com:88
 }

Manually modifying the krb5.conf file allows you to define lists of KDCs for realms. However, configuring more than one KDC does not provide a load-balancing mechanism. Only the first entry will be used unless it is not available. Supported formats for supplying KDCs are fqdn|IP(:port).

For example:

kdc = dc1.us.examplekdc.com kdc = 10.152.9.5 kdc = 10.152.25.5:88

Note: F5 does not recommend modifying the krb5.conf file, unless you have no other options to accomplish your intended outcome. Back up working files to a separate location before making changes.

Refer to the https://support.f5.com/csp/article/K59350434 article for additional information on troubleshooting issues with Kerberos SSO constrained delegation.

Kerberos SSO - Error Messages¶

Error Message: Client not found in Kerberos database¶

When you give an incorrect user name during the authentication process, the following error message is displayed in Access logs:

Client not found in Kerberos database, principal name: <username>@<domain>. Please verify Active Directory and DNS configuration

Cause

Possible AD items that may cause Access to report this error include the following:

Incorrect user name.
Incorrect configuration of constrained delegation.
Using multiple delegation accounts for the same realm.
Incorrect configuration of AD delegation account.

Resolution

Use the correct user name and ensure that the constrained delegation is correctly configured for the account. If you are still unable to authenticate, contact your Active Directory administrator to resolve the issue.