Configure token counting and enforcement

Overview

When you route AI inference requests through F5 BIG-IP Next for Kubernetes (BNK), you can track how many tokens each user consumes and set limits to control usage. Token counting and enforcement lets you:

• Count input and output tokens per user, per model, and across all inference requests.

• Identify users by API key ID or source IP address.

• Set rate limits to cap token usage within a time window.

• Limit concurrent active sessions per user.

• Report usage data through High Speed Logging (HSL) to external metering and billing tools such as OpenMeter.

Token counting operates at the gateway level. You configure this feature through infrastructure annotations on the Gateway custom resource (CR), the same way you configure AI semantic caching.

Before you begin

• BNK installed with Gateway API custom resources available in your cluster.

• Access to modify Gateway API resource manifests in your Kubernetes cluster.

• The distributed Shared State Manager (dSSM) pod deployed in your cluster for multi-pod environments.

• A logging pool configured, if you plan to use HSL reporting.

How token counting works

The request and response flow helps you plan your rate limit strategy and troubleshoot issues.

User identification

The system identifies users based on the user_id_source configuration, which supports these modes:

• IP address (ip): The system uses the client IP address directly as the user identity.

• Header (header): The system uses the raw value of the request header specified by user_id_header as the user identity. If the header is missing or empty and fallback_to_ip is set to true, the system uses the client IP address as a fallback.

• API key (api_key): The system extracts the user identity from the request header specified by user_id_header. If user_id_header is set to Authorization, the system parses the bearer token from the Authorization: Bearer <token> header. For all other headers (for example, x-api-key), the system uses the raw header value directly. If the resolved value is empty and fallback_to_ip is set to true, the system uses the client IP address as a fallback.

Note

If the user identity is empty and fallback_to_ip is set to false, the system rejects the request with the 401 user_identity_missing error.

Request path

1. The client sends an inference request to the BNK virtual server.

2. The system extracts the API key ID, the client IP address, and the raw value of the configured request header. If no API key is found, the system falls back to the client source IP address when fallback_to_ip is set to true.

3. The system queries the dSSM pod for the user’s current token count.

4. When the user exceeds their rate limit and the action is set to drop, the system returns a 429 Too Many Requests response. When the action is set to log_only, the system logs the request and allows it.

5. When the user is within their limit, the system forwards the request to the backend large language model (LLM).

Response path for non-streaming requests

1. The system parses the JSON response to extract usage.prompt_tokens, usage.completion_tokens, and usage.total_tokens.

2. The system updates the dSSM pod with the new token counts: the user total, the user-model count, and the global counters.

3. The system generates an HSL log entry with the usage data.

4. The system returns the response to the client.

Response path for streaming requests

1. The system processes Server-Sent Events (SSE) chunks and accumulates the content.

2. The system extracts the final token count from the terminal chunk using the finish_reason attribute (for example, stop, length, content_filter, tool_calls, or function_call). The system then checks whether usage.prompt_tokens and usage.completion_tokens are configured. If the usage object is not configured, the system uses the estimate.

3. After the stream completes, the system updates the dSSM pod and generates an HSL log entry.

Rate limiting policies

You can apply rate limits at different levels:

• Per user: Sets a single token limit that applies to all of a user’s requests, regardless of model.

• Per model: Sets token limits based on the model used, regardless of user.

• Per user, per model: Sets different token limits for different models for each user. Use wildcard patterns to match model names (for example, gpt-4*).

• Global: Sets a single token limit shared across all users.

Enforcement actions

When a user exceeds their token limit, the system takes the action you configure:

• drop: Returns a 429 Too Many Requests response and rejects the request.

• log_only: Logs the violation but allows the request to proceed.

Concurrent session limiting

You can set a maximum number of active sessions per user. This limit prevents a single user from consuming too many gateway resources at once.

Configure basic token counting

To monitor token usage without enforcing limits:

1. Open your Gateway CR manifest.

2. Add the k8s.f5.com/ai-token-counting annotation under spec.infrastructure.annotations.

3. Set these parameters:

   • Set token_counting to enabled.

   • Set user_id_source to api_key.

   • Set user_id_header to the header that contains the API key (for example, Authorization).

   • Set fallback_to_ip to true to fall back to the source IP address when no key is found.

   • Set hsl_publisher to the name of your logging pool.

4. Apply the manifest to your cluster.

Example:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: llm-gateway
  namespace: nim
spec:
  gatewayClassName: f5-gateway-class
  listeners:
    - name: http
      protocol: HTTP
      port: 8000
  infrastructure:
    annotations:
      k8s.f5.com/ai-token-counting: |
        token_counting=enabled,
        user_id_source=api_key,
        user_id_header=Authorization,
        fallback_to_ip=true,
        hsl_publisher=hsl-logging-pool

This configuration turns on token counting for all inference requests, including streaming, non-streaming, and batch requests. It also sends usage data to the hsl-logging-pool logging pool.

Configure token counting with per-user rate limiting

To enforce a single token limit for each user across all models:

1. Open your Gateway CR manifest.

2. Set enforcement to enabled.

3. Set rate_limit_policy to per_user.

4. Set rate_limit_tokens, rate_limit_window, and rate_limit_action to the values you want.

5. Apply the manifest to your cluster.

Example:

infrastructure:
  annotations:
    k8s.f5.com/ai-token-counting: |
      token_counting=enabled,
      user_id_source=api_key,
      user_id_header=x-api-key,
      enforcement=enabled,
      rate_limit_policy=per_user,
      rate_limit_tokens=100000,
      rate_limit_window=3600,
      rate_limit_action=drop

Configure token counting with per-model rate limiting

To enforce different token limits for different models, use the per_user_per_model policy with the rate_limits array:

1. Open your Gateway CR manifest.

2. Set enforcement to enabled.

3. Set rate_limit_policy to per_user_per_model.

4. Define per-model limits in the rate_limits array.

5. Apply the manifest to your cluster.

Example:

infrastructure:
  annotations:
    k8s.f5.com/ai-token-counting: |
      token_counting=enabled,
      user_id_source=api_key,
      enforcement=enabled,
      rate_limit_policy=per_user_per_model,
      rate_limits=[
        {"model":"gpt-4*","tokens":10000,"window":3600},
        {"model":"gpt-3.5*","tokens":100000,"window":3600},
        {"model":"*","tokens":500000,"window":86400}
      ],
      concurrent_sessions_limit=5,
      rate_limit_action=drop,
      hsl_publisher=usage-logging-pool

This configuration does the following:

• Limits requests to models matching gpt-4* to 10,000 tokens per hour.

• Limits requests to models matching gpt-3.5* to 100,000 tokens per hour.

• Sets a default limit of 500,000 tokens per day for all other models.

• Limits each user to five concurrent active sessions.

• Returns a 429 Too Many Requests response when any limit is exceeded.

• Sends usage data to the usage-logging-pool logging pool.

Use wildcard patterns in the model field to match model name prefixes. The * entry acts as a default for any model that doesn’t match a more specific rule.

Annotation parameters reference

Use these parameters in the k8s.f5.com/ai-token-counting annotation to customize token counting and enforcement for your environment.

Token counting parameters

Parameter	Default	Description
token_counting	disabled	Required. Set to enabled to turn on token counting, or disabled to turn it off.

User identification parameters

Parameter	Default	Description
user_id_source	api_key	Optional. Choose how the system identifies users. Set to api_key to extract the user identity from a request header, ip to use the source IP address, or header to use a custom header value.
user_id_header	Authorization	Optional. Set to the name of the request header that contains the API key, for example, Authorization or x-api-key.
fallback_to_ip	true	Optional. Set to true to fall back to the source IP address when no API key is found.

Enforcement parameters

Parameter	Default	Description
enforcement	disabled	Optional. Set to enabled to turn on rate limit enforcement, or disabled to turn it off.
rate_limit_policy	None	Required when enforcement is enabled. Choose a rate limiting strategy. Set to per_user to apply limits per user across all models, per_model to apply limits per model across all users, per_user_per_model to apply limits per user for each model separately, or global to apply a single limit shared across all users.
rate_limit_tokens	None	Optional. Set the total number of tokens allowed in the rate limit window.
rate_limit_input_tokens	None	Optional. Set the maximum number of input tokens allowed in the rate limit window.
rate_limit_output_tokens	None	Optional. Set the maximum number of output tokens allowed in the rate limit window.
rate_limit_window	3600	Optional. Set the time window for rate limiting, in seconds.
rate_limits	None	Required when rate_limit_policy is per_user_per_model. Define model-specific limits. Each object requires a model (name or wildcard pattern), tokens (maximum token count), in (maximum input token count), out (maximum output token count), and window (time window, in seconds).
concurrent_sessions_limit	None	Optional. Set the maximum number of concurrent active sessions per user. Set to 0 for no limit.
rate_limit_action	None	Optional. Choose the action the system takes when a limit is exceeded. Set to drop to return a 429 Too Many Requests response, or log_only to log the violation but allow the request.
timeout	86400	Optional. Set the entry idle timeout for the non-rate-limit state, in seconds.
lifetime	86400	Optional. Set the absolute lifetime for the non-rate-limit state, in seconds.

Logging and estimation parameters

Parameter	Default	Description
hsl_publisher	None	Optional. Set to the pool object ID created by the F5BigLogHslpub CR. Use this format: {F5BigLogHslpub namespace}-{F5BigLogHslpub name}-{spec.syslog[].pool}-pool.
token_source	auto	Optional. Choose where the system gets token counts. Set to response to use the token counts from the backend response, estimate to estimate token counts based on character length, or auto to use response counts when available and fall back to estimation.
token_estimate_ratio	4.0	Optional. Set the number of characters per token used for estimation.

When token_source is set to estimate, the system estimates token counts based on character length. The estimation logic splits text by character class and applies a different token ratio to each class:

• ASCII and English characters use the token_estimate_ratio value. The default is 4.0 characters per token.

• Non-ASCII and UTF characters use a tighter ratio, about half the characters per token of ASCII. The same string length produces a higher token count.

• Whitespace acts as a separator, and the system handles it independently.

After the system calculates the initial estimate, the gateway applies a 25% discount to both the estimated prompt and completion token counts. The final value is about 75% of the raw estimate. The system applies this estimation logic to both non-streaming and streaming requests whenever it uses the estimation path or a fallback is required.