Configure Token Counting and Enforcement

Overview

When you route AI inference requests through F5 BIG-IP Next for Kubernetes, you can track how many tokensearch user consumes and set limits to control usage. Token counting and enforcement lets you:

• Count input and output tokens per user, per model, and across all inference requests.

• Identify users by API key ID or source IP address.

• Set rate limits to cap token usage within a time window.

• Limit concurrent active sessions per user.

• Report usage data through High Speed Logging (HSL) to external metering and billing tools such as OpenMeter.

Token counting operates at the gateway level. You configure this feature through infrastructure annotations on the Gateway custom resource (CR), following the same pattern as AI semantic caching.

Before you begin

• You have a running F5 BIG-IP Next for Kubernetes deployment with AI Gateway configured.

• You have access to modify Gateway CR manifests in your Kubernetes cluster.

• The Distributed Shared State Manager (DSSM) pod is deployed in your cluster for multi-pod environments.

• If you plan to use HSL reporting, you have a logging pool configured.

How Token Counting Works

Understanding the request and response flow helps you plan your rate limit strategy and troubleshoot issues.

User Identification

The system identifies users based on the user_id_source configuration, which supports the following modes:

• IP address (ip): The client IP address is used directly as the user identity.

• Header (header): The raw value of the request header specified by user_id_header is used as the user identity. If the header is missing or empty and fallback_to_ip is set to true, the client IP address is used as a fallback.

• API key (api_key): The user identity is extracted from the request header specified by user_id_header. If user_id_header is set to Authorization, the Bearer token is parsed from the Authorization: Bearer <token> format. For all other headers (for example, x-api-key), the raw header value is used directly. If the resolved value is empty and fallback_to_ip is set to true, the client IP address is used as a fallback.

Note: If the user identity is empty and the value of fallback_to_ip is set as false, the request is rejected with the 401 user_identity_missing error.

Request Path

1. The client sends an inference request to the BIG-IP Next for Kubernetes virtual server.

2. The system extracts the API key ID, or the client IP address, and the raw value of the request header from the configured header. If no API key is found, it falls back to the client source IP address (if the fallback_to_ip is set as true).

3. The system queries the DSSM pod for the user’s current token count.

4. If the user has exceeded their rate limit, the system immediately returns a 429 Too Many Requests response if the action is set as drop. If the action is set as log_only, the system logs and allows request.

5. If the user is within their limit, the request is forwarded to the backend large language model (LLM).

Response Path — Non-streaming

1. The system parses the JSON response to extract usage.prompt_tokens, usage.completion_tokens, and usage.total_tokens.

2. The system updates the DSSM pod with the new token counts (user total, user-model, and global counters).

3. The system generates an HSL log entry with the usage data.

4. The response is returned to the client.

Response Path — Streaming

1. The system processes Server-Sent Events (SSE) chunks and accumulates the content.

2. The system extracts the final tokens count from the terminal chunk using the finish_reason attribute(for example, stop/length/content_filter, tool_calls, function_call), and then checks if the usage.prompt_tokens and usage.completion_tokens are configured. If the usage object is not configured, the estimate is used.

3. After the stream completes, the system updates the DSSM pod and generates an HSL log entry.

Rate Limiting Policies

You can apply rate limits at different levels:

• Per user: Sets a single token limit that applies to all of a user’s requests, regardless of model.

• Per model: Sets token limits based on the model used, regardless of user.

• Per user, per model: Sets different token limits for different models on a per-user basis. Use wildcard patterns to match model names (for example, gpt-4*).

• Global: Sets a single token limit shared across all users.

Enforcement Actions

When a user exceeds their token limit, the system takes the action you configure:

• drop: Returns a 429 Too Many Requests response and rejects the request.

• log_only: Logs the violation but allows the request to proceed.

Concurrent Session Limiting

You can set a maximum number of active sessions per user. This prevents a single user from consuming too many gateway resources at once.

Configure Basic Token Counting

To monitor token usage without enforcing limits:

1. Open your Gateway CR manifest.

2. Add the k8s.f5.com/ai-token-counting annotation under spec.infrastructure.annotations.

3. Set the following parameters:

   • Set token_counting to enabled.

   • Set user_id_source to api_key.

   • Set user_id_header to the header that contains the API key (for example, Authorization).

   • Set fallback_to_ip to true to fall back to the source IP when no key is found.

   • Set hsl_publisher to the name of your logging pool.

4. Apply the manifest to your cluster.

Example:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: ai-gateway
  namespace: nim
spec:
  gatewayClassName: f5-gateway-class
  listeners:
    - name: http
      protocol: HTTP
      port: 8000
  infrastructure:
    annotations:
      k8s.f5.com/ai-token-counting: |
        token_counting=enabled,
        user_id_source=api_key,
        user_id_header=Authorization,
        fallback_to_ip=true,
        hsl_publisher=hsl-logging-pool

This configuration enables token counting for all inference requests — including streaming, non-streaming, and batch — and sends usage data to the hsl-logging-pool logging pool.

Configure Token Counting with Rate Limiting

• Enable enforcement

• Set:

  • rate_limit_policy=per_user

  • rate_limit_tokens

  • rate_limit_window

  • rate_limit_action

Example:

infrastructure:
  annotations:
    k8s.f5.com/ai-token-counting: |
      token_counting=enabled,
      user_id_source=api_key,
      user_id_header=x-api-key,
      enforcement=enabled,
      rate_limit_policy=per_user,
      rate_limit_tokens=100000,
      rate_limit_window=3600,
      rate_limit_action=drop

Configure Token Counting with Rate Limiting

Example:

infrastructure:
  annotations:
    k8s.f5.com/ai-token-counting: |
      token_counting=enabled,
      user_id_source=api_key,
      enforcement=enabled,
      rate_limit_policy=per_user_per_model,
      rate_limits=[
        {"model":"gpt-4*","tokens":10000,"window":3600},
        {"model":"gpt-3.5*","tokens":100000,"window":3600},
        {"model":"*","tokens":500000,"window":86400}
      ],
      concurrent_sessions_limit=5,
      rate_limit_action=drop,
      hsl_publisher=usage-logging-pool

This configuration: • Limits requests to models matching gpt-4* to 10,000 tokens per hour. • Limits requests to models matching gpt-3.5* to 100,000 tokens per hour. • Sets a catch-all limit of 500,000 tokens per day for all other models. • Limits each user to five concurrent active sessions. • Returns a 429 Too Many Requests response when any limit is exceeded. • Sends usage data to the usage-logging-pool logging pool.

Use wildcard patterns in the model field to match model name prefixes. The * entry acts as a default for any model that doesn’t match a more specific rule.

Configure the Annotation Parameters

Use the following parameters in the k8s.f5.com/ai-token-counting annotation to customize token counting and enforcement for your environment.

Required parameters:

Set these parameters in every token counting configuration:

• token_counting (string): Set to enabled to turn on token counting, or disabled to turn it off. The default value is disabled.

Optional parameters — user identification

• user_id_source (string): Choose how the system identifies users. Set to one of:

  • api_key — Extracts the user identity from a request header. This is the default value.

  • ip — Uses the source IP address.

  • header — Uses a custom header value.

• user_id_header (string): Set to the name of the request header that contains the API key. For example, Authorization or x-api-key. Default: Authorization.

• fallback_to_ip (boolean): Set to true to fall back to the source IP address when no API key is found. Default: true.

Optional parameters — enforcement

• enforcement (string)**: Set to enabled to turn on rate limit enforcement, or disabled to turn it off. The default value is disabled.

• rate_limit_policy (string): Choose a rate limiting strategy. Set to one of:

  • per_user — Applies limits per user across all models.

  • per_model — Applies limits per model across all users.

  • per_user_per_model — Applies limits per user for each model separately.

  • global — Applies a single limit shared across all users.

• rate_limit_tokens (integer)**: Set the total limit of the number of tokens allowed in the rate limit window.

• rate_limit_input_tokens (integer): Set the maximum number of input tokens allowed in the rate limit window.

• rate_limit_output_tokens (integer): Set the maximum number of output tokens allowed in the rate limit window.

• rate_limit_window (integer): Set the time window for rate limiting, in seconds. Default: 3600.

• rate_limits (array): Define model-specific limits. Each object in the array requires:

  • model — A model name or wildcard pattern.

  • tokens — The maximum token count.

  • in — The maximum input token count.

  • out — The maximum output token count.

  • window — The time window in seconds.

• concurrent_sessions_limit (integer): Set the maximum number of concurrent active sessions per user. If the concurrent_sessions_limit is set to 0, there are no limits.

• rate_limit_action (string): Choose the action the system takes when a limit is exceeded. Set to one of:

  • drop — Returns a 429 Too Many Requests response.

  • log_only — Logs the violation but allows the request.

• timeout (integer): Set the entry idle timeout for the non-rate-limit state. The default value is set to 86400 seconds.

• lifetime (integer): Set the absolute lifetime for the non-rate-limit state. The default value is set to 86400 seconds.

Optional parameters — logging and estimation

• hsl_publisher (string): The value must be the pool object id created by the F5BigLogHslpub CR, following the format: {F5BigLogHslpub namespace}-{F5BigLogHslpub name}-{spec.syslog[].pool}-pool.

• token_source (string): Choose where token counts are sourced. Set to one of:

  • response — Uses the token counts from the backend response.

  • estimate — Estimates token counts based on character length. The token estimation logic uses a character-class-based split strategy, distinguishing between ASCII and non-ASCII (UTF) characters, each with different token ratios.

    • ASCII/English characters use the token_estimate_ratio (default: 4.0 characters ≈ 1 token).

    • Non-ASCII/UTF characters apply a tighter ratio — effectively half the characters-per-token of ASCII — resulting in a higher token count for the same string length.

    • Whitespace is treated as a separator and is handled independently. After the initial estimate is calculated, the gateway applies a 25% discount to both the estimated prompt and completion token counts, meaning only ~75% of the raw estimate is used as the final value. This estimation logic is applied consistently across both non-streaming and streaming flows, whenever the estimation path is selected or a fallback is required.

  • auto — Uses response counts when available, falls back to estimation. Default: auto.

• token_estimate_ratio (float): Set the number of characters per token used for estimation. Default: 4.0.