Manage statistics with tenant-aware configuration¶

Overview¶

F5 BIG-IP Next for Kubernetes supports tenant-aware statistics management for both single-tenant and multi-tenant Kubernetes deployments through a set of Kubernetes Custom Resources (CRs). This feature lets each tenant or namespace independently control which metrics they collect, how often they collect them, and how they label and export the data — all without affecting other tenants in the same cluster.

TMM instances generate metrics organized into tables. Each table contains metrics and labels. Labels identify the entity being measured, such as a virtual server’s name or destination address. When multiple TMM instances report the same metric with same labels, the system combines them into a single exported value instead of sending a separate data point for each TMM.

For more details on metrics and tables, see OpenTelemetry Metric Details

In this document, tenant means the Kubernetes namespace where the traffic configuration CRs and metric collection CRs are applied.

Why you need this feature¶

Tenant-aware statistics management gives each tenant independent control over their monitoring configuration. This allows,

Independent-interval metric scraping
Selective metric collection and export
Custom metric labeling and relabeling
Configurable metric aggregation modes (Aggregated, SemiAggregated, Diagnostic)

Note

In single-tenant deployments, these features can simplify monitoring by separating export policies, scrape intervals, and aggregation behavior for different workloads or namespaces.

Workflow¶

alt text

The telemetry pipeline uses a gRPC-based flow to collect and export data efficiently. The Operator watches for Scrape custom resources and schedules collection jobs when they are valid. The Observer then sends gRPC requests to each target receiver and collects the data. To save resources, the Operator merges overlapping intervals. For example, if two tenants request data from the same table at two-second and three-second intervals, the Operator schedules a single scrape at the shortest interval. The Observer filters the results and sends them to each tenant’s endpoint at the requested frequency. The pipeline then exports the processed data to OpenTelemetry (OTel).

Use cases¶

Use case	Description
Namespace or Tenant-specific metric collection	Each tenant selects only the metrics that matter to their workloads, reducing noise and resource usage.
Custom scrape intervals per workload or tenant	Different tenants can collect the same metric at different frequencies — for example, one tenant every 10 seconds and another every 60 seconds.
Aggregation mode selection	Tenants choose how metrics from multiple Traffic Management Microkernel (TMM) instances are combined: `Aggregated`, `SemiAggregated`, or `Diagnostic`.
Label customization	Tenants can relabel or extract new labels from existing metrics to match their own indexing and dashboarding requirements.
Namespace-based isolation and workload separation	Each tenant operates in their own Kubernetes namespace. Configuration of namespace cannot interfere with another.
Administrator-controlled policies	Cluster administrators define which namespaces and metrics each tenant can access, enforcing security and governance boundaries.
Narrowing export volume	Tenants can limit jobs to a subset of metrics within a table using `metricNames`, reducing OpenTelemetry (OTel) export volume.
Single-tenant workload separation	Use the same CR framework to isolate monitoring policies for different applications or teams within a single cluster.

Understand tenant-aware statistics management¶

The statistics infrastructure uses three custom resource kinds — ScrapeClass, ScrapeTemplate, and Scrape — to give administrators policy-level control while granting tenants the flexibility to define their own monitoring workflows.

The three custom resources¶

Three Custom Resource kinds work together to manage tenant-aware statistics:

Resource	API version	Scope	Purpose
ScrapeClass	`metrics.f5.net/v1alpha1`	Cluster	Defines which namespaces or instances a `scrape` can target. Referenced by both `ScrapeTemplate` and `Scrape`.
ScrapeTemplate	`metrics.f5.net/v1alpha1`	Namespaced	Declares which tables to export, at what interval, and with what aggregation mode Auto-generates a `Scrape` CR when applied.
Scrape	`metrics.f5.net/v1alpha1`	Namespaced	Holds the PromQL queries that define what metrics to export. Exposes a per-job lifecycle and status.

Default behavior¶

A ScrapeClass named default-scrape-class and a ScrapeTemplate named default-scrape-template are pre-installed with the product. Together they export all available tables once per minute with aggregation enabled.

Note

The pre-installed default-scrape-class is not annotated with scrapeclass.io.observer/is-default-class: "true". It will not be automatically resolved when a Scrape or ScrapeTemplate omits the scrapeClass field.

CR parameters¶

ScrapeClass CR fields¶

Field	Type	Required	Default	Description
`spec.scope`	`Cluster` \| `Namespaced`	No	`Cluster`	Controls whether this ScrapeClass allows Scrape CRs to export metrics cluster-wide from all namespaces, or only from the namespace where the `Scrape` is applied. See Scope.
`spec.allowedNamespaces`	`[]string`	No	—	A list of namespaces permitted to reference this ScrapeClass. Scrapes or ScrapeTemplates applied outside this list will not be scheduled.
`spec.matchers`	`[]Matcher`	No	—	Defines global label filtering rules that are automatically applied to every scrape/job referencing this ScrapeClass. See Matchers.
`spec.controllerNamespace`	`string`	No	—	Restricts this ScrapeClass to a specific Observer controller (Operator) namespace. In multi-deployment setups with multiple Operators, only the Operator in the specified namespace will process Scrapes referencing this class. If not set, any Operator can use this ScrapeClass.

Scope¶

The scope field controls where a scrape job collects metrics from—either across all tenant namespaces in the cluster or only within the namespace where the Scrape resource is defined. This is especially important in multi-tenant environments to ensure proper data isolation.

Value	Behavior
Cluster (default)	Scrape jobs collect metrics from all namespaces in the cluster. Recommended for single-tenant deployments or when global visibility is required.
Namespaced	Scrape jobs collect metrics only from the namespace where the `Scrape` or `ScrapeTemplate` is defined. Recommended for multi-tenant environments to restrict tenant visibility.

Note

Some TMM metric tables (for example, tmm_stat) represent system-wide data and are not associated with any tenant namespace. Metrics such as f5.tmm.cpu_utilization.1min apply to the entire TMM instance. These metrics require Cluster scope, because a Namespaced scrape cannot match data without a namespace association.

Matchers¶

Matchers provide additional filtering by applying label-based conditions to scrape jobs. They help control which TMM instances or metric streams are included when a ScrapeClass is used.

Field	Description
name	Label to evaluate (for example, `__bucket_name__`, `__bucket_namespace__`, or `__bucket_group__`).
operator	Type of comparison applied to the label value.
value	Value to compare against. For `Regexp` and `NotRegexp`, this is a regular expression.

Supported Operators

Operator	Description
Equal	Matches when the label value equals the specified value.
NotEqual	Matches when the label value does not equal the specified value.
Regexp	Matches when the label value matches the regular expression.
NotRegexp	Matches when the label value does not match the regular expression.

For configuration see Configure metrics For examples see CR Samples

ScrapeTemplate CR fields¶

Field	Type	Required	Default	Description
`spec.scrapeClass`	`string`	No	(default class)	Name of the ScrapeClass to use. If you omit this field, the system uses the default ScrapeClass. Note: You can define the default `ScrapeClass` by adding the annotation `scrapeclass.io.observer/is-default-class: "true"` to a ScrapeClass.
`spec.jobs`	`[]JobTemplate`	Yes	—	List of scrape jobs. You must include at least one job.
`spec.jobs[].name`	`string`	Yes	—	A unique name for the scrape job.
`spec.jobs[].table`	`string`	Yes	—	The descriptor table name to scrape (for example, `virtual_server_stat`).
`spec.jobs[].every`	`duration`	Yes	—	Defines how often to run the scrape job (for example, `10s`, `30s`, `1m`, `5m`).
`spec.jobs[].metricNames`	`[]string`	No	(all metrics)	Limits the job to a subset of metrics within the table. Use alias names from the descriptor.
`spec.jobs[].aggregation`	`string`	No	`Aggregated`	Controls how metrics from multiple TMM instances are combined: `Aggregated`, `SemiAggregated`, or `Diagnostic`. See Aggregation modes
`spec.jobs[].scrapeInterval`	`duration`	No	same value specified in `every`	Defines how often the receiver polls TMM for new metrics. For shorter export intervals, set this slightly lower than `every` (for example, `scrapeInterval: 4.5s` with `every: 5s`) to ensure each export cycle uses the most recent data.
`spec.suspend`	`bool`	No	`false`	When `true`, the generated Scrape is suspended and no metrics are exported.

Note

The minimum recommended value for every and scrapeInterval is 3s. Setting below this threshold may cause high resource usage or unstable behavior.

Aggregation modes¶

When multiple TMM instances report the same metric with identical label values, the system combines them before exporting. The aggregation field on a ScrapeTemplate job controls how that combination works.

Mode	Behavior
`Aggregated` (default)	Metrics are summed across all TMM instances, including those that have been removed.
`SemiAggregated`	Metrics are summed across currently active TMM instances only. Removed instances are excluded.
`Diagnostic`	No aggregation. Each TMM instance’s metrics are exported individually.

For configuration see Configure metrics For examples see CR Samples

Scrape CR fields¶

Note

ScrapeTemplate auto-generates a Scrape CR with default queries to export metrics. To override template and customize the PromQL queries, create a Scrape CR.

Field	Type	Required	Default	Description
`spec.scrapeClass`	`string`	No	(default class)	Name of the ScrapeClass to use. If you omit this field, the system uses the default ScrapeClass.
`spec.jobs`	`[]Job`	Yes	—	List of scrape jobs. Each job includes a name, interval, and PromQL query.
`spec.jobs[].name`	`string`	Yes	—	A unique name for the scrape job.
`spec.jobs[].every`	`duration`	Yes	—	Defines how often to run the scrape job (for example, `10s`, `30s`, `1m`).
`spec.jobs[].query`	`string`	Yes	—	The PromQL query that defines which metrics to collect.
`spec.jobs[].scrapeInterval`	`duration`	No	same value specified in `every`	Defines how often the receiver polls TMM for new metrics. For shorter export intervals, set this slightly lower than `every` (for example, `scrapeInterval: 4.5s` with `every: 5s`) to ensure each export cycle uses the most recent data.
`spec.suspend`	`bool`	No	`false`	When `true`, pauses all jobs and removes their scheduled executions.

Note

The minimum recommended value for every and scrapeInterval is 3s. Setting below this threshold may cause high resource usage or unstable behavior.

Understanding PromQL queries¶

The query field in a Scrape job defines a PromQL expression that selects which metrics to export. PromQL (Prometheus Query Language) is used to filter, transform, and aggregate time-series metrics.

When a Scrape is created from a ScrapeTemplate, the query is automatically generated based on:
- Table name
- Aggregation mode
- Optional metric filters
When creating a Scrape manually, you must define the query yourself.

The system validates the query at scheduling time. If the query is invalid, the Scrape moves to a Failed phase with a QueryError reason.

Note

The Observer runs queries internally using PromQL-compatible logic.
Only instant queries are supported. Range queries and historical metrics are not supported.

Basic queries¶

Basic queries use the __name__ label to match metrics by name. These are useful when aggregation and label transformations are not required. You can use basic queries for,

Simple metric selection
Debugging
Lightweight exports

Advanced queries with aliases and internal labels¶

Aliases Metrics and labels use standardized alias names (dot‑separated and table‑prefixed), which are used directly in queries, where the metric alias is the __name__ value and label aliases appear as regular labels.

Examples:

# Select all metrics matching a pattern
query: "{__name__=~\"f5.virtual_server.*\"}"

# Select a specific metric
query: "{__name__=\"f5.virtual_server.clientside.received.bytes\"}"

Note

Always use alias names exactly as defined in the tables reference.

Internal labels Internal labels are automatically attached during metric processing. These labels start with __ and are removed from exported metrics unless explicitly renamed.

Label	Description
`__name__`	Metric alias name
`__fqn__`	Fully qualified metric identifier
`__table__`	Source metrics table
`__namespace__`	Kubernetes namespace of configuration
`__bucket_namespace__`	Namespace of the TMM instance
`__bucket_name__`	Name of the TMM instance
`__bucket_group__`	Group of the TMM instance

Custom functions¶

The system extends standard PromQL with custom functions to help you combine, modify, and enrich metric data during queries.

Use these when you need to:

Combine multiple metrics
Rename labels for export or standardisation
Add metadata to metrics

Functions

Function	Syntax	Description
coalesce	coalesce(expr1, expr2, …)	Combines all samples from input vectors into a single result set to enable querying multiple metrics from the same table in one query.
label_move	label_move(v, src1, dst1, …)	Renames existing labels, often used to expose or standardise internal label names.
label_add	label_add(v, key1, val1, …)	Adds constant labels to all samples in the metric vector.

Aggregation and Grouping¶

Aggregation and grouping help you summarise and organise metrics effectively when writing queries. By selecting the right labels and controls, you can ensure accurate results and avoid mixing unrelated data.

Best Practices

Topic	Details
Include required labels	Always include essential labels such as `__namespace__` and `__bucket_namespace__`. Add other relevant label aliases as needed.
Use `__fqn__` when necessary	Include `__fqn__` in grouping when metrics share the same alias but differ by additional labels.
Control aggregation scope	To exclude data from removed or merged TMM instances, use: `text __bucket_name__ != "merged"`
Use aggregation operators	Common operators include `sum`, `min`, and `max` to calculate totals or identify minimum and maximum values.

PromQL query examples¶

Aggregated — sum across all TMM instances including removed ones

This example demonstrates adding a custom label to the exported metrics.

query: |-
  label_add(
    label_move(
      coalesce(
        sum by (__name__, __namespace__, __bucket_namespace__,
          "f5.virtual_server.destination", "f5.virtual_server.name", "f5.virtual_server.source") (
          {__name__="f5.virtual_server.clientside.received.bytes", __table__="virtual_server_stat"}
        ),
        sum by (__name__, __namespace__, __bucket_namespace__,
          "f5.virtual_server.destination", "f5.virtual_server.name", "f5.virtual_server.source") (
          {__name__="f5.virtual_server.clientside.transmitted.bytes", __table__="virtual_server_stat"}
        )
      ),
      "__namespace__", "k8s.namespace.name",
      "__bucket_name__", "target.name",
      "__bucket_namespace__", "target.namespace"
    ),
    "my.custom.label", "my-value"
  )

Diagnostic — raw per-TMM, no aggregation

This example for TMM instance exports its metrics as separate data points. The target.name field identifies the specific TMM instance that generated each metric sample.

query: |-
  label_move(
    coalesce(
      {__name__="f5.virtual_server.clientside.received.bytes", __table__="virtual_server_stat"},
      {__name__="f5.virtual_server.clientside.transmitted.bytes", __table__="virtual_server_stat"}
    ),
    "__namespace__", "k8s.namespace.name",
    "__bucket_name__", "target.name",
    "__bucket_namespace__", "target.namespace"
  )

Configure metrics¶

Create a ScrapeClass. A ScrapeClass defines the access policy for a tenant. Set scope: Namespaced to restrict tenants to metrics in their own namespace.

Note

You can define the default ScrapeClass by adding the annotation scrapeclass.io.observer/is-default-class: "true". When a Scrape or ScrapeTemplate omits the scrapeClass field, the default ScrapeClass is resolved automatically. Only one default ScrapeClass may exist at a time.

Example
```
apiVersion: metrics.f5.net/v1alpha1
kind: ScrapeClass
metadata:
  name: tenant-a-class
spec:
  scope: Namespaced
  allowedNamespaces:
    - tenant-a
```

Apply a ScrapeTemplate. A ScrapeTemplate declares which tables to export and at what interval. Apply the template in the tenant’s namespace.

Example

apiVersion: metrics.f5.net/v1alpha1
kind: ScrapeTemplate
metadata:
  name: tenant-a-template
  namespace: tenant-a
spec:
  scrapeClass: tenant-a-class
  jobs:
    - name: virtual-server-job
      table: virtual_server_stat
      every: 10s

(Optional) Create a direct Scrape if you need fine-grained control beyond what a ScrapeTemplate provides with specific PromQL queries. This overrides the template layer entirely.

Example

apiVersion: metrics.f5.net/v1alpha1
kind: Scrape
metadata:
  name: custom-scrape
  namespace: default
spec:
  scrapeClass: my-scrape-class
  jobs:
    - name: vs-received-bytes
      every: 30s
      query: "{__name__=\"f5.virtual_server.clientside.received.bytes\"}"
    - name: vs-transmitted-bytes
      every: 30s
      query: "{__name__=\"f5.virtual_server.clientside.transmitted.bytes\"}"

Verification¶

# Check the ScrapeTemplate status
$ kubectl get scrapetemplate <template-name> -n <namespace>

ScrapeTemplate status

After you apply a ScrapeTemplate, the Operator updates status to indicate whether a Scrape resource was successfully generated.

Field	Description
`status.scrapeName`	Name of the generated `Scrape` resource.
`status.conditions[].type: Ready`	Indicates whether the Scrape was generated successfully. Set to `True` on success, or `False` message on failure is included.

# Check the generated Scrape CR and its phase
$ kubectl get scrape -n <namespace>

# View detailed status of a Scrape CR
kubectl describe scrape <scrape-name> -n <namespace>

Scrape status

Field	Description
`status.phase`	Current lifecycle phase: `Pending`, `Running`, `Error`, or `Failed`.
`status.ready`	Number of ready jobs out of total (for example, `2/3`).
`status.conditions[].type: JobsScheduled`	Set to `True` once all jobs are accepted for scheduling. Set to `False` with a reason and message if scheduling is blocked.
`status.conditions[].type: Ready`	Set to `True` when all jobs are running successfully. Set to `False` with a list of unready job names otherwise.
`status.jobStatuses`	Per-job status list including job name, `ready` (boolean), and a message describing the current state or last error. When a job is ready, `ready: true` is shown with no message. When not ready, `ready` is omitted and the message contains the reason.

Lifecycle

A Scrape moves through the following phases:

Phase	Description
`Pending`	The Scrape is waiting to be scheduled. This can happen if the referenced ScrapeClass is missing, `spec.suspend` is set to `true`, or the jobs have not completed their first run. The system retries automatically every minute.
`Running`	All jobs are scheduled and have completed at least one successful run.
`Error`	Jobs are running, but one or more jobs fail during execution. The Scrape continues to retry.
`Failed`	A configuration issue prevents scheduling. The Scrape does not retry. Check `status.conditions` for details. Common causes include an invalid job query, restricted namespace access, or a configuration mismatch.

# Check the reason for `Pending`, `Error`, or `Failed`
kubectl describe scrape <name>

CR Samples¶

Cluster-scoped ScrapeClass¶

apiVersion: metrics.f5.net/v1alpha1
kind: ScrapeClass
metadata:
  name: my-scrape-class
spec:
  scope: Cluster

Restrict scraping using matchers¶

This example excludes metrics from the beta namespace while allowing all others.

apiVersion: metrics.f5.net/v1alpha1
kind: ScrapeClass
metadata:
  name: test-class
spec:
  scope: Cluster
  matchers:
    - name: "__bucket_namespace__"
      operator: "NotEqual"
      value: "beta"

Multi-tenant isolation¶

In this example, two tenants each have their own namespace, ScrapeClass, and ScrapeTemplate. Tenant A collects virtual_server_stat every minute. Tenant B collects the same table every 10 seconds.

Tenant A:

apiVersion: metrics.f5.net/v1alpha1
kind: ScrapeClass
metadata:
  name: tenant-a-class
spec:
  scope: Namespaced
  allowedNamespaces:
    - tenant-a
---
apiVersion: metrics.f5.net/v1alpha1
kind: ScrapeTemplate
metadata:
  name: tenant-a-template
  namespace: tenant-a
spec:
  scrapeClass: tenant-a-class
  jobs:
    - name: virtual-server-job
      table: virtual_server_stat
      every: 1m

Tenant B:

apiVersion: metrics.f5.net/v1alpha1
kind: ScrapeClass
metadata:
  name: tenant-b-class
spec:
  scope: Namespaced
  allowedNamespaces:
    - tenant-b
---
apiVersion: metrics.f5.net/v1alpha1
kind: ScrapeTemplate
metadata:
  name: tenant-b-template
  namespace: tenant-b
spec:
  scrapeClass: tenant-b-class
  jobs:
    - name: virtual-server-job
      table: virtual_server_stat
      every: 10s

A ScrapeTemplate applied in tenant-b cannot reference tenant-a-class. If a namespace restriction is violated, the generated Scrape enters a Failed phase.

Narrowing export to specific metrics¶

To reduce OTel export volume, use metricNames to limit a job to a subset of metrics within a table:

apiVersion: metrics.f5.net/v1alpha1
kind: ScrapeTemplate
metadata:
  name: vs-bytes-only
  namespace: default
spec:
  scrapeClass: my-scrape-class
  jobs:
    - name: vs-bytes-job
      table: virtual_server_stat
      every: 5m
      metricNames:
        - f5.virtual_server.clientside.received.bytes
        - f5.virtual_server.clientside.transmitted.bytes

Note

Adding additional Scrape/ScrapeTemplate resources increases the load on the OpenTelemetry (OTel) Collector. In deployments where the OTel Collector resources cannot be adjusted, the collector may reach its capacity and metric export can fail with the following error:

rpc error: code = Unavailable desc = sending queue is full

To avoid this issue, reduce the total number of scrape jobs across all configured Scrape/ScrapeTemplate resources. This can be achieved by removing unnecessary Scrape/ScrapeTemplate resources, reducing the number of jobs defined, or removing the default Scrape/ScrapeTemplate if it’s not required. See Exporter Helper for more information on exporter configuration.

Cluster-wide scraping¶

For system-wide observability, an administrator can create a cluster-scoped ScrapeClass:

apiVersion: metrics.f5.net/v1alpha1
kind: ScrapeClass
metadata:
  name: my-scrape-class
spec:
  scope: Cluster
---
apiVersion: metrics.f5.net/v1alpha1
kind: ScrapeTemplate
metadata:
  name: my-scrape-template
  namespace: default
spec:
  scrapeClass: my-scrape-class
  jobs:
    - name: virtual-server-job
      table: virtual_server_stat
      every: 1m

For more details see, Distributed Telemetry and Operations Data Aggregation (TODA) for statistics.