Integrating NVIDIA NIM Metrics with Prometheus for GPU Workload Monitoring

This guide explains how to collect and monitor GPU metrics from NVIDIA NIM (NVIDIA Inference Microservices) using Prometheus in Kubernetes environments.

Overview

NIM metrics are embedded within NIM inference containers, providing application-level observability for Large Language Model (LLM) workloads.

Prerequisites

• Kubernetes cluster with GPU nodes.

• NVIDIA GPUs with compute capability 6.0 or higher.

• NIM-based LLM containers deployed.

Prometheus Configuration

Note: The code snippets below are examples and may need adjustments to match your architecture.

Option 1: With Prometheus Operator (Recommended) - ServiceMonitor

If Prometheus Operator is installed, use a ServiceMonitor for automatic service discovery.

NIM Metrics Collection

Endpoint: NIM containers expose metrics on port 8000 at /metrics (default configuration). Since NIM containers already include built-in observability, configure Prometheus to scrape your existing NIM services:

Note: Prometheus 2.53+ defaults to /metrics. Earlier versions used /v1/metrics.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nim-metrics
  namespace: your-nim-namespace
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: nim-llm  # Replace with your NIM service labels
  endpoints:
  - port: http-openai  # default port name your NIM service uses
    interval: 30s
    path: /metrics

Option 2: Without Prometheus Operator - Native Kubernetes Service Discovery

If Prometheus Operator is not installed, configure native Prometheus with Kubernetes service discovery:

NIM Metrics Collection

scrape_configs:
  # NIM Services Discovery  
  - job_name: 'nim-metrics'
    kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names: [nim]  # Replace with your NIM namespace
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
      action: keep
      regex: nim-llm # This should match a common label type among all your NIM LLMs
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      action: keep
      regex: http-openai

Note: You can optionally add target_label and other source labels to enrich NIM metrics for Prometheus.

---

HTTPS Configuration

Certificate Management with cert-manager

For production environments requiring secure HTTPS connections, use cert-manager to automate certificate lifecycle management. This eliminates manual handling and provides automatic certificate renewal.

Prerequisites

• cert-manager installed in your cluster.

• ClusterIssuer configured.

• Certificate authority (CA) set up.

• TLS certificate requirements defined

NIM HTTP Overview

• By default, NIM serves API endpoints (including metrics) over HTTP on port 8000.

• Built-in SSL/TLS support can be configured via environment variables.

NIM supports three SSL modes through environment variables:

• DISABLED (default): HTTP only (default)

• TLS: HTTPS with server-side TLS only

• MTLS: HTTPS with mutual TLS (client certificate required)

1. Create a NIM cert using cert-manager and mount certificate/secret in NIM deployment.

2. Add these environment variables to NIM deployment to enable HTTPS. See documentation for more details: Configuring NIM

env:
  # Enable TLS mode
  - name: NIM_SSL_MODE
    value: "TLS"
  # Path to server certificate
  - name: NIM_SSL_CERTS_PATH
    value: "/etc/nim-certs/tls.crt"
  # Path to server private key
  - name: NIM_SSL_KEY_PATH
    value: "/etc/nim-certs/tls.key"

Client-Side Prometheus HTTPS Configuration

1. Mount the CA certificate in the Prometheus deployment to validate the server certificate.

Example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
spec:
  template:
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus:latest
        volumeMounts:
        - name: prometheus-config
          mountPath: /etc/prometheus
        # Mount CA certificates for HTTPS validation
        - name: ca-certificates
          mountPath: /etc/ssl/certs/prometheus
          readOnly: true
      volumes:
      - name: prometheus-config
        configMap:
          name: prometheus-config
      # CA bundle for validating server certificates
      - name: ca-certificates
        secret:
          secretName: prometheus-ca-bundle

---

Key Metrics

NIM Metrics

NIM containers provide application-level metrics including:

• Request latency and throughput

• Queue depth and processing time

• Model-specific performance metrics

• Token generation rates

Complete NIM Metrics Reference: NVIDIA NIM Observability Documentation

---

Verification

Check NIM Metrics:

# Port forward to your NIM service
kubectl port-forward -n your-nim-namespace service/your-nim-service 8000:8000

# Check metrics
curl http://localhost:8000/metrics

---

Troubleshooting

NIM Issues:

• No metrics endpoint: Verify NIM container configuration includes metrics exposure.

• Connection refused: Ensure NIM service is exposed correctly and port is configured..

• Missing metrics: Verify NIM container version supports observability features.

---

Additional Resources

• NVIDIA GPU Operator Documentation

• NIM Observability Guide

• Prometheus Configuration