Securing F5 CIS BIG-IP Credentials Using the External Secrets Operator with HashiCorp Vault

Overview

F5 BIG-IP Container Ingress Services (CIS) authenticates to the BIG-IP system using a username and password supplied through a native Kubernetes Secret (f5-bigip-ctlr-login) mounted into the controller pod via --credentials-directory. By default this Secret is authored directly as a base64-encoded manifest, which means the plaintext BIG-IP credential must either exist in version control or be applied manually — an undesirable practice for any production environment.

This guide describes how to eliminate that requirement by storing the BIG-IP credential in HashiCorp Vault and using the External Secrets Operator (ESO) to synchronize it into the native Secret that CIS already expects. CIS requires no code changes and no CIS-specific configuration to work with ESO, it simply reads the same Secret object it always has. ESO is solely responsible for creating and refreshing that Secret from Vault.

At a high level, Vault holds the credential as the source of truth, ESO reads from Vault and materializes a native Kubernetes Secret, and CIS consumes that Secret exactly as before — unaware that Vault or ESO exist.

Validated Component Versions

This integration was validated with the following component versions:

Component	Validated Version
Kubernetes	v1.31 (kubeadm)
F5 CIS	2.20.3 (Helm chart f5-bigip-ctlr), Custom Resource (CRD) mode
AS3	3.56.0
External Secrets Operator	2.8.0 (CRDs served at external-secrets.io/v1)
HashiCorp Vault	1.x (Helm chart vault), standalone mode with raft (persistent) storage
BIG-IP	17.5.1.6, LTM provisioned

Important

Vault storage

Deploy Vault with persistent storage (Integrated Storage / raft, or an external storage backend). A dev-mode Vault (server.dev.enabled=true) holds its configuration in memory only, restarting the Vault pod wipes the Kubernetes auth method, policies, roles, and any KV secrets, which breaks the ESO integration with 403 InvalidProviderConfig errors.

This failure was confirmed during validation. Dev mode is not a supported configuration for anything beyond a disposable demonstration.

Prerequisites

• A Kubernetes cluster (v1.13 or later) with Helm 3 installed.
• A BIG-IP system reachable from the cluster nodes on TCP/443, with AS3 3.13 or later installed.
• A dedicated BIG-IP partition for CIS (for example, k8s-cis). Do not use /Common.
• A BIG-IP user account for CIS with the Administrator role, created before CIS starts.

Note

Create the BIG-IP user first

Create the BIG-IP user that CIS will authenticate as before installing CIS or writing the credential to Vault. If CIS starts with no valid credential available, or if the account does not yet exist, recovery may require direct tmsh or iControl REST access to the BIG-IP to reset the account.

Follow these step-by-step instructions to configure the HashiCorp Vault integration with the External Secrets Operator for F5 CIS.

1. Create the BIG-IP User

Create the dedicated partition and the CIS service account on the BIG-IP:

tmsh create auth partition k8s-cis
tmsh create auth user cis-ctlr \
    partition-access add { k8s-cis { role administrator } } \
    password '**<strong-password>**'

Important

Password character Restrictions

Avoid special characters such as # in the password. During validation, a password containing # allowed HTTP Basic authentication to succeed (HTTP 200) but caused iControl REST token authentication to fail (HTTP 401, restjavad AuthnWorker: failed to login tmos/local). Because CIS uses token authentication, this failure mode is not apparent from a simple curl -u test against the BIG-IP.

2. Install HashiCorp Vault with Persistent Storage

Create the Vault Helm values file, vault-values.yaml:

server:
  standalone:
    enabled: true
    config: |
      ui = true
      listener "tcp" {
        tls_disable      = 1
        address          = "[::]:8200"
        cluster_address  = "[::]:8201"
      }
      storage "raft" {
        path = "/vault/data"
      }
  dataStorage:
    enabled: true
    size: 1Gi
    storageClass: <your-storage-class>

Add the Helm repository and install Vault:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault -n vault --create-namespace -f vault-values.yaml

Initialize and unseal Vault, then store the BIG-IP credential in the KV v2 secrets engine:

kubectl exec -n vault vault-0 -- vault operator init
kubectl exec -n vault vault-0 -- vault operator unseal <unseal-key-1>
# repeat unseal with additional keys per your unseal threshold

kubectl exec -n vault vault-0 -- vault kv put secret/f5-cis \
    username=cis-ctlr \
    password='**<strong-password>**'

3. Configure the Vault Kubernetes Auth Method

Create a ServiceAccount and the RBAC binding that Vault needs to validate tokens via the Kubernetes TokenReview API:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: eso-vault-auth
  namespace: eso
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: eso-vault-auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: eso-vault-auth
  namespace: eso

Apply the manifest, enable the Kubernetes auth method, and configure it:

kubectl apply -f eso-vault-auth-sa.yaml
kubectl exec -n vault vault-0 -- vault auth enable kubernetes

kubectl exec -n vault vault-0 -- vault write auth/kubernetes/config \
    token_reviewer_jwt="$(kubectl create token eso-vault-auth -n eso --duration=8760h)" \
    kubernetes_host="https://<kubernetes-api-server>:6443" \
    kubernetes_ca_cert=@/path/to/ca.crt

Restrict access to only the BIG-IP secret path, then bind a role to the ServiceAccount:

kubectl exec -n vault vault-0 -- vault policy write f5-cis-secret-policy - <<EOF
path "secret/data/f5-cis" {
  capabilities = ["read"]
}
EOF

kubectl exec -n vault vault-0 -- vault write auth/kubernetes/role/f5-cis-vault-role \
    bound_service_account_names=eso-vault-auth \
    bound_service_account_namespaces=eso \
    policies=f5-cis-secret-policy \
    ttl=1h

Note

Validate the login before proceeding

Confirm the Vault login path works end to end before wiring up ESO:

kubectl run vault-auth-test --rm -it --image=curlimages/curl \
    --overrides='{"spec":{"serviceAccountName":"eso-vault-auth"}}' -n eso -- sh

# inside the pod:
# TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
# curl -s -X POST http://vault.vault.svc:8200/v1/auth/kubernetes/login \
#     -d "{\"role\": \"f5-cis-vault-role\", \"jwt\": \"$TOKEN\"}"

A successful response returns a Vault client token with "policies": ["default", "f5-cis-secret-policy"] and no 403.

4. Install the External Secrets Operator

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
    -n external-secrets --create-namespace

Note

ESO is a Cluster-wide singleton

The ESO controller watches ExternalSecret and ClusterSecretStore objects across all namespaces. Install only one ESO release per cluster. Do not deploy a second controller for isolation purposes — two controllers reconciling the same custom resources will race and produce conflicting writes.

5. Create the ClusterSecretStore and ExternalSecret

Define the ClusterSecretStore that points ESO at Vault using the Kubernetes auth path configured in Step 3:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-bigip
spec:
  provider:
    vault:
      server: "http://vault.vault.svc:8200"
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: f5-cis-vault-role
          serviceAccountRef:
            name: eso-vault-auth
            namespace: eso

Define the ExternalSecret that materializes the native Secret CIS expects, f5-bigip-ctlr-login:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: f5-bigip-credentials-sync
  namespace: kube-system
spec:
  refreshInterval: "1m"
  secretStoreRef:
    name: vault-bigip
    kind: ClusterSecretStore
  target:
    name: f5-bigip-ctlr-login
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: f5-cis
      property: username
  - secretKey: password
    remoteRef:
      key: f5-cis
      property: password

Apply both manifests and verify their status:

kubectl apply -f clustersecretstore.yaml
kubectl apply -f externalsecret.yaml

kubectl get clustersecretstore vault-bigip
kubectl get externalsecret -n kube-system f5-bigip-credentials-sync

Expect Status: Valid for the ClusterSecretStore, and STATUS: SecretSynced / READY: True for the ExternalSecret.

6. Install CIS to Consume the ESO-Managed Secret

Create the CIS Helm values file, cis-values.yaml:

namespace: kube-system
args:
  bigip_url: <bigip-mgmt-ip>
  bigip_partition: k8s-cis
  custom_resource_mode: true
  pool_member_type: nodeport
bigip_login_secret: true
version: 2.20.3

Add the F5 chart repository and install CIS:

helm repo add f5-stable https://f5networks.github.io/charts/stable
helm install cis f5-stable/f5-bigip-ctlr -n kube-system -f cis-values.yaml

No credential is referenced by name in the Helm values. CIS mounts whichever Secret named f5-bigip-ctlr-login exists in its namespace via --credentials-directory the Secret that ESO already created in Step 5.

Important

Multi contoller namespace scoping

By default, CIS watches VirtualServer, TransportServer, and Ingress resources in all namespaces (--namespace defaults to All). If more than one CIS controller runs in the cluster — for example, during a staged migration or a side-by-side upgrade test each instance must be scoped with an explicit --namespace=<ns> argument. Without this, two controllers can both reconcile the same custom resource into two different BIG-IP partitions, and BIG-IP will reject the second attempt with an AS3 422 error such as: 0107176c:3: Invalid Virtual Address, the IP address <address> already exists. This occurs because a given virtual-server IP address cannot be reused across partitions on the same route domain.

Validation

Verify the Secret

Confirm the Secret exists and is managed by ESO rather than a hand-authored manifest:

kubectl get secret f5-bigip-ctlr-login -n kube-system
kubectl get secret f5-bigip-ctlr-login -n kube-system -o yaml

Expect ownerReferences[].kind: ExternalSecret and the label reconcile.externalsecrets.io/managed: "true".

Verify CIS Connectivity

kubectl logs deploy/<cis-release-name>-f5-bigip-ctlr -n kube-system

Note

CIS DOES NOT LOG AN EXPLICIT AUTHENTICATION SUCCESS

CIS does not print an explicit “authentication succeeded” message. Look instead for AS3 success lines:

[AS3][BigIP] post resulted in SUCCESS or [AS3][POST] SUCCESS: code: 200 --- tenant:<partition> --- message: ....

A 401/403 response, or the controller pod entering CrashLoopBackOff shortly after startup, indicates a credential problem. Because iControl REST rejects unauthenticated or invalid-credential requests before evaluating the AS3 declaration body, an HTTP 200 AS3 response is reliable evidence that authentication succeeded.

Credential Chain-of-Custody Check (Optional)

To confirm that the credential Vault holds is the one mounted into the CIS pod, trace the value through each stage of the pipeline:

• A human or automation writes the secret into Vault (vault kv put secret/f5-cis ...). Vault is the source of truth; ESO does not create anything in Vault.
• ESO reads from Vault via the ClusterSecretStore, and creates or syncs a native Kubernetes Secret (f5-bigip-ctlr-login).
• CIS reads that native Secret, mounted as files via --credentials-directory (for example, /tmp/creds/username, /tmp/creds/password). CIS never communicates with ESO or Vault directly.

Vault  (source of truth)
   │   ESO reads from Vault
   ▼
ESO  → creates / syncs native K8s Secret
   │   CIS reads the Secret (mounted file)
   ▼
CIS

Compare a hash of the password at each stage; all three should be identical:

# 1. Value in Vault
kubectl exec -n vault vault-0 -- vault kv get -field=password secret/f5-cis | sha256sum

# 2. Value in the native Secret ESO created
kubectl get secret f5-bigip-ctlr-login -n kube-system \
    -o jsonpath='{.data.password}' | base64 -d | sha256sum

# 3. Value mounted inside the CIS pod
kubectl exec -n kube-system deploy/<cis-release-name>-f5-bigip-ctlr -- \
    cat /tmp/creds/password | sha256sum

Demonstrate Credential Rotation

Update the credential in Vault; ESO propagates the change automatically within the ExternalSecret’s refreshInterval:

kubectl exec -n vault vault-0 -- vault kv put secret/f5-cis \
    username=cis-ctlr password='<new-password>'

Warning

Token caching during rotation

CIS’s BIG-IP auth token has a limited TTL (20 minutes by default), and CIS reuses that token until it expires or is invalidated. It does not proactively re-authenticate the moment the mounted credential file changes.

If the BIG-IP-side password is rotated independently of Vault (so the two are briefly out of sync), CIS continues operating on its cached token until that token is invalidated, at which point it re-authenticates using whatever credential is currently mounted. A CIS pod restart always forces immediate re-authentication using the currently mounted credential.

Troubleshooting

Symptom	Cause	Resolution
ClusterSecretStore shows InvalidProviderConfig / 403	Vault was deployed in dev mode and its pod restarted, wiping the Kubernetes auth method, policy, role, and KV secret.	Redeploy Vault with persistent (raft or external) storage. See the Important note in Overview.
CIS pod CrashLoopBackOff shortly after startup; log shows 401 Maximum number of login attempts exceeded	The mounted credential does not match the current BIG-IP password (for example, during a rotation test, or a stale Vault value).	Confirm the credential chain of custody, correct the value in Vault, and restart the CIS pod. Repeated failed attempts can trigger a brief BIG-IP rate limit that clears on its own; it is not a permanent lockout.
AS3 422 Invalid Virtual Address … already exists	More than one CIS controller is reconciling the same custom resource into different BIG-IP partitions because neither is namespace-scoped.	Add an explicit –namespace=<own-namespace> argument to each CIS deployment so controllers watch only their own resources.
kubectl logs deploy/f5-bigip-ctlr returns NotFound	The Deployment name depends on the Helm release name used at install (a release named cis produces cis-f5-bigip-ctlr).	Run kubectl get deploy -n kube-system to find the actual Deployment name for your release.

Production Hardening

• Use TLS between ESO and Vault (tls_disable = 0 with an appropriate CA bundle) rather than the plaintext listener shown in this guide.
• Scope the Vault policy as narrowly as possible — read-only access to the single KV path used by CIS.
• Rotate the BIG-IP credential regularly using Vault’s versioned KV secrets engine, and restart the CIS pod (or wait for its BIG-IP token to expire) to force prompt pickup of the new value.
• Use a dedicated BIG-IP partition per CIS instance; never point CIS at /Common.