Consuming Aspen Mesh certificates via Kubernetes Secrets API#

Aspen Mesh ships with the secure Secret Discovery Service (SDS) mode enabled in release 1.6 and higher as the primary means for provisioning TLS credentials (keys and certificates) for workloads within the mesh. In SDS mode, the TLS credentials are fetched from the Aspen Mesh control plane via APIs without relying on Kubernetes secrets API or any file mounts thereby providing additional security. This SDS provisioning system works well in cases where the sidecar proxies can be injected into the workload pods but can be challenging for legacy and non-Kubernetes native workloads. If you’re using Aspen Mesh as the primary certificate provisioner for all workloads (Kubernetes and legacy) within your cluster, this guide describes how you can securely access Aspen Mesh TLS credentials using Kubernetes secrets API with the default SDS mode enabled. This functionality is useful in scenarios where workloads without sidecar proxies are communicating over TLS with certificates minted by Aspen Mesh with services with sidecar proxies injected or Aspen Mesh gateways.

When this functionality is enabled, Aspen Mesh generates Kubernetes secrets with TLS credentials and automatically manages the lifecycle and rotation of these credentials. Your application workloads can then use these secrets by mounting them as file mounts as described here or by consuming the secrets directly from the Kubernetes API. The generated secret names are based on the Service Account name associated with the workload pod. For instance, if httpbin workload is deployed with an associated service account of httpbin-service-account in the default namespace, the Kubernetes secret istio.httpbin-service-account will be created in the default namespace with TLS credentials.

Setup#

Warning

This feature requires you to provide your existing Certificate Authority (CA) credentials to be used by Aspen Mesh control plane components (istiod and Citadel).

  1. Follow the instructions on configuring certificates to plug in your own CA certificates.

  2. Continue with your installation of Aspen Mesh, but refer back to this documentation have installation has succeeded.

  3. You will need to add explicit namespace annotations to instruct Aspen Mesh Citadel to generate secrets for workloads in that namespace:

      kubectl label --overwrite namespace <NAMESPACE> ca.istio.io/override=true

Note

This annotation is independent of the istio-injection annotation used on namespaces to automatically inject the sidecar proxies which implies that you can use Aspen Mesh generated credentials using this feature for workloads without the sidecar proxies.

  1. You can verify that the Aspen Mesh Citdadel has created credentials by inspecting the generated secrets in your namespace:

      kubectl get secret -n <NAMESPACE> istio.default -o yaml | yq r - 'data."cert"' | base64 --decode | openssl x509 -noout -text

Step-by-step example (Do not use in production)#

In this set-by-step example you will deploy 2 applications, one without sidecar proxy which will use the credentials generated by Aspen Mesh Citadel and another injected with sidecar proxy which will use credentials via SDS mode and enable traffic flow between these applications both using Aspen Mesh generated credentials underneath.

  1. Install Aspen Mesh from the installation guide following the guide to plug in your own CA certificates.

Note

Make sure to add this to your overrides file! We will also be enabling Mesh Workload Certificates with SAN DNS and URI entries.

```yaml
citadel:
   enabled: true

global:
  certificateCustomFields: true
```

  1. Create a test namespace called without-sidecar-injection where secrets will be generated by Aspen Mesh Citadel

      kubectl create namespace without-sidecar-injection

  2. Add to the namespace the label ca.istio.io/override

      kubectl label --overwrite namespace without-sidecar-injection ca.istio.io/override=true

  3. Deploy the following sleep workload into the without-sidecar-injection namespace

     cat <<EOF | kubectl create -n without-sidecar-injection -f -
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: sleep
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: sleep
    labels:
      app: sleep
  spec:
    ports:
    - port: 80
      name: http
    selector:
      app: sleep
  ---
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: sleep
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: sleep
    template:
      metadata:
        labels:
          app: sleep
      spec:
        serviceAccountName: sleep
        containers:
        - name: sleep
          image: governmentpaas/curl-ssl
          command: ["/bin/sleep", "3650d"]
          imagePullPolicy: IfNotPresent
          volumeMounts:
          - mountPath: /etc/sleep/tls
            name: secret-volume
          - mountPath: /etc/sleep/citadel
            name: citadel-volume
        volumes:
        - name: secret-volume
          secret:
            secretName: sleep-secret
            optional: true
        - name: citadel-volume
          secret:
            secretName: istio.sleep
            optional: true
  EOF

Note

The above is a modified version of the samples/sleep/sleep.yaml file that has a volume using the istio.sleep credential created by Aspen Mesh Citadel!

  1. Store the sleep pod name as a variable

      export SLEEP_POD=$(kubectl get pod -n without-sidecar-injection -l app=sleep -o jsonpath='{.items[0].metadata.name}')

  2. Verify the certificate istio.sleep is created

      kubectl get secret -n without-sidecar-injection istio.sleep -o yaml | yq r - 'data."cert-chain.pem"' | base64 --decode | openssl x509 -noout -text

    Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            bc:1d:f4:6f:c3:e7:d3:88:4f:94:72:46:dc:95:3c:bd
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: O=Juju org
        Validity
            Not Before: Oct 29 22:50:41 2020 GMT
            Not After : Jan 27 22:50:41 2021 GMT
        Subject:
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    ...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Alternative Name: critical
                URI:spiffe://cluster.local/ns/without-sidecar-injection/sa/sleep, DNS:sleep.without-sidecar-injection.svc.cluster.local
    Signature Algorithm: sha256WithRSAEncryption
         ...

  3. Create a namespace called with-sidecar-injection

      kubectl create namespace with-sidecar-injection

  4. Add the appropriate labels for istio-injection set properly to automatically inject sidecars. We will not be adding the Citadel annotations to create secrets in this namespace. The sidecar proxy injected will use SDS mode to provision TLS credentials.

      kubectl label --overwrite namespace with-sidecar-injection istio-injection=enabled

  5. Deploy the httpbin application to the with-sidecar-injection namespace

     cat <<EOF | kubectl apply -n with-sidecar-injection -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
  annotations:
    "certificate.aspenmesh.io/customFields": '{ "SAN": { "DNS": [ "httpbin.with-sidecar-injection.svc.cluster.local" ] } }'
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80
  EOF

  6. Store the details pod name as a variable

      export HTTPBIN_POD=$(kubectl get pod -n with-sidecar-injection -l app=httpbin -o jsonpath='{.items[0].metadata.name}')

  7. Inspect the certificate the httpbin pod is serving to its clients over SDS

      ./bin/istioctl proxy-config secret ${HTTPBIN_POD}.with-sidecar-injection -o json | jq '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | sed 's/"//g' | base64 --decode | openssl x509 -noout -text

    Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            3b:6b:04:00:23:58:a4:c0:2b:ae:25:64:d2:32:77:fe
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: O=Juju org
        Validity
            Not Before: Oct 29 23:05:12 2020 GMT
            Not After : Oct 30 23:05:12 2020 GMT
        Subject:
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    ...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Alternative Name: critical
                DNS:httpbin.with-sidecar-injection.svc.cluster.local, URI:spiffe://cluster.local/ns/with-sidecar-injection/sa/httpbin
    Signature Algorithm: sha256WithRSAEncryption
        ...

  8. Using the certificate minted in the without-sidecar-injection namespace for the sleep pod we will have it communicate with the httpbin service deployed in the httpbin namespace.

      kubectl exec -it -n without-sidecar-injection $SLEEP_POD -- curl --key /etc/sleep/citadel/key.pem --cacert /etc/sleep/citadel/root-cert.pem --cert /etc/sleep/citadel/cert-chain.pem https://httpbin.with-sidecar-injection.svc.cluster.local:8000/status/200 -vvvv

    *   Trying 100.67.121.12:8000...
* Connected to httpbin.with-sidecar-injection.svc.cluster.local (100.67.121.12) port 8000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/sleep/citadel/root-cert.pem
  CApath: none
...
* Server certificate:
*  subject: [NONE]
*  start date: Oct 29 23:05:12 2020 GMT
*  expire date: Oct 30 23:05:12 2020 GMT
*  subjectAltName: host "httpbin.with-sidecar-injection.svc.cluster.local" matched cert's "httpbin.with-sidecar-injection.svc.cluster.local"
*  issuer: O=Juju org
*  SSL certificate verify ok.
...
> GET /status/200 HTTP/2
> Host: httpbin.with-sidecar-injection.svc.cluster.local:8000
...
< HTTP/2 200
< server: istio-envoy
...

  9. Congratulations! Services deployed in your Kubernetes cluster without a sidecar can now communicate over TLS with a service with a sidecar proxy.