Consuming Aspen Mesh certificates via Kubernetes Secrets API

Aspen Mesh ships with the secure Secret Discovery Service (SDS) mechanism in release 1.6 and higher as the primary means for provisioning TLS credentials (keys and certificates) for workloads within the mesh. In SDS mode, the TLS credentials are fetched from the Istio control plane via APIs without relying on Kubernetes secrets API or any file mounts thereby providing additional security. This SDS provisioning system works well in cases where the sidecar proxies can be injected into the workload pods but can be challenging for legacy and non-Kubernetes native workloads. If you’re using Aspen Mesh as the primary certificate provisioner for all workloads (Kubernetes and legacy) within your cluster, this guide describes how you can securely access Aspen Mesh TLS credentials using the Kubernetes secrets API. This functionality is useful in scenarios where workloads without sidecar proxies are communicating over TLS with certificates minted by Aspen Mesh with services with sidecar proxies injected or Aspen Mesh gateways.

When this functionality is enabled, Aspen Mesh generates Kubernetes secrets with TLS credentials and automatically manages the lifecycle and rotation of these credentials. Your application workloads can then use these secrets by mounting them as file mounts as described below. The generated secret names are based on the Service Account name associated with the workload pod. For instance, if httpbin workload is deployed with an associated service account of httpbin-service-account in the default namespace, the Kubernetes secret istio.httpbin-service-account will be created in the default namespace with TLS credentials.

Setup

Warning

This feature requires you to provide your existing Certificate Authority (CA) credentials to be used by Aspen Mesh control plane components (istiod and Citadel).

1. You will need to add explicit namespace annotations to instruct Aspen Mesh Citadel to generate secrets for workloads in that namespace:

   $ kubectl label --overwrite namespace <NAMESPACE> ca.istio.io/override=true
   

   Note

   This annotation is independent of the istio-injection annotation used on namespaces to automatically inject the sidecar proxies which implies that you can use Aspen Mesh generated credentials using this feature for workloads without the sidecar proxies.

2. You can verify that the Aspen Mesh Citadel has created credentials by inspecting the generated secrets in your namespace:

   $ kubectl get secret -n <NAMESPACE> istio.default -o yaml | \
       yq r - 'data."cert"' | base64 --decode | openssl x509 -noout -text
   

3. To mount the Aspen Mesh certificates, add /etc/certs volume to the pod definition:

   volumeMounts:
   - name: istio-certs
     mountPath: /etc/certs
   

4. In the spec, add volume for the certificates

   volumes:
   - name: istio-certs
     secret:
       defaultMode: 420
       optional: true
       secretName: istio.<ServiceAccountName>
   

Step-by-step example (Do not use in production)

In this set-by-step example you will deploy 2 applications, one without sidecar proxy which will use the credentials generated by Aspen Mesh Citadel and another injected with sidecar proxy which will use credentials via SDS mode and enable traffic flow between these applications both using Aspen Mesh generated credentials underneath.

1. Follow the clean-installation instructions following the guide to plug in your own CA certificates.

   Note

   Make sure to add this to your overrides file! We will also be enabling Mesh Workload Certificates with SAN DNS and URI entries.

   global:
     certificateCustomFields: true
   

2. Install Citadel.

3. Create a test namespace called without-sidecar-injection where secrets will be generated by Aspen Mesh Citadel

   $ kubectl create namespace without-sidecar-injection
   

4. Add to the namespace the label ca.istio.io/override

   $ kubectl label --overwrite namespace without-sidecar-injection ca.istio.io/override=true
   

5. Deploy the following sleep workload into the without-sidecar-injection namespace

   $ cat <<EOF | kubectl create -n without-sidecar-injection -f -
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: sleep
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: sleep
     labels:
       app: sleep
   spec:
     ports:
     - port: 80
       name: http
     selector:
       app: sleep
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: sleep
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: sleep
     template:
       metadata:
         labels:
           app: sleep
       spec:
         serviceAccountName: sleep
         containers:
         - name: sleep
           image: governmentpaas/curl-ssl
           command: ["/bin/sleep", "3650d"]
           imagePullPolicy: IfNotPresent
           volumeMounts:
           - mountPath: /etc/sleep/tls
             name: secret-volume
           - mountPath: /etc/sleep/citadel
             name: citadel-volume
         volumes:
         - name: secret-volume
           secret:
             secretName: sleep-secret
             optional: true
         - name: citadel-volume
           secret:
             secretName: istio.sleep
             optional: true
   EOF
   

Note

The above is a modified version of the samples/sleep/sleep.yaml file that has a volume using the istio.sleep credential created by Aspen Mesh Citadel!

1. Store the sleep pod name as a variable

   $ export SLEEP_POD=$(kubectl get pod -n without-sidecar-injection \
       -l app=sleep -o jsonpath='{.items[0].metadata.name}')
   

2. Verify the certificate istio.sleep is created

   $ kubectl get secret -n without-sidecar-injection istio.sleep -o yaml | \
       yq r - 'data."cert-chain.pem"' | base64 --decode | openssl x509 -noout -text
   

   Certificate:
       Data:
           Version: 3 (0x2)
           Serial Number:
               bc:1d:f4:6f:c3:e7:d3:88:4f:94:72:46:dc:95:3c:bd
       Signature Algorithm: sha256WithRSAEncryption
           Issuer: O=Juju org
           Validity
               Not Before: Oct 29 22:50:41 2020 GMT
               Not After : Jan 27 22:50:41 2021 GMT
           Subject:
           Subject Public Key Info:
               Public Key Algorithm: rsaEncryption
                   Public-Key: (2048 bit)
                   Modulus:
                       ...
                   Exponent: 65537 (0x10001)
           X509v3 extensions:
               X509v3 Key Usage: critical
                   Digital Signature, Key Encipherment
               X509v3 Extended Key Usage:
                   TLS Web Server Authentication, TLS Web Client Authentication
               X509v3 Basic Constraints: critical
                   CA:FALSE
               X509v3 Subject Alternative Name: critical
                   URI:spiffe://cluster.local/ns/without-sidecar-injection/sa/sleep, DNS:sleep.without-sidecar-injection.svc.cluster.local
       Signature Algorithm: sha256WithRSAEncryption
            ...
   

3. Create a namespace called with-sidecar-injection

   $ kubectl create namespace with-sidecar-injection
   

4. Add the appropriate labels for istio-injection set properly to automatically inject sidecars. We will not be adding the Citadel annotations to create secrets in this namespace. The sidecar proxy injected will use SDS mode to provision TLS credentials.

   $ kubectl label --overwrite namespace with-sidecar-injection istio-injection=enabled
   

5. Deploy the httpbin application to the with-sidecar-injection namespace

   $ cat <<EOF | kubectl apply -n with-sidecar-injection -f -
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: httpbin
     annotations:
       "certificate.aspenmesh.io/customFields": '{ "SAN": { "DNS": [ "httpbin.with-sidecar-injection.svc.cluster.local" ] } }'
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: httpbin
     labels:
       app: httpbin
   spec:
     ports:
     - name: http
       port: 8000
       targetPort: 80
     selector:
       app: httpbin
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: httpbin
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: httpbin
         version: v1
     template:
       metadata:
         labels:
           app: httpbin
           version: v1
       spec:
         serviceAccountName: httpbin
         containers:
         - image: docker.io/kennethreitz/httpbin
           imagePullPolicy: IfNotPresent
           name: httpbin
           ports:
           - containerPort: 80
   EOF
   

6. Store the details pod name as a variable

   $ export HTTPBIN_POD=$(kubectl get pod -n with-sidecar-injection -l app=httpbin \
       -o jsonpath='{.items[0].metadata.name}')
   

7. Inspect the certificate the httpbin pod is serving to its clients over SDS

   $ ./bin/istioctl proxy-config secret ${HTTPBIN_POD}.with-sidecar-injection -o json | \
       jq '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | \
       sed 's/"//g' | base64 --decode | openssl x509 -noout -text
   

   Certificate:
       Data:
           Version: 3 (0x2)
           Serial Number:
               3b:6b:04:00:23:58:a4:c0:2b:ae:25:64:d2:32:77:fe
       Signature Algorithm: sha256WithRSAEncryption
           Issuer: O=Juju org
           Validity
               Not Before: Oct 29 23:05:12 2020 GMT
               Not After : Oct 30 23:05:12 2020 GMT
           Subject:
           Subject Public Key Info:
               Public Key Algorithm: rsaEncryption
                   Public-Key: (2048 bit)
                   Modulus:
                       ...
                   Exponent: 65537 (0x10001)
           X509v3 extensions:
               X509v3 Key Usage: critical
                   Digital Signature, Key Encipherment
               X509v3 Extended Key Usage:
                   TLS Web Server Authentication, TLS Web Client Authentication
               X509v3 Basic Constraints: critical
                   CA:FALSE
               X509v3 Subject Alternative Name: critical
                   DNS:httpbin.with-sidecar-injection.svc.cluster.local, URI:spiffe://cluster.local/ns/with-sidecar-injection/sa/httpbin
       Signature Algorithm: sha256WithRSAEncryption
   ...
   

8. Using the certificate minted in the without-sidecar-injection namespace for the sleep pod we will have it communicate with the httpbin service deployed in the httpbin namespace.

   $ kubectl exec -it -n without-sidecar-injection $SLEEP_POD -- \
       curl --key /etc/sleep/citadel/key.pem \
         --cacert /etc/sleep/citadel/root-cert.pem \
         --cert /etc/sleep/citadel/cert-chain.pem \
         https://httpbin.with-sidecar-injection.svc.cluster.local:8000/status/200 -vvvv
   

   *   Trying 100.67.121.12:8000...
   * Connected to httpbin.with-sidecar-injection.svc.cluster.local (100.67.121.12) port 8000 (#0)
   * ALPN, offering h2
   * ALPN, offering http/1.1
   * successfully set certificate verify locations:
   *   CAfile: /etc/sleep/citadel/root-cert.pem
     CApath: none
   ...
   * Server certificate:
   *  subject: [NONE]
   *  start date: Oct 29 23:05:12 2020 GMT
   *  expire date: Oct 30 23:05:12 2020 GMT
   *  subjectAltName: host "httpbin.with-sidecar-injection.svc.cluster.local" matched cert's "httpbin.with-sidecar-injection.svc.cluster.local"
   *  issuer: O=Juju org
   *  SSL certificate verify ok.
   ...
   > GET /status/200 HTTP/2
   > Host: httpbin.with-sidecar-injection.svc.cluster.local:8000
   ...
   < HTTP/2 200
   < server: istio-envoy
   ...
   

9. Congratulations! Services deployed in your Kubernetes cluster without a sidecar can now communicate over TLS with a service with a sidecar proxy.

Sample code

The following code demonstrates the use of the certificate for a simple HTTP server:

keyPair, err := tls.LoadX509KeyPair("/etc/certs/cert-chain.pem", "/etc/certs/key.pem")
if err != nil {
	panic(err)
}

s := &http.Server{
	Addr:      "0.0.0.0:8080",
	TLSConfig: &tls.Config{Certificates: []tls.Certificate{keyPair}},
	Handler:   http.HandlerFunc(func(w ResponseWriter, r *Request) {
		fmt.Fprintf(w, "Hello, %s!", r.URL.Path[1:])
	}),
}

s.ListenAndServeTLS("/etc/certs/cert-chain.pem", "/etc/certs/key.pem")