Debugging distroless images

This page describes how to configure your cluster to debug pods with Aspen Mesh distroless images.

Background

We ensure that only necessary software is included in the container image. Reducing the attack surface of binaries and tools available in the container enhances security. While additional software is not exposed by default, if an attacker finds a vulnerability that they can use to activate software inside the container, they will no longer have tools or shells to activate. This approach is often called “distroless”—we remove the Linux distribution and include only the bare minimum in each container.

This reduces or eliminates security vulnerabilities reported by image container scanners such as Achore, Snyk, Amazon ECR, Quay and similar.

To ensure that this is the case for images in the Aspen Mesh distribution the following container images have been converted to distroless: proxyv2, citadel, pilot, and mixer.

Distroless improves the security posture of Aspen Mesh. However, since tools have been removed from the containers, you need to understand these additional steps to take if you need to use tools to troubleshoot an issue.

Enabling debug containers

Enabling ephemeral containers allows you to add containers to an already-running pod. You can add a debug container that has the tools that you need for troubleshooting.

Depending on how you deploy your Kubernetes cluster, follow one of the following sections. After enabling debug containers, please click here for instructions and examples on how to use them.

Note

This will work only on Kubernetes release 1.18 and above.

KOPS

Edit your cluster configuration: kops edit cluster you.cluster.name

Under the spec field add the following to your YAML configuration:

spec:
  kubelet:
    featureGates:
      EphemeralContainers: "true"
  kubeAPIServer:
    featureGates:
      EphemeralContainers: "true"

Then apply the configuration:

kops update cluster --name your.cluster.name --yes

kops rolling-update cluster your.cluster.name --yes

After the rolling update finish, you now can debug your pods with ephemeral containers.

Other Kubernetes distributions

If you have access to where the manifest for the static pods are stored: /etc/kubernetes/manifest/kube-apiserver.yaml

You can edit the manifest below the “- command” field and add the following flag: "--feature-gates=EphemeralContainers=true"

- command:
    - kube-apiserver
    - --advertise-address=10.132.0.48
    - --allow-privileged=true
    ...
    - --feature-gates=EphemeralContainers=true

After edit the apiserver pod, it will take the new configuration if not just delete the pod to restart it and it will recreate a new one.

At this point is necessary to update the Kubelet configuration as well. We need to add an extra argument, there is an existent env file located at: /etc/sysconfig/kubelet

KUBELET_EXTRA_ARGS="--feature-gates=EphemeralContainers=true"

Perform this change in all your nodes and restart Kubelet agent.