Collect debug information from sidecar proxies#

Introduction#

To troubleshoot problems with a pod’s Istio sidecar proxy, you can collect debug information from the sidecar proxy and provide it to Aspen Mesh for analysis.

Types of debug information#

There are two types of debug information you can collect from a pod’s sidecar proxy:

  • Core files

  • Heap-usage information

You may need to collect core files or heap-usage information from more than one pod.

When to collect core files#

Collect core files from a pod’s sidecar proxy if it exhibits a segmentation fault.

When to collect heap-usage information#

Collect heap-usage information from a pod’s sidecar proxy if you suspect it has a memory leak.

Prerequisite#

To collect debug information, you must install the debug sidecar proxy. After you finish collecting debug information, uninstall the debug sidecar proxy.

Install the debug sidecar proxy#

Collect core files from a pod’s sidecar proxy#

  1. Create a file to use as a script to collect core files from the pod’s sidecar proxy:

    $ touch get-core-files.sh

  2. Open the file in a text editor.

  3. Copy the following code and paste it into the file:

    #!/bin/bash

export POD=<appPodName>
export NS=<appNamespaceName>
kubectl cp -n "$NS" "$POD":/var/lib/istio/data /tmp/envoy-"$POD" -c istio-proxy
kubectl cp -n "$NS" "$POD":/lib/x86_64-linux-gnu /tmp/envoy-"$POD"/lib -c istio-proxy
kubectl cp -n "$NS" "$POD":/usr/local/bin/envoy /tmp/envoy-"$POD"/lib/envoy -c istio-proxy

  4. Replace <appPodName> with the name of the application’s pod, and replace <appNamespaceName> with the name of the namespace that the pod is in.

  5. Save and close the file.

  6. Make the file executable:

    $ chmod +x get-core-files.sh

  7. Execute the script:

    $ ./get-core-files.sh

  8. Create a compressed archive file of the core files, making sure to replace <appPodName> with the name of the application’s pod and <date> with the date the files were collected in the form yyyy-mm-dd:

    $ tar czvf <appPodName>-core-<date>.tar.gz /tmp/envoy-<appPodName>

Collect heap-usage information from a pod’s sidecar proxy#

  1. Create a file to use as a script to collect heap-usage information from the pod’s sidecar proxy:

    $ touch get-heap-usage-info.sh

  2. Open the file in a text editor.

  3. Copy the following code and paste it into the file:

    #!/bin/bash

export POD=<appPodName>
export NS=<appNamespaceName>
export PROFILER="heap"
kubectl exec -n "$NS" "$POD" -c istio-proxy -- curl -X POST -s "http://localhost:15000/${PROFILER}profiler?enable=y"
sleep 600
kubectl exec -n "$NS" "$POD" -c istio-proxy -- curl -X POST -s "http://localhost:15000/${PROFILER}profiler?enable=n"
kubectl cp -n "$NS" "$POD":/var/lib/istio/data /tmp/envoy-"$POD" -c istio-proxy
kubectl cp -n "$NS" "$POD":/lib/x86_64-linux-gnu /tmp/envoy-"$POD"/lib -c istio-proxy
kubectl cp -n "$NS" "$POD":/usr/local/bin/envoy /tmp/envoy-"$POD"/lib/envoy -c istio-proxy

  4. Replace <appPodName> with the name of the application’s pod, and replace <appNamespaceName> with the name of the namespace that the pod is in.

  5. Save and close the file.

  6. Make the file executable:

    $ chmod +x get-heap-usage-info.sh

  7. Execute the script (note that it will run for a while):

    $ ./get-heap-usage-info.sh

  8. Create a compressed archive file of the heap-usage information, making sure to replace <appPodName> with the name of the application’s pod and <date> with the date the information was collected in the form yyyy-mm-dd:

    $ tar czvf <appPodName>-heap-<date>.tar.gz /tmp/envoy-<appPodName>

Provide the debug information to Aspen Mesh for analysis#

  • After you finish collecting all debug information, attach the compressed archive files to your support request.

Uninstall the debug sidecar proxy#