Use Packet Inspector 1#

Packet Inspector 1 captures and streams network traffic from Envoy proxies, aggregates it, and forwards it to an analysis service.

Note

This feature is available only in Aspen Mesh Carrier-Grade.

About Packet Inspector 1#

Introduction#

Capturing network packets via tools like tcpdump and analyzing them in Wireshark is one of the most commonly used techniques to troubleshoot failures and understand application behavior in distributed and microservices-based applications. However, using this technique in cloud-native environments like Kubernetes is challenging due to the zero-trust security model and encrypted traffic between all workloads with perfect forward secrecy.

Packet Inspector 1 solves these challenges by capturing packets in plaintext at the sidecar proxies before sent packets are encrypted or after received packets are decrypted and proxied to the services in your cluster. Packet Inspector 1 performs captures at the application layer (Layer 7) instead of providing a TCP (Layer 4) level capture.

The captures contain rich contextual cloud-native data like your pod and service name and namespace, making it possible to associate the captured packets with workloads running in your Kubernetes cluster. Additionally, the packet captures are serialized in the well known libpcap format and can be transported to an analysis service and analyzed in Wireshark.

In Aspen Mesh 1.11, Packet Inspector 1 captures traffic from all pods with sidecars, regardless of namespace.

Supported protocols#

Packet Inspector 1 supports the following protocols:

Diameter (Aspen Mesh 1.11.8-am1.2, 1.11.8-am1.3, and 1.11.8-am1.4 only)
HTTP (1.1 and 2), including gRPC

Note

Diameter packet captures from ingress and egress gateways are not supported.

Packet Inspector 1 block diagram#

This diagram shows the components of Packet Inspector 1 and how data travels among them:

Packet Inspector 1 components#

Packet Inspector 1 consists of the following components:

Component	Description
Filters	Extensions of Envoy proxies (whether used as sidecar proxies or in a gateway) that capture and forward data
Aggregator service	A daemon set that receives capture data and finalizes it in the `pcap` format before transporting it to the analysis service

Types of analysis services#

You can send captured traffic to either of the following types of analysis services:

Analysis service	Purpose
A third-party network analysis tool	For use in production environments to analyze captured data
The Aspen Mesh analysis emulator	For use in non-production environments to demonstrate or test Packet Inspector 1

About the analysis emulator#

Types of files written by the analysis emulator#

For each capture, the analysis emulator writes two types of files to the /tmp directory:

The raw-request-and-metadata (.bson) file
The corresponding .pcap file

Analysis-emulator HTTP filename formats#

HTTP files written by the analysis emulator use one of the following filename formats:

1.11.8-am2 and earlier#

Role	Filename format	Examples
Consumer	`<podName>-consumer-<xTraceId>-<timestamp>`	`sleep-856f4f69cd-wkxw6-consumer-ba9f745f4018e235dc32f0e6b6321286-1604430235431770462.bson` `sleep-856f4f69cd-wkxw6-consumer-ba9f745f4018e235dc32f0e6b6321286-1604430235431770462.pcap`
Producer	`<apiPath>-producer-<xTraceId>-<timestamp>`	`flasgger_static-producer-ba9f745f4018e235dc32f0e6b6321286-1604430235278545932.bson` `flasgger_static-producer-ba9f745f4018e235dc32f0e6b6321286-1604430235278545932.pcap`

1.11.8-am3 and later (including 1.11.8-am1.2, 1.11.8-am1.3, and 1.11.8-am1.4)#

Role	Filename format	Examples
Consumer	`<podName>-consumer-<xTraceId>-<sequenceNumber>`	`sleep-856f4f69cd-wkxw6-consumer-ba9f745f4018e235dc32f0e6b6321286-1.bson` `sleep-856f4f69cd-wkxw6-consumer-ba9f745f4018e235dc32f0e6b6321286-1.pcap`
Producer	`<apiPath>-producer-<xTraceId>-<sequenceNumber>`	`flasgger_static-producer-ba9f745f4018e235dc32f0e6b6321286-1.bson` `flasgger_static-producer-ba9f745f4018e235dc32f0e6b6321286-1.pcap`

Analysis-emulator Diameter filename formats#

Diameter files written by the analysis emulator use the following filename format:

<podName>-<endToEndId>-<hopByHopId>-<timestamp>

Examples

diameter-client-6464c65b66-4c8jl-1339708294-1857878061-1675782061987009574.bson
diameter-client-6464c65b66-4c8jl-1339708294-1857878061-1675782061987009574.pcap

Prepare to capture traffic with Packet Inspector 1#

Introduction#

To prepare to capture traffic using Packet Inspector 1, you need to:

Plug in certificate-authority (CA) certificates
Enable traditional DNS-name SAN extensions in workload certificates
Enable Citadel
Add the certificate-secret label to the aggregator-service namespace
Install the network analysis tool or the analysis emulator

When to follow these instructions#

Follow the instructions below only if you want to start using Packet Inspector 1 to capture traffic and only after you’ve installed Aspen Mesh. Perform the tasks in order.

Why you must plug in CA certificates#

Mutual TLS (mTLS) communication is used between the Packet Inspector 1 aggregator service (off mesh) and the sidecars (on mesh), so their certificates must be generated and signed by the same certificate-authority (CA) certificate. To ensure the CA certificate used by Citadel (to generate and sign certificates for the aggregator service) and the Aspen Mesh CA (to generate and sign certificates for the sidecars) is the same, you must plug in CA certificates.

Plug in CA certificates#

Follow the instructions on configuring certificates to plug in CA certificates.

Why Packet Inspector 1 requires Citadel#

TLS communication is used between the Packet Inspector 1 aggregator service and analysis service, both of which are off mesh. To enable TLS communication between these services, you must enable Citadel and apply a certificate-secret label to the aggregator-service and analysis-service namespaces so Citadel generates and signs certificates (in the form of Kubernetes secrets) for use by those services. Learn more about using the Citadel certificate-secret label.

Why Packet Inspector 1 requires certificates with traditional DNS-name SAN extensions#

The analysis service is required to use a certificate with a traditional DNS-name Subject Alternative Name (SAN) extension for communication with the aggregator service. Learn about the format of the DNS-name SAN extension.

Enable Citadel and enable traditional DNS-name SAN extensions in workload certificates#

Open your Aspen Mesh override values file (for example, aspen-mesh-override-values.yaml) in a text editor.

Copy the following information and paste it into the file:

citadel:
  enabled: true

global:
  # Enable traditional DNS-name and non-SPIFFE-URI SAN extensions in
  # Aspen Mesh certificate-authority (CA) workload certificates and
  # Citadel certificate secrets.
  certificateCustomFields: true

Save and close the file.
Perform an upgrade of Aspen Mesh from the current version to the same version.

Add the certificate-secret label to the aggregator-service namespace#

Add the certificate-secret label to the istio-system namespace so Citadel generates a certificate (in the form of a Kubernetes secret) that allows the aggregator service to communicate over TLS with the analysis service:
```
$ kubectl label namespace istio-system \
    ca.istio.io/override=true --overwrite
```

Install the analysis service#

Do you want to use a network analysis tool (not the analysis emulator) as your analysis service?
- Yes:
  - Install the network analysis tool.
- No:
  - Install the analysis emulator.

Install the network analysis tool#

Note

Perform this task only when other instructions tell you to do so.

Make sure the following requirements are met:
- Add the certificate-secret label to the namespace in which you’ll install the network analysis tool so Citadel generates a certificate (in the form of a Kubernetes secret) that allows the network analysis tool to communicate over TLS with the aggregator service:
```
apiVersion: v1
kind: Namespace
metadata:
  name: <networkAnalysisToolNamespaceName>
  labels:
    ca.istio.io/override: "true"
```
- Use the same name for the network analysis tool’s service and service account.
  
  When connecting to the analysis service (either the network analysis tool or the analysis emulator), the Packet Inspector 1 aggregator service checks the analysis service’s certificate for a DNS-name SAN extension that matches the address of the analysis service. The DNS-name SAN extension that Citadel adds to the analysis service’s certificate is a fully qualified domain name (FQDN) based on the analysis service’s service-account name; however, the address of the analysis service is based on the analysis service’s service name. To make sure the check is successful, the network analysis tool’s service and service account must have the same name.
- In the pod spec for the network analysis tool, add the istio-certs volume for the certificate secret:
```
volumes:
- name: istio-certs
  secret:
    defaultMode: 420
    optional: true
    secretName: istio.<networkAnalysisToolServiceAccountName>
```
- In the container spec for the network analysis tool, add a volume mount that references the istio-certs volume and mounts it at /etc/analysis/certs:
```
volumeMounts:
- name: istio-certs
  mountPath: /etc/analysis/certs
```
Follow the installation instructions provided with the network analysis tool.

Install the analysis emulator#

Note

Perform this task only when other instructions tell you to do so.

Change to the Aspen Mesh release directory.

Create a namespace named analysis-emulator:

$ kubectl create namespace analysis-emulator

Add the certificate-secret label to the analysis-emulator namespace so Citadel generates a certificate (in the form of a Kubernetes secret) that allows the analysis emulator to communicate over TLS with the aggregator service:
```
$ kubectl label namespace analysis-emulator \
    ca.istio.io/override=true --overwrite
```

Install the analysis emulator:

$ helm install pi1-analysis-emulator samples/aspenmesh/analysis-emulator \
    --namespace analysis-emulator

Capture traffic using Packet Inspector 1#

Introduction#

Before you can capture traffic using Packet Inspector 1, you need to:

Configure Packet Inspector 1
Enable Packet Inspector 1

After you capture traffic, you need to disable Packet Inspector 1 to prevent it from lowering the performance of your cluster.

When to follow these instructions#

Follow the instructions below only if you want to capture traffic using Packet Inspector 1 and only after you’ve prepared to capture traffic with Packet Inspector 1. Perform the tasks in order.

Configure Packet Inspector 1 (1.11.8-am1.3 and 1.11.8-am1.4 only)#

Note

Perform this task only if your Aspen Mesh version is 1.11.8-am1.3 or 1.11.8-am1.4.

Open your Aspen Mesh override values file (for example, aspen-mesh-override-values.yaml) in a text editor.

Copy the following information and paste it into the file:

aspen-mesh-packet-inspector:
  enabled: false
  analysis:
    address:
  httpCapture:
    enabled: false
    excludeNamespaces: []
    excludePods: []
  diameterCapture:
    enabled: false
    port:

Do you want to send captured traffic to a network analysis tool (not the analysis emulator)?

Yes:

Set the value of .aspen-mesh-packet-inspector.analysis.address to:

<networkAnalysisToolServiceName>.<networkAnalysisToolNamespaceName>.svc.cluster.local:<port>

Replace the placeholders:

Replace this placeholder	With
`<networkAnalysisToolServiceName>`	The network analysis tool’s Kubernetes service name
`<networkAnalysisToolNamespaceName>`	The namespace in which the network analysis tool is installed
`<port>`	The port on which the network analysis tool is listening

Go to step 4.

No:
- Set the value of .aspen-mesh-packet-inspector.analysis.address to:
  
  aspen-mesh-analysis-emulator.analysis-emulator.svc.cluster.local:21000
- Go to step 4.

Do you want to prevent Packet Inspector 1 from capturing HTTP packets from specific namespaces?
- Yes:
  - Change the value of .aspen-mesh-packet-inspector.httpCapture.excludeNamespaces from [] to a list of one or more namespace specifiers, each of which can be a namespace name or a regular expression.
    
    Example
    
    In this example, Packet Inspector 1 will not capture HTTP packets from any namespace whose name is or begins with dont-capture-me- and will not capture HTTP packets from any namespace named capture-me-not:
    aspen-mesh-packet-inspector: httpCapture: excludeNamespaces: - dont-capture-me-.* - capture-me-not
  - Go to step 5.
- No:
  - Go to step 5.
Do you want to prevent Packet Inspector 1 from capturing HTTP packets from specific pods?
- Yes:
  - Change the value of .aspen-mesh-packet-inspector.httpCapture.excludePods from [] to a list of one or more pod specifiers, each of which can be a pod name or a regular expression.
    
    Example
    
    In this example, Packet Inspector 1 will not capture HTTP packets from any pod named stealthy-pod (regardless of namespace) and will not capture HTTP packets from any pod whose name is or begins with under-the-radar-app (regardless of namespace):
    aspen-mesh-packet-inspector: httpCapture: excludePods: - stealthy-pod - under-the-radar-app.*
  - Go to step 6.
- No:
  - Go to step 6.
Save and close the file.

Configure Packet Inspector 1 (any version other than 1.11.8-am1.3 or 1.11.8-am1.4)#

Note

Perform this task only if your Aspen Mesh version is not 1.11.8-am1.3 or 1.11.8-am1.4.

Open your Aspen Mesh override values file (for example, aspen-mesh-override-values.yaml) in a text editor.

Is your current version of Aspen Mesh 1.11.8-am1.2?

Yes:

Copy the following information and paste it into the file:

aspen-mesh-packet-inspector:
  enabled: false
  analysis:
    address:
  httpCapture:
    enabled: false
  diameterCapture:
    enabled: false
    port:

Go to step 3.

No:

Copy the following information and paste it into the file:

aspen-mesh-packet-inspector:
  enabled: false
  analysis:
    address:

Go to step 3.

Do you want to send captured traffic to a network analysis tool (not the analysis emulator)?

Yes:

Set the value of .aspen-mesh-packet-inspector.analysis.address to:

<networkAnalysisToolServiceName>.<networkAnalysisToolNamespaceName>.svc.cluster.local:<port>

Replace the placeholders:

Replace this placeholder	With
`<networkAnalysisToolServiceName>`	The network analysis tool’s Kubernetes service name
`<networkAnalysisToolNamespaceName>`	The namespace in which the network analysis tool is installed
`<port>`	The port on which the network analysis tool is listening

Go to step 4.

No:
- Set the value of .aspen-mesh-packet-inspector.analysis.address to:
  
  aspen-mesh-analysis-emulator.analysis-emulator.svc.cluster.local:21000
- Go to step 4.

Save and close the file.

Enable Packet Inspector 1 (1.11.8-am1.2, 1.11.8-am1.3, and 1.11.8-am1.4 only)#

Note

Perform this task only if your Aspen Mesh version is 1.11.8-am1.2, 1.11.8-am1.3, or 1.11.8-am1.4.

Open your Aspen Mesh override values file (for example, aspen-mesh-override-values.yaml) in a text editor.
Enable the aggregator service by setting the value of .aspen-mesh-packet-inspector.enabled to true.
Do you want to capture HTTP traffic?
- Yes:
  - Enable the HTTP filter by setting the value of .aspen-mesh-packet-inspector.httpCapture.enabled to true.
  - Go to step 4.
- No:
  - Go to step 4.
Do you want to capture Diameter traffic?
- Yes:
  - Enable the Diameter filter by setting the value of .aspen-mesh-packet-inspector.diameterCapture.enabled to true.
  - Make sure .aspen-mesh-packet-inspector.diameterCapture.port specifies the port being used for Diameter traffic (typically 3868).
  - Go to step 5.
- No:
  - Go to step 5.
Save and close the file.
Perform an upgrade of Aspen Mesh from the current version to the same version.

Enable Packet Inspector 1 (any version other than 1.11.8-am1.2, 1.11.8-am1.3, or 1.11.8-am1.4)#

Note

Perform this task only if your Aspen Mesh version is not 1.11.8-am1.2, 1.11.8-am1.3, or 1.11.8-am1.4.

Open your Aspen Mesh override values file (for example, aspen-mesh-override-values.yaml) in a text editor.
Enable the aggregator service and the HTTP filter by setting the value of .aspen-mesh-packet-inspector.enabled to true.
Save and close the file.
Perform an upgrade of Aspen Mesh from the current version to the same version.

Capture traffic#

Use the analysis service (either the network analysis tool or the analysis emulator) to capture traffic using Packet Inspector 1.

Disable Packet Inspector 1#

Important

Perform this task after you’re finished capturing traffic.

Open your Aspen Mesh override values file (for example, aspen-mesh-override-values.yaml) in a text editor.
Set the value of .aspen-mesh-packet-inspector.enabled to false.
If present, set the values of .aspen-mesh-packet-inspector.httpCapture.enabled and .aspen-mesh-packet-inspector.diameterCapture.enabled to false.
Save and close the file.
Perform an upgrade of Aspen Mesh from the current version to the same version.

Troubleshooting#

Troubleshooting guide#

The aggregator service doesn’t start.

Verify that the address of the analysis service is configured properly in your Aspen Mesh override values file, and ensure that the analysis service is running.
There are no packets being captured.

Do the following:
- Verify that sidecar proxies are injected in workloads that should be capturing packets.
- Inspect the sidecar proxy (istio-proxy) container logs for any communication errors with the aggregator service.
- Inspect the aggregator service for errors related to packets getting dropped or issues communicating with the analysis service.
The aggregator service crashed (OOM killed).

When there is a high enough load, it can exceed the aggregator service’s memory. Adjust the CPU limit (.resources.limits.cpu) and memory limit (.resources.limits.memory) for the aggregator service in the Aspen Mesh override values file as required for your workloads. (If the CPU limit is too low, more memory is needed to buffer data.)

Uninstall Packet Inspector 1#

When to follow these instructions#

Follow the instructions below in the following cases:

When you no longer plan to use Packet Inspector 1
Before you upgrade to a later version of Aspen Mesh

Perform the tasks in order.

Delete the Packet Inspector 1 HTTP filter#

Delete the Packet Inspector 1 HTTP filter:

$ kubectl delete envoyfilter http-capture-filter \
    --namespace istio-system

Make sure Packet Inspector 1 is disabled#

Is Packet Inspector 1 enabled?

You can use the following command to determine whether the Packet Inspector 1 aggregator service is installed, which indicates that Packet Inspector 1 is enabled:
```
$ kubectl get daemonset aspen-mesh-packet-inspector \
        --namespace istio-system
```
- Yes: Disable Packet Inspector 1.
- No: Stop. Packet Inspector 1 is already disabled.

Uninstall the analysis emulator#

Uninstall the analysis emulator if you previously installed it:

Note

Previous versions of this documentation specified a Helm release name of analysisemulator. If you installed the analysis emulator using that name, replace pi1-analysis-emulator with analysisemulator in the command below.
```
$ helm uninstall pi1-analysis-emulator --namespace analysis-emulator
```