Use Packet Inspector 1#

About Packet Inspector 1#

Introduction#

Capturing network packets via tools like tcpdump and analyzing them in Wireshark is one of the most commonly used techniques to troubleshoot failures and understand application behavior in distributed and microservices-based applications. However, using this technique in cloud-native environments like Kubernetes is challenging due to the zero-trust security model and encrypted traffic between all workloads with perfect forward secrecy.

Packet Inspector 1 solves these challenges by capturing packets in plaintext at the sidecar proxies before sent packets are encrypted or after received packets are decrypted and proxied to the services in your cluster. Packet Inspector 1 performs captures at the application layer (Layer 7) instead of providing a TCP (Layer 4) level capture.

The captures contain rich contextual cloud-native data like your pod and service name and namespace, making it possible to associate the captured packets with workloads running in your Kubernetes cluster. Additionally, the packet captures are serialized in the well known libpcap format and can be transported to an analysis service and analyzed in Wireshark.

Supported protocols#

Packet Inspector 1 supports the following protocols:

Diameter (Aspen Mesh 1.14.6-am1 and later)
HTTP (1.1 and 2), including gRPC

Note

Diameter packet captures from ingress and egress gateways are not supported.

Packet Inspector 1 block diagram#

This diagram shows the components of Packet Inspector 1 and how data travels among them:

Packet Inspector 1 components#

Packet Inspector 1 consists of the following components:

Component	Description
Filters	Extensions of Envoy proxies (whether used as sidecar proxies or in a gateway) that capture and forward data
Aggregator service	A daemon set that receives capture data and finalizes it in the `pcap` format before transporting it to the analysis service

Types of analysis services#

You can send captured traffic to either of the following types of analysis services:

Analysis service	Purpose
A third-party network analysis tool	For use in production environments to analyze captured data
The Aspen Mesh analysis emulator	For use in non-production environments to demonstrate or test Packet Inspector 1

About the analysis emulator#

Types of files written by the analysis emulator#

For each capture, the analysis emulator writes two types of files to the /tmp directory:

The raw-request-and-metadata (.bson) file
The corresponding .pcap file

Analysis-emulator HTTP filename formats#

HTTP files written by the analysis emulator use the following filename formats:

Role	Filename format	Examples
Consumer	`<podName>-consumer-<xTraceId>-<sequenceNumber>`	`sleep-856f4f69cd-wkxw6-consumer-ba9f745f4018e235dc32f0e6b6321286-1.bson` `sleep-856f4f69cd-wkxw6-consumer-ba9f745f4018e235dc32f0e6b6321286-1.pcap`
Producer	`<apiPath>-producer-<xTraceId>-<sequenceNumber>`	`flasgger_static-producer-ba9f745f4018e235dc32f0e6b6321286-1.bson` `flasgger_static-producer-ba9f745f4018e235dc32f0e6b6321286-1.pcap`

Analysis-emulator Diameter filename formats#

Diameter files written by the analysis emulator use the following filename format:

<podName>-<endToEndId>-<hopByHopId>-<timestamp>

Examples

diameter-client-6464c65b66-4c8jl-1339708294-1857878061-1675782061987009574.bson
diameter-client-6464c65b66-4c8jl-1339708294-1857878061-1675782061987009574.pcap

Prepare to capture traffic with Packet Inspector 1#

Introduction#

To prepare to capture traffic using Packet Inspector 1, you need to:

Plug in certificate-authority (CA) certificates
Enable traditional DNS-name SAN extensions in workload certificates
Install Citadel
Add the certificate-secret label to the aggregator-service namespace
Install the network analysis tool or the analysis emulator
Create an override values file for the Packet Inspector 1 aggregator
Install the Packet Inspector 1 aggregator

When to follow these instructions#

Follow the instructions below only if you want to start using Packet Inspector 1 to capture traffic and only after you’ve installed Aspen Mesh. Perform the tasks in order.

Why you must plug in CA certificates#

Mutual TLS (mTLS) communication is used between the Packet Inspector 1 aggregator service (off mesh) and the sidecars (on mesh), so their certificates must be generated and signed by the same certificate-authority (CA) certificate. To ensure the CA certificate used by Citadel (to generate and sign certificates for the aggregator service) and the Aspen Mesh CA (to generate and sign certificates for the sidecars) is the same, you must plug in CA certificates.

Plug in CA certificates#

Follow the instructions on configuring certificates to plug in CA certificates.

Why Packet Inspector 1 requires certificates with traditional DNS-name SAN extensions#

The analysis service is required to use a certificate with a traditional DNS-name Subject Alternative Name (SAN) extension for communication with the aggregator service. Learn about the format of the DNS-name SAN extension.

Enable traditional DNS-name SAN extensions in workload certificates#

Open your Aspen Mesh override values file (for example, aspen-mesh-override-values.yaml) in a text editor.

Copy the following information and paste it into the file:

global:
  # Enable traditional DNS-name and non-SPIFFE-URI SAN extensions in
  # Aspen Mesh certificate-authority (CA) workload certificates and
  # Citadel certificate secrets.
  certificateCustomFields: true

Save and close the file.
Perform an upgrade of Aspen Mesh from the current version to the same version.

Why Packet Inspector 1 requires Citadel#

TLS communication is used between the Packet Inspector 1 aggregator service and analysis service, both of which are off mesh. To enable TLS communication between these services, you must enable Citadel and apply a certificate-secret label to the aggregator-service and analysis-service namespaces so Citadel generates and signs certificates (in the form of Kubernetes secrets) for use by those services. Learn more about using the Citadel certificate-secret label.

Install Citadel#

Install Citadel.

Add the certificate-secret label to the aggregator-service namespace#

Add the certificate-secret label to the istio-system namespace so Citadel generates a certificate (in the form of a Kubernetes secret) that allows the aggregator service to communicate over TLS with the analysis service:
```
$ kubectl label namespace istio-system \
    ca.istio.io/override=true --overwrite
```

Install the analysis service#

Do you want to use a network analysis tool (not the analysis emulator) as your analysis service?
- Yes:
  - Install the network analysis tool.
- No:
  - Install the analysis emulator.

Install the network analysis tool#

Note

Perform this task only when other instructions tell you to do so.

Make sure the following requirements are met:
- Add the certificate-secret label to the namespace in which you’ll install the network analysis tool so Citadel generates a certificate (in the form of a Kubernetes secret) that allows the network analysis tool to communicate over TLS with the aggregator service:
```
apiVersion: v1
kind: Namespace
metadata:
  name: <networkAnalysisToolNamespaceName>
  labels:
    ca.istio.io/override: "true"
```
- Use the same name for the network analysis tool’s service and service account.
  
  When connecting to the analysis service (either the network analysis tool or the analysis emulator), the Packet Inspector 1 aggregator service checks the analysis service’s certificate for a DNS-name SAN extension that matches the address of the analysis service. The DNS-name SAN extension that Citadel adds to the analysis service’s certificate is a fully qualified domain name (FQDN) based on the analysis service’s service-account name; however, the address of the analysis service is based on the analysis service’s service name. To make sure the check is successful, the network analysis tool’s service and service account must have the same name.
- In the pod spec for the network analysis tool, add the istio-certs volume for the certificate secret:
```
volumes:
- name: istio-certs
  secret:
    defaultMode: 420
    optional: true
    secretName: istio.<networkAnalysisToolServiceAccountName>
```
- In the container spec for the network analysis tool, add a volume mount that references the istio-certs volume and mounts it at /etc/analysis/certs:
```
volumeMounts:
- name: istio-certs
  mountPath: /etc/analysis/certs
```
Follow the installation instructions provided with the network analysis tool.

Install the analysis emulator#

Note

Perform this task only when other instructions tell you to do so.

Change to the Aspen Mesh release directory.

Create a namespace named analysis-emulator:

$ kubectl create namespace analysis-emulator

Add the certificate-secret label to the analysis-emulator namespace so Citadel generates a certificate (in the form of a Kubernetes secret) that allows the analysis emulator to communicate over TLS with the aggregator service:
```
$ kubectl label namespace analysis-emulator \
    ca.istio.io/override=true --overwrite
```

Install the analysis emulator:

$ helm install pi1-analysis-emulator \
    samples/aspenmesh/packet-inspector-1-analysis-emulator \
    --namespace analysis-emulator

About Packet Inspector 1 aggregator configuration#

The Helm chart for the Packet Inspector 1 aggregator includes a values.yaml file that contains fields that Helm uses to generate the manifest files during an installation or an upgrade. You can change the default value of a field by doing both of the following:

Include the field in the override values file for the Packet Inspector 1 aggregator and specify a different value.
Specify the override values file and a namespace when installing the Packet Inspector 1 aggregator.

Create an override values file for the Packet Inspector 1 aggregator#

Change to the Aspen Mesh release directory.
Change to the directory for the Packet Inspector 1 aggregator:
```
$ cd manifests/charts/packet-inspector
```
Create an override values file for the Packet Inspector 1 aggregator by copying the Packet Inspector 1 aggregator values.yaml file to a new file (for example, pi1-aggregator-override-values.yaml):
```
$ cp values.yaml <packetInspector1AggregatorOverrideValuesFilename>
```
Open the override values file for the Packet Inspector 1 aggregator in a text editor.

Do you want to send captured traffic to a network analysis tool (not the analysis emulator)?

Yes:

Set the value of .analysis.address to:

<networkAnalysisToolServiceName>.<networkAnalysisToolNamespaceName>.svc.cluster.local:<port>

Replace the placeholders:

Replace this placeholder	With
`<networkAnalysisToolServiceName>`	The network analysis tool’s Kubernetes service name
`<networkAnalysisToolNamespaceName>`	The namespace in which the network analysis tool is installed
`<port>`	The port on which the network analysis tool is listening

Go to step 6.

No:
- Set the value of .analysis.address to:
  
  packet-inspector-1-analysis-emulator.analysis-emulator.svc.cluster.local:21000
- Go to step 6.

Is Aspen Mesh running in IPv4/IPv6 dual-stack mode in a dual-stack cluster (a cluster with IPv4/IPv6 dual-stack networking enabled)?
- Yes:
  - Change the value of the .service.ipFamilyPolicy and .service.ipFamilies fields so they specify that the Packet Inspector 1 aggregator should also run in IPv4/IPv6 dual-stack mode:
    service: ipFamilyPolicy: RequireDualStack ipFamilies: - IPv4 - IPv6
  - Go to step 7.
- No:
  - Go to step 7.
Read the comments for each remaining field and comment out or remove any unrequired fields that you don’t want to override.
Change the values of the remaining fields.
Save and close the file.
Move the override values file for the Packet Inspector 1 aggregator to a permanent location.

You’ll need access to this file when installing the Packet Inspector 1 aggregator, and you should keep the file under version control.

Install the Packet Inspector 1 aggregator#

Add the certificate-secret label to the istio-system namespace so Citadel generates a certificate (in the form of a Kubernetes secret) that allows the aggregator service to communicate over TLS with the analysis service:
```
$ kubectl label namespace istio-system \
    ca.istio.io/override=true --overwrite
```

Install the aggregator:

$ helm install packet-inspector-1-aggregator \
    manifests/charts/packet-inspector \
    --namespace istio-system \
    --values <aggregatorOverrideValuesFilename>

Capture traffic using a Packet Inspector 1 filter instance#

Introduction#

Before you can capture traffic using a Packet Inspector 1 filter instance, you need to:

(Optional) Label the set of pods you want to capture traffic from
Create an override values file for the Packet Inspector 1 filter instance
Install the Packet Inspector 1 filter instance

After you capture traffic, you need to uninstall the Packet Inspector 1 filter instance to prevent it from lowering the performance of your cluster.

When to follow these instructions#

Follow the instructions below only if you want to capture traffic using Packet Inspector 1 and only after you’ve prepared to capture traffic with Packet Inspector 1. Perform the tasks in order.

About Packet Inspector 1 filter configuration#

The Helm chart for the Packet Inspector 1 filter includes a values.yaml file that contains fields that Helm uses to generate the manifest files during an installation or an upgrade. You can change the default value of a field by doing both of the following:

Include the field in the override values file for a Packet Inspector 1 filter instance and specify a different value.
Specify the override values file and a namespace when installing the Packet Inspector 1 filter instance.

Configuration and installation overview for a Packet Inspector 1 filter instance#

This table provides an overview of how you’ll configure and install a Packet Inspector 1 filter instance, depending on what traffic you want to capture.

If you want to capture traffic from	Then you’ll do this
All pods with sidecars, regardless of namespace	- Install the filter instance in the `istio-system` namespace. Note: In the override values file for the filter instance, do not add a key/value pair under `.workloadSelector.labels`.
All pods with sidecars in a namespace	- Install the filter instance in the namespace. Note: - In the override values file for the filter instance, do not add a key/value pair under `.workloadSelector.labels`. - Do not install the filter instance in the `istio-system` namespace.
A set of labeled pods with sidecars, regardless of namespace	- Label each pod in the set with a unique key/value pair for the set. - In the override values file for the filter instance, add the key/value pair under `.workloadSelector.labels`. - Install the filter instance in the `istio-system` namespace.
A set of labeled pods with sidecars in a namespace	- Label each pod in the set with a unique key/value pair for the set. - In the override values file for the filter instance, add the key/value pair under `.workloadSelector.labels`. - Install the filter instance in the namespace that contains the labeled pods. Note: Do not install the filter instance in the `istio-system` namespace.

Label the set of pods you want to capture traffic from#

Important

Perform this task only if you want to capture traffic from a set of labeled pods.

Label each pod in the set with a unique key/value pair (for example, pi1-bookinfo-app-reviews2 = true).

Create an override values file for the Packet Inspector 1 filter instance (1.14.6-am1 or later)#

Note

Perform this task only if your Aspen Mesh version is 1.14.6-am1 or later.

Change to the Aspen Mesh release directory.
Change to the directory for the Packet Inspector 1 filter chart:
```
$ cd manifests/charts/packet-inspector-1-filter
```
Create an override values file for the Packet Inspector 1 filter instance by copying the Packet Inspector 1 filter values.yaml file to a new file (for example, pi1-filter-override-values-bookinfo-app-reviews2.yaml):
```
$ cp values.yaml <instanceOverrideValuesFilename>
```
Open the override values file for the Packet Inspector 1 filter instance in a text editor.
Do you want to capture HTTP traffic?
- Yes:
  - Make sure the value of .httpCapture.enabled is true.
  - Go to step 6.
- No:
  - Make sure the value of .httpCapture.enabled is false.
  - Go to step 6.
Do you want to capture Diameter traffic?
- Yes:
  - Make sure the value of .diameterCapture.enabled is true.
  - Make sure .diameterCapture.port specifies the port being used for Diameter traffic (typically 3868).
  - Go to step 7.
- No:
  - Make sure the value of .diameterCapture.enabled is false.
  - Go to step 7.
Read the comments for each remaining field and comment out or remove any unrequired fields that you don’t want to override.

Remember, if you want the Packet Inspector 1 filter instance to capture traffic from a set of labeled pods, you must add the key/value pair for the set (for example, pi1-bookinfo-app-reviews2 = true) under .workloadSelector.labels, as mentioned in Configuration and installation overview for a Packet Inspector 1 filter instance.
Change the values of the remaining fields.
Save and close the file.
Move the override values file for the Packet Inspector 1 filter instance to a permanent location.

You’ll need access to this file when installing the Packet Inspector 1 filter instance, and you should keep the file under version control.

Create an override values file for the Packet Inspector 1 filter instance (1.14.5-am1)#

Note

Perform this task only if your Aspen Mesh version is 1.14.5-am1.

Change to the Aspen Mesh release directory.
Change to the directory for the Packet Inspector 1 filter chart:
```
$ cd manifests/charts/packet-inspector-1-filter
```
Create an override values file for the Packet Inspector 1 filter instance by copying the Packet Inspector 1 filter values.yaml file to a new file (for example, pi1-filter-override-values-bookinfo-app-reviews2.yaml):
```
$ cp values.yaml <instanceOverrideValuesFilename>
```
Open the override values file for the Packet Inspector 1 filter instance in a text editor.
Read the comments for each field and comment out or remove any unrequired fields that you don’t want to override.

Remember, if you want the Packet Inspector 1 filter instance to capture traffic from a set of labeled pods, you must add the key/value pair for the set (for example, pi1-bookinfo-app-reviews2 = true) under .workloadSelector.labels, as mentioned in Configuration and installation overview for a Packet Inspector 1 filter instance.
Change the values of the remaining fields.
Save and close the file.
Move the override values file for the Packet Inspector 1 filter instance to a permanent location.

You’ll need access to this file when installing the Packet Inspector 1 filter instance, and you should keep the file under version control.

Install the Packet Inspector 1 filter instance#

Change to the Aspen Mesh release directory.

Install the Packet Inspector 1 filter instance:

$ helm install <packetInspector1FilterInstanceName> \
    manifests/charts/packet-inspector-1-filter \
    --namespace <instanceNamespaceName> \
    --values <instanceOverrideValuesFilename>

Capture traffic using the Packet Inspector 1 filter instance#

Use the analysis service (either the network analysis tool or the analysis emulator) to capture traffic using the Packet Inspector 1 filter instance.

Uninstall the Packet Inspector 1 filter instance#

Important

Perform this task after you’re finished capturing traffic.

Uninstall the Packet Inspector 1 filter instance:

$ helm uninstall <packetInspector1FilterInstanceName> \
    --namespace <instanceNamespaceName>

Troubleshooting#

Troubleshooting guide#

The aggregator service doesn’t start.

Verify that the address of the analysis service is configured properly in the override values file for the Packet Inspector 1 aggregator, and ensure that the analysis service is running.
There are no packets being captured.

Do the following:
- Verify that sidecar proxies are injected in workloads that should be capturing packets.
- Inspect the sidecar proxy (istio-proxy) container logs for any communication errors with the aggregator service.
- Inspect the aggregator service for errors related to packets getting dropped or issues communicating with the analysis service.
The aggregator service crashed (OOM killed).

When there is a high enough load, it can exceed the aggregator service’s memory. Adjust the CPU limit (.resources.limits.cpu) and memory limit (.resources.limits.memory) for the aggregator service in the Packet Inspector 1 aggregator override values file as required for your workloads. (If the CPU limit is too low, more memory is needed to buffer data.)

Uninstall Packet Inspector 1#

When to follow these instructions#

Follow the instructions below in the following cases:

When you no longer plan to use Packet Inspector 1
Before you upgrade to a later version of Aspen Mesh

Perform the tasks in order.

Uninstall all Packet Inspector 1 filter instances#

Are any Packet Inspector 1 filter instances installed?

You can use the following command to determine whether any Packet Inspector 1 filter instances are installed:
```
$ kubectl get envoyfilters --all-namespaces | grep packet-inspector-1-filter-
```
- Yes: Uninstall each Packet Inspector 1 filter instance.
- No: Stop. All Packet Inspector 1 filter instances are uninstalled.

Uninstall the Packet Inspector 1 aggregator#

Uninstall the aggregator:

$ helm uninstall packet-inspector-1-aggregator \
    --namespace istio-system

Uninstall the analysis emulator#

Uninstall the analysis emulator if you previously installed it:

$ helm uninstall pi1-analysis-emulator --namespace analysis-emulator