Use Packet Inspector 1#

About Packet Inspector 1#

Introduction#

Capturing network packets via tools like tcpdump and analyzing them in Wireshark is one of the most commonly used techniques to troubleshoot failures and understand application behavior in distributed and microservices-based applications. However, using this technique in cloud-native environments like Kubernetes is challenging due to the zero-trust security model and encrypted traffic between all workloads with perfect forward secrecy.

Packet Inspector 1 solves these challenges by capturing packets in plaintext at the sidecar proxies before sent packets are encrypted or after received packets are decrypted and proxied to the services in your cluster. Packet Inspector 1 performs captures at the application layer (Layer 7) instead of providing a TCP (Layer 4) level capture.

The captures contain rich contextual cloud-native data like your pod and service name and namespace, making it possible to associate the captured packets with workloads running in your Kubernetes cluster. Additionally, the packet captures are serialized in the well known libpcap format and can be transported to an analysis service and analyzed in Wireshark.

Supported protocols#

Packet Inspector 1 supports the following protocols:

Diameter
HTTP (1.1 and 2), including gRPC

Note

Diameter packet captures from ingress and egress gateways are not supported.

Packet Inspector 1 block diagram#

This diagram shows the components of Packet Inspector 1 and how data travels among them:

Packet Inspector 1 components#

Packet Inspector 1 consists of the following components:

Component	Description
Filters	Extensions of Istio proxies (whether used as sidecars or in a gateway) that capture and forward data
Aggregator service	A daemon set that receives capture data and finalizes it in the `pcap` format before transporting it to the analysis service

Types of analysis services#

You can send captured traffic to either of the following types of analysis services:

Analysis service	Purpose
A third-party network analysis tool	For use in production environments to analyze captured data
The Aspen Mesh analysis emulator	For use in non-production environments to demonstrate or test Packet Inspector 1

About the analysis emulator#

Types of files written by the analysis emulator#

For each capture, the analysis emulator writes two types of files to the /tmp directory:

The raw-request-and-metadata (.bson) file
The corresponding .pcap file

Analysis-emulator HTTP filename formats#

HTTP files written by the analysis emulator use the following filename formats:

Role	Filename format	Examples
Consumer	`<podName>-consumer-<xTraceId>-<sequenceNumber>`	`sleep-856f4f69cd-wkxw6-consumer-ba9f745f4018e235dc32f0e6b6321286-1.bson` `sleep-856f4f69cd-wkxw6-consumer-ba9f745f4018e235dc32f0e6b6321286-1.pcap`
Producer	`<apiPath>-producer-<xTraceId>-<sequenceNumber>`	`flasgger_static-producer-ba9f745f4018e235dc32f0e6b6321286-1.bson` `flasgger_static-producer-ba9f745f4018e235dc32f0e6b6321286-1.pcap`

Analysis-emulator Diameter filename formats#

Diameter files written by the analysis emulator use the following filename format:

<podName>-<endToEndId>-<hopByHopId>-<timestamp>

Examples

diameter-client-6464c65b66-4c8jl-1339708294-1857878061-1675782061987009574.bson
diameter-client-6464c65b66-4c8jl-1339708294-1857878061-1675782061987009574.pcap

Prepare to capture traffic with Packet Inspector 1#

Introduction#

To prepare to capture traffic using Packet Inspector 1, you need to:

Perform a clean installation of Citadel
Enable Citadel workload certificates for the aggregator-service namespace
Install the network analysis tool or the analysis emulator
Create an override values file for the Packet Inspector 1 aggregator
Install the Packet Inspector 1 aggregator

When to follow these instructions#

Follow the instructions below only if you want to start using Packet Inspector 1 to capture traffic and only after you’ve installed Aspen Mesh. Perform the tasks in order.

Perform a clean installation of Citadel#

Perform a clean installation of Citadel.

Note

As part of performing a clean installation of Citadel, you’ll do the following:

Plug in CA certificates so on-mesh non-gateway pods (those with sidecars) can communicate over an encrypted connection with the Packet Inspector 1 aggregator service (off mesh).
Install Citadel so the Packet Inspector 1 aggregator service (off mesh) can communicate over an encrypted connection with the analysis service (also off mesh).

Enable Citadel workload certificates for the aggregator-service namespace#

Enable Citadel workload certificates for the istio-system namespace so Citadel creates and maintains a workload certificate (in the form of a secret named istio.<aggregatorServiceServiceAccountName>) for the aggregator service’s service account, which allows the analysis service to perform identity validation of the aggregator service:
```
$ kubectl label namespace istio-system \
    ca.istio.io/override=true --overwrite
```

Install the analysis service#

Do you want to use a network analysis tool (not the analysis emulator) as your analysis service?
- Yes:
  - Install the network analysis tool.
- No:
  - Install the analysis emulator.

Install the network analysis tool#

Note

Perform this task only when other instructions tell you to do so.

Make sure the following requirements are met:
- Use the same name for the network analysis tool’s service and service account.
  
  When connecting to the network analysis tool, the Packet Inspector 1 aggregator service checks the network analysis tool’s certificate for a DNS name in the SAN field that matches the address of the network analysis tool. By default, the DNS name that Citadel adds to the network analysis tool’s certificate is a fully qualified domain name (FQDN) based on the network analysis tool’s service-account name (<networkAnalysisToolServiceAccountName>.<namespaceName>.svc.cluster.local); however, the address of the network analysis tool is based on the network analysis tool’s service name (<networkAnalysisToolServiceName>.<namespaceName>.svc.cluster.local). To make sure the check is successful, the network analysis tool’s service and service account must have the same name.
- Enable Citadel workload certificates for the namespace in which you’ll install the network analysis tool so Citadel creates and maintains a workload certificate (in the form of a secret named istio.<networkAnalysisToolServiceAccountName>) for the network analysis tool’s service account:
```
apiVersion: v1
kind: Namespace
metadata:
  name: <networkAnalysisToolNamespaceName>
  labels:
    ca.istio.io/override: "true"
```
- In the pod spec for the network analysis tool, add an istio-certs volume and set .spec.volumes[].secret.secretName for that volume to istio.<networkAnalysisToolServiceAccountName>, where istio.<networkAnalysisToolServiceAccountName> is the network analysis tool’s service account’s Citadel workload certificate (in the form of a secret) that’s in the same namespace as the network analysis tool:
```
volumes:
- name: istio-certs
  secret:
    defaultMode: 420
    optional: true
    secretName: istio.<networkAnalysisToolServiceAccountName>
```
- In the container spec for the network analysis tool, add a volume mount that references the istio-certs volume and mounts it at /etc/analysis/certs: In the container spec for the network analysis tool, add a volume mount that references the istio-certs volume and mounts it at the directory in which the network analysis tool expects to find the credential files from the Citadel workload certificate (for example, /etc/analysis/certs):
  
  Example
```
volumeMounts:
- name: istio-certs
  mountPath: /etc/analysis/certs
```
Follow the installation instructions provided with the network analysis tool.

Install the analysis emulator#

Note

Perform this task only when other instructions tell you to do so.

Change to the Aspen Mesh release directory.

Create a namespace named analysis-emulator:

$ kubectl create namespace analysis-emulator

Enable Citadel workload certificates for the analysis-emulator namespace so Citadel creates and maintains a workload certificate (in the form of a secret named istio.<analysisEmulatorServiceAccountName>) for the analysis emulator’s service account, which allows the aggregator service to perform identity validation of the analysis emulator:
```
$ kubectl label namespace analysis-emulator \
    ca.istio.io/override=true --overwrite
```

Install the analysis emulator:

$ helm install pi1-analysis-emulator \
    samples/aspenmesh/packet-inspector-1-analysis-emulator \
    --namespace analysis-emulator \
    --wait

About Packet Inspector 1 aggregator configuration#

The Helm chart for the Packet Inspector 1 aggregator includes a values.yaml file that contains fields that Helm uses to generate the manifest files during an installation or an upgrade. You can change the default value of a field by doing both of the following:

Include the field in the override values file for the Packet Inspector 1 aggregator and specify a different value.
Specify the override values file and a namespace when installing the Packet Inspector 1 aggregator.

Create an override values file for the Packet Inspector 1 aggregator#

Change to the Aspen Mesh release directory.
Change to the directory for the Packet Inspector 1 aggregator:
```
$ cd manifests/charts/packet-inspector
```
Create an override values file for the Packet Inspector 1 aggregator by copying the Packet Inspector 1 aggregator values.yaml file to a new file (for example, pi1-aggregator-override-values.yaml):
```
$ cp values.yaml <packetInspector1AggregatorOverrideValuesFilename>
```
Open the override values file for the Packet Inspector 1 aggregator in a text editor.

Do you want to send captured traffic to a network analysis tool (not the analysis emulator)?

Yes:

Set the value of .analysis.address to:

<networkAnalysisToolServiceName>.<networkAnalysisToolNamespaceName>.svc.cluster.local:<port>

Replace the placeholders:

Replace this placeholder	With
`<networkAnalysisToolServiceName>`	The network analysis tool’s Kubernetes service name
`<networkAnalysisToolNamespaceName>`	The namespace in which the network analysis tool is installed
`<port>`	The port on which the network analysis tool is listening

Go to step 6.

No:
- Set the value of .analysis.address to:
  
  packet-inspector-1-analysis-emulator.analysis-emulator.svc.cluster.local:21000
- Go to step 6.

Is Aspen Mesh running in IPv4/IPv6 dual-stack mode in a dual-stack cluster (a cluster with IPv4/IPv6 dual-stack networking enabled)?
- Yes:
  - Change the value of the .service.ipFamilyPolicy and .service.ipFamilies fields so they specify that the Packet Inspector 1 aggregator should also run in IPv4/IPv6 dual-stack mode:
    service: ipFamilyPolicy: RequireDualStack ipFamilies: - IPv4 - IPv6
  - Go to step 7.
- No:
  - Go to step 7.
Read the comments for each remaining field and comment out or remove any unrequired fields that you don’t want to override.
Change the values of the remaining fields.
Save and close the file.
Move the override values file for the Packet Inspector 1 aggregator to a permanent location.

You’ll need access to this file when installing the Packet Inspector 1 aggregator, and you should keep the file under version control.

Install the Packet Inspector 1 aggregator#

Install the aggregator:

$ helm install packet-inspector-1-aggregator \
    manifests/charts/packet-inspector \
    --namespace istio-system \
    --values <aggregatorOverrideValuesFilename> \
    --wait

Capture traffic using a Packet Inspector 1 filter instance#

Introduction#

Before you can capture traffic using a Packet Inspector 1 filter instance, you need to:

(Optional) Label the set of pods you want to capture traffic from
Create an override values file for the Packet Inspector 1 filter instance
Install the Packet Inspector 1 filter instance

After you capture traffic, you need to uninstall the Packet Inspector 1 filter instance to prevent it from lowering the performance of your cluster.

When to follow these instructions#

Follow the instructions below only if you want to capture traffic using Packet Inspector 1 and only after you’ve prepared to capture traffic with Packet Inspector 1. Perform the tasks in order.

About Packet Inspector 1 filter configuration#

The Helm chart for the Packet Inspector 1 filter includes a values.yaml file that contains fields that Helm uses to generate the manifest files during an installation or an upgrade. You can change the default value of a field by doing both of the following:

Include the field in the override values file for a Packet Inspector 1 filter instance and specify a different value.
Specify the override values file and a namespace when installing the Packet Inspector 1 filter instance.

Configuration overview for a Packet Inspector 1 filter instance#

This table provides an overview of how to configure a Packet Inspector 1 filter instance, depending on what traffic you want to capture.

To capture traffic from	Do this
All pods with sidecars, regardless of namespace	- In the override values file for the Packet Inspector 1 filter instance, do not add a key/value pair under `.workloadSelector.labels`. - Install the Packet Inspector 1 filter instance in the `istio-system` namespace.
All pods with sidecars in a namespace	- In the override values file for the Packet Inspector 1 filter instance, do not add a key/value pair under `.workloadSelector.labels`. - Install the Packet Inspector 1 filter instance in the namespace. Important: The namespace can be any namespace other than the `istio-system` namespace.
A set of labeled pods with sidecars, regardless of namespace	- Label each pod in the set with a unique key/value pair for the set. - In the override values file for the Packet Inspector 1 filter instance, add the key/value pair under `.workloadSelector.labels`. - Install the Packet Inspector 1 filter instance in the `istio-system` namespace.
A set of labeled pods with sidecars in a namespace	- Label each pod in the set with a unique key/value pair for the set. - In the override values file for the Packet Inspector 1 filter instance, add the key/value pair under `.workloadSelector.labels`. - Install the Packet Inspector 1 filter instance in the namespace that contains the labeled pods.

Label the set of pods you want to capture traffic from#

Important

Perform this task only if you want to capture traffic from a set of labeled pods.

Label each pod in the set with a unique key/value pair (for example, pi1-bookinfo-app-reviews2 = true).

Create an override values file for the Packet Inspector 1 filter instance#

Change to the Aspen Mesh release directory.
Change to the directory for the Packet Inspector 1 filter chart:
```
$ cd manifests/charts/packet-inspector-1-filter
```
Create an override values file for the Packet Inspector 1 filter instance by copying the Packet Inspector 1 filter values.yaml file to a new file (for example, pi1-filter-override-values-bookinfo-app-reviews2.yaml):
```
$ cp values.yaml <instanceOverrideValuesFilename>
```
Open the override values file for the Packet Inspector 1 filter instance in a text editor.
Do you want to capture HTTP traffic?
- Yes:
  - Make sure the value of .httpCapture.enabled is true.
  - Go to step 6.
- No:
  - Make sure the value of .httpCapture.enabled is false.
  - Go to step 6.
Do you want to capture Diameter traffic?
- Yes:
  - Make sure the value of .diameterCapture.enabled is true.
  - Make sure .diameterCapture.port specifies the port being used for Diameter traffic (typically 3868).
  - Go to step 7.
- No:
  - Make sure the value of .diameterCapture.enabled is false.
  - Go to step 7.
Read the comments for each remaining field and comment out or remove any unrequired fields that you don’t want to override.

Remember, if you want the Packet Inspector 1 filter instance to capture traffic from a set of labeled pods, you must add the key/value pair for the set (for example, pi1-bookinfo-app-reviews2 = true) under .workloadSelector.labels, as mentioned in Configuration overview for a Packet Inspector 1 filter instance.
Change the values of the remaining fields.
Save and close the file.
Move the override values file for the Packet Inspector 1 filter instance to a permanent location.

You’ll need access to this file when installing the Packet Inspector 1 filter instance, and you should keep the file under version control.

Install the Packet Inspector 1 filter instance#

Change to the Aspen Mesh release directory.

Install the Packet Inspector 1 filter instance:

$ helm install <packetInspector1FilterInstanceName> \
    manifests/charts/packet-inspector-1-filter \
    --namespace <instanceNamespaceName> \
    --values <instanceOverrideValuesFilename> \
    --wait

Capture traffic using the Packet Inspector 1 filter instance#

Use the analysis service (either the network analysis tool or the analysis emulator) to capture traffic using the Packet Inspector 1 filter instance.

Uninstall the Packet Inspector 1 filter instance#

Important

Perform this task after you’re finished capturing traffic.

Uninstall the Packet Inspector 1 filter instance:

$ helm uninstall <packetInspector1FilterInstanceName> \
    --namespace <instanceNamespaceName> \
    --wait

Troubleshooting#

Troubleshooting guide#

The aggregator service doesn’t start.

Verify that the address of the analysis service is configured properly in the override values file for the Packet Inspector 1 aggregator, and ensure that the analysis service is running.
There are no packets being captured.

Do the following:
- Verify that sidecar proxies are injected in workloads that should be capturing packets.
- Inspect the sidecar proxy (istio-proxy) container logs for any communication errors with the aggregator service.
- Inspect the aggregator service for errors related to packets getting dropped or issues communicating with the analysis service.
The aggregator service crashed (OOM killed).

When there is a high enough load, it can exceed the aggregator service’s memory. Adjust the CPU limit (.resources.limits.cpu) and memory limit (.resources.limits.memory) for the aggregator service in the Packet Inspector 1 aggregator override values file as required for your workloads. (If the CPU limit is too low, more memory is needed to buffer data.)

Uninstall Packet Inspector 1#

When to follow these instructions#

Follow the instructions below in the following cases:

When you no longer plan to use Packet Inspector 1
Before you upgrade to a later version of Aspen Mesh

Perform the tasks in order.

Uninstall all Packet Inspector 1 filter instances#

Are any Packet Inspector 1 filter instances installed?

You can use the following command to determine whether any Packet Inspector 1 filter instances are installed:
```
$ kubectl get envoyfilters --all-namespaces | grep packet-inspector-1-filter-
```
- Yes: Uninstall each Packet Inspector 1 filter instance.
- No: Stop. All Packet Inspector 1 filter instances are uninstalled.

Uninstall the Packet Inspector 1 aggregator#

Uninstall the aggregator:

$ helm uninstall packet-inspector-1-aggregator \
    --namespace istio-system \
    --wait

Uninstall the analysis emulator#

Uninstall the analysis emulator if you previously installed it:

$ helm uninstall pi1-analysis-emulator --namespace analysis-emulator --wait