Use Packet Inspector 1#

About Packet Inspector 1#

Introduction#

Capturing network packets via tools like tcpdump and analyzing them in Wireshark is one of the most commonly used techniques to troubleshoot failures and understand application behavior in distributed and microservices-based applications. However, using this technique in cloud-native environments like Kubernetes is challenging due to the zero-trust security model and encrypted traffic between all workloads with perfect forward secrecy.

Packet Inspector 1 solves these challenges by capturing packets in plaintext at the sidecar proxies before sent packets are encrypted or after received packets are decrypted and proxied to the services in your cluster. Packet Inspector 1 performs captures at the application layer (Layer 7) instead of providing a TCP (Layer 4) level capture.

The captures contain rich contextual cloud-native data like your pod and service name and namespace, making it possible to associate the captured packets with workloads running in your Kubernetes cluster. Additionally, the packet captures are serialized in the well known libpcap format and can be transported to an analysis service and analyzed in Wireshark.

Supported protocols#

Packet Inspector 1 supports the following protocols:

Diameter
HTTP (1.1 and 2), including gRPC

Note

Diameter packet captures from ingress and egress gateways are not supported.

Packet Inspector 1 block diagram#

This diagram shows the components of Packet Inspector 1 and how data travels among them:

Packet Inspector 1 components#

Packet Inspector 1 consists of the following components:

Component	Description
Filters	Extensions of Istio proxies (whether used as sidecars or in a gateway) that capture and forward data
Aggregator service	A daemon set that receives capture data and finalizes it in the `pcap` format before transporting it to the analysis service

Types of analysis services#

You can send captured traffic to either of the following types of analysis services:

Analysis service	Purpose
A third-party network analysis tool	For use in production environments to analyze captured data
The Packet Inspector 1 analysis emulator	For use in non-production environments to demonstrate or test Packet Inspector 1

About the Packet Inspector 1 analysis emulator#

Types of files written by the Packet Inspector 1 analysis emulator#

For each capture, the Packet Inspector 1 analysis emulator writes two types of files to the /tmp directory:

The raw-request-and-metadata (.bson) file
The corresponding .pcap file

Packet Inspector 1 analysis-emulator HTTP filename formats#

HTTP files written by the Packet Inspector 1 analysis emulator use the following filename formats:

Role	Filename format	Examples
Consumer	`<podName>-consumer-<xTraceId>-<sequenceNumber>`	`sleep-856f4f69cd-wkxw6-consumer-ba9f745f4018e235dc32f0e6b6321286-1.bson` `sleep-856f4f69cd-wkxw6-consumer-ba9f745f4018e235dc32f0e6b6321286-1.pcap`
Producer	`<apiPath>-producer-<xTraceId>-<sequenceNumber>`	`flasgger_static-producer-ba9f745f4018e235dc32f0e6b6321286-1.bson` `flasgger_static-producer-ba9f745f4018e235dc32f0e6b6321286-1.pcap`

Packet Inspector 1 analysis-emulator Diameter filename formats#

Diameter files written by the Packet Inspector 1 analysis emulator use the following filename format:

<podName>-<endToEndId>-<hopByHopId>-<timestamp>

Examples

diameter-client-6464c65b66-4c8jl-1339708294-1857878061-1675782061987009574.bson
diameter-client-6464c65b66-4c8jl-1339708294-1857878061-1675782061987009574.pcap

Prepare to capture traffic with Packet Inspector 1#

Introduction#

To prepare to capture traffic using Packet Inspector 1, you need to:

Perform a clean installation of Citadel
Enable Citadel workload certificates for the aggregator-service namespace
Install the analysis service (either the network analysis tool or the Packet Inspector 1 analysis emulator)
Create an override values file for the Packet Inspector 1 aggregator
(Private artifact registry only) Override the default repository address and the default artifact registry’s pull-secret name for the Packet Inspector 1 aggregator
Install the Packet Inspector 1 aggregator

When to follow these instructions#

Follow the instructions below only if you want to start using Packet Inspector 1 to capture traffic and only after you’ve installed Aspen Mesh. Perform the tasks in order.

Perform a clean installation of Citadel#

Perform a clean installation of Citadel.

Note

As part of performing a clean installation of Citadel, you’ll do the following:

Plug in CA certificates so on-mesh non-gateway pods (those with sidecars) can communicate over an encrypted connection with the Packet Inspector 1 aggregator service (off mesh).
Install Citadel so the Packet Inspector 1 aggregator service (off mesh) can communicate over an encrypted connection with the analysis service (also off mesh).

Enable Citadel workload certificates for the aggregator-service namespace#

Enable Citadel workload certificates for the istio-system namespace so Citadel creates and maintains a workload certificate (in the form of a secret named istio.<aggregatorServiceServiceAccountName>) for the aggregator service’s service account, which allows the analysis service to perform identity validation of the aggregator service:
```
$ kubectl label namespace istio-system \
    ca.istio.io/override=true --overwrite
```

Install the analysis service#

Do you want to use a network analysis tool (not the Packet Inspector 1 analysis emulator) as your analysis service?
- Yes:
  - Install the network analysis tool.
- No:

Install the network analysis tool#

Note

Perform this task only when other instructions tell you to do so.

Make sure the following requirements are met:
- Use the same name for the network analysis tool’s service and service account.
  
  When connecting to the network analysis tool, the Packet Inspector 1 aggregator service checks the network analysis tool’s certificate for a DNS name in the SAN field that matches the address of the network analysis tool. By default, the DNS name that Citadel adds to the network analysis tool’s certificate is a fully qualified domain name (FQDN) based on the network analysis tool’s service-account name (<networkAnalysisToolServiceAccountName>.<namespaceName>.svc.cluster.local); however, the address of the network analysis tool is based on the network analysis tool’s service name (<networkAnalysisToolServiceName>.<namespaceName>.svc.cluster.local). To make sure the check is successful, the network analysis tool’s service and service account must have the same name.
- Enable Citadel workload certificates for the namespace in which you’ll install the network analysis tool so Citadel creates and maintains a workload certificate (in the form of a secret named istio.<networkAnalysisToolServiceAccountName>) for the network analysis tool’s service account:
```
apiVersion: v1
kind: Namespace
metadata:
  name: <networkAnalysisToolNamespaceName>
  labels:
    ca.istio.io/override: "true"
```
- In the pod spec for the network analysis tool, add an istio-certs volume and set .spec.volumes[].secret.secretName for that volume to istio.<networkAnalysisToolServiceAccountName>, where istio.<networkAnalysisToolServiceAccountName> is the network analysis tool’s service account’s Citadel workload certificate (in the form of a secret) that’s in the same namespace as the network analysis tool:
```
volumes:
- name: istio-certs
  secret:
    defaultMode: 420
    optional: true
    secretName: istio.<networkAnalysisToolServiceAccountName>
```
- In the container spec for the network analysis tool, add a volume mount that references the istio-certs volume and mounts it at the directory in which the network analysis tool expects to find the credential files from the Citadel workload certificate (for example, /etc/analysis/certs):
  
  Example
```
volumeMounts:
- name: istio-certs
  mountPath: /etc/analysis/certs
```
Follow the installation instructions provided with the network analysis tool.

Create an override values file for the Packet Inspector 1 analysis emulator#

Note

Perform this task only when other instructions tell you to do so.

In a terminal window on your computer, make sure you’re in the Aspen Mesh Platform chart directory.
Create a file to use as as the override values file for the Packet Inspector 1 analysis emulator:
```
$ touch pi1-analysis-emulator-override-values.yaml
```
You’ll need access to this file when installing the Packet Inspector 1 analysis emulator and should keep the file under version control.

(Private artifact registry only) Override the default repository address and the default artifact registry’s pull-secret name for the Packet Inspector 1 analysis emulator#

Important

Perform this task only when other instructions tell you to do so and only if you stored Helm charts and Docker container images in a private artifact registry.

In a terminal window on your computer, make sure you’re in the Aspen Mesh Platform chart directory.
Open the override values file for the Packet Inspector 1 analysis emulator (pi1-analysis-emulator-override-values.yaml) in a text editor.
Does your private artifact registry require authentication?
- Yes:
  - Add the following information at the top level of the hierarchy:
    hub: <privateRepositoryAddress> imagePullSecrets: - name: <privateRegistryPullSecretName>
  - Replace <privateRegistryPullSecretName> with the name of the private artifact registry’s pull secret (for example, private-registry-pull-secret).
- No:
  - Add the following information at the top level of the hierarchy:
    hub: <privateRepositoryAddress> imagePullSecrets: []
Replace <privateRepositoryAddress> with the address of the Aspen Mesh repository in the private artifact registry (for example, registry.company.com/aspenmesh).
Save and close the file.

Install the Packet Inspector 1 analysis emulator#

Note

Perform this task only when other instructions tell you to do so.

Create a namespace named analysis-emulator:

$ kubectl create namespace analysis-emulator

Enable Citadel workload certificates for the analysis-emulator namespace so Citadel creates and maintains a workload certificate (in the form of a secret named istio.<analysisEmulatorServiceAccountName>) for the analysis emulator’s service account, which allows the aggregator service to perform identity validation of the analysis emulator:
```
$ kubectl label namespace analysis-emulator \
    ca.istio.io/override=true --overwrite
```
Log in to the appropriate artifact registry with Helm and set the repository address.
Set the AM_TAG environment variable to the version of the Packet Inspector 1 analysis emulator you want to install.

Example
```
$ export AM_TAG=1.21.6-am1
```
If you’re installing from a private artifact registry that requires authentication (that is, you must provide credentials to access the registry), create the private artifact registry’s pull secret (for example, private-registry-pull-secret) in the analysis-emulator namespace:
```
$ kubectl apply -f <privateRegistryPullSecretManifestFilename> \
    --namespace analysis-emulator
```

Install the Packet Inspector 1 analysis emulator:

$ helm install pi1-analysis-emulator \
    oci://$AM_REPO/charts/packet-inspector-1-analysis-emulator --version $AM_TAG \
    --namespace analysis-emulator \
    --values pi1-analysis-emulator-override-values.yaml \
    --wait

About Packet Inspector 1 aggregator configuration#

The Helm chart for the Packet Inspector 1 aggregator includes a values.yaml file that contains fields that Helm uses to generate the manifest files during an installation or an upgrade. You can change the default value of a field by doing both of the following:

Include the field in the override values file for the Packet Inspector 1 aggregator and specify a different value.
Specify the override values file when installing the Packet Inspector 1 aggregator.

Create an override values file for the Packet Inspector 1 aggregator#

In a terminal window on your computer, make sure you’re in the Aspen Mesh Platform chart directory.
Set the AM_TAG environment variable to the version of the Packet Inspector 1 aggregator you want to install.

Example
```
$ export AM_TAG=1.21.6-am1
```
Create an override values file for the Packet Inspector 1 aggregator by copying the Packet Inspector 1 aggregator values.yaml file to a new file (for example, pi1-aggregator-override-values.yaml):
```
$ helm show values oci://$AM_REPO/charts/packet-inspector --version $AM_TAG > <packetInspector1AggregatorOverrideValuesFilename>
```
Open the override values file for the Packet Inspector 1 aggregator in a text editor.

Do you want to send captured traffic to a network analysis tool (not the analysis emulator)?

Yes:

Set the value of .analysis.address to:

<networkAnalysisToolServiceName>.<networkAnalysisToolNamespaceName>.svc.cluster.local:<port>

Replace the placeholders:

Replace this placeholder	With
`<networkAnalysisToolServiceName>`	The network analysis tool’s Kubernetes service name
`<networkAnalysisToolNamespaceName>`	The namespace in which the network analysis tool is installed
`<port>`	The port on which the network analysis tool is listening

Go to step 6.

No:
- Set the value of .analysis.address to:
  
  packet-inspector-1-analysis-emulator.analysis-emulator.svc.cluster.local:21000
- Go to step 6.

If Aspen Mesh is running in IPv4/IPv6 dual-stack mode in a dual-stack cluster (a cluster with IPv4/IPv6 dual-stack networking enabled), change the value of the .service.ipFamilyPolicy and .service.ipFamilies fields so they specify that the Packet Inspector 1 aggregator should also run in IPv4/IPv6 dual-stack mode:
```
service:
  ipFamilyPolicy: RequireDualStack
  ipFamilies:
  - IPv4
  - IPv6
```
Read the comments for each remaining field and comment out or remove any unrequired fields that you don’t want to override.
Change the values of the remaining fields.
Save and close the file.

You’ll need access to this file when installing the Packet Inspector 1 aggregator, and you should keep the file under version control.

(Private artifact registry only) Override the default repository address and the default artifact registry’s pull-secret name for the Packet Inspector 1 aggregator#

Important

Perform this task only if you stored Helm charts and Docker container images in a private artifact registry.

In a terminal window on your computer, make sure you’re in the Aspen Mesh Platform chart directory.
Open the override values file for the Packet Inspector 1 aggregator in a text editor.
Does your private artifact registry require authentication?
- Yes:
  - Add the following information at the top level of the hierarchy:
    hub: <privateRepositoryAddress> imagePullSecrets: - name: <privateRegistryPullSecretName>
  - Replace <privateRegistryPullSecretName> with the name of the private artifact registry’s pull secret (for example, private-registry-pull-secret).
- No:
  - Add the following information at the top level of the hierarchy:
    hub: <privateRepositoryAddress> imagePullSecrets: []
Replace <privateRepositoryAddress> with the address of the Aspen Mesh repository in the private artifact registry (for example, registry.company.com/aspenmesh).
Save and close the file.

Install the Packet Inspector 1 aggregator#

In a terminal window on your computer, make sure you’re in the Aspen Mesh Platform chart directory.
Set the AM_TAG environment variable to the version of the Packet Inspector 1 aggregator you want to install.

Example
```
$ export AM_TAG=1.21.6-am1
```
If you’re installing from a private artifact registry that requires authentication (that is, you must provide credentials to access the registry), create the private artifact registry’s pull secret (for example, private-registry-pull-secret) in the istio-system namespace:
```
$ kubectl apply -f <privateRegistryPullSecretManifestFilename> \
    --namespace istio-system
```

Install the Packet Inspector 1 aggregator:

$ helm install packet-inspector-1-aggregator \
    oci://$AM_REPO/charts/packet-inspector --version $AM_TAG \
    --namespace istio-system \
    --values <aggregatorOverrideValuesFilename> \
    --wait

Capture traffic using a Packet Inspector 1 filter instance#

Introduction#

Before you can capture traffic using a Packet Inspector 1 filter instance, you need to:

(Optional) Label the set of pods you want to capture traffic from
Log in to the appropriate artifact registry with Helm and set the repository address
Create an override values file for the Packet Inspector 1 filter instance
Install the Packet Inspector 1 filter instance

After you capture traffic, you need to uninstall the Packet Inspector 1 filter instance to prevent it from lowering the performance of your cluster.

When to follow these instructions#

Follow the instructions below only if you want to capture traffic using Packet Inspector 1 and only after you’ve prepared to capture traffic with Packet Inspector 1. Perform the tasks in order.

About Packet Inspector 1 filter configuration#

The Helm chart for the Packet Inspector 1 filter includes a values.yaml file that contains fields that Helm uses to generate the manifest files during an installation or an upgrade. You can change the default value of a field by doing both of the following:

Include the field in the override values file for a Packet Inspector 1 filter instance and specify a different value.
Specify the override values file when installing the Packet Inspector 1 filter instance.

Configuration and installation overview for a Packet Inspector 1 filter instance#

This table provides an overview of how you’ll configure and install a Packet Inspector 1 filter instance, depending on what traffic you want to capture.

If you want to capture traffic from	Then you’ll do this
All pods with sidecars, regardless of namespace	- Install the filter instance in the `istio-system` namespace. Note: In the override values file for the filter instance, do not add a key/value pair under `.workloadSelector.labels`.
All pods with sidecars in a namespace	- Install the filter instance in the namespace. Note: - In the override values file for the filter instance, do not add a key/value pair under `.workloadSelector.labels`. - Do not install the filter instance in the `istio-system` namespace.
A set of labeled pods with sidecars, regardless of namespace	- Label each pod in the set with a unique key/value pair for the set. - In the override values file for the filter instance, add the key/value pair under `.workloadSelector.labels`. - Install the filter instance in the `istio-system` namespace.
A set of labeled pods with sidecars in a namespace	- Label each pod in the set with a unique key/value pair for the set. - In the override values file for the filter instance, add the key/value pair under `.workloadSelector.labels`. - Install the filter instance in the namespace that contains the labeled pods. Note: Do not install the filter instance in the `istio-system` namespace.

Label the set of pods you want to capture traffic from#

Important

Perform this task only if you want to capture traffic from a set of labeled pods.

Label each pod in the set with a unique key/value pair (for example, pi1-bookinfo-app-reviews2 = true).

Log in to the appropriate artifact registry with Helm and set the repository address#

Create an override values file for the Packet Inspector 1 filter instance#

In a terminal window on your computer, make sure you’re in the Aspen Mesh Platform chart directory.
Set the AM_TAG environment variable to the version of the Packet Inspector 1 filter instance you want to install.

Example
```
$ export AM_TAG=1.21.6-am1
```
Create an override values file for the Packet Inspector 1 filter instance by copying the Packet Inspector 1 filter values.yaml file to a new file (for example, pi1-filter-override-values-bookinfo-app-reviews2.yaml):
```
$ helm show values oci://$AM_REPO/charts/packet-inspector-1-filter --version $AM_TAG > <instanceOverrideValuesFilename>
```
Open the override values file for the Packet Inspector 1 filter instance in a text editor.
Do you want to capture HTTP traffic?
- Yes:
  - Make sure the value of .httpCapture.enabled is true.
  - Go to step 6.
- No:
  - Make sure the value of .httpCapture.enabled is false.
  - Go to step 6.
Do you want to capture Diameter traffic?
- Yes:
  - Make sure the value of .diameterCapture.enabled is true.
  - Make sure .diameterCapture.port specifies the port being used for Diameter traffic (typically 3868).
  - Go to step 7.
- No:
  - Make sure the value of .diameterCapture.enabled is false.
  - Go to step 7.
Read the comments for each remaining field and comment out or remove any unrequired fields that you don’t want to override.

Remember, if you want the Packet Inspector 1 filter instance to capture traffic from a set of labeled pods, you must add the key/value pair for the set (for example, pi1-bookinfo-app-reviews2 = true) under .workloadSelector.labels, as mentioned in Configuration and installation overview for a Packet Inspector 1 filter instance.
Change the values of the remaining fields.
Save and close the file.

You’ll need access to this file when installing the Packet Inspector 1 filter instance, and you should keep the file under version control.

Install the Packet Inspector 1 filter instance#

In a terminal window on your computer, make sure you’re in the Aspen Mesh Platform chart directory.
Set the AM_TAG environment variable to the version of the Packet Inspector 1 filter instance you want to install.

Example
```
$ export AM_TAG=1.21.6-am1
```

Install the Packet Inspector 1 filter instance:

$ helm install <packetInspector1FilterInstanceName> \
    oci://$AM_REPO/charts/packet-inspector-1-filter --version $AM_TAG \
    --namespace <instanceNamespaceName> \
    --values <instanceOverrideValuesFilename> \
    --wait

Capture traffic using the Packet Inspector 1 filter instance#

Use the analysis service (either the network analysis tool or the Packet Inspector 1 analysis emulator) to capture traffic using the Packet Inspector 1 filter instance.

Uninstall the Packet Inspector 1 filter instance#

Important

Perform this task after you’re finished capturing traffic.

Uninstall the Packet Inspector 1 filter instance:

$ helm uninstall <packetInspector1FilterInstanceName> \
    --namespace <instanceNamespaceName> \
    --wait

Troubleshooting#

Troubleshooting guide#

The aggregator service doesn’t start.

Verify that the address of the analysis service is configured properly in the override values file for the Packet Inspector 1 aggregator, and ensure that the analysis service is running.
There are no packets being captured.

Do the following:
- Verify that sidecar proxies are injected in workloads that should be capturing packets.
- Inspect the sidecar proxy (istio-proxy) container logs for any communication errors with the aggregator service.
- Inspect the aggregator service for errors related to packets getting dropped or issues communicating with the analysis service.
The aggregator service crashed (OOM killed).

When there is a high enough load, it can exceed the aggregator service’s memory. Adjust the CPU limit (.resources.limits.cpu) and memory limit (.resources.limits.memory) for the aggregator service in the Packet Inspector 1 aggregator override values file as required for your workloads. (If the CPU limit is too low, more memory is needed to buffer data.)

Uninstall Packet Inspector 1#

When to follow these instructions#

Follow the instructions below in the following cases:

When you no longer plan to use Packet Inspector 1
Before you upgrade to a later version of Aspen Mesh

Perform the tasks in order.

Uninstall all Packet Inspector 1 filter instances#

If any Packet Inspector 1 filter instances are installed, uninstall each Packet Inspector 1 filter instance.

You can use the following command to determine whether any Packet Inspector 1 filter instances are installed:
```
$ kubectl get envoyfilters --all-namespaces | grep packet-inspector-1-filter-
```

Uninstall the Packet Inspector 1 aggregator#

Uninstall the Packet Inspector 1 aggregator:

$ helm uninstall packet-inspector-1-aggregator \
    --namespace istio-system \
    --wait

Uninstall the Packet Inspector 1 analysis emulator#

Uninstall the Packet Inspector 1 analysis emulator if you previously installed it:

$ helm uninstall pi1-analysis-emulator --namespace analysis-emulator --wait