Use multi-destination#
About multi-destination packet forwarding#
Introduction#
The Packet Inspector 1 Aggregator can send the same captured packet data to multiple analysis backends at the same time. This feature is called multi-destination packet forwarding.
With multi-destination mode, you can configure up to five total destinations: one primary and up to four secondaries. Each destination receives a byte-for-byte identical copy of every captured packet.
Note
Multi-destination applies to Packet Inspector 1 only. Packet Inspector 2 is not supported in this release.
Use cases#
Multi-destination packet forwarding supports several common scenarios:
Packet loss investigation — Send identical captures to different analysis tools. Compare results to identify which component is dropping packets.
High availability — If the primary destination goes down, secondary destinations still have the data.
Parallel analysis — Send captures to multiple tools (such as NetScout, Elasticsearch, or a machine learning pipeline) without duplicating your capture infrastructure.
Canary testing — Deploy a new version of your analysis backend as a secondary destination. Validate it against production traffic before promoting it to primary.
Compliance and audit — Send data to both an operational monitoring system and a separate audit system.
Multi-tenant distribution — Give different teams their own analysis instance while using the same capture data.
How multi-destination delivery works#
The aggregator uses a delivery model:
The aggregator sends packets to all configured destinations in parallel.
Healthy destinations receive their data even when another destination fails.
If any destination fails, the combined send operation returns an error.
If the error is retryable (connection errors or HTTP 500, 502, 503, or 504 responses), the aggregator retries the entire capture group. The default maximum retry count is three.
Healthy destinations may receive duplicate data during retries.
Note
A primary failure does not stop the aggregator from sending to secondaries. All destinations are attempted independently in each send cycle.
Single-destination fast path#
When no secondary destinations are configured, the aggregator sends directly to the primary destination. There is no packet cloning, no parallel processing, and no extra overhead. Existing single-destination deployments work exactly the same way after upgrading.
TLS certificate handling#
By default, the aggregator reuses the primary TLS certificates for all secondary destinations. This works when all destinations share the same Istio trust domain and Certificate Authority (CA).
If a secondary destination uses a different CA or trust domain, you must provide per-destination certificate paths. See Configure multi-destination packet forwarding for details.
Each destination has its own TLS configuration watcher. Certificate rotation happens automatically — the watcher detects file changes on disk and updates the TLS configuration without a restart.
Automatic reconnection#
The aggregator periodically reconnects to all destinations (primary and secondaries). The default reconnection interval is five minutes. You can configure this interval from one minute to 24 hours using the analysis.reconnectTime Helm value or the --analysis-reconnect-time CLI flag.
If a destination becomes unreachable, sends to that destination fail until the next reconnection cycle restores connectivity. Healthy destinations continue receiving data independently.
Note
Adding or removing destinations requires a rolling update of the aggregator (
helm upgrade).Retry logic applies to the entire capture group, not to individual destinations.
No prebuilt Grafana dashboards or Prometheus alerting rules are included for multi-destination metrics. You can create your own using the exposed metrics.
Cross-cluster destinations require routable DNS/network connectivity and certificates whose trust and Subject Alternative Name (SAN) validation match the destination. Cross-cluster destinations are not covered by integration tests.
Configure multi-destination packet forwarding#
Introduction#
This section walks you through setting up the Packet Inspector 1 Aggregator to send identical packet captures to multiple analysis destinations. By the end of this procedure, you have a primary destination and one or more secondary destinations receiving the same data.
Note
Multi-destination requires Aspen Mesh version 1.21.6-am5 or later.
Before you begin#
Make sure the following components are installed and running:
Istio base and control plane
Citadel (for certificate generation)
A primary analysis emulator in its own namespace
The Packet Inspector 1 Aggregator Helm chart version 1.21.6-am5 or later
The istio-system namespace and each analysis-emulator namespace must have the label ca.istio.io/override=true applied. Without this label, Istio does not generate the TLS certificates that the emulator needs.
Install a secondary analysis emulator#
Repeat these steps for each secondary emulator you want to add (up to four).
Create a new namespace for the secondary emulator:
$ kubectl create namespace <secondaryEmulatorNamespaceName>
Example
$ kubectl create namespace analysis-emulator2
Label the namespace for certificate generation. Apply the Istio certificate override label:
$ kubectl label namespace <secondaryEmulatorNamespaceName> \ ca.istio.io/override=true --overwrite
Important
This label is required. Without it, the emulator pod cannot obtain TLS certificates and remains in a
ContainerCreatingstate.Ensure Citadel workload certificates for the
istio-systemnamespace is enabled for Citadel to create and maintain a workload certificate (in the form of a secret namedistio.<aggregatorServiceServiceAccountName>) for the aggregator service’s service account, which allows the analysis service to perform identity validation of the aggregator service.Ensure Citadel workload certificates for the
analysis-emulatornamespace is enableds for Citadel to create and maintain a workload certificate (in the form of a secret namedistio.<analysisEmulatorServiceAccountName>) for the analysis emulator’s service account, which allows the aggregator service to perform identity validation of the analysis emulator.Log in to the appropriate artifact registry with Helm and set the repository address.
Set the
AM_TAGenvironment variable to the version of the Packet Inspector 1 analysis emulator you want to install.Example
$ export AM_TAG=1.21.6-am5
If you’re installing from a private artifact registry that requires authentication (that is, you must provide credentials to access the registry), create the private artifact registry’s pull secret (for example,
private-registry-pull-secret) in theanalysis-emulatornamespace:$ kubectl apply -f <privateRegistryPullSecretManifestFilename> \ --namespace <secondaryEmulatorNamespaceName>
Wait for the certificate secret. Before you deploy the emulator, confirm that Istio has generated the certificate secret for the
ServiceAccount:$ kubectl wait --for=create secret/istio.<secondaryEmulatorServiceAccountName> \ -n <secondaryEmulatorNamespaceName> \ --timeout=60s
Install the secondary emulator. Install the emulator using Helm. Set a unique instanceName to avoid naming conflicts with the primary:
$ helm install <secondaryEmulatorReleaseName> \ oci://$AM_REPO/charts/packet-inspector-1-analysis-emulator --version $AM_TAG \ --namespace <secondaryEmulatorNamespaceName> \ --set instanceName=<secondaryEmulatorInstanceName> \ --values pi1-analysis-emulator-override-values.yaml \ --wait
Example
$ helm install pi1-analysis-emulator2 \ oci://$AM_REPO/charts/packet-inspector-1-analysis-emulator --version $AM_TAG \ --namespace analysis-emulator2 \ --set instanceName=packet-inspector-1-analysis-emulator2 \ --values pi1-analysis-emulator-override-values.yaml \ --wait
Note
Each secondary emulator must have a unique
instanceName. If you omit this value, the resource names collide with the primary emulator.Verify the emulator is running.
$ kubectl wait --for=condition=available \ deployment/<secondaryEmulatorInstanceName> \ -n <secondaryEmulatorNamespaceName> \ --timeout=60s
Configure the aggregator for multi-destination#
After you deploy one or more secondary emulators, update the Packet Inspector 1 aggregator override values file to include the secondary addresses.
Shared certificates (same trust domain)
If all destinations share the same Istio trust domain, add only the secondary addresses. The aggregator automatically reuses the primary TLS certificates.
Add the following to your aggregator override values file:
analysis:
addressesSecondary:
- "<secondaryEmulatorInstanceName1>.<secondaryEmulatorNamespaceName1>.svc.cluster.local:21000"
- "<secondaryEmulatorInstanceName2>.<secondaryEmulatorNamespaceName2>.svc.cluster.local:21000"
Example
analysis:
addressesSecondary:
- "packet-inspector-1-analysis-emulator2.analysis-emulator2.svc.cluster.local:21000"
- "packet-inspector-1-analysis-emulator3.analysis-emulator3.svc.cluster.local:21000"
Per-destination certificates (different trust domains)
If a secondary destination uses a different CA, provide explicit certificate paths. The number of entries in each certificate list must match the number of entries in addressesSecondary.
analysis:
addressesSecondary:
- "broker.analysis.svc.cluster.local:21000"
- "external.partner.com:21000"
certPathsSecondary:
- "/etc/analysis/certs/root-cert.pem"
- "/etc/external/certs/ca.pem"
clientCertPathsSecondary:
- "/etc/analysis/certs/cert-chain.pem"
- "/etc/external/certs/client.pem"
clientKeyPathsSecondary:
- "/etc/analysis/certs/key.pem"
- "/etc/external/certs/key.pem"
Apply the multi-destination configuration#
For new installations Install the aggregator with your updated override values file:
```shell
$ helm install packet-inspector-1-aggregator \
oci://$AM_REPO/charts/packet-inspector --version $AM_TAG \
--namespace istio-system \
--values <aggregatorOverrideValuesFilename> \
--wait
```
For existing installations (upgrade in place)
Export your current Helm values:
$ helm get values packet-inspector-1-aggregator -n istio-system > current-values.yaml
Add the
addressesSecondary(and certificate paths if needed) tocurrent-values.yaml. Upgrade the release:$ helm upgrade packet-inspector-1-aggregator \ oci://$AM_REPO/charts/packet-inspector --version $AM_TAG \ --namespace istio-system \ --values current-values.yaml \ --wait
Verify the DaemonSet rollout:
$ kubectl rollout status daemonset/packet-inspector-aggregator \ -n istio-system \ --timeout=120s
Verify multi-destination is active#
Check the aggregator logs for the startup confirmation:
$ AGG_POD=$(kubectl get pods -n istio-system \
-l istio=packet-inspector-aggregator -o name | head -1)
$ kubectl logs -n istio-system $AGG_POD | grep "Multi-destination mode"
When multi-destination is active, you see:
INFO Multi-destination mode enabled - all destinations must succeed
{"primary": "...:21000", "secondary_count": 1, "secondaries": ["...:21000"]}
Verify the secondary addresses are passed to the DaemonSet:
$ kubectl get ds -n istio-system packet-inspector-aggregator \
-o jsonpath='{.spec.template.spec.containers[0].args}' | grep secondary
Revert to single destination#
To remove all secondaries and return to single-destination mode:
Remove the
addressesSecondaryentries (or set them to an empty list) in your aggregator override values file:analysis: addressesSecondary: []
Upgrade the Helm release:
$ helm upgrade packet-inspector-1-aggregator \ oci://$AM_REPO/charts/packet-inspector --version $AM_TAG \ --namespace istio-system \ --values <aggregatorOverrideValuesFilename> \ --wait
Verify the rollout:
$ kubectl rollout status daemonset/packet-inspector-aggregator \ -n istio-system \ --timeout=120s
The aggregator returns to the single-destination fast path with no cloning overhead.
Multi-destination configuration reference#
Helm values
All multi-destination values are under the analysis section in the aggregator values.yaml.
Value |
Type |
Required |
Default |
Description |
|---|---|---|---|---|
|
string |
Yes |
None |
Primary analysis destination in |
|
list of strings |
No |
|
Secondary analysis destinations. Maximum four entries. Format: |
|
list of strings |
No |
|
CA certificate paths for each secondary. Must match the length of |
|
list of strings |
No |
|
Client certificate paths for each secondary. Must match the length of |
|
list of strings |
No |
|
Client key paths for each secondary. Must match the length of |
|
integer (minutes) |
No |
|
How often the aggregator reconnects to all destinations. Valid range: 1–1440 (24 hours) |
|
integer |
No |
|
Maximum number of retries for a failed capture group. Applies to the combined send operation, not per-destination. Valid range: 1–10 |
|
string (duration) |
No |
|
How long the aggregator waits for a capture group’s final marker before flushing. CLI default is |
CLI flags
These flags are passed to the aggregator binary. In most deployments, you set them through Helm values. The Helm chart template renders the flags automatically.
Flag |
Type |
Default |
Description |
|---|---|---|---|
|
comma-separated string |
Empty |
Secondary destinations. Comma-separated, no spaces. Maximum four |
|
comma-separated string |
Empty (inherits primary) |
CA certificate paths for secondaries |
|
comma-separated string |
Empty (inherits primary) |
Client certificate paths for secondaries |
|
comma-separated string |
Empty (inherits primary) |
Client key paths for secondaries |
|
integer (minutes) |
|
Reconnection interval for all destinations. Range: 1–1440 |
|
integer |
|
Maximum retries for failed capture groups. Range: 1–10 |
|
duration |
|
Wait time before flushing incomplete capture groups. The Helm chart overrides this to |
|
integer (bytes) |
|
Maximum gRPC receive message size. Larger messages increase cloning overhead in multi-destination mode |
Startup validation
The aggregator validates multi-destination configuration at startup and exits with a panic if any of these conditions are met:
Condition |
Error message |
|---|---|
More than four secondary addresses |
|
|
|
|
|
|
|
Malformed secondary addresses (unparseable host or port) are logged as warnings. That destination is skipped, but the aggregator continues to start. A secondary that is unreachable at startup is warned about during its version check; the aggregator does not fail startup solely because a secondary is unavailable.