CNFs Coremond¶

Overview¶

The Coremond pod collects and processes core files for Cloud-Native Network Functions (CNFs). A core file is a snapshot of the memory and register state of a process or a program when it terminates unexpectedly due to an uncertain or unexpected event that triggers default signal handling. Root-cause analysis can be performed on the core file. The core files are generated either by a third party or by the kernel itself.

Coremond monitors /var/crash folder mapped to a volume to detect updates to core files as the Coremond pod does not have access to the operating system. When Coremond starts, it reads the core_pattern from /proc/sys/kernel/ to decide if the configured core_pattern is supported.

CNFs Robin.io platform store all the core files generated in a single directory on the host at /home/crash/f5. This directory is not created by default. You can create one or enable it through installation to store the core files. F5 recommends to enable the directory during installation.

To prevent the core files overload, Coremond rotates the core files for each process and keeps only the three latest files. Coremond also detects continuous process crashes within the same time frame and ignores coredump writing in such scenarios.

Prerequisites¶

Ensure you have the following:

A working cluster with Robin/Tanzu platform.
A Linux-based workstation
A core_pattern file located at /proc/sys/kernel/core_pattern. Some of the supported core patterns are:
- By default, for Openshift and Tanzu platforms, the core dump used by the system is systemd-coredump with xz, lz4 or zst extension, such as (|/usr/lib/systemd/systemd-coredumps %P %u %g %s %t 9223372036854775808 %h)
- In Robin.io, the native Kernel must be /var/crash/core.%e.%p.%h.%t otherwise, an error is returned.
  
  Specifier
  
  Description
  
  %h
  
  Hostname
  
  %e
  
  Executable filename
  
  %p
  
  pid of the process
  
  %t
  
  UNIX time of dump
- When Coremond is not compatible with the core_pattern, enable the Coremond init container to override the core_pattern. Refer Optional settings: section in procedure for details.
Note

F5 recommends to install the Coremond first before installing any other F5 components. This is suggested as if there are any other components installed prior to Coremond, they may generate the core files.

Specifier	Description
`%h`	Hostname
`%e`	Executable filename
`%p`	pid of the process
`%t`	UNIX time of dump

Configure Rotation and Retention¶

This section outlines the environment variables used to configure the core file retention, rotation, and cleanup of Coremond. These variables allow you to manage retention durations, set file limits per process, and define rotation policies.

Environment Variable	Default Value	Description
`COREMON_RETENTION_INTERVAL`	5m	Specifies the time frame to ignore additional core dumps from the same process once `COREMON_CORES_MAX_FILES` limit is reached.
`COREMON_CORES_MAX_FILES`	3	Specifies the maximum number of core files allowed for the same process. This parameter is used to prevent continuous crashes and rotations.
`COREMON_RETENTION`	0	Specifies the duration to keep core files before deletion. This also applies to the final core file copied to the volume. To disable the retention, set the value of this parameter to 0.
`COREMON_CORES_INTERVAL`	5m	Specifies the interval or duration at which, Coremond schedules scanning and deletion of core files exceeding the `COREMON_RETENTION` period.
`COREMON_DELETE_SRC`	true	Specifies to delete source core files from the host path `/home/crash/f5` generated by the kernel. This setting is not applicable on OpenShift.
`COREMON_ROTATE`	false	Allows to replace old core files with the new ones, when `COREMON_CORES_MAX_FILES` limit is reached. This only occurs if `COREMON_RETENTION_INTERVAL` limit is elapsed and the Coremond continues processing core files for that process.

Coremond Installation using FLO¶

For the installation of Coremond, refer to the FLO section.

Follow the steps below to allow the Coremond init container override the core_pattern and enable core file generation in containers.

Configure core dump handling to override the kernel core_pattern with the desired pattern.
Apply SELinux labelling on the crash directory using chcon.
Adjust container security contexts via FLO:
- If using FLO, set the Coremond container SecurityContext to privileged and allow privilege escalation.

Store core files on Host instead of PVCs¶

Coremond supports storing core files directly on the host directory instead of using Persistent Volumes (PVCs), eliminating the need for ReadWriteMany volumes and shared storage when multiple Coremond pods are deployed. By default, this option is disabled and PVs are used.

To change the default and store cores on the Host machine instead of PVs, enable the following in the Coremond advanced section of the CNEInstance CR:

# FLO: cneinstance CR
apiVersion: k8s.f5.com/v1
kind: CNEInstance
metadata:
  name: cneinstance-sample
spec:
 advanced:
  coremon:
    hostPath: true
    env:
    - name: "COREMOND_OVERRIDE_CORE_PATTERN"
      value: "true" 

Coremond Installation using HELM¶

Obtain the [TAG/Version] from the CNFs 2.1.0 tarball.

Install the Coremond by using the following syntax on Robin platform:

helm install coremond tar/<helm-chart>.tgz \ -f <values>.yaml -n <project>

For example:

helm install coremond tar/coremond-0.10.0-0.2.3.tgz -n coremond

You can edit the values.yaml file as per usecase and requirement. Following are some of the mandatory and optional settings that can done by editing the values.yaml file:

Run the following command to install through Helm:
```
echo 'platformType: "robin"' >> values.yaml
helm install coremond f5-coremond-0.10.0-0.2.3.tgz
```
You can edit the values.yaml file as per usecase and requirement. Following are some of the mandatory and optional settings that can done by editing the values.yaml file:

a. Mandatory settings:
- Override the image settings by specifying the custom image values:
```
image:
repository: repo.f5.com/images/f5-toda-docker
name: f5-coremond
tag: v
pullPolicy: IfNotPresent
```
- Coremond supports the usage of node selectors and node affinity to specify the nodes. For this, a Coremond pod should be scheduled in a Kubernetes cluster. By default, Coremond runs on all worker nodes.
  To run the pod on the worker-node node, configure both the nodeSelector and affinity as shown in the following example.
```
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - worker-node
  nodeSelector:
    kubernetes.io/hostname: worker-node
```
b. Optional settings:
- Coremond supports storing core files directly on the host directory instead of using Persistent Volumes (PVCs), eliminating the need for ReadWriteMany volumes and shared storage when multiple Coremond pods are deployed. By default, this option is disabled and PVs are used.
  
  To change the default and store cores on the Host machine instead of PVs, set the following value to true in values.yaml file:
```
useHostPath: true
```
  When useHostPath is enabled, core files are written to a directory on the host node. The default host path is /home/crash/f5-processed. To customize the destination directory, set coreDstHostPath in your values.yaml:
```
useHostPath: true
coreDstHostPath: "/your/custom/path"
```
  If the host kernel is configured with a core_pattern that directs core dumps to a specific directory, Coremond must be configured to locate them accordingly. Set the global.core.mountPath parameter to correspond with the directory specified in your kernel’s core_pattern.
  
  For example, to a kernel core pattern of core_pattern=/var/crash/core.%e.%p.%h.%t, set the mount path to the directory portion:
```
global:
  core:
    mountPath: "/var/crash"
```
  The default value is /var/crash.
- To adjust the Log level setting in COREMON_LOG_LEVEL value, add the following in values.yaml file:
```
env:  
- name: COREMON_LOG_LEVEL
  value: "debug"
```
- Coremond requires a PV with RWX access and if the default storage class does not support that, it may cause the Coremond to remain pending. To avoid this, override the storageClass parameter with RWX through values.yaml file.
  
  Following is an example to override the file:
```
persistence:
  accessMode: ReadWriteMany
  storageClass: your-rwx
```
- To override the resources settings, specify the custom resources values in the values.yaml file as shown in the following example:
```
resources: 
    limits:
    cpu: 100m
    memory: 128Mi
    requests:
    cpu: 100m
    memory: 128Mi
```
- To disable the qkview process, set the enabled value to false in the values.yaml file:
```
f5_csm_qkview:
  enabled: false
```
- To override the fluentbit_sidecar image settings, specify the custom image values as shown in the following example:
```
fluentbit_sidecar:  
  image:  
      repository: repo.f5.com/images/f5-toda-docker
      name: f5-fluentbit
      tag: v
      pullPolicy: IfNotPresent
```
- To override the fluentbit_sidecar resources settings, specify the custom resources values as shown in the following example:
```
fluentbit_sidecar:  
  resources:  
    limits:
      cpu: "0.5"
      memory: "512Mi"
    requests:
      cpu: "0.25"
      memory: "256Mi"
```
- To override the fluentbit_sidecar security context settings, specify the custom securityContext values as shown in the following example:
```
fluentbit_sidecar:
  securityContext:
    allowPrivilegeEscalation: false
    # runAsUser: 10000
```
- To override the fluentbit_sidecar additional settings, specify the custom fluentbit values as shown in the following example:
```
fluentbit_sidecar:
  fluentbit:
    # Interval to flush output (seconds)
    flush_interval: 1
    # Error/warning/info/debug/trace
    logLevel: debug
    # Pipe reading parameters
    input:
      pipes:
        bufSize: 8096
        intervalSec: 1
        intervalNsec: 0
    tls:
      enabled: false
      # TLS debug verbosity level, values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational) and 4 (Verbose)
      debug: 1
      # Force certificate validation
      verify: Off
      # key string known by the remote Fluentd used for authorization.
      shared_key: f5-toda-shared-key
fluentd:
  host: '127.0.0.1'
  port: 54321
```
- To disable fluentbit_sidecar container, set the fluentbit_sidecar value to false in values.yaml file:
```
fluentbit_sidecar:
  enabled: false
```
- Follow the steps below to let the Coremond init container override the core_pattern and enable core file generation in containers.
  - Configure core dump handling to override the kernel core_pattern with the desired pattern.
  - Apply SELinux labelling on the crash directory using chcon.
  - Adjust container security contexts via Helm values:
    - Set the init container’s security context to allow writing core files.
    Example for Helm:
    # Helm charts: values.yaml # - corePattern overrides /proc/sys/kernel/core_pattern content # - chcon is applied to the crash directory (e.g., hostPath/PV mount) # - securityContext controls initContainer privileges initContainers: enabled: true corePattern: "|/usr/bin/crashhandler %P %U %G %s %t %e" chcon: "-Rt sandbox_file_t" securityContext: privileged: true

How to generate a core file¶

Following are the steps to generate a core file:

Run the following command to get the list of pods.

kubectl get pods 

Sample output with the list of pods:

NAME                                          READY   STATUS    RESTARTS   AGE
client                                        1/1     Running   0          2m28s
dssm-f5-dssm-db-0                             2/2     Running   0          2m26s
dssm-f5-dssm-db-1                             2/2     Running   0          96s
dssm-f5-dssm-sentinel-0                       2/2     Running   0          2m26s
dssm-f5-dssm-sentinel-1                       2/2     Running   0          90s
f5-cert-manager-84f857f786-gk6xq              1/1     Running   0          4m10s
f5-cert-manager-cainjector-695866d7ff-m2h2g   1/1     Running   0          4m10s
f5-cert-manager-webhook-8554fd5b58-xc89x      1/1     Running   0          4m10s
f5-coremond-7gqfp                             2/2     Running   0          2m54s
f5-crdconversion-7df678d8fc-2vplv             1/1     Running   0          2m51s
f5-rabbit-f9c58487c-vhtw2                     1/1     Running   0          2m53s
f5-spk-cwc-669f8c9dc-ptjb2                    2/2     Running   0          2m52s
f5-tmm-7b685cd57c-lp7cl                       0/4     Pending   0          2m9s
f5-tmm-7b685cd57c-rq92s                       4/4     Running   0          2m9s
f5-toda-fluentd-6bc5cb8bfb-wqsvx              1/1     Running   0          2m11s
f5-toda-observer-788ddcd596-6qjpg             2/2     Running   0          2m12s
f5-toda-stats-77cb79c44d-4cn4x                2/2     Running   0          2m25s
otel-collector-5f48b7ccf7-s6wx7               1/1     Running   0          2m9s
router                                        2/2     Running   0          2m27s
server                                        1/1     Running   0          2m28s
spk-f5ingress-797bdbb59-zssd6                 4/4     Running   0          2m9s

To get the process list, run the following command:

kubectl exec <pod-name> -- ps aux

Example:

Defaulted container "f5-toda-observer" out of: f5-toda-observer, fluentbit
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
f5docker       1  0.0  0.0 711880  3024 ?        Ssl  09:45   0:00 /init
f5docker      25  0.0  0.0   3024  1200 ?        S    09:45   0:00 s6-svscan -c30 -t0 /var/run/s6/services
f5docker      27  0.0  0.0   3036  1264 ?        S    09:45   0:00 s6-supervise observer
f5docker      28  0.0  0.0   3036  1268 ?        S    09:45   0:00 s6-supervise qkview-collect-daemon
f5docker      29  1.2  0.3 1270624 49540 ?       Ssl  09:45   0:01 observer
f5docker      30  0.0  0.0 1235736 9892 ?        Ssl  09:45   0:00 /usr/bin/qkview-collect-daemon
f5docker     212  0.0  0.0   7072  1592 ?        Rs   09:47   0:00 ps aux

To kill a process and generate the core dumps, run the following command:

kubectl exec <pod-name> -- kill -11 <process-id> 

Sample output:

Defaulted container "f5-toda-observer" out of: f5-toda-observer, fluentbit

How to validate the core file¶

To verify that the core file is created, do the following:

Identify a Coremond pod.

kubectl get pods | grep coremond 

Sample output:

NAME                                          READY   STATUS    RESTARTS      AGE
f5-coremond-2llc8                             2/2     Running   0             49m
f5-coremond-wfrv5                             2/2     Running   0             49m

Check Coremond logs to confirm it detected and processed the core.

kubectl logs <coremond_pod>

Sample output:

"ts"="2026-04-29 10:56:18.277"|"l"="info"|"m"="new core file detected"|"lt"="A"|"id"="06110015"|"file"="/var/crashagent/cores/core.observer.26.f5-observer-0.1777460178084395302"|"pod"="f5-coremond-wfrv5"|"ct"="f5-coremond"|"cv"="v0.16.4"|"ns"="f5-utils"|"v"="1.0"
"ts"="2026-04-29 10:56:18.277"|"l"="info"|"m"="creating coredump"|"lt"="A"|"id"="06110021"|"src"="/var/crashagent/cores/core.observer.26.f5-observer-0.1777460178084395302"|"dst"="/var/cores/datkube-worker/f5-utils/core.f5-observer-0.f5-observer.observer.26.1777460178084395302"|"pod"="f5-coremond-wfrv5"|"ct"="f5-coremond"|"cv"="v0.16.4"|"ns"="f5-utils"|"v"="1.0"

Validate the core file exist under /var/cores.

kubectl exec <coremond_pod> -- ls /var/cores

Sample output:

core.f5-observer-0.f5-observer.observer.26.1777460178084395302.gz
core.f5-observer-0.f5-observer.observer.26.1777460178084395302.gz.crc

Core File Streaming Support¶

Core file handling has been enhanced to support streaming core files directly from crashing containers to Coremond, eliminating the dependency on a shared host volume.

Note

Streaming is only supported when core files are written inside the container, either by the kernel or Apport. When using core_pattern with systemd, the core monitor must mount the host volume in order to consume core files generated by systemd.

Note

Core file streaming is configurable and optional.

With this change, Coremond can optionally run as a Deployment (instead of a DaemonSet), receiving streamed core files from multiple nodes. This introduces an east-west network bandwidth dependency, as streaming requires sufficient intra-cluster throughput.

Note

It is recommended to deploy Coremond as a DaemonSet to ensure that a crashing pod can transfer its core file to a Coremond instance operating on the same node.

In Robin.io environments, the kernel writes core files directly to the container filesystem. The crash agent detects the core pattern and monitors the directory where core files are written inside the container. Once the kernel opens a core file for writing, the crash agent detects the operation and, in parallel, starts consuming the chunks as the kernel produces them. It either streams the chunks (if streaming is enabled) or writes them to a file in a shared volume. The crash agent also appends a crash report, if available, to the end of the core file. Additionally, it attaches metadata: when streaming, the metadata is sent as part of the gRPC metadata; when using a shared volume, it is stored as extended attributes (xattr).

Core File Streaming Configuration for Specific Helm Release¶

Streaming is enabled by default and uses the /var/crash directory, where Crashagent monitors generated core files.

To disable streaming, set it to false in the values.yaml file:

global:
  core:
    streaming: false

To use a different directory for core file monitoring:

global:
  core:
    streaming: true
    mountPath: /var/corefiles

Note

When streaming is enabled, coremonAddr should be set to a reachable address of Coremond. It may need to be adjusted if, for example, Coremond is deployed in a different namespace. Otherwise, the crashagent process within a container won’t be able to connect to Coremond and stream its core files.

global:
  core:
    streaming: true
    coremonAddr: f5-coremond.f5-utils:8088

Core File Streaming FLO Configuration per Cluster¶

Streaming is enabled by default and uses the /var/crash directory, where Crashagent monitors generated core files.

To disable streaming, set the COREMOND_STREAMING to false in the CNEInstance CR:

spec:
  advanced:
    coremon:
        env:
        - name: COREMOND_STREAMING
          value: "false" 

To use a different directory for core file monitoring, set the COREMOND_SRC_PATH to the desired value in the CNEInstance CR:

spec:
  advanced:
    coremon:
        env:
        - name: COREMOND_STREAMING
          value: "true"
        - name: COREMOND_SRC_PATH
          value: "/var/corefiles"

The namespace is injected dynamically to the CRASHAGENT_COREMON_ADDR value. The final coremond address is constructed automatically at runtime using the correct namespace.

Core File Validation for Streaming¶

Trigger a core dump generation by killing a process:

kubectl -n <namespace> exec <pod_name> -c <container> -- kill -11 <process_id>

Check the logs in the container where the core dump was generated:
```
kubectl -n <namespace> logs <pod_name> -c <container>
```

In contrast to non-streaming mode, where the destination is a local path (“dest”=”/var/crashagent/cores”), in streaming mode the destination is the Coremond address (“dest”=”f5-coremond.f5-utils:8088”), meaning the core file is sent over the network. Log example:

"ts"="2026-04-28 12:32:00.726"|"l"="info"|"m"="core file written to destination"|"lt"="A"|"id"="06110054"|"proc"="crashagent"|"dest"="f5-coremond.f5-utils:8088"|"exe"="s6-supervise"|"pid"="22"|"size"="499712"|"crc32c"="3311527668"|"pod"="f5-spk-cwc-6c4956c65d-m6z4q"|"ct"="spkcwc"|"cv"="v0.41.4"|"v"="1.0"

Validate the core files existence under /var/cores:

kubectl -n <namespace> exec <coremond_pod> -- ls /var/cores

Previous Next