TMM Core Files¶

Overview¶

Core files are typically produced to diagnose chronic issues such as memory leaks, high CPU usage, and intermittent networking issues. The Debug sidecar’s core-tmm utility creates a diagnostic core file of the Service Proxy Traffic Managment Microkernel (TMM) process. Once obtained, the core file can be provided to F5 support for further analysis.

This document describes how to create, and obtain a TMM core file in an OpenShift orchestration environment.

Requirements¶

Ensure you have:

A Linux cluster Node using systemd-coredump.
A working OpenShift cluster.
A Linux based workstation.
Installed the Debug Sidecar

Configure Rotation and Retention¶

This section outlines the environment variables used to configure the core file retention, rotation, and cleanup of Coremond. These variables allow you to manage retention durations, set file limits per process, and define rotation policies.

Environment Variable	Default Value	Description
`COREMON_RETENTION_INTERVAL`	5m	Specifies the time frame to ignore additional core dumps from the same process once `COREMON_CORES_MAX_FILES` limit is reached.
`COREMON_CORES_MAX_FILES`	3	Specifies the maximum number of core files allowed for the same process. This parameter is used to prevent continuous crashes and rotations.
`COREMON_RETENTION`	0	Specifies the duration to keep core files before deletion. This also applies to the final core file copied to the volume. To disable the retention, set the value of this parameter to 0.
`COREMON_CORES_INTERVAL`	5m	Specifies the interval or duration at which, Coremond schedules scanning and deletion of core files exceeding the `COREMON_RETENTION` period.
`COREMON_ROTATE`	false	Allows to replace old core files with the new ones, when `COREMON_CORES_MAX_FILES` limit is reached. This only occurs if `COREMON_RETENTION_INTERVAL` limit is elapsed and the Coremond continues processing core files for that process.

Procedures¶

Generate the core file¶

Use the following steps to connect to the Service Proxy Pod’s debug container, and generate a core file using the core-tmm command.

Connect to the debug container:

oc exec -it deploy/f5-tmm -c debug -n <project> -- bash

In this example, the debug sidecar is in the spk-ingress Project:

oc exec -it deploy/f5-tmm -c debug -n spk-ingress -- bash

Generate the TMM core file:

Note: It may be helpful to note the time the core is being generated.
```
core-tmm
```
```
Floating point exception (core dumped)
```

Obtain the core file¶

Use these steps to launch an oc debug Pod, and Secure Copy (SCP) the TMM core file to a remote server.

Obtain the name of the worker node that the TMM Pod is running on:

oc get pods -n <project> -o wide | grep f5-tmm

In this example, the TMM Pod named f5-tmm-7cd5b85bdb-7c4b7 is in the spk-ingress Project, and is running on worker-2.ocp.f5.com:

oc get pods -n spk-ingress -o wide | grep f5-tmm

NAME                        READY   STATUS    IP             NODE          
f5-tmm-7cd5b85bdb-7c4b7     3/3     Running   10.244.2.107   worker-2.ocp.f5.com
f5-tmm-7cd5b85bdb-b7rgb     3/3     Running   10.244.3.90    worker-1.ocp.f5.com

Launch the oc debug Pod:

Note: The oc debug command creates a new Pod, and opens a command shell.

oc debug node/<node name>

In this example, we create a copy of the worker-2.ocp.f5.com Pod:

oc debug node/worker-2.ocp.f5.com

Creating debug namespace/openshift-debug-node-m7f8z ...
Starting pod/worker-2ocpf5com-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.144.2.107
If you don't see a command prompt, try pressing enter.

To use the host binaries run:
```
chroot /host 
```

List the core files written to the journal:

coredumpctl list

In this example, note the TIME the file was created and the PID (process ID):

TIME                         PID      UID   GID SIG COREFILE   EXE
Mon 2021-01-01 12:00:00 UTC  590091   0     0   8   truncated  /usr/bin/tmm64.no_pgo

Change into the core file directory, and list the core file on the file system:

cd /var/lib/systemd/coredump; ls -1

In this example, the PID 590091 from the previous step identifies the bottom core file:

cd /var/lib/systemd/coredump; ls -1
'core.tmm\x2e0.0.951073d306bb4465a3d784e29da99995.1004628.1617028629000000.lz4'
'core.tmm\x2e0.0.951073d306bb4465a3d784e29da99995.2442391.1617019721000000.lz4'
'core.tmm\x2e0.0.951073d306bb4465a3d784e29da99995.590091.1617133773000000.lz4'

Create an MD5 signature of the core file to ensure file integrity:
```
md5sum <core_file> > <file_name>
```
In this example, an MD5 signature is obtained of the TMM core file, and saved to a file named tmm_core.md5:
```
md5sum core.tmm\x2e0.0.951073d306bb4465a3d784e29da99995.590091.1617133773000000.lz4 \
> tmm_core.md5
```

Secure Copy (SCP) the TMM core file to the remote server:

scp <tmm_core> <username>@<ip address>:<directory> 

In this example, the file is copied using the ocadmin user, to the remote server with IP address 10.244.4.10:

scp core.tmm\x2e0.0.951073d306bb4465a3d784e29da99995.590091.1617133773000000.lz4 \
ocadmin@10.244.4.10:/var/tmp/

Secure Copy (SCP) the MD5 file to the remote server:

scp <md5_file> <username>@<ip address>:<directory> 

For example:

scp tmm_core.md5 ocadmin@10.244.4.10:/var/tmp/

Feedback¶

Provide feedback to improve this document by emailing spkdocs@f5.com.

Previous Next