K15569543: Producing a TMM Diagnostic Core File

K15569543: Producing a TMM diagnostic core file

Support Solution

Original Publication Date: Oct 22, 2019

Update Date: Aug 12, 2020

Details

Description

F5 Support may ask you to produce a Traffic Management Microkernel (TMM) core file on the BIG-IP system to establish the state of TMM process during an event. You should perform this task only when asked to do so by F5 Support.

Important: The BIG-IP system will temporarily fail to process traffic as it recovers from TMM restarting, and devices configured as a high availability (HA) pair may fail over. Additionally, on platforms with a large memory capacity, the system will respond slowly while it writes several gigabytes of data to the disk and compresses the core file. This process may take several minutes to complete. When the core file compression is complete, the unit returns to normal operation and restarts the TMMs.

Environment

BIG-IP F5 Support has requested a TMM diagnostic core file Adequate amount of disk space for the diagnostic core file K14403: Maintaining disk space on the BIG-IP system

Cause

None

Recommended Actions

To produce a TMM diagnostic core file, perform the following:

1. Disable the watchdog process K13539: Disabling the watchdog process to perform a core dump (11.x - 15.x)

2. Log in to the BIG-IP command line

3. Determine which tmm processes are running on the system, enter the following command: ps ax | grep -i tmm

Note: Most BIG-IP platforms are enabled with Clustered Multiprocessing (CMP) and run multiple TMM 3.

instances.

Example: If a system has two instances of tmm running, output appears similar to the following:

5736 ? S 0:00 runsv tmm 12325 ? S 0:00 /usr/bin/tmipsecd --tmmcount 12 12327 ? S 0:00 /etc/bigstart/scripts/tmm.start /var/run 12 6 0 0 --platform E101 -m -s 21853 16379 ? SLl 17:22 tmm.0 -T 6 --tmid 0 --npus 12 --platform E101 -m -s 21853 16380 ? SLl 17:31 tmm.6 -T 6 --tmid 6 --npus 12 --platform E101 -m -s 21853 17707 pts/0 S+ 0:00 grep -i tmm

Note: In this example, the two tmm instances are tmm.0 and tmm.6

4. Use the killall command to signal the tmm process to produce a diagnostic core file for each instance. To do so, use the following command syntax: killall -v -s 6 ...

You should replace the placeholders with the tmm instances from step 3.

For example, to signal both tmm.0 and tmm.6 to produce diagnostic core files in the /shared/core directory, enter the following command:

killall -v -s 6 tmm.0 tmm.6

Alternatively, you can run the commands individually by entering both of the following commands:

killall -v -s 6 tmm.0 killall -v -s 6 tmm.6

Note: You can automate steps 1 through 3 and produce a core file for all tmm processes simultaneously by entering the following command:

killall -v -s 6 $(pgrep -l 'tmm\.[0-9]' | awk '{print $2}')

Additionally, when you produce a diagnostic core file on a multi-blade system, you may want to simultaneously produce core files on all blades using the clsh command:

clsh killall -v -s 6 $(pgrep -l 'tmm\.[0-9]' | awk '{print $2}')

5. The diagnostic core files are written to the /shared/core directory, and the file names appear similar to the following examples: tmm.0.bld0.0.606.core.gz tmm.6.bld0.0.606.core.gz

Note: It may take several minutes for the system to complete writing the core files to disk.

6. 6. Once the diagnostic core file creation has completed, re-enable the watchdog process K13539: Disabling the watchdog process to perform a core dump (11.x - 15.x)

Additional Information

For information about generating an md5sum and providing the TMM diagnostic core file to F5 Support, refer to K13002225: Providing diagnostic core files to F5 Support (12.x and later)