January 6, 2021
Estimated Post Reading Time ~

How can I take thread dumps from a JVM on Linux, UNIX or Windows?

A thread dump is a list of all the Java threads that are currently active in a Java Virtual Machine (JVM).

There are several ways to take thread dumps from a JVM. It is highly recommended to take more than 1 thread dump. A good practice is to take 10 thread dumps at a regular interval (for example, one thread dump every ten seconds).

Step 1: Get the PID of your Java process
The first piece of information you will need to be able to obtain a thread dump is your Java process's PID.

The Java JDK ships with the jps command which lists all Java process ids. You can run this command like this:

jps -l 70660 sun.tools.jps.Jps 70305
Note: In Linux and UNIX, you may have to run this command as sudo -u user jps -l, where "user" is the username of the user that the Java process is running as.

If this doesn't work or you still cannot find your Java process, (path not set, JDK not installed, or older Java version), use UNIX, Linux, and Mac OS X: ps -el | grep java
Windows: Press Ctrl+Shift+Esc to open the task manager and find the PID of the Java process

Step 2: Request a thread dump from the JVM

jstack
If installed/available, we recommend using the jstack tool. It prints thread dumps to the command line console.

To obtain a thread dump using jstack, run the following command:
jstack -l <pid>

You can output consecutive thread dumps to a file by using the console output redirect/append directive:
jstack -l <pid> >> threaddumps.log

Notes:
The jstack tool is available since JDK 1.5 (for JVM on Windows, it's available in some versions of JDK 1.5 and JDK 1.6 only).
jstack works even if the -Xrs jvm parameter is enabled

It's not possible to use the jstack tool from JDK 1.6 to take threaddumps from a process running on JDK 1.5.

In Linux and UNIX, you need to run the command as the user that owns the java process:
sudo -u java-user jstack -l <pid>
(<java-user> should be replaced with the id of the user that the Java process is running as)

In Windows, if you run jstack and get the error "Not enough storage is available to process this command" then you must run jstack as the windows SYSTEM user or the user that owns the java process. You can do this by using psexec which you can download here. To run jstack as SYSTEM user, use a command like this:
psexec -s jstack <pid> >> threaddumps.log

If you are unable to install psexec on the server, then you can create a .bat file containing the command and run it using the Windows task scheduler (as a different user).

If the java process isn't responding, then it can sometimes help to use the option -J-d64 (on 64 bit systems), for example:
jstack -J-d64 -l <pid> >> threaddumps.log

If the jstack command (jstack -l <pid> >> threaddumps.log) throws the error [1] below, then run the command as the user that owns the java process. For example:
sudo -u sling jstack -l <pid> >> threaddumps.log

[1]
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 22893: Operation not permitted
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 22893: Operation not permitted
...
Caused by: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 22893: Operation not permitted
...
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:138)

JSTACK SCRIPT
Here's a script (adapted from the one on eclipse.org) that will take a series of thread dumps using jstack. It also takes the thread level cpu usage using top command as well.

Just run it like this:
sudo -u <user> jstackSeries.sh <pid> <aemserveruser> <count> <delay>

For example:
sudo -u aemuser jstackSeries.sh 1234 aemserveruser 10 3
1234 is the pid of the Java process
cq5serveruser is the Linux or UNIX user that the Java process runs as
10 is how many thread dumps to take
3 is the delay between each dump

Note: The top output has the native thread id in decimal format while the jstack output has the nid in hexadecimal. You can match the high cpu thread from the top output to the jstack output by converting the thread id to hexadecimal.

In addition to the script above, we also have a similar Windows Powershell script and an Adobe AEM specific script on github.

Thread Dump Tool for Adobe Experience Manager
If you are using Adobe Experience Manager product then you can install this tool to have a simple UI for generating thread dumps.

ALTERNATIVE WAYS TO OBTAIN A THREAD DUMP
If the jstack tool is not available to you then you can take thread dumps as follows:

Note: Some tools cannot take thread dumps from the JVM if the commandline parameter -Xrs is enabled. If you are having trouble taking thread dumps then please see if this option is enabled.
UNIX, Mac OS X, and Linux (JDK 1.4 or lesser version)

On UNIX, Mac OS X, and Linux, you can send a QUIT signal to the Java process to tell it to output a thread dump to standard output.

Run this command to do this: kill -QUIT <pid>
You may need to run this command as sudo -u user kill -QUIT <pid> where "user" is the user that the Java process is running as.

If you are starting CQSE using the crx-quickstart/server/start script then your thread dumps will be output to crx-quickstart/server/logs/startup.log. If you are using a third-party application server such as JBoss, WebSphere, Tomcat, or other, see the server's documentation to find out which file the standard output is directed to.

Windows:
JDK 1.X

  1. Download javadump.exe (attached below).
  2. Start the JVM with these three arguments (they must be in the right order): -XX:+UnlockDiagnosticVMOptions -XX:+LogVMOutput -XX:LogFile=C:\temp\jvmoutput.log
  3. Press Ctrl+Shift+Esc to open the Task Manager.
  4. Find the PID of the Java process.
  5. From the command line, run javadump.exe <pid>
  6. The thread dump will appear in the jvmoutput.log file mentioned in step 2.
JDK 1.6
Get a thread dump from jconsole tool, by using a plugin: [0]

Here's how you can request a thread dump:

  1. Add the following parameter to the jvm running Communique : -Dcom.sun.management.jmxremote
  2. Download and install JDK 1.6 (if not done yet).
  3. Download and extract the Thread Dump Analyzer utility. [1]
  4. Run jconsole.exe of JDK 1.6:
  5. jconsole.exe -pluginpath /path/to/file/tda.jar
  6. Click the Thread dumps tab.
  7. Click the Request Thread Dump link.
Note: If you are running AEM 6.* and want to observe the running threads, you can request http://<host>:<port>/system/console/status-Threads to get a thread list. However, please note that these thread dumps will not work with thread dump analysis tools such as samurai or tda.
Applies to

All Adobe Products running in a JVM

Thread dump analysis tools:
[0] http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx
[1] https://github.com/irockel/tda
[2] https://fastthread.io/
[3] https://www.ibm.com/support/knowledgecenter/en/SSLLVC_5.0.0/com.ibm.esupport.tool.tmda.doc/docs/readme.htm


By aem4beginner

No comments:

Post a Comment

If you have any doubts or questions, please let us know.