“top threads” plugin for JConsole

When working with large (server side) java application, sometimes it would be nice if you could look inside, to see what thread is taking up so much cpu time, and why. Something similar to the Unix top command, but then showing all threads in one (java) application, instead of all processes in the system.

When I was looking for such a monitoring application, I came accross the 2.0 version of MC4J that provides a “Thread Info” panel that displays threads together with CPU usage; exactly what I needed. Unfortunately, there is only an alpha release of this MC4J version, that is not yet perfectly stable. Moreover, the thread info panel doesn’t handle applications with large amounts of threads very well. As the source code of this version of MC4J is not (yet) publically available, this option turned out to be a dead end.

To my surprise, other applications with such functionality are hard to find. There are probably enough profiling applications that can do the job, but I wanted something simple, something JMX-based, that can used also to monitor applications running in production.

There is however something called JTop, which is a plugin for JConsole. It’s actually a demo for the new (since Java 6) JConsole plugin API, that does show CPU usage per thread. It’s fairly basic and only shows total CPU usage, which is not very usefull. You would expect that (after a year), somebody would have extended the demo to something more useful, but as I couldn’t find anything like that, I thought I should give it a try myself.

The result is a JConsole plugin that displays the top threads, sorted by cpu usage in the last sampling period. It also displays cpu usage history, and an average over the last 10 sampling periods.

To avoid ending up with an unresponsive user interface when monitoring applications with large number of threads, I took a few precautions. First of all, the plugin has it’s own refresh rate. It’s independent from the JConsole poll interval, which is 4 seconds by default. For applications with large amounts of threads, this is way too short: only retrieving all thread information can already take 4 or 5 seconds! Although you can change the JConsole poll interval with a command line option, I thought it would be more convenient to be able to change it from the monitoring panel. It’s default set to 10 seconds, which I think is reasonable in most cases. If you notice that cpu usage measurement takes too much of the application under test, just increase the sample period and see the RMI Connection thread that processes these request, sink in the list.

Another precaution was not to list all threads in the table. Displaying thousands of rows in a table is quite useless in any case, and I was afraid it would seriously harm performance. Eventually, diplaying that many rows turned out to be not much of a problem; I guess I still suffer from an prejudice with
respect to Swing performance…

Using MX4J also showed me that in a continuously refreshing table, it’s hard to select a thread in order to see it’s stacktrace. Therefore, in this plugin, tracing a thread is “sticky”: when you click a row in the table, the stacktrace of that thread is shown immediately and is refreshed with each new sample, until you deselect it or select another thread.

Even though having threads sorted by cpu usage is the logical thing to do, it’s not always convenient when you’re studying the results, as rows keeping moving with each refresh. To lock the rows to there current position, click the “fix order” button. The topmost rows (actually all rows with a non-zero cpu usage), will stay where they are. Rows that had a cpu usage of zero, but have a non-zero value in the next sampling periods, will appear just below these rows, to avoid that you oversee any thread that suddenly takes a large amount of cpu time.

You can run the plugin by downloading the jar file and passing it to JConsole with the plugin option:
jconsole -pluginpath topthreads.jar. When JConsole connects, it should have a seventh tab named “top threads”.

update: there’s a new version, see http://lsd.luminis.eu/new_version_topthreads_jconsole_plugin

, ,

  1. #1 door Peter Doornbosch om 3 maart 2008

    Thanks for all the feedback!

    Alexander: you can simply copy-and-paste from the stacktrace panel, can’t you?
    Why would you need (or want) a copy-to-clipboard button?

    Mike: i’ve tested all kinds of scenario’s, but i never got a hanging JConsole.
    Could you please provide me with more details? E.g. what Java version is used
    for JConsole
    and what for the application that is monitored?

    Mike: i’ve noticed a small memory leak in the non-heap memory too. As you
    suggest, this might be related to bug 6434648 or bug 6469701, although those seem to be fixed in the JKD you are using.
    Anyway, a leak in the monitored app is out of our (JConsole’s) control….

    Peter Paz: good point. I made a new version that is Java 5 compatible (and fixes a bug in the computation of the average, and has a new color scheme).

    Additionally: i like the suggestion of adding thread priority, but this is not
    trivial as the ThreadInfo returned by the
    ThreadMXBean does not contain thread priority (see
    bug 6588467). I’m working on a solution with a separate “thread-priority-mbean”… more on that later.

  2. #2 door David Mulligan om 16 maart 2010

    Thank you very much for this invaluable plugin Peter. It is amazing the number of times that I’ve been able to get some clue as to what a long running process is up to or stuck doing using this! I’ve used your Topthreads plugin to see which threads are using up a whole cpu as well as to see where a long running thread currently is. Basically to give me a view into the VM when I’d rather not slow down the app with a profiler.

    I have a request though. Please make it possible to keep the stack trace pane scrolled to the top when the page is refreshed.

    It might also be interesting to see which lines of the stack trace have not changed since the last refresh. Perhaps a slightly different font colour or some other relatively unobtrusive indicator like that.

  3. #3 door Aravindhan om 25 juni 2010

    The Java thread id is available in the java.lang.management.ThreadInfo but there seems to be no way to get the nid (Native Thread ID ) for a Java thread – is there any way to do this from Java code?

    Following are my use case,

    When CPU utilization of a m/c is too high due to a java application, I can get the exact problematic thread by using following command in linux,

    top -b -H -n1 -d1 | head -n 20

    The PID listed in the above command is directly mapped to native id. I can easily identify the problematic trace which causing high cpu by converting PID to HEX and match it in the thread dump. For easy debugging, I would like to high light the problematic thread trace which taken using JMX.

    Is there any way to get native ID from JMX ? or any other work around to get native ID….

    I greatly appreciate your help on the above requirement.

  4. #4 door Lauro om 9 september 2010

    It saved my neck! :)

    Thank you!

  5. #5 door Bob om 13 oktober 2010

    Thanks so much!

    This has helped me solve an issue i have been struggling with for weeks.

    Nice work.

  6. #6 door Bernd om 12 januari 2011


    im trying to connect to a remote server with jmxremote.ssl and jmxremote.authenticate enabled.

    I’m getting a SecurityException in the Java Console:
    Exception in thread “AWT-EventQueue-0″ java.lang.SecurityException: Access denied! Invalid access level for requested MBeanServer operation.

    at net.luminis.jmx.topthreads.TopThreadsPanel.setMBeanServerConnection(Unknown Source)

    Any clue what the problem here could be?


  7. #7 door Peter Doornbosch om 26 januari 2011

    Hi Bernd,

    This indicates that you don’t have enough rights (on the target VM) to retrieve the thread info. Apparently, the default “monitorRole” does not allow this, see http://download.oracle.com/javase/1.5.0/docs/guide/management/agent.html

    I suggest trying to connect with the “controlRole”. Also, you might try the new version of the topthreads plugin, that does not try to set the “threadCpuTimeEnabled” property when it is enabled already (avoiding a possible SecurityException), and handles security exceptions a little better.

    More info on the latest greatest can be found here.

  8. #8 door Peter Reyes om 18 juni 2011

    Thank you so much…
    It has been very helpful to me and saves me more time.

    Peter Reyes
    internet marketing vancouver bc

  9. #9 door Andre om 14 juli 2011

    Hi Bernd,

    excellent tool! I was desiring something like that for a long time. Didn’t know it was possible…. Thanks a lot!

    Kind regards

  10. #10 door Keir om 20 december 2011

    Worked straight out of the box which makes a pleasent change

    Nice Job

  11. #11 door Zakir om 3 januari 2012

    A great tool – helped us a lot in debugging a baffling problem. Thank you so very much!

  12. #12 door Bob om 12 maart 2012

    Under which license do you publish this? can it be used commercially (I mean: in a company, not for sale or as service) ?


  13. #13 door Eike Pögel om 20 mei 2012

    very efficient tool – helps to detect CPU top scorerer threads easily – very helpful to find out the “bad guys”, when running an Apache Servicemix ESB with bundles from different parties.

    One thing: Using it unattended in VisualVMs jconsole plugin over a long time modal “refresh interval is modifidied, because it was too small” dialog(s), left unanswered because of AFK, requires the VisualVM to be killed via Procexp/TaskManager: You can no longer click OK in this dialog(s) – no more access to the whole VisualVM possible

    Anyway: Thanks a lot, your tool helped me to solve some big problems.

  14. #14 door Rahul om 29 augustus 2012

    This plugin is very valuable in trying to debug java application.
    So I am trying to connect it to JBoss AS7 and running into the below exception.
    Any suggestions on how to make it work with JBoss7?


    Aug 28, 2012 3:26:05 PM org.xnio.Xnio
    INFO: XNIO Version 3.0.3.GA
    Aug 28, 2012 3:26:05 PM org.xnio.nio.NioXnio
    INFO: XNIO NIO Implementation Version 3.0.3.GA
    Aug 28, 2012 3:26:05 PM org.jboss.remoting3.EndpointImpl
    INFO: JBoss Remoting version 3.2.3.GA
    Exception in thread “AWT-EventQueue-0″ java.lang.reflect.UndeclaredThrowableException
    at $Proxy2.setThreadCpuTimeEnabled(Unknown Source)
    at net.luminis.jmx.topthreads.TopThreadsPanel.setMBeanServerConnection(Unknown Source)
    at net.luminis.jmx.topthreads.PluginAdapter$ConnectionListener.checkConnection(Unknown Source)
    at net.luminis.jmx.topthreads.PluginAdapter.getTabs(Unknown Source)
    at sun.tools.jconsole.VMPanel.createPluginTabs(VMPanel.java:641)
    at sun.tools.jconsole.VMPanel.propertyChange(VMPanel.java:315)
    at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:339)
    at javax.swing.event.SwingPropertyChangeSupport.firePropertyChange(SwingPropertyChangeSupport.java:75)
    at javax.swing.event.SwingPropertyChangeSupport$1.run(SwingPropertyChangeSupport.java:80)
    at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:597)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
    at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
    Caused by: java.io.IOException: java.lang.ClassNotFoundException: java.lang.ReflectiveOperationException
    at org.jboss.remotingjmx.protocol.v1.ClientConnection$BaseResponseHandler.handle(ClientConnection.java:1768)
    at org.jboss.remotingjmx.protocol.v1.ClientConnection$MessageReceiver$1.run(ClientConnection.java:453)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)
    Caused by: java.lang.ClassNotFoundException: java.lang.ReflectiveOperationException
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:135)
    at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:116)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:892)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:905)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1204)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
    at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1677)
    at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1593)
    at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1557)
    at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1557)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1235)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
    at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:72)
    at org.jboss.remotingjmx.protocol.v1.ClientConnection$BaseResponseHandler.handle(ClientConnection.java:1759)
    … 4 more

  15. #15 door Peter Doornbosch om 20 november 2012

    Hi Bob,

    You can use the tool freely for anything you like. I shared it in the hope that it would be useful for others; if this is the case, i’m a happy dude ;-) .

  16. #16 door Peter Doornbosch om 20 november 2012

    Hi Eike,

    Thanks for the feedback. You’re issue with the plugin in VisualVM must have been very irritating or frustrating, i’ll look into it.

  17. #17 door Peter Doornbosch om 20 november 2012

    Hi Rahul,

    Sounds like JBoss is sending serialized classes that JConsole cannot resolve. I’ll try to reproduce and find out what’s going on.

  18. #18 door Dennis Homann om 30 januari 2013

    Hi Rahul,

    the cause is “java.lang.ClassNotFoundException: java.lang.ReflectiveOperationException”.

    java.lang.ReflectiveOperationException is new in Java 7. My guess is that JBoss is running Java 7, but you are running JConsole on Java 6. Try 7 on both sides.


  19. #19 door Jaime Romaguera om 31 juli 2013


    Having issues with downloading the TopThread.jar (can’t find it) When I click on the link it sends me to http://arnhem.luminis.eu/blog/

    Would you be able to provide the link where I can download TopThread plugin.

    Many Thanks



  20. #20 door Peter Doornbosch om 31 augustus 2013

    You can download topthreads here: http://lsd.luminis.eu/wp-content/uploads/2011/01/topthreads-1.1.jar . Sorry for the inconvenience.

    Also note there is an update to this blog: http://lsd.luminis.eu/new_version_topthreads_jconsole_plugin/

(wordt niet gepubliceerd)
  1. Nog geen trackbacks.