World is full of patterns

Starting from the fine grain of salt to complex cosmos there are patterns everywhere. A right pattern withstands time. They are the changeless in the ever changing world. It requires lot of hard labor, rich experience (both good and bad), laser focus and perseverance to crystalize and create patterns from the noise. Fortunately our computing world has been blessed to have such wonderful patterns created.

One of the classic example is software design patterns – Singleton, Factory, Visitor, Observer, Memento … created by Erich Gamma, Richard Helm, Ralph Johnson and John Vlissides. These four engineers have encapsulated their years and years of hard learnt lessons, refined them and passed it on to us as easy to understand patterns. These patterns are universally applicable, irrespective of what programming language you use (Java, C, C++, PHP, Ruby…), irrespective of what technology stack you run (JEE, .NET, LAMP,…), irrespective of what type of application you build (Mobile, Web, SOA, Micro-services, Batch…).

Thread dump analysis Patterns

Inspired by those four great engineers in my limited capacity, in a humble way I have crystalized, refined & created my years and years of production battle fought experience as thread dump analysis patterns.

Thread dumps are vital artifact to do RCA (Root Cause Analysis). Did your application become unresponsive all of sudden? Did your application’s CPU started to spike up without increase in traffic, without making any code changes or any environmental changes? Did your application’s response start to degrade? Did your application start to experience memory problems after running for multiple days/weeks? Answers to several such complex problems are present in the thread dumps. But they buried inside a gamete of details. To give visibility on those hidden answers, I have created thread dump analysis patterns.

In this article let me introduce you to ‘Leprechaun trap’ pattern. Objects that have finalize() method are treated differently during Garbage collection process than the ones which don’t have them. During garbage collection phase, object with finalize() aren’t immediately evicted from the memory. Instead as first step, those objects are added to an internal queue of java.lang.ref.Finalizer object. There is a low priority JVM thread by name ‘Finalizer’ that executes finalize() method of each object in the queue. Only after the execution of finalize() method, object becomes eligible for Garbage Collection. Because of poor implementation of finalize() method if Finalizer thread gets blocked then it will have a severe detrimental cascading effect on the JVM.

If Finalizer thread gets blocked then internal queue of java.lang.ref.Finalize will start to grow. It would cause JVM’s memory consumption to grow rapidly. It would result in OutOfMemoryError, jeopardizing entire JVM’s availability. Thus when analyzing thread dumps it’s highly recommended to study the stack trace of Finalizer thread.

Real-world example

Here is a sample stack trace of a Finalizer thread which got blocked in a finalize() method:

"Finalizer" daemon prio=10 tid=0x00007fb2dc32b000 nid=0x7a21 waiting for monitor entry [0x00007fb2cdcb6000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at net.sourceforge.jtds.jdbc.JtdsConnection.releaseTds(JtdsConnection.java:2024)
	- waiting to lock 0x00000007d50d98f0 (a net.sourceforge.jtds.jdbc.JtdsConnection)
	at net.sourceforge.jtds.jdbc.JtdsStatement.close(JtdsStatement.java:972)
	at net.sourceforge.jtds.jdbc.JtdsStatement.finalize(JtdsStatement.java:219)
	at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
	at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101)
	at java.lang.ref.Finalizer.access$100(Finalizer.java:32)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:178)

Above stack trace was captured from a JVM which was using one of the older versions of JTDS JDBC Driver. Apparently this version of driver had an issue; you can see finalize() method in the net.sourceforge.jtds.jdbc.JtdsStatement object calling JtdsConnection#releaseTds() method. Apparently, this method got blocked and never returned back. Thus Finalizer thread got stuck indefinitely in the JtdsConnection#releaseTds() method. Due to that Finalizer thread wasn’t able to work on the other objects that had finalize() method. Due to that application started to suffer from OutOfMemoryError. In the latest version of JTDS JDBC Driver this issue was fixed. Thus when you are implementing finalize() method be very careful.

Why named as Leprechaun Trap?

Kids in western countries build Leprechaun Trap as part of St. Patrick’s day celebration. Leprechaun is a fairy character, basically a very tiny old man, wearing a green coat & hat who is in search for gold coins. Kids build creative traps for this Laprechaun, luring him with gold coins. Similarly anxious Finalizer thread is always in search of objects that has finalize() method to execute them. In case if finalize() method is wrongly implemented, it can trap the Finalizer thread. Because of this similarity we have named it as Leprechaun Trap.

You can learn about other patterns here: https://blog.fastthread.io/category/thread-dump-patterns/