Leprechaun Trap

image: Shutterstock
World is full of patterns
Starting from the fine grain of salt to complex cosmos there are patterns everywhere. A right pattern withstands time. They are the changeless in the ever changing world. It requires lot of hard labor, rich experience (both good and bad), laser focus and perseverance to crystalize and create patterns from the noise. Fortunately our computing world has been blessed to have such wonderful patterns created.
One of the classic example is software design patterns – Singleton, Factory, Visitor, Observer, Memento … created by Erich Gamma, Richard Helm, Ralph Johnson and John Vlissides. These four engineers have encapsulated their years and years of hard learnt lessons, refined them and passed it on to us as easy to understand patterns. These patterns are universally applicable, irrespective of what programming language you use (Java, C, C++, PHP, Ruby…), irrespective of what technology stack you run (JEE, .NET, LAMP,…), irrespective of what type of application you build (Mobile, Web, SOA, Micro-services, Batch…).
Thread dump analysis Patterns
Inspired by those four great engineers in my limited capacity, in a humble way I have crystalized, refined & created my years and years of production battle fought experience as thread dump analysis patterns.
Thread dumps are vital artifact to do RCA (Root Cause Analysis). Did your application become unresponsive all of sudden? Did your application’s CPU started to spike up without increase in traffic, without making any code changes or any environmental changes? Did your application’s response start to degrade? Did your application start to experience memory problems after running for multiple days/weeks? Answers to several such complex problems are present in the thread dumps. But they buried inside a gamete of details. To give visibility on those hidden answers, I have created thread dump analysis patterns.
In this article let me introduce you to ‘Leprechaun trap’ pattern. Objects that have finalize() method are treated differently during Garbage collection process than the ones which don’t have them. During garbage collection phase, object with finalize() aren’t immediately evicted from the memory. Instead as first step, those objects are added to an internal queue of java.lang.ref.Finalizer object. There is a low priority JVM thread by name ‘Finalizer’ that executes finalize() method of each object in the queue. Only after the execution of finalize() method, object becomes eligible for Garbage Collection. Because of poor implementation of finalize() method if Finalizer thread gets blocked then it will have a severe detrimental cascading effect on the JVM.
If Finalizer thread gets blocked then internal queue of java.lang.ref.Finalize will start to grow. It would cause JVM’s memory consumption to grow rapidly. It would result in OutOfMemoryError, jeopardizing entire JVM’s availability. Thus when analyzing thread dumps it’s highly recommended to study the stack trace of Finalizer thread.
Real-world example
Here is a sample stack trace of a Finalizer thread which got blocked in a finalize() method:
"Finalizer" daemon prio=10 tid=0x00007fb2dc32b000 nid=0x7a21 waiting for monitor entry [0x00007fb2cdcb6000] java.lang.Thread.State: BLOCKED (on object monitor) at net.sourceforge.jtds.jdbc.JtdsConnection.releaseTds(JtdsConnection.java:2024) - waiting to lock 0x00000007d50d98f0 (a net.sourceforge.jtds.jdbc.JtdsConnection) at net.sourceforge.jtds.jdbc.JtdsStatement.close(JtdsStatement.java:972) at net.sourceforge.jtds.jdbc.JtdsStatement.finalize(JtdsStatement.java:219) at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method) at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101) at java.lang.ref.Finalizer.access$100(Finalizer.java:32) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:178)
Above stack trace was captured from a JVM which was using one of the older versions of JTDS JDBC Driver. Apparently this version of driver had an issue; you can see finalize() method in the net.sourceforge.jtds.jdbc.JtdsStatement object calling JtdsConnection#releaseTds() method. Apparently, this method got blocked and never returned back. Thus Finalizer thread got stuck indefinitely in the JtdsConnection#releaseTds() method. Due to that Finalizer thread wasn’t able to work on the other objects that had finalize() method. Due to that application started to suffer from OutOfMemoryError. In the latest version of JTDS JDBC Driver this issue was fixed. Thus when you are implementing finalize() method be very careful.
Why named as Leprechaun Trap?
Kids in western countries build Leprechaun Trap as part of St. Patrick’s day celebration. Leprechaun is a fairy character, basically a very tiny old man, wearing a green coat & hat who is in search for gold coins. Kids build creative traps for this Laprechaun, luring him with gold coins. Similarly anxious Finalizer thread is always in search of objects that has finalize() method to execute them. In case if finalize() method is wrongly implemented, it can trap the Finalizer thread. Because of this similarity we have named it as Leprechaun Trap.
You can learn about other patterns here: https://blog.fastthread.io/category/thread-dump-patterns/
You actually make it seem so easy with your presentation but I find
this matter to be actually something that I think I would never understand.
It seems too complicated and extremely broad
for me. I am looking forward for your next post, I will
try to get the hang of it!
Thanks for your marvelous posting! I truly enjoyed reading it, you could be a
great author.I will be sure to bookmark your blog and may come back in the future.
I want to encourage you continue your great job, have a nice
weekend!
Does your website have a contact page? I’m having a tough time locating it but,
I’d like to send you an email. I’ve got some creative
ideas for your blog you might be interested in hearing. Either way,
great website and I look forward to seeing it develop over time.
Hello there! This is my first comment here so I just wanted to give a quick shout out and
tell you I truly enjoy reading your blog posts.
Can you recommend any other blogs/websites/forums that deal with the same
topics? Thanks a ton!
I’m not that much of a internet reader to be honest but your sites really nice,
keep it up! I’ll go ahead and bookmark your
website to come back later on. All the best
My coder is trying to convince me to move to .net from PHP.
I have always disliked the idea because of the expenses.
But he’s tryiong none the less. I’ve been using Movable-type on a variety of websites for about a year and am concerned
about switching to another platform. I have heard excellent things about blogengine.net.
Is there a way I can transfer all my wordpress content into it?
Any help would be really appreciated!
When I initially commented I clicked the “Notify me when new comments are added” checkbox
and now each time a comment is added I get several e-mails with the same comment.
Is there any way you can remove people from that service?
Thanks!
Nice post. I was checking constantly this blog and I’m impressed!
Very helpful information specifically the last part 🙂 I care for such information much.
I was seeking this certain info for a very long time.
Thank you and good luck.
That is really attention-grabbing, You are an excessively professional blogger.
I’ve joined your feed and look ahead to looking for extra of
your magnificent post. Also, I’ve shared your web site in my social
networks
I was recommended this web site by my cousin. I am not sure whether this post
is written by him as no one else know such detailed about my problem.
You are amazing! Thanks!
Its not my first time to visit this website, i am browsing this website dailly and get pleasant facts
from here every day.
Usually I do not learn post on blogs, however I would like
to say that this write-up very compelled me to take a look at and
do so! Your writing taste has been amazed me. Thanks, quite great post.
Hi! I’ve been reading your web site for a while now and finally got the bravery
to go ahead and give you a shout out from
Humble Tx! Just wanted to tell you keep up the fantastic job!
Great beat ! I wish to apprentice while you amend your site, how
could i subscribe for a blog website? The account aided
me a acceptable deal. I had been tiny bit acquainted of
this your broadcast offered bright clear idea
Thanks a bunch for sharing this with all of us you
really realize what you’re talking about! Bookmarked. Please also talk over with my web site
=). We may have a hyperlink alternate contract among us
What’s up, always i used to check weblog posts here early in the daylight,
because i love to find out more and more.