JAX FINANCE BLOG

JAX Finance, 3-6 April 2017
The Conference for Technology in Finance
21
Feb

Leprechaun Trap

How is St. Patrick’s day related to coding? In this article, JAX Finance speaker Ram Lakshmanan introduces the ‘Leprechaun trap’ pattern for optimized Garbage collection processes.

World is full of patterns

Starting from the fine grain of salt to complex cosmos there are patterns everywhere. A right pattern withstands time. They are the changeless in the ever changing world. It requires lot of hard labor, rich experience (both good and bad), laser focus and perseverance to crystalize and create patterns from the noise. Fortunately our computing world has been blessed to have such wonderful patterns created.

One of the classic example is software design patterns – Singleton, Factory, Visitor, Observer, Memento … created by Erich Gamma, Richard Helm, Ralph Johnson and John Vlissides. These four engineers have encapsulated their years and years of hard learnt lessons, refined them and passed it on to us as easy to understand patterns. These patterns are universally applicable, irrespective of what programming language you use (Java, C, C++, PHP, Ruby…), irrespective of what technology stack you run (JEE, .NET, LAMP,…), irrespective of what type of application you build (Mobile, Web, SOA, Micro-services, Batch…).

Thread dump analysis Patterns

Inspired by those four great engineers in my limited capacity, in a humble way I have crystalized, refined & created my years and years of production battle fought experience as thread dump analysis patterns.

Thread dumps are vital artifact to do RCA (Root Cause Analysis). Did your application become unresponsive all of sudden? Did your application’s CPU started to spike up without increase in traffic, without making any code changes or any environmental changes? Did your application’s response start to degrade? Did your application start to experience memory problems after running for multiple days/weeks? Answers to several such complex problems are present in the thread dumps. But they buried inside a gamete of details. To give visibility on those hidden answers, I have created thread dump analysis patterns.

In this article let me introduce you to ‘Leprechaun trap’ pattern. Objects that have finalize() method are treated differently during Garbage collection process than the ones which don’t have them. During garbage collection phase, object with finalize() aren’t immediately evicted from the memory. Instead as first step, those objects are added to an internal queue of java.lang.ref.Finalizer object. There is a low priority JVM thread by name ‘Finalizer’ that executes finalize() method of each object in the queue. Only after the execution of finalize() method, object becomes eligible for Garbage Collection. Because of poor implementation of finalize() method if Finalizer thread gets blocked then it will have a severe detrimental cascading effect on the JVM.

If Finalizer thread gets blocked then internal queue of java.lang.ref.Finalize will start to grow. It would cause JVM’s memory consumption to grow rapidly. It would result in OutOfMemoryError, jeopardizing entire JVM’s availability. Thus when analyzing thread dumps it’s highly recommended to study the stack trace of Finalizer thread.

Real-world example

Here is a sample stack trace of a Finalizer thread which got blocked in a finalize() method:

"Finalizer" daemon prio=10 tid=0x00007fb2dc32b000 nid=0x7a21 waiting for monitor entry [0x00007fb2cdcb6000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at net.sourceforge.jtds.jdbc.JtdsConnection.releaseTds(JtdsConnection.java:2024)
	- waiting to lock 0x00000007d50d98f0 (a net.sourceforge.jtds.jdbc.JtdsConnection)
	at net.sourceforge.jtds.jdbc.JtdsStatement.close(JtdsStatement.java:972)
	at net.sourceforge.jtds.jdbc.JtdsStatement.finalize(JtdsStatement.java:219)
	at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
	at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101)
	at java.lang.ref.Finalizer.access$100(Finalizer.java:32)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:178)

Above stack trace was captured from a JVM which was using one of the older versions of JTDS JDBC Driver. Apparently this version of driver had an issue; you can see finalize() method in the net.sourceforge.jtds.jdbc.JtdsStatement object calling JtdsConnection#releaseTds() method. Apparently, this method got blocked and never returned back. Thus Finalizer thread got stuck indefinitely in the JtdsConnection#releaseTds() method. Due to that Finalizer thread wasn’t able to work on the other objects that had finalize() method. Due to that application started to suffer from OutOfMemoryError. In the latest version of JTDS JDBC Driver this issue was fixed. Thus when you are implementing finalize() method be very careful.

Why named as Leprechaun Trap?

Kids in western countries build Leprechaun Trap as part of St. Patrick’s day celebration. Leprechaun is a fairy character, basically a very tiny old man, wearing a green coat & hat who is in search for gold coins. Kids build creative traps for this Laprechaun, luring him with gold coins. Similarly anxious Finalizer thread is always in search of objects that has finalize() method to execute them. In case if finalize() method is wrongly implemented, it can trap the Finalizer thread. Because of this similarity we have named it as Leprechaun Trap.

You can learn about other patterns here: https://blog.fastthread.io/category/thread-dump-patterns/

JAX DevOps talks by Ram Lakshmanan:
Become a garbage collection hero

Tue. 04 APR 2017,
17:30 – 18:20
Don’t dump the thread dump

Wed. 05 APR 2017,
09:00 – 09:50

16 Responses

  1. You actually make it seem so easy with your presentation but I find
    this matter to be actually something that I think I would never understand.
    It seems too complicated and extremely broad
    for me. I am looking forward for your next post, I will
    try to get the hang of it!

  2. Does your website have a contact page? I’m having a tough time locating it but,
    I’d like to send you an email. I’ve got some creative
    ideas for your blog you might be interested in hearing. Either way,
    great website and I look forward to seeing it develop over time.

  3. Hello there! This is my first comment here so I just wanted to give a quick shout out and
    tell you I truly enjoy reading your blog posts.
    Can you recommend any other blogs/websites/forums that deal with the same
    topics? Thanks a ton!

  4. My coder is trying to convince me to move to .net from PHP.
    I have always disliked the idea because of the expenses.
    But he’s tryiong none the less. I’ve been using Movable-type on a variety of websites for about a year and am concerned
    about switching to another platform. I have heard excellent things about blogengine.net.
    Is there a way I can transfer all my wordpress content into it?
    Any help would be really appreciated!

  5. When I initially commented I clicked the “Notify me when new comments are added” checkbox
    and now each time a comment is added I get several e-mails with the same comment.

    Is there any way you can remove people from that service?

    Thanks!

  6. Nice post. I was checking constantly this blog and I’m impressed!

    Very helpful information specifically the last part 🙂 I care for such information much.
    I was seeking this certain info for a very long time.
    Thank you and good luck.

  7. That is really attention-grabbing, You are an excessively professional blogger.

    I’ve joined your feed and look ahead to looking for extra of
    your magnificent post. Also, I’ve shared your web site in my social
    networks

  8. Usually I do not learn post on blogs, however I would like
    to say that this write-up very compelled me to take a look at and
    do so! Your writing taste has been amazed me. Thanks, quite great post.

  9. Hi! I’ve been reading your web site for a while now and finally got the bravery
    to go ahead and give you a shout out from
    Humble Tx! Just wanted to tell you keep up the fantastic job!

  10. Great beat ! I wish to apprentice while you amend your site, how
    could i subscribe for a blog website? The account aided
    me a acceptable deal. I had been tiny bit acquainted of
    this your broadcast offered bright clear idea

BEHIND THE TRACKS

CONTINUOUS DELIVERY & AUTOMATION
Automate delivery and boost productivity
CORE JAVA & PROGRAMMINGv
The joy of being a software engineer
AGILE & COMPANY CULTURE
The process of becoming fully agile
SOFTWARE ARCHITECTURE
the dos and don’ts
BLOCKCHAIN TECHNOLOGY & SOLUTIONS
Get started with the tech and build apps on the Ethereum blockchain.
FINTECH & INNOVATION
Insight into the FinTech movement
HIGH PERFORMANCE & LOW LATENCY
Squeeze the best performance from your system
Agile & Company Culture

Agile & Company Culture

FinTech & Innovation

FinTech & Innovation

Continuous Delivery & DevOps

Continuous Delivery & DevOps

Software Architecture

Software Architecture

Core Java & Programming

Core Java & Programming

High Performance & Low Latency

High Performance & Low Latency

Blockchain Technology & Solutions

Blockchain Technology & Solutions