JAX Finance, 3-6 April 2017
The Conference for Technology in Finance

After acceptance: reason about what your system is producing with data testing

JAX Finance speaker Dr. Stefanos Zachariadis offers a sneak peek into his session and shares some techniques for making sense of what your system is producing and detecting issues before customers do.


after acceptance: reason about what your system is producing with data testing

It’s 2017. You’re asked to implement a feature in your team. The feature is distilled into a story with acceptance criteria. You start by writing some honest end to end acceptance tests and then test drive the development of the implementation. An iteration later, all the tests are passing and the feature goes into production. Where it proceeds to fail. Maybe exceptions are thrown left right and centre. Sounds familiar? The development process above sounds almost textbook. So what could have gone wrong?

setUp to fail

Let’s assume we’re developing a piece of software for retail banking. What could have gone wrong with our release that our acceptance tests in our CI environment did not catch?

making our data invalid

Let’s assume that we model bank accounts as follows:

enum AccountType {

public class Account {
private final String name;
private final AccountType type;

Now, our bank decides to get out of the private banking business. We then proceed to engage ourselves in one of the most satisfying experiences a developer can have: deleting code. Our code no longer supports the creation of PRIVATE AccountTypes and we deleted the enum value and supporting code. And of course, any acceptance test that used to work with private banking accounts is gone. Hey, we don’t support this type of account any more.

Only that we forgot that there are still private banking accounts in production, so as soon as we release the code, reporting starts failing with exceptions!

nobody expects that value there!

In our domain, we’re assuming that every account has a name. That’s the name you see when you log in to your online banking portal. Armed with this assumption, we add an equals() method to our Account. Unfortunately, one of the previous revisions had a bug that under some circumstances created accounts with null names; we have fixed the bug a long time ago now, so we can no longer create null name accounts in our CI environment. Or perhaps an account name was optional in the distant past. Either way, there’s now accounts in production with null name fields which are impossible to create in our CI environment. As soon as we release our code and equals() is run on the right account, it fails with a NullPointerException.

did our migration work?

Our bank is expanding! It used to be the case that every account was implicitly a USD account – but we now are going to support accounts of different currencies. We modify our account to look like this:

public class Account {
private final String name;
private final AccountType type;
private final CurrencyCode currency;

We migrate (or perhaps have our DBA migrate) all production accounts to have a default currency code of USD, as they are all accounts that were created when the only account currency supported was USD. How do we actually know that our migration worked? How can we rely on the fact that all our accounts have a non null USD currency code? After all, we cannot create null currency code accounts in our CI environment.

This is just a small sample of problems that traditional CI may miss. So how can we avoid them?

the problem with tests

Software tests are the best means of defence we have against bugs and regressions. And in modern software development, passing tests around a new feature is how we know that feature is done and working. But tests have limitations. Here’s how a typical test for our banking software could look like, assuming it’s an end to end test:

public void getsTheCorrectBalanceAfterADeposit() {
Account account = new Account();


assertEquals(200, account.getBalance());

Basically, given an account, when I deposit $200 into it, then the balance of the account is $200. This is nice and simple, right?

software is complicated

Unfortunately, real systems are rarely that simple. Take an example system that has 200 boolean variables. Even such a trivial system has 2^200 possible states it can be in. This is a huge number – larger than the number of stars in the known universe. A system also does not stay static; We all like frequent releases and for very good reasons, but frequent releases also mean frequent exposure to change and therefore new risk.

Our hypothetical system with the 200 booleans had 200 booleans that are interpreted in a certain way at a particular release. So as if 2^200 possible states wasn’t hard enough on its own, we now need to add a time dimension to that complexity. This is because most systems produce durable data and that data needs to be read and dealt with sensibly in any future release. After all, you wouldn’t want to lose the money in your bank account when your bank releases a new version of their software.

continuous delivery & durable data makes it even harder

Our given, when, then test above is showing us that the system works when I’ve just created a bank account and deposited $200. In reality, the balance in your account is going to be a result of a sequence of operations you’ve instigated over a long period of time, actions other actors (e.g. direct debits) have performed, system releases and data migrations. What this means is that real data (your bank account) is a result of a very complex set of interactions, that testing is unlikely to include. This is because testing cannot be exhaustive because our systems are way too complex and produce and amend data over long time periods and continuous releases.

Durable data over multiple releases means that data is being produced and added to by multiple versions of your software. A CI environment and continuous delivery pipeline by definition tests a single snapshot in time, that is the software revision / commit that we’re determining whether to release to production. This means that even if somehow you do think you can do exhaustive testing, it’s actually impossible to address the time dimension using standard CI acceptance tests, since if you’d wanted to address it you’d need to run your acceptance tests for your nth release on top of data generated by tests from the previous n – 1 releases. Clearly this does not scale. This means that your new release is going to touch data that have been generated by sequences of operations that are not tested in your CI environment no matter what you do!

what we’d like to test

There’s three data attributes we’d like to reason about:

* Data validity
Validity is an indication of the system correctness. An invalid state can lead our system to catastrophic failure (e.g. Exceptions) or undefined behaviour that’s not predicted by our tests. There’s two aspects to validity:

  1. All data that could be loaded at runtime should be able to be loaded without error. This tests for things like unexpected (even corrupt) data that the system can no longer read.
  2. Each application has different business specific criteria of what it means to be valid, which could be asserted on. In our fictional banking software, it may be that no account can have a balance that’s negative and less than some overdraft limit.

These tests could be expressed rather simply:

public void allAccountsAreReadable() {
final AccountDao accountDao = new AccountDao();
for(User user : users) {
public void allAccountsHaveABalanceThatsAboveTheOverdraftLimit() {
final AccountDao accountDao = new AccountDao();
for(User user : users) {
Account account = accountDao.loadAccountFor(user);
assertTrue(account.getBalance() > NEGATIVE_OVERDRAFT_LIMIT);

* Data invariance
The state of the system before and after a release should be the same (there may be some exceptions of course). Given that data is a result of a complex set of operations that we have no hope of fully replicating in our CI environment, we can’t really make concrete assertions on what individual data values could be. To put this another way, if you’re a customer of our fictional bank, we can’t write an assertion on what your balance should exactly be. But what we could do, is that if we have revision X of our banking software out in production and we’re taking revision Y through our continuous delivery pipeline, then revision X should give me the same balance as revision Y in a controlled environment. In code, this could look as follows:

public void accountBalanceIsMaintained() {
final AccountDao accountDao = new AccountDao();
for(User user : users) {
Account account = accountDao.loadAccountFor(user);

BigDecimal productionBalance = dataOfProductionRevision.getBalance(account.getId());

assertEqual(productionBalance, account.getBalance());

More generally, invariant testing involves the following:

  1. capture invariants
  2. perform action
  3. verify invariants

In our case, the action is deploy the new version of the system in a controlled environment.

* Migration integrity
At some point in a system’s life, we’re going to change the way the data is stored. To paraphrase, a continuously delivered system cannot have data persistence that’s set in stone. One way or another we’re going to have to migrate our data from the current representation in production to a different representation. If our bank is expanding to include accounts in different currencies as explained above, how can we make sure that the migration of all existing accounts to be explicitly USD accounts happens correctly? Well, here’s a test:

public void allAccountsAreInUSD() {
final AccountDao accountDao = new AccountDao();
for(User user : users) {
Account account = accountDao.loadAccountFor(user);
assertEqual(CurrencyCode.USD, account.getCurrency())

What’s stopping us from writing these kind of tests? The missing link is data – real, production data. Despite agile development techniques, continuous delivery, iterating over requirements and implementation, test driven development, CI and the like, our code tends to touch production data at the time of release. This is as late as possible and smells an awful lot like waterfall development.

if it’s painful, do it early and often

If we were able to get production data into a CI environment, we could import it into a database or a running system and write these kind of tests.

Unfortunately, this would probably be illegal. If you’re a customer of our fictional bank, you wouldn’t want your details to be available in a CI environment for all devs to see. But what if we could get something as close as possible to production data into CI, but something without personally identifying information?

data sanitisation to the rescue

The solution to this problem is to include a data sanitisation service and ship it to production along with the rest of the code. The service, a cleanser, is responsible for taking a copy of production data, removing personally identifying information, and shipping it to the CI environment. In a database environment, this could be done as follows:

  • take a database slave out of the pool
  • run code that removes personally identifying information. For example all names could change to variations of First name, Last name. This is the sanitisation or cleansing process
  • archive the cleansed result and upload it to CI
  • undo / rollback the changes and return the database to the pool

The cleanser is an integral part of our application and should be deployed with it. This is because new versions of the application may include new types of sensitive data and the cleanser needs to be updated to sanitise it.

the migration blues

A cleanser will allow us to ship our data from production to our CI environment. But can the latest version of our software in CI read that data? It may be able to, but what if the current version assumes a different data structure, because, for example, we’re now supporting multiple account currencies in our retail banking software? If only we could migrate the data to the latest schema. The solution presents itself:

Data is owned by the application and therefore data migration is integral to it. This means that the migration process is owned by the application, and new migrations should ship with the application and be performed with the deployment of every new version. This allows for a CI process that does the following:

  1. import the sanitised data that the cleanser produced
  2. capture invariants (e.g. account balances)
  3. migrate it to the latest version of the application
  4. run the tests

volumes and frequency

This approach sounds good in principle, but how often should you do it? When should the data be refreshed? How frequently should the data tests run in CI? What do you do if you you have terabytes of data?

As is often the case with difficult questions, the answer is it depends. In principle, we want to do these things as often as possible. This includes refreshing data and running the tests in CI. After all, frequent refreshes and test runs have a chance of catching bugs earlier. But often we have to be pragmatic. Here’s some techniques to consider:

  • Sampling: If your data volumes are just too large, consider making the cleanser sample the data that it sends over, rather than sending the whole lot.
  • Incremental updates: Rather than sending the complete volume of data every time, consider sending an incremental update, through a diff mechanism.
  • Rolling back: Consider rolling back after a test run – this should make the next run much faster.

a note on staging

If you’ve implemented this approach, consider using it in your staging environment too. Once you have real (sanitised) production data, why not import them into your staging environment prior to release? That way you will end up with an environment that’s as close as possible to production topology and data-wise, running the code that you wish to release to production.


Data testing can help you catch a whole new category of bugs and errors before your users do. We hope that this article outlined some approaches for doing this. This will be talked (or has been talked about, depending on when you read this) in more detail in JAX Finance in April 2017. Hopefully see you there (or maybe you were there already :-).

26 Responses

  1. Does your website have a contact page? I’m having problems locating it but, I’d like to shoot you an email.
    I’ve got some creative ideas for your blog you might be interested
    in hearing. Either way, great site and I look forward to seeing it grow over time.

  2. My developer is trying to convince me to move to .net from PHP.
    I have always disliked the idea because of the expenses.
    But he’s tryiong none the less. I’ve been using WordPress on a number of websites
    for about a year and am nervous about switching to another platform.
    I have heard great things about blogengine.net. Is there a
    way I can import all my wordpress content into it?
    Any kind of help would be really appreciated!

  3. I like the valuable info you provide in your articles.
    I’ll bookmark your blog and check again here regularly. I’m quite
    sure I’ll learn many new stuff right here! Good luck for the next!

  4. Simply wish to say your article is as astonishing. The clarity
    in your post is just nice and i can assume you’re an expert
    on this subject. Well with your permission allow me to grab your feed to keep up to date with forthcoming post.
    Thanks a million and please keep up the rewarding work.

  5. Wow, superb weblog structure! How lengthy have you ever been running a blog for?
    you make blogging glance easy. The full look
    of your site is great, let alone the content!

  6. When I originally commented I clicked the “Notify me when new comments are added” checkbox and now each
    time a comment is added I get four emails with the same comment.
    Is there any way you can remove me from that service? Appreciate it!

  7. I seriously love your website.. Very nice colors & theme.
    Did you build this amazing site yourself? Please reply back as
    I’m hoping to create my very own site and want to learn where you got this from
    or exactly what the theme is called. Cheers!

  8. Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point.
    You clearly know what youre talking about, why throw away your
    intelligence on just posting videos to your blog when you could be giving
    us something enlightening to read?

  9. I really like what you guys are usually up too.
    Such clever work and exposure! Keep up the superb works guys I’ve incorporated you guys to blogroll.

  10. Hello this is somewhat of off topic but I was wondering if blogs
    use WYSIWYG editors or if you have to manually code with HTML.
    I’m starting a blog soon but have no coding knowledge so I wanted to get guidance from someone with
    experience. Any help would be enormously appreciated!

  11. continuously i used to read smaller articles that also clear
    their motive, and that is also happening
    with this post which I am reading at this time.

  12. Hi there to every one, the contents existing at this web page are truly remarkable for
    people experience, well, keep up the good work fellows.

  13. Admiring the hard work you put into your blog and detailed information you present.
    It’s nice to come across a blog every once in a while that
    isn’t the same old rehashed material. Fantastic read! I’ve bookmarked your site and I’m adding your RSS feeds to my Google account.

  14. When I initially commented I clicked the “Notify me when new comments are added” checkbox
    and now each time a comment is added I get three emails
    with the same comment. Is there any way you can remove me from that
    service? Thanks a lot!

  15. That is very attention-grabbing, You are an excessively professional blogger.

    I’ve joined your rss feed and look ahead to in the
    hunt for more of your fantastic post. Also, I have
    shared your site in my social networks

  16. I do not even know how I ended up here, but I thought this put up was
    good. I don’t understand who you’re but definitely you are going to a well-known blogger should you are not already.


  17. Woah! I’m really enjoying the template/theme of this
    website. It’s simple, yet effective. A lot of times it’s hard to get that “perfect balance” between usability and visual appeal.
    I must say you have done a superb job with this. In addition, the blog
    loads extremely fast for me on Opera. Excellent Blog!


Automate delivery and boost productivity
The joy of being a software engineer
The process of becoming fully agile
the dos and don’ts
Get started with the tech and build apps on the Ethereum blockchain.
Insight into the FinTech movement
Squeeze the best performance from your system
Agile & Company Culture

Agile & Company Culture

FinTech & Innovation

FinTech & Innovation

Continuous Delivery & DevOps

Continuous Delivery & DevOps

Software Architecture

Software Architecture

Core Java & Programming

Core Java & Programming

High Performance & Low Latency

High Performance & Low Latency

Blockchain Technology & Solutions

Blockchain Technology & Solutions