Integrated Tests Are a Scam

December 29, 2014

Reading time ~3 minutes

This is an excellent conference talk laying out the core problem with reliance on integrated tests1. I've seen this problem at several companies. Here are the steps to reproduce:

  1. Build a complicated system organically.
  2. Observe quality degrade.
  3. Declare testing a panacea.
  4. Build a large, QA-lead integrated test suite at great cost.

Now a huge, brittle test suite exists which provides no direct value to development, where it's needed most, and doesn't address the root problem: poor system design.

Integrated test hell.

J.B. Rainsberger describes the problem well. If your project has a monolithic, external test suite which relies on the entire architecture to run you are in this special hell right now. The clearest representation I can think of is his explanation of how many tests you'd have to write to get value out of an integrated tests versus isolated tests2.

A software architecture with a few interconnected components (lets say, for example: a database, REST API, UI, Job Queue) requires tests for each function of each component. If we're relying on integrated tests we have to write tests to exercise every function of every component, and every combination of connections between every function of every component. As you write that software you need to write integrated tests. You can't write that many tests. You don't have enough time to write enough tests to have confidence in that system. That's scary. That's impossible.

The cycle.

I love this because it's clear and true. We've all seen this run-on sentence loop:

100% of our integrated tests pass but there's a mistake3 in our software; so we write more integrated tests to fill in the cracks which allows us to design more sloppily and gives us more opportunities for mistakes, and spending time on integrated tests means less time for isolated tests which increases the likelihood that 100% of the tests pass but we still have mistakes.

What should you do about this?

There is a strong correlation between large numbers of integrated tests and design problems. Integrated tests don't offer any pressure to improve our designs. Isolated tests do. Stop pretending integrated tests are helping you. Write isolated tests.

To quote J.B.:

The real benefit of isolated tests — testing one function at a time — is that those tests put tremendous pressure on our designs. Those tests are the ones that make it most clear where our design problems are. Remember that the whole point of test driven development is not to do testing; it's to learn about the quality of our design. We use the theory that if our design has problems then the tests will be hard to write. The tests will be hard to understand. It'll be difficult to write these small, isolated tests to check one thing at a time.

If you have more isolated tests than integrated tests chances are you have a decent design with clear interfaces and contracts between collaborating systems. This path is cheaper, faster, less likely to allow mistakes, and provides high-bandwith feedback on the quality of your software design. As you write this software you need to write isolated tests.

You don't have to multiply the code paths in your system to get thorough coverage, you can just add them. You go from a combinatorial explosion of tests-to-code-paths to a linear increase in tests. That's possible.

There's a lot of gold in this talk. Watch it. Twice!

  1. Not to be confused with integration tests (referred to in this talk as collaboration tests). Integrated tests require a complete, integrated architecture to run. Integration tests simply test the collaboration between independent components. 

  2. Isolated tests is a good name for tests operating on a specific function within a software architecture which exercise that function directly, in isolation, independent of any external collaborators in the architecture. 

  3. Sometimes we call these defects or bugs; I agree with the speaker that's too abstract. It's a human error (more likely a series of human errors). Everywhere else in the world we call those mistakes. 

Run a Minecraft Bedrock Server on Kubernetes

Run Minecraft Bedrock Server on Kubernetes reliably with persistent storage. Continue reading

Redefining Culture Fit

Published on November 06, 2015

The cloud-native future

Published on September 02, 2015