Writing tests that earn their keep
Tests are code, and like all code they have a cost: someone has to write them, run them, and maintain them as the system changes. A test suite can be an asset that lets you change software fearlessly, or a liability that makes every change twice as much work. The difference is not how many tests you have; it is which tests you wrote. We think a lot about that distinction.
We don't chase coverage
Coverage percentages are easy to measure and easy to game, which makes them a tempting target and a poor one. It is entirely possible to have ninety per cent coverage and almost no confidence, because the tests assert nothing meaningful, or to have sixty per cent coverage that protects exactly the parts that matter. We use coverage as a rough hint about what we might have missed, never as a goal in its own right. The real question is not "how much of the code runs during the tests?" but "if I break something important, will a test tell me?"
Test behaviour, not implementation
The most valuable tests describe what the software should do, from the outside, in terms a user or a calling system would recognise. The least valuable tests pin down how the code happens to be written today. The trouble with the second kind is that they break every time you refactor, even when the behaviour is unchanged — so they punish exactly the cleanup work that keeps a codebase healthy. We aim our tests at observable behaviour and give the implementation room to change underneath them.
A good test fails when the behaviour is wrong and stays green when you simply tidy the code. A bad test does the opposite.
Mind the shape of the suite
We think in terms of the familiar testing pyramid, and we follow it loosely. The base is a large number of fast, focused tests around the logic — the calculations, the rules, the edge cases — that can run in milliseconds because they do not touch the database, the network, or the filesystem. These are cheap to write and cheap to run, so we have many of them.
Above that sits a smaller layer of integration tests that check the pieces work together: that a request really does write the right row, that two components agree on the shape of the data passing between them. These are slower and more involved, so we are selective. At the very top sits a thin layer of end-to-end tests that exercise the whole system through its real interface. These are the most realistic and the most expensive — slow, occasionally flaky, and costly to maintain — so we reserve them for the handful of critical journeys that absolutely must work.
The most valuable tests follow the bugs
When something does break in production, the fix is not complete until there is a test that would have caught it. These regression tests are some of the most valuable in the whole suite, because they are grounded in something that actually went wrong rather than something we imagined might. Over time, a codebase accumulates a kind of scar tissue of these tests, and each one is a promise that a particular painful mistake will not happen twice.
Edge cases are where the value hides
The happy path usually works; it is what everyone tries first. The bugs live at the edges: the empty list, the enormous input, the date that lands on a leap day, the two requests that arrive at the same instant, the user who clicks the button twice. When we write tests, we spend most of our attention on these boundaries, because that is where confidence is actually won or lost.
A test you don't trust is worse than no test
A flaky test — one that fails intermittently for no real reason — does real damage, because it teaches the team to ignore red builds. Once people start re-running the suite until it passes, the suite has stopped doing its job. We treat flakiness as a serious bug in its own right and fix it promptly, even if that means deleting a test we cannot make reliable. A smaller suite that everyone trusts is far more useful than a large one that people have learned to shrug at.
Tests as a design tool
There is a final benefit that has nothing to do with catching bugs. Writing a test forces you to use your own code from the outside, and awkward-to-test code is almost always awkward-to-use code. When a test is painful to write, it is usually telling us something about the design — that a function is doing too much, or that two concerns are tangled together. Listening to that discomfort, rather than working around it, has improved our designs as much as it has improved our reliability. The tests earn their keep twice over.
Written by the Arcwell engineering team. If you're wrestling with something similar, we're happy to compare notes.