Learning from incidents without pointing fingers
If you run software for long enough, it will break. A deployment will go wrong, a dependency will fail, a clever edge case nobody imagined will arrive at the worst possible moment. The mark of a good team is not that incidents never happen — that is impossible — but what it does with them afterwards. A well-run review turns a bad day into a lasting improvement. A badly-run one turns it into a search for someone to blame, and teaches everyone to hide problems next time.
Blame is the enemy of learning
The single most important thing about how we review incidents is that they are blameless. The goal is never to find the person who made the mistake; it is to understand the conditions that allowed the mistake to cause harm. This is not about being soft. It is about being effective. The moment people fear being blamed, they stop sharing the very details a review depends on. They hedge, they omit, and the team learns nothing. When people trust that honesty is safe, they tell you exactly what happened, and you can actually fix it.
Ask "what about the system let this happen?" rather than "who did this?" The first question makes things better. The second just makes people quiet.
People don't fail in a vacuum
When someone deploys a bad change, the interesting questions are not about them. Why did the tests not catch it? Why did the deployment process let it through? Why was it possible to make that mistake so easily, and why did it take so long to notice? Almost every incident that looks like human error turns out, on inspection, to be a system that made the error easy to commit and hard to catch. Those are the things we can actually change. Telling someone to "be more careful" changes nothing; redesigning the process so the mistake cannot happen changes everything.
Write down what actually happened
For any incident worth learning from, we write a short, factual account: what happened, when, what the impact was, and how it was resolved. We build a timeline, because the sequence of events almost always reveals more than any single moment. Often the most valuable insight is not the original trigger but something further along — a missing alert that delayed detection, a runbook that was out of date, a dashboard that pointed in the wrong direction. The write-up is not a formality; it is the artefact the whole process produces.
Separate the trigger from the cause
Every incident has a trigger — the specific thing that set it off — but the trigger is rarely the real story. A single failing server should not take down a whole system; if it did, the deeper cause is the lack of redundancy, not the unlucky hardware. We push past the immediate trigger to the conditions that let it escalate, because fixing only the trigger leaves you exposed to the next one that looks slightly different. The question we keep asking is not just "what broke?" but "why did that break cause this much harm?"
Actions, with owners and dates
A review that ends in shared understanding but no change is a missed opportunity. Every review produces a small number of concrete improvements, each with a person responsible and a realistic date. We are deliberately modest about the number — a handful of changes that actually get made is worth far more than a long list that gathers dust. And we track them to completion, because an action item that is written down and forgotten is just a different way of learning nothing.
Share what you learned
The lessons from one incident are usually relevant well beyond the team that lived through it. We share our reviews openly within the studio, because the same class of mistake tends to lurk in more than one place. There is no shame attached, only the expectation that we all get a little wiser each time. Over the years this has built up a shared memory of how things go wrong and how to prevent them — which, more than any tool or process, is what makes a team genuinely reliable.
Reliability is built from incidents
It sounds backwards, but the most reliable systems we run are reliable precisely because they have had incidents and learned from each one. Every outage, handled well, removes a whole category of future failure. The teams that improve fastest are not the ones that never stumble; they are the ones that treat every stumble as information, examine it without fear, and quietly make sure that particular thing never hurts them again.
Written by the Arcwell engineering team. If you're wrestling with something similar, we're happy to compare notes.