Observability for small teams, without the overwhelm
Observability has a reputation for being expensive and complicated — a wall of dashboards, a firehose of metrics, an alerting system that pages someone every hour. For a small team, that is not just unaffordable; it is counterproductive. The goal is not to collect everything. It is to be able to answer two questions quickly: is the system healthy, and if not, why not? You can get a long way toward both with surprisingly little.
Start with the signals users feel
The most useful place to begin is with the things your users actually experience: are requests succeeding, and are they fast? A handful of high-level signals — the rate of requests, the proportion that are failing, and how long they take — tell you most of what you need to know about whether the system is doing its job. If errors are low and response times are healthy, the system is probably fine, whatever the deeper metrics say. If they are not, you know there is a problem before your users have to tell you.
We deliberately start at this level rather than drowning in low-level detail. Internal metrics like memory use and queue depth matter, but they are clues for diagnosis, not the headline. The headline is the experience.
Logs are for the story, metrics are for the trend
Metrics and logs do different jobs, and confusing them leads to waste. Metrics are good at answering "how much" and "how often" over time — they are cheap to store and quick to chart. Logs are good at telling the story of a single request when something has gone wrong. We use metrics to notice that error rates have climbed, and logs to understand why. Trying to reconstruct trends from logs is slow and expensive; trying to debug a specific failure from metrics alone is impossible.
Metrics tell you that something is wrong. Logs and traces tell you what. You need both, but for different moments.
Make logs worth reading
A log that just says "error occurred" is almost useless at three in the morning. We write logs that carry context — which user, which request, which operation, what was being attempted — so that a single line can point us toward the cause. Structured logging, where each entry has consistent fields rather than free-form text, turns logs from something you scroll through into something you can search and filter. The extra discipline at writing time pays off enormously at debugging time.
Connect the request to itself
In any system made of more than one component, the hardest question is often "what happened to this one request as it moved through everything?" Attaching a single identifier to a request and carrying it through every component and every log line makes that question answerable. You do not need a heavyweight tracing platform to benefit; even just threading a request ID through your logs lets you follow a single journey across services, which is often exactly what you need when something has gone wrong.
Alert on symptoms, not causes
The fastest way to make a team ignore its alerts is to send too many. We are ruthless about this. An alert should mean "a human needs to do something now", and it should fire on things users actually feel — elevated errors, requests timing out, a queue backing up beyond recovery — rather than on every internal fluctuation. A server briefly hitting eighty per cent of its memory is usually not worth waking anyone; users getting errors always is. Every alert that fires and turns out not to matter erodes trust in all the others.
The dashboard you actually look at
It is tempting to build elaborate dashboards with dozens of panels, but a dashboard nobody reads is just decoration. We keep one simple overview per system that shows the handful of signals that matter, the kind of thing you can glance at and immediately know whether things are healthy. The detailed dashboards have their place for digging in, but the day-to-day view should fit on one screen and tell the truth at a glance.
Good enough, consistently
For a small team, the right amount of observability is the amount you will actually maintain and act on. A modest setup — a few golden signals, structured logs with request IDs, and a short list of meaningful alerts — applied consistently across every system beats an elaborate platform that only one person understands. The aim is calm operations: knowing the system is healthy without staring at it, and being pointed quickly at the cause on the rare occasions it is not.
Written by the Arcwell engineering team. If you're wrestling with something similar, we're happy to compare notes.