Engineering
Observability is being able to answer what a system is doing without deploying new code to find out. It is logging, metrics and alerting designed together rather than added after an incident.
Context
The time between something breaking and someone understanding why is the part of an incident a business actually feels.
Choices
Every choice below has a reason attached. A stack without reasons is a logo wall.
The questions we work through before recommending anything in this area.
Recommending something everywhere is the same as having no opinion.
Where we apply it
Monitoring tells you a known thing broke. Observability lets you investigate an unknown failure without shipping new code to gather the data, which is the situation most real incidents present.
Structured logs with a request identifier and error reporting with context. Together they cover most of the diagnostic gap, and both are cheap relative to a full monitoring platform.
Observability is being able to answer what a system is doing without deploying new code to find out. It is logging, metrics and alerting designed together rather than added after an incident.
The time between something breaking and someone understanding why is the part of an incident a business actually feels.
A static site with no server-side behaviour worth observing; Buying an expensive platform before anyone has agreed what would constitute an incident; and As a substitute for fixing a known recurring failure, which monitoring will simply document
How long the last incident took to diagnose, and why; Whether anyone would know about a failure before a customer reported it; and Whether logs can be searched, or only read
Logs as data rather than prose, with one identifier threading a single request across services. Without that thread, diagnosing anything distributed means guessing which lines belong together.
Error rate, latency, throughput and saturation, per service. Dashboards with forty panels are read by nobody during the incident they were built for.
Connected
Bring the existing system and the constraints around it. We will tell you where this fits, where it would be the wrong tool, and what we would leave exactly as it is.
