Five real services are running inside this Space, calling each other. Break one of them and the engine has to work out which one you broke, using nothing but the OpenTelemetry traces they emit. It is never told the answer.
Breaking a leaf is the interesting case: everything above it goes red too.
Each one produces a different signature in the telemetry.
inventory is called from two directions, which is what makes it worth breaking.
When inventory breaks, catalog breaks because it calls inventory, and
gateway breaks because it calls catalog. Three services throwing errors, one actual
problem. And because services higher up handle more requests, they produce
more errors than the service that actually failed. So sorting by error count points
almost exactly away from the fault.
The signal that separates doing work from waiting for work:
A service stuck waiting on a slow dependency has a big total duration but its self time does
not move, because it is not doing anything extra. Here is a real run with 120ms added to
inventory and nothing else touched:
| service | total time | self time | |
|---|---|---|---|
| gateway | 3.8x | 0.9x | victim |
| orders | 3.4x | 0.9x | victim |
| catalog | 7.3x | 0.9x | victim |
| payments | 1.0x | 1.0x | untouched |
| inventory | 10.9x | 11.0x | the actual fault |
catalog got 7.3 times slower without doing one bit of extra work. Total time says
investigate catalog. Self time says catalog never did anything unusual.
Error origins. Walking each trace, a span only counts as an origin if it failed and none
of its children failed. A span that failed because something underneath it failed is just
relaying bad news, and its service gets no blame.
Downstream-clean. A service is only a candidate if it looks unhealthy and none of the
things it depends on look unhealthy. If its dependency is broken too, it is probably a victim.
No language model is involved in any of that. It is trace-tree traversal and statistics, which is what makes it measurable. Full detail in docs/method.md.
The benchmark covers eight fault types. This demo offers four of them, and the omissions are deliberate rather than accidental.
Network partition and total outage. Severing a link, or failing essentially every request, trips the callers' circuit breakers within a second or two. Once those latch open the traffic stops flowing and the entire subtree below goes silent, which leaves nothing in the telemetry to reason about. The engine scores 100% on both classes in the benchmark, where the hardware was fast enough that some traffic still got through, but on a shared-CPU Space the cascade completes and the answer becomes unreliable. I would rather say that here than give you a button that returns the wrong answer.
That is the circuit breaker blind spot, and it is the most interesting thing this project turned up: a safety mechanism working exactly as designed erases the evidence needed to diagnose what it was protecting against. There is a hand-written postmortem about it in the repository.
Concurrent faults. Every experiment here breaks exactly one thing. Real incidents do not, and handling two overlapping faults would need the engine to return a set rather than a single culprit. That is the first thing I would build next.
33 injected faults across 8 fault types, plus 7 controls where nothing is broken.
Same telemetry, scored four other ways.
| method | AC@1 | MRR |
|---|---|---|
| this engine | 100.0% | 1.000 |
| most errors | 33.3% | 0.449 |
| largest latency increase | 33.3% | 0.474 |
| random guess | 21.2% | 0.474 |
| highest total latency | 6.1% | 0.414 |
Sorting by total latency scores below random guessing. It is not random, it is consistently wrong: it picks whichever service is sitting directly above the fault, waiting.
I tuned the detection thresholds by looking at which scenarios were failing, so 100% on that set is a fitted number. It shows the method can work, not that it generalises. For an honest estimate I wrote a second set of 12 scenarios with fault sizes that appear nowhere in the first set, and did not look at them while tuning anything.
| development | held out | |
|---|---|---|
| faults / controls | 33 / 7 | 9 / 3 |
| AC@1 | 100.0% | 100.0% |
| classification | 100.0% | 88.9% |
| false positives | 14.3% | 33.3% |
9 of 9 on the held-out set. With only 9 samples the 95% confidence interval runs from roughly 66% to 100%, so this is consistent with the method being genuinely good and also with it being merely okay. I would want a few hundred scenarios before claiming more.