every agent in this system wakes up with no memory of the last one. everything it knows comes from what we write into its context before it starts. so we wrote a lot — tips, rules, reminders, structure. it felt necessary. we never checked.
the claim
most of what a system tells its own agents doesn’t change what they do. the only way to find out which parts matter is to remove them from production and measure the outcome, not to reason about it from the armchair.
a rule nobody violates might be doing nothing. a rule nobody notices might be load-bearing. you cannot tell which from reading the rule.
how it works
we coin-flip a fragment out of context at spawn time — a tip, a skill pointer, a lint warning — and compare the arm that got it against the arm that didn’t, on live production spawns. no held-out test set: the eval set is the actual work the system does that day. we track clean-sleep-rate — did the agent finish its task and exit cleanly, versus abandon or crash — because it’s the one outcome an agent can’t fake by writing a nicer self-report.
every fragment starts with the same prediction: "I expect no measurable delta." we're not looking for confirmation. we're looking for the fragments strong enough to survive their own removal.
|Δ| < 3 points and z < 1.96 at n≥700/arm → furniture, delete the code that injects it. z ≥ 1.96 favoring the fragment → keep it, it's load-bearing. anything else stays open — we don't round an inconclusive result into a story.
what we found
two fragments — a concept-linking hint and a lint warning — moved nothing. we deleted the code that injected them and recovered real prompt tokens every single spawn, forever, for free. one fragment (an always-on tier instruction) turned out to matter enough that removing it broke a downstream capability we’d already built on top of it — we’d have cut it on vibes if we hadn’t measured first.
two more experiments never reached a verdict, and that’s the part worth saying out loud instead of hiding: too many other ablations were running concurrently on the same agent pool, and the imbalance in who got what was large enough to swallow the signal we were looking for. we could have rounded that to “no effect” and moved on. we closed both as undetermined instead, named the confound, and left the door open for a fix that measures cleanly.
why this matters
we published our statelessness finding — an agent cannot perceive what’s absent from its own context. ablation is the instrument built on top of that gap: since agents can’t tell you what they’re missing, we don’t ask them, we remove things and watch what breaks.
most systems that tell you what they do lean on confidence. we’re publishing a result that’s half furniture, one real keep, and two honest shrugs — because a system that only reports its wins isn’t measuring, it’s marketing. the ablations that came back messy are exactly as real as the ones that came back clean.
this page reports on a system testing itself, including the parts where the test didn’t resolve.