Transmute

The problem

Companies running agents in production are renting the same general intelligence for every single call, at frontier prices, even though most of their traffic is the same task repeated over and over. Rented intelligence is exactly the same on day 1,000 as it was on day one.

What struck me is that the evidence of what the agent actually does already exists. There are thousands of traces, and there's usually an oracle hiding in the traffic itself: output schemas, review queues, task outcomes. Nothing turns that evidence into software the company owns, and that's the gap I wanted to close.

What I built

I started by proving it could work at all, by hand. I took a GPT-5.2 retail agent from Sierra's tau^2-bench and compiled it into a version that makes one small-model call per turn, with everything else deterministic, and hands back to the frontier model whenever the compiled path isn't confident.

Then I built the compiler itself. The first version reads the frontier agent's traces and the task oracle, writes a small delta-only artifact for a model at roughly a tenth of the cost, evaluates it, looks at exactly where it failed, and only accepts a change if it clears a fixed margin. No human writes or reads anything domain-specific along the way. My favourite lesson from this: the very first artifact just restated the policy and actually made the model worse. The instruction that fixed it was “write only the delta, and make every line traceable to a concrete failure.” Specialists are deltas, not smaller generalists.

$ transmute compile --benchmark tau2-retail --receipt
[compile]      120 traces → delta-only specialist
[holdout]      180 sealed tasks × 4 trials
[pass^1]       93.9% compiled / 94.3% frontier
[pass^4]       86.1% compiled / 81.1% frontier
[cost]         ~18× lower | frontier calls −99.1%
[constructor]  240 held-out tasks, four trials each
[cheap model]  88.5% → 92.4% | pass^4 66.3% → 74.6%
[paired]       96 wins / 59 losses | p = 0.0037
[human rules]  0
[guard]        frontier fallback for unfamiliar cases
[claim]        parity on single attempts; repeatability is higher

Results

The hand-compiled agent matches the frontier model on 180 held-out tasks it had never seen (94% vs 94%), and it's actually more repeatable: it succeeds on all four attempts 86% of the time versus 81%. It removes 99% of frontier calls and costs about 18 times less.

The control experiment matters as much as the headline to me. If you just swap a model at a tenth of the cost into the same agent loop, you lose 8 points of accuracy and 20 points of repeatability on the very same tasks. Changing the structure of the agent recovers both, at even lower cost. It's construction, not substitution.

The automated compiler was tested on four separate sets of held-out tasks it had never seen, 240 tasks with four trials each. The cheap model goes from 89% to 92% and gets more reliable (it succeeds on all four attempts 75% of the time instead of 66%), and the gain points in the same direction on all four sets (96 won versus 59 lost, p = 0.004). The frontier model scored 95% on the one set where it was run, at about 9 times the cost. The whole artifact was built from 120 traces for $16, completely unattended.

I also want to be upfront about the negative result. A version of the compiler that had no access to the ground truth produced no gain at all. The transcript judge it relied on has 100% recall but only 43% precision, so the compiler ended up optimizing for failures that never happened. That is exactly the research problem the company exists to solve, and I now have the instrument to measure it.

What I'd do differently

My first version tried to distill the whole agent in one pass, and it broke exactly where it mattered most: the rare branches with almost no trace coverage. What made it work in the end was treating verification as the product rather than the safety net. The fallback path is what lets the compiled stack be aggressive everywhere else. If I were starting again, I'd build the adversarial verifier before the generator, and I'd measure per-branch coverage from day one instead of looking at aggregate pass rates.

I'd also start talking to buyers earlier. Everything above is evidence that the technology works, and none of it is evidence that someone will pay for it. I know the difference, and closing that gap is what I'm working on now.