Parity
Parity is a Stripe Billing simulator for testing agents without using the real API. It includes subscriptions, proration, invoices, signed webhooks and test clocks, and checks its behavior against real sandbox recordings. I built the working system in one week using coding agents.
- Role
- Designed and built using coding agents
- Timeline
- September 2026 — ongoing
- Stack
- TypeScript, Node, Stripe sandbox, GitHub Actions, MCP
- Status
- Working pre-release simulator, benchmark and nightly alignment loop
The problem
Agents need somewhere to practice and be tested before they touch real billing state. A mock that returns the right shape isn't enough: subscriptions, invoices and payments affect each other, and an agent makes a sequence of decisions. I wanted an environment whose differences from the real API were measured, rather than assumed away.
What I built
A stateful TypeScript emulator of Stripe Billing and a benchmark that replays multi-step scenarios. It compares HTTP status, response shape, normalized values, and whole sequences including webhook events. The scores include 95% confidence intervals and a separate measurement of how much the real sandbox varies against itself.
Each night, the pipeline refreshes part of the sandbox recordings, scores the simulator, and looks for differences. Deterministic data fixes can be applied automatically; an LLM proposes code patches in a network-isolated sandbox, with a held-out gate and review before merge. The simulator also works as an MCP server and an RL environment.
Results
In the September 30 comparison, the LLM agent passed 80/80 tasks on Parity and 79/80 on Stripe. Its outcomes agreed on 79/80 tasks, or 98.8%, with a 95% confidence interval of 93.3–99.8%. A second Parity run agreed on all 80. This measures task outcomes, not whether every API response is identical.
Stripe's stripe-mock passed 0/600 whole multi-step scenarios in the benchmark. Parity's first reviewed code patch raised held-out response-value agreement from 18.1% to 39.1% on the same sandbox recordings. The nightly loop had also landed 180 automatic data patches by that point.
The sandbox's measured response-value noise floor was 0.7% on an independent pair of recordings. Whole-sequence agreement has its own noise measurement. Those controls matter because otherwise a changing timestamp or event order can look like a bug in the simulator.
What I'd do differently
What I like most is that the environment has a way to check and improve itself. I'd keep expanding the multi-step scenarios and measuring task outcomes alongside response fidelity. The next thing I want to test is whether practicing in an environment like this makes an agent better when it moves to the real API.