Editing constraint models with LLMs
Can an LLM edit a formal constraint program when the requirements change, instead of regenerating the whole thing? There was no benchmark for this, so I defined the task, built one, and showed GPT-4o reaching 95.8% compile accuracy and 100% semantic accuracy on it. I presented it as a first-author paper at the LLMSolve workshop at CP 2025 in Glasgow.
- Role
- Research intern, then first author. With Phong Le and Nguyen Dang, University of St Andrews
- Timeline
- January — August 2025
- Stack
- Python, GPT-4o and GPT-4o-mini, constraint solvers
The problem
Constraint models encode rules, and rules change. Regenerating a whole model from scratch every time a requirement moves is unreliable, and it throws away the parts that already worked. Nobody had defined what it means for an LLM to edit a model well, let alone measured it.
What I built
I formalized the task and a taxonomy of eight edit categories, built a public benchmark, and proposed and implemented compile and semantic-equivalence metrics for evaluating it. Then I built a standalone open-source pipeline for running LLMs on the benchmark and did a systematic evaluation of GPT-4o and GPT-4o-mini in zero-shot and few-shot settings.
Results
Editing turned out to be more reliable than regenerating from scratch. GPT-4o compiled 23 of the 24 edits and reached full semantic equivalence on all 24.
What I'd do differently
The internship officially ended in June, but I kept working on it unpaid through the summer because I really thought the result was worth publishing. Looking back, 24 edits is a small benchmark. What I'd want now is a few hundred, with harder categories that actually break the frontier model. The bigger lesson stuck with me, though: non-deterministic models working with deterministic rules is exactly where reliability gets decided, and that's the layer I keep coming back to.