Agent harnesses for cloud infrastructure
A research project at Michigan on the part around the model: its tools, feedback, and the loop that lets it try again. I started by studying xAI's Grok Build, then built a small Terraform harness of my own. On 50 held-out tasks, Qwen 27B reached the earlier Max baseline on policy checks and judged quality.
- Role
- Graduate researcher, with Professor Ang Chen's group
- Timeline
- September 2026 — ongoing
- Stack
- Python, Terraform, mocked AWS, OPA, Qwen, Grok Build
- Status
- Testing on larger, multi-file Terraform projects
The problem
A model can write Terraform that looks plausible and still fails validation, or deploys something that doesn't meet the requirements. I wanted to separate what the model knows from what a harness helps it do. Does a small model need more parameters, or does it need a better way to check its work?
What I built
I mapped Grok Build's agent loop across about 1.2 million lines of Rust and set up an independent evaluation: terraform validate, offline plans with mocked AWS, and OPA policy checks. Then I built a 180-line ReAct harness, so I could compare a small tool loop with one-shot generation and the much larger harness.
The next version adds offline Terraform documentation the agent can search, a summary of the plan after each edit, and a final check that maps each requirement to a resource. I kept development tasks separate from the 50 held-out tasks, froze the harness before testing, and ran two repeats for the 27B model.
Results
On the first 50-task evaluation, the minimal loop raised Qwen 27B's valid configurations from 13/50 to 49/50. For Qwen 9B, it reached 45/50, compared with 39/50 for Grok Build, using about a quarter to a third as many tokens.
On the held-out set, the Terraform-specific harness raised Qwen 27B's policy pass rate from 18% to 31% and its Claude-judged score from 0.45 to 0.58, averaged over two repeats. Its judged score improved on 35 of 50 tasks. For Qwen 9B, successful plans went from 82% to 98%, and policy passes from 0/50 to 7/50.
Qwen 27B reached the earlier Max baseline on the 36 tasks Max completed. That comparison is encouraging, but Max's runs predate changes to the policies and evaluation setup. A fresh Max run with the new harness is still needed.
What I'd do differently
The useful surprise was that searchable provider documentation helped, while adding another 27B model as a reviewer didn't measurably improve the result. I'd rather keep testing which pieces actually help than make the harness bigger by default. The next test is on larger, multi-file projects, where getting one file to validate won't be enough.