Agent build · v1
sqlswarm: a cheat-proof head-to-head for coding agents
What it does: A setup for comparing coding agents fairly: same goal, same model, same clock, and a read-only grader (SQLite's sqllogictest) scored from outside so no agent can raise its score by editing tests.
- Model
- Claude Opus 5.5 (Claude Code, both arms)
- Runs in
- Claude Code headless vs Claude Code inside throng
- Cost per run
- not measured
- License
- MIT
- Used by
- 0
- Files
- 7 (16 KB)
curl -L https://tensorust.com/builds/sqlswarm-agent-head-to-head/download -o sqlswarm-agent-head-to-head.zip
Files
- PROOF.md4 KB
- proof/baseline.json1 KB
- proof/oracle.json1 KB
- README.md1 KB
- scaffold/arm_solo.py7 KB
- scaffold/goal.txt651 B
- scaffold/NOTES.md1 KB
# sqlswarm: a fair head-to-head for coding agents An experiment setup for answering one question honestly: does a swarm of Claude Code agents inside throng beat one plain Claude Code session on the same hard task? The task is writing a SQL engine in Rust from a 20-line stub, graded by SQLite's own sqllogictest suite. The design is the part worth copying. The grader, tests and corpus are written before any agent starts and are read-only to the agents; scores are always computed from the untouched scaffold plus the agent's src/, so an agent that edits tests cannot raise its score. ## Files ``` PROOF.md the claim, who wrote what, how scoring works, how to check every number scaffold/goal.txt the exact goal both arms get scaffold/arm_solo.py Arm A runner: headless Claude Code for a fixed wall clock, graded from outside every 5 minutes scaffold/NOTES.md every change to a non-src/ path, with its reason proof/baseline.json the stub's score: 30 of 11,185 records proof/oracle.json real SQLite on the same grader: 11,183 of 11,185 ``` The grader (slt/), generated tests and the Rust score example are not included here; they are Rust source.
README
sqlswarm is an experiment design for one question: does a swarm of Claude Code agents inside throng beat one plain Claude Code session on the same hard task? The task is writing a SQL database engine in Rust, standard library only, from a 20-line stub. The grader is SQLite's own sqllogictest suite: 17 files, 11,185 records.
What makes it fair, and worth copying for your own agent comparisons:
The grader, the tests and the corpus are written before any agent starts, tagged as a scaffold commit, and are read-only. Scores are always computed on the untouched scaffold plus the agent's src/, so editing tests or the grader cannot raise a score.
The grader is checked against the real thing: run on real SQLite 3.46.0 it passes 11,183 of 11,185 records, and the two misses are a known SQLite version change (proof/oracle.json).
The corpus is copied unmodified from a public mirror at a pinned commit, with a hash of every file (proof/corpus.sha256).
Both arms get the same goal text (scaffold/goal.txt), the same model (Claude Opus 5.5 through Claude Code), the same neutral Claude Code config (hooks, skills and MCP servers off, no internet) and the same 30-minute clock, run one after the other so they never share rate limits.
The solo arm is graded from outside every 5 minutes and never sees its grades.
Files: README.md, PROOF.md (the full design and how to check every claim), scaffold/goal.txt, scaffold/arm_solo.py (the solo-arm runner), scaffold/NOTES.md (every change outside src/ with its reason), and the proof JSON files.
How to run the solo arm on your own task:
python scaffold/arm_solo.py --repo <path-to-your-repo> --minutes 30 --model opus --goal-file scaffold/goal.txtIt starts headless Claude Code (claude -p with stream-json output), resumes it with "keep going" if it stops early, and commits a checkpoint with the score every 5 minutes. It uses throng grade for scoring; swap in your own grader command if you do not use throng.
What to change: the goal text, the grader, the clock and the model. Keep the rules: graders and tests written first and read-only, scores computed from outside, identical config for every arm.
Status: the stub scores 30 of 11,185 records (proof/baseline.json). The head-to-head results are not in yet.