Agent build · v1
sqlswarm: a cheat-proof head-to-head for coding agents
What it does: A setup for comparing coding agents fairly: same goal, same model, same clock, and a read-only grader (SQLite's sqllogictest) scored from outside so no agent can raise its score by editing tests.
- Model
- Claude Opus 5.5 (Claude Code, both arms)
- Runs in
- Claude Code headless vs Claude Code inside throng
- Cost per run
- not measured
- License
- MIT
- Used by
- 0
- Files
- 7 (16 KB)
curl -L https://tensorust.com/builds/sqlswarm-agent-head-to-head/download -o sqlswarm-agent-head-to-head.zip
Files
- PROOF.md4 KB
- proof/baseline.json1 KB
- proof/oracle.json1 KB
- README.md1 KB
- scaffold/arm_solo.py7 KB
- scaffold/goal.txt651 B
- scaffold/NOTES.md1 KB
{"passed":30,"total":11185,"files":[{"file":"evidence/in1.test","passed":0,"total":204},{"file":"evidence/in2.test","passed":4,"total":53},{"file":"evidence/slt_lang_aggfunc.test","passed":2,"total":80},{"file":"evidence/slt_lang_createtrigger.test","passed":1,"total":26},{"file":"evidence/slt_lang_createview.test","passed":10,"total":23},{"file":"evidence/slt_lang_dropindex.test","passed":2,"total":8},{"file":"evidence/slt_lang_droptable.test","passed":3,"total":12},{"file":"evidence/slt_lang_droptrigger.test","passed":3,"total":12},{"file":"evidence/slt_lang_dropview.test","passed":2,"total":13},{"file":"evidence/slt_lang_reindex.test","passed":1,"total":7},{"file":"evidence/slt_lang_replace.test","passed":0,"total":14},{"file":"evidence/slt_lang_update.test","passed":2,"total":27},{"file":"select1.test","passed":0,"total":1031},{"file":"select2.test","passed":0,"total":1031},{"file":"select3.test","passed":0,"total":3351},{"file":"select4.test","passed":0,"total":3857},{"file":"select5.test","passed":0,"total":1436}]}
README
sqlswarm is an experiment design for one question: does a swarm of Claude Code agents inside throng beat one plain Claude Code session on the same hard task? The task is writing a SQL database engine in Rust, standard library only, from a 20-line stub. The grader is SQLite's own sqllogictest suite: 17 files, 11,185 records.
What makes it fair, and worth copying for your own agent comparisons:
The grader, the tests and the corpus are written before any agent starts, tagged as a scaffold commit, and are read-only. Scores are always computed on the untouched scaffold plus the agent's src/, so editing tests or the grader cannot raise a score.
The grader is checked against the real thing: run on real SQLite 3.46.0 it passes 11,183 of 11,185 records, and the two misses are a known SQLite version change (proof/oracle.json).
The corpus is copied unmodified from a public mirror at a pinned commit, with a hash of every file (proof/corpus.sha256).
Both arms get the same goal text (scaffold/goal.txt), the same model (Claude Opus 5.5 through Claude Code), the same neutral Claude Code config (hooks, skills and MCP servers off, no internet) and the same 30-minute clock, run one after the other so they never share rate limits.
The solo arm is graded from outside every 5 minutes and never sees its grades.
Files: README.md, PROOF.md (the full design and how to check every claim), scaffold/goal.txt, scaffold/arm_solo.py (the solo-arm runner), scaffold/NOTES.md (every change outside src/ with its reason), and the proof JSON files.
How to run the solo arm on your own task:
python scaffold/arm_solo.py --repo <path-to-your-repo> --minutes 30 --model opus --goal-file scaffold/goal.txtIt starts headless Claude Code (claude -p with stream-json output), resumes it with "keep going" if it stops early, and commits a checkpoint with the score every 5 minutes. It uses throng grade for scoring; swap in your own grader command if you do not use throng.
What to change: the goal text, the grader, the clock and the model. Keep the rules: graders and tests written first and read-only, scores computed from outside, identical config for every arm.
Status: the stub scores 30 of 11,185 records (proof/baseline.json). The head-to-head results are not in yet.