Agent build · v1
sqlswarm: a cheat-proof head-to-head for coding agents
What it does: A setup for comparing coding agents fairly: same goal, same model, same clock, and a read-only grader (SQLite's sqllogictest) scored from outside so no agent can raise its score by editing tests.
- Model
- Claude Opus 5.5 (Claude Code, both arms)
- Runs in
- Claude Code headless vs Claude Code inside throng
- Cost per run
- not measured
- License
- MIT
- Used by
- 0
- Files
- 7 (16 KB)
curl -L https://tensorust.com/builds/sqlswarm-agent-head-to-head/download -o sqlswarm-agent-head-to-head.zip
Files
- PROOF.md4 KB
- proof/baseline.json1 KB
- proof/oracle.json1 KB
- README.md1 KB
- scaffold/arm_solo.py7 KB
- scaffold/goal.txt651 B
- scaffold/NOTES.md1 KB
Scaffolding log (human/assistant written, before and between swarm runs). Every change to a non-src/ path is listed here with its reason. - scaffold: grader (slt/), generated tests, score example, SQLite oracle, tier-1 corpus (select1-5, evidence/), proof files. Oracle 11,183/11,185. - experiment: head-to-head design (PROOF.md), shared goal (scaffold/goal.txt), solo-arm runner (scaffold/arm_solo.py). No change to the grader, tests or corpus. - fix: solo runner launches claude.exe directly (the PATH entry is a .cmd wrapper); found in a 90 s dry run on a throwaway clone. - aborted launch 01:57: a test launch of Arm B from a terminal ran ~20 s and was killed by a timeout; its round-1 commit (1 -> 181) is kept on branch aborted-launch-0157. It exposed two throng issues, fixed before the real Arm B: Claude Code workers could outlive a killed throng (now tied to a job object that dies with throng; verified), and failure groups keyed on query text (457 groups of ~2) instead of the failure reason (fixed, with a test).
README
sqlswarm is an experiment design for one question: does a swarm of Claude Code agents inside throng beat one plain Claude Code session on the same hard task? The task is writing a SQL database engine in Rust, standard library only, from a 20-line stub. The grader is SQLite's own sqllogictest suite: 17 files, 11,185 records.
What makes it fair, and worth copying for your own agent comparisons:
The grader, the tests and the corpus are written before any agent starts, tagged as a scaffold commit, and are read-only. Scores are always computed on the untouched scaffold plus the agent's src/, so editing tests or the grader cannot raise a score.
The grader is checked against the real thing: run on real SQLite 3.46.0 it passes 11,183 of 11,185 records, and the two misses are a known SQLite version change (proof/oracle.json).
The corpus is copied unmodified from a public mirror at a pinned commit, with a hash of every file (proof/corpus.sha256).
Both arms get the same goal text (scaffold/goal.txt), the same model (Claude Opus 5.5 through Claude Code), the same neutral Claude Code config (hooks, skills and MCP servers off, no internet) and the same 30-minute clock, run one after the other so they never share rate limits.
The solo arm is graded from outside every 5 minutes and never sees its grades.
Files: README.md, PROOF.md (the full design and how to check every claim), scaffold/goal.txt, scaffold/arm_solo.py (the solo-arm runner), scaffold/NOTES.md (every change outside src/ with its reason), and the proof JSON files.
How to run the solo arm on your own task:
python scaffold/arm_solo.py --repo <path-to-your-repo> --minutes 30 --model opus --goal-file scaffold/goal.txtIt starts headless Claude Code (claude -p with stream-json output), resumes it with "keep going" if it stops early, and commits a checkpoint with the score every 5 minutes. It uses throng grade for scoring; swap in your own grader command if you do not use throng.
What to change: the goal text, the grader, the clock and the model. Keep the rules: graders and tests written first and read-only, scores computed from outside, identical config for every arm.
Status: the stub scores 30 of 11,185 records (proof/baseline.json). The head-to-head results are not in yet.