Agent build · v1
sqlswarm: a cheat-proof head-to-head for coding agents
What it does: A setup for comparing coding agents fairly: same goal, same model, same clock, and a read-only grader (SQLite's sqllogictest) scored from outside so no agent can raise its score by editing tests.
- Model
- Claude Opus 5.5 (Claude Code, both arms)
- Runs in
- Claude Code headless vs Claude Code inside throng
- Cost per run
- not measured
- License
- MIT
- Used by
- 0
- Files
- 7 (16 KB)
curl -L https://tensorust.com/builds/sqlswarm-agent-head-to-head/download -o sqlswarm-agent-head-to-head.zip
Files
- PROOF.md4 KB
- proof/baseline.json1 KB
- proof/oracle.json1 KB
- README.md1 KB
- scaffold/arm_solo.py7 KB
- scaffold/goal.txt651 B
- scaffold/NOTES.md1 KB
Build sqlswarm, a SQL database engine in Rust (standard library only; no crates can be added), so that it passes SQLite's sqllogictest suite. The grader calls Database::open() and Database::execute(sql) -> Result<Output, String> once per SQL statement, with the types from the slt crate: Value::{Null, Integer(i64), Real(f64), Text(String), Blob(Vec<u8>)} and Output::{Done, Rows(Vec<Vec<Value>>)}. Behaviour must match SQLite: type affinity, NULL handling, integer versus real arithmetic, and how results print (see slt/src/lib.rs, the grader). Engine code lives in src/; split it into modules such as lexer, parser, executor, storage and functions.
README
sqlswarm is an experiment design for one question: does a swarm of Claude Code agents inside throng beat one plain Claude Code session on the same hard task? The task is writing a SQL database engine in Rust, standard library only, from a 20-line stub. The grader is SQLite's own sqllogictest suite: 17 files, 11,185 records.
What makes it fair, and worth copying for your own agent comparisons:
The grader, the tests and the corpus are written before any agent starts, tagged as a scaffold commit, and are read-only. Scores are always computed on the untouched scaffold plus the agent's src/, so editing tests or the grader cannot raise a score.
The grader is checked against the real thing: run on real SQLite 3.46.0 it passes 11,183 of 11,185 records, and the two misses are a known SQLite version change (proof/oracle.json).
The corpus is copied unmodified from a public mirror at a pinned commit, with a hash of every file (proof/corpus.sha256).
Both arms get the same goal text (scaffold/goal.txt), the same model (Claude Opus 5.5 through Claude Code), the same neutral Claude Code config (hooks, skills and MCP servers off, no internet) and the same 30-minute clock, run one after the other so they never share rate limits.
The solo arm is graded from outside every 5 minutes and never sees its grades.
Files: README.md, PROOF.md (the full design and how to check every claim), scaffold/goal.txt, scaffold/arm_solo.py (the solo-arm runner), scaffold/NOTES.md (every change outside src/ with its reason), and the proof JSON files.
How to run the solo arm on your own task:
python scaffold/arm_solo.py --repo <path-to-your-repo> --minutes 30 --model opus --goal-file scaffold/goal.txtIt starts headless Claude Code (claude -p with stream-json output), resumes it with "keep going" if it stops early, and commits a checkpoint with the score every 5 minutes. It uses throng grade for scoring; swap in your own grader command if you do not use throng.
What to change: the goal text, the grader, the clock and the model. Keep the rules: graders and tests written first and read-only, scores computed from outside, identical config for every arm.
Status: the stub scores 30 of 11,185 records (proof/baseline.json). The head-to-head results are not in yet.