Agent build · v1
sqlswarm: a cheat-proof head-to-head for coding agents
What it does: A setup for comparing coding agents fairly: same goal, same model, same clock, and a read-only grader (SQLite's sqllogictest) scored from outside so no agent can raise its score by editing tests.
- Model
- Claude Opus 5.5 (Claude Code, both arms)
- Runs in
- Claude Code headless vs Claude Code inside throng
- Cost per run
- not measured
- License
- MIT
- Used by
- 0
- Files
- 7 (16 KB)
curl -L https://tensorust.com/builds/sqlswarm-agent-head-to-head/download -o sqlswarm-agent-head-to-head.zip
Files
- PROOF.md4 KB
- proof/baseline.json1 KB
- proof/oracle.json1 KB
- README.md1 KB
- scaffold/arm_solo.py7 KB
- scaffold/goal.txt651 B
- scaffold/NOTES.md1 KB
# How this experiment works, and how to check it **Claim under test:** a swarm of AI agents run by throng can write a SQL database engine in Rust from an empty stub, graded by SQLite's own `sqllogictest` suite. ## What was written by whom | Path | Author | Can the swarm change it? | |---|---|---| | `slt/` (grader), `tests/`, `examples/`, `corpus/`, `oracle/`, `proof/`, `scaffold/`, `Cargo.toml`, this file | Scaffolding, written before the swarm started (commit tagged `scaffold`) | **No.** throng rejects any agent change outside `src/`. | | `src/` | Started as a 20-line stub that fails every statement. Everything after the `scaffold` commit is swarm output. | Yes, and only here. | The engine may use the Rust standard library only: `Cargo.toml` has no dependencies besides the grader, and the swarm cannot add any. ## The grader is honest `slt/` reimplements SQLite's sqllogictest runner (record parsing, `skipif`/`onlyif`, value formatting, `rowsort`/`valuesort`, MD5 result hashing). To show it grades the way SQLite's own runner does, `oracle/` runs the same grader against real SQLite (3.46.0 via rusqlite): - **11,183 of 11,185 records pass (99.98%)**: `proof/oracle.json`. - The two misses are `evidence/slt_lang_aggfunc.test` lines 480 and 484: SQLite 3.46 raises "integer overflow" for `sum()` where the older SQLite that recorded the corpus returned a value. That is a SQLite version change, not a grader error. ## The test corpus `corpus/` is copied unmodified from the sqllogictest mirror <https://github.com/gregrahn/sqllogictest>, commit `c67f97bf3ca7e590d12e073408bcacaf2ff0f3a0`. `proof/corpus.sha256` lists a hash of every file so anyone can check nothing was edited. Tier 1 (this corpus) is `select1-5.test` and `evidence/*.test`: 17 files, 11,185 records. ## Scoring - **`cargo test`**: 1,128 tests, each grading 10 consecutive records of one file (earlier statements in the file are replayed first). This is what throng sees; a change is kept only if more tests pass and no passing test breaks. - **`cargo run --release --example score`**: the canonical sqllogictest way, every file run top to bottom on one database, counting passing records. Starting point (stub): **30 / 11,185 records (0.27%)**, 1 / 1,128 tests , `proof/baseline.json`. The 30 are `statement error` records, which a stub that rejects everything passes trivially. ## The record of what happened - **Git history.** Every round the swarm completes is one commit on `main`, with trailers naming the throng run, round, model, agents, tokens and the test score before and after. - **`proof/ledger.jsonl`**: one line per round with the same numbers. - **`proof/runs/`**: throng's full event log of each run (every tool call, check and result each agent made). ## The head-to-head Same task, same model (Claude Opus 5.5 through Claude Code on a Claude subscription, no API), same neutral Claude Code configuration (hooks, skills and MCP servers off; no internet), same goal text (`scaffold/goal.txt`), same 30-minute wall clock, both starting from the `scaffold` commit. Run one after the other so neither shares rate limits with the other. - **Arm A, plain Claude Code** (repo `sqlswarm-solo`): one Claude Code session with its normal tools, including `cargo test` and its own subagents, resumed with "keep going" if it stops early (`scaffold/arm_solo.py`). Graded from outside every 5 minutes and at the end; the agent never sees the grades. - **Arm B, Claude Code agents inside throng** (this repo): `throng run --pattern target --provider claude-code`. throng groups failing tests by cause, runs parallel Claude Code agents in private copies, gives them `throng grade` (the shared, cached verification engine), keeps only work that raises the score without breaking a passing test, and merges it round by round. Both arms are scored with the same grader on the untouched scaffold plus the arm's `src/`, so changes to tests or the grader cannot raise a score. Reported: tests passing over time, tests that broke along the way, and Claude usage. Full session transcripts of both arms are in `proof/runs/`. ## Reproduce it ``` git clone <this repo> && cd sqlswarm cargo test # graded tests, pass count cargo run --release --example score # records passed, per file cd oracle && cargo run --release # real SQLite on the same grader sha256sum -c proof/corpus.sha256 # corpus unmodified ```
README
sqlswarm is an experiment design for one question: does a swarm of Claude Code agents inside throng beat one plain Claude Code session on the same hard task? The task is writing a SQL database engine in Rust, standard library only, from a 20-line stub. The grader is SQLite's own sqllogictest suite: 17 files, 11,185 records.
What makes it fair, and worth copying for your own agent comparisons:
The grader, the tests and the corpus are written before any agent starts, tagged as a scaffold commit, and are read-only. Scores are always computed on the untouched scaffold plus the agent's src/, so editing tests or the grader cannot raise a score.
The grader is checked against the real thing: run on real SQLite 3.46.0 it passes 11,183 of 11,185 records, and the two misses are a known SQLite version change (proof/oracle.json).
The corpus is copied unmodified from a public mirror at a pinned commit, with a hash of every file (proof/corpus.sha256).
Both arms get the same goal text (scaffold/goal.txt), the same model (Claude Opus 5.5 through Claude Code), the same neutral Claude Code config (hooks, skills and MCP servers off, no internet) and the same 30-minute clock, run one after the other so they never share rate limits.
The solo arm is graded from outside every 5 minutes and never sees its grades.
Files: README.md, PROOF.md (the full design and how to check every claim), scaffold/goal.txt, scaffold/arm_solo.py (the solo-arm runner), scaffold/NOTES.md (every change outside src/ with its reason), and the proof JSON files.
How to run the solo arm on your own task:
python scaffold/arm_solo.py --repo <path-to-your-repo> --minutes 30 --model opus --goal-file scaffold/goal.txtIt starts headless Claude Code (claude -p with stream-json output), resumes it with "keep going" if it stops early, and commits a checkpoint with the score every 5 minutes. It uses throng grade for scoring; swap in your own grader command if you do not use throng.
What to change: the goal text, the grader, the clock and the model. Keep the rules: graders and tests written first and read-only, scores computed from outside, identical config for every arm.
Status: the stub scores 30 of 11,185 records (proof/baseline.json). The head-to-head results are not in yet.