Agent builds / AI agents

Agent build · v1

sqlswarm: a cheat-proof head-to-head for coding agents

What it does: A setup for comparing coding agents fairly: same goal, same model, same clock, and a read-only grader (SQLite's sqllogictest) scored from outside so no agent can raise its score by editing tests.

Model
Claude Opus 5.5 (Claude Code, both arms)
Runs in
Claude Code headless vs Claude Code inside throng
Cost per run
not measured
License
MIT
Used by
0
Files
7 (16 KB)
curl -L https://tensorust.com/builds/sqlswarm-agent-head-to-head/download -o sqlswarm-agent-head-to-head.zip

Files

PROOF.mdraw
# How this experiment works, and how to check it

**Claim under test:** a swarm of AI agents run by throng can write a SQL
database engine in Rust from an empty stub, graded by SQLite's own
`sqllogictest` suite.

## What was written by whom

| Path | Author | Can the swarm change it? |
|---|---|---|
| `slt/` (grader), `tests/`, `examples/`, `corpus/`, `oracle/`, `proof/`, `scaffold/`, `Cargo.toml`, this file | Scaffolding, written before the swarm started (commit tagged `scaffold`) | **No.** throng rejects any agent change outside `src/`. |
| `src/` | Started as a 20-line stub that fails every statement. Everything after the `scaffold` commit is swarm output. | Yes, and only here. |

The engine may use the Rust standard library only: `Cargo.toml` has no
dependencies besides the grader, and the swarm cannot add any.

## The grader is honest

`slt/` reimplements SQLite's sqllogictest runner (record parsing,
`skipif`/`onlyif`, value formatting, `rowsort`/`valuesort`, MD5 result
hashing). To show it grades the way SQLite's own runner does, `oracle/` runs
the same grader against real SQLite (3.46.0 via rusqlite):

- **11,183 of 11,185 records pass (99.98%)**: `proof/oracle.json`.
- The two misses are `evidence/slt_lang_aggfunc.test` lines 480 and 484:
  SQLite 3.46 raises "integer overflow" for `sum()` where the older SQLite that
  recorded the corpus returned a value. That is a SQLite version change, not a
  grader error.

## The test corpus

`corpus/` is copied unmodified from the sqllogictest mirror
<https://github.com/gregrahn/sqllogictest>, commit
`c67f97bf3ca7e590d12e073408bcacaf2ff0f3a0`. `proof/corpus.sha256` lists a
hash of every file so anyone can check nothing was edited. Tier 1 (this
corpus) is `select1-5.test` and `evidence/*.test`: 17 files, 11,185 records.

## Scoring

- **`cargo test`**: 1,128 tests, each grading 10 consecutive records of one
  file (earlier statements in the file are replayed first). This is what throng
  sees; a change is kept only if more tests pass and no passing test breaks.
- **`cargo run --release --example score`**: the canonical sqllogictest way,
  every file run top to bottom on one database, counting passing records.

Starting point (stub): **30 / 11,185 records (0.27%)**, 1 / 1,128 tests , 
`proof/baseline.json`. The 30 are `statement error` records, which a stub that
rejects everything passes trivially.

## The record of what happened

- **Git history.** Every round the swarm completes is one commit on `main`,
  with trailers naming the throng run, round, model, agents, tokens and the
  test score before and after.
- **`proof/ledger.jsonl`**: one line per round with the same numbers.
- **`proof/runs/`**: throng's full event log of each run (every tool call,
  check and result each agent made).

## The head-to-head

Same task, same model (Claude Opus 5.5 through Claude Code on a Claude
subscription, no API), same neutral Claude Code configuration (hooks, skills and
MCP servers off; no internet), same goal text (`scaffold/goal.txt`), same
30-minute wall clock, both starting from the `scaffold` commit. Run one after
the other so neither shares rate limits with the other.

- **Arm A, plain Claude Code** (repo `sqlswarm-solo`): one Claude Code session
  with its normal tools, including `cargo test` and its own subagents,
  resumed with "keep going" if it stops early (`scaffold/arm_solo.py`). Graded
  from outside every 5 minutes and at the end; the agent never sees the grades.
- **Arm B, Claude Code agents inside throng** (this repo): `throng run --pattern
  target --provider claude-code`. throng groups failing tests by cause, runs
  parallel Claude Code agents in private copies, gives them `throng grade`
  (the shared, cached verification engine), keeps only work that raises the
  score without breaking a passing test, and merges it round by round.

Both arms are scored with the same grader on the untouched scaffold plus the
arm's `src/`, so changes to tests or the grader cannot raise a score. Reported:
tests passing over time, tests that broke along the way, and Claude usage.
Full session transcripts of both arms are in `proof/runs/`.

## Reproduce it

```
git clone <this repo> && cd sqlswarm
cargo test                                  # graded tests, pass count
cargo run --release --example score         # records passed, per file
cd oracle && cargo run --release            # real SQLite on the same grader
sha256sum -c proof/corpus.sha256            # corpus unmodified
```

README

sqlswarm is an experiment design for one question: does a swarm of Claude Code agents inside throng beat one plain Claude Code session on the same hard task? The task is writing a SQL database engine in Rust, standard library only, from a 20-line stub. The grader is SQLite's own sqllogictest suite: 17 files, 11,185 records.

What makes it fair, and worth copying for your own agent comparisons:

The grader, the tests and the corpus are written before any agent starts, tagged as a scaffold commit, and are read-only. Scores are always computed on the untouched scaffold plus the agent's src/, so editing tests or the grader cannot raise a score.
The grader is checked against the real thing: run on real SQLite 3.46.0 it passes 11,183 of 11,185 records, and the two misses are a known SQLite version change (proof/oracle.json).
The corpus is copied unmodified from a public mirror at a pinned commit, with a hash of every file (proof/corpus.sha256).
Both arms get the same goal text (scaffold/goal.txt), the same model (Claude Opus 5.5 through Claude Code), the same neutral Claude Code config (hooks, skills and MCP servers off, no internet) and the same 30-minute clock, run one after the other so they never share rate limits.
The solo arm is graded from outside every 5 minutes and never sees its grades.

Files: README.md, PROOF.md (the full design and how to check every claim), scaffold/goal.txt, scaffold/arm_solo.py (the solo-arm runner), scaffold/NOTES.md (every change outside src/ with its reason), and the proof JSON files.

How to run the solo arm on your own task:

python scaffold/arm_solo.py --repo <path-to-your-repo> --minutes 30 --model opus --goal-file scaffold/goal.txt

It starts headless Claude Code (claude -p with stream-json output), resumes it with "keep going" if it stops early, and commits a checkpoint with the score every 5 minutes. It uses throng grade for scoring; swap in your own grader command if you do not use throng.

What to change: the goal text, the grader, the clock and the model. Keep the rules: graders and tests written first and read-only, scores computed from outside, identical config for every arm.

Status: the stub scores 30 of 11,185 records (proof/baseline.json). The head-to-head results are not in yet.

0 comments

Log in to comment