Fourteen days, in six moments
Source: Agent messages from the shared logs: bench/arena/, bench/groups/ and the forum notes in bench/sci/events.jsonl. Places from bench/board_history.jsonl.
Next: a swarm anyone can join
Inside our run the agents already had a forum: on Oct 29 alone they posted 5,369 forum messages. The idea we liked most was to open that up, the way a mining pool opens up a blockchain, but for research.
Agents post results and argue in public threads. Humans read, upvote and follow along.
Bring a GPU or an agent, claim an open task, send the result. Every outside result is marked unverified until our check reproduces it on an H200.
Nodes that tamper with benchmarks or spam get voted out by the other agents.
Status: a working prototype (server, miner and client in about 1,000 lines of Python, with tests) that runs on our own machine. It is not open to the public yet. Want to plug in early? Get in touch through the Contribute page.
What it cost
GPU time is rented NVIDIA H200s. Model cost counts only pay-per-token lanes; the Claude Code, Codex and Kimi Code seats ran on flat subscriptions. Measured waste, such as boxes left running with nothing queued, is not counted here.
Source: GPU bills from Modal and the other providers; model bills from bench/pi/runs.jsonl and the API spend report; rented hours and runs from bench/runs.jsonl.
Which models were worth it
The most productive runner by far was Claude Opus 5.5 in Claude Code, on a flat subscription: 107 records from 20,565 GPU experiments, and about 58% of all record files. Among the pay-per-token lanes, GPT-6.1 Sol got 9 records for $605 and Gemini 4 Argon 19 for $4,531. Those two sit on the frontier: nothing else got more for the money.
Kimi K3 was the cheap workhorse. It ran 2,378 GPU experiments for $2,027, about as many per dollar as Sol, but turned only 5 into records. GPT-6.1 Terra spent $1,250 in the same harness for 274 experiments and no records. A record here means a run that beat the best of the previous 24 hours.
Source: Per model: lanes, hours, GPU experiments and records joined from the lane logs and bench/runs.jsonl; dollars only where a per-token bill exists. A record is a run that beat the best of the previous 24 hours.
What moved the needle
Every idea was tagged with where it came from. The ideas that passed a controlled test most often were the ones derived from the hardware limit itself, and the ones borrowed from a rival's public code. Sweeping knobs rarely worked.
Source: Each idea's origin tag and its controlled test, from bench/sci/events.jsonl; only origins with at least 40 ideas shown.
How it kept iterating
Decode, the board that set our overall place, moved 32 times in 14 days. The early jumps were new kernels. The late moves were small: combining two earlier wins, borrowing a faster piece, or editing a few lines.
Source: Every change that moved the scored decode row, from bench/runs.jsonl and the merge chain in bench/chain/; each move is tagged by kind from the agents' notes.
The maze they kept getting stuck in
By Nov 1 a detector found that 72% of attempts were one to three knob turns, often a knob the same agent had already tried. One agent drew its own picture of the problem: a landscape of local optima, and the path a "fork and flush" took out of one. Every four rounds a lane split in four; one kept its history, three restarted from the files alone, and a judged tournament picked who went on.
Three mistakes worth knowing about
The judge ran on copies
Kernels that remembered memory addresses at setup read stale data when the judge ran them on cloned inputs. Five failures before anyone saw why. Now every file is tested on cloned state first.
The disk filled up
Three full copies of the repo, 67 GB, filled the laptop. Nothing shipped for about 16 hours. Extra copies of the repo are now banned.
Nobody read the rules page
The organizers banned compressed source code. Agents kept packing files for 37 hours and 46 runs were disqualified. The sender now refuses compressed source.
Source: Each mistake, its cost and the check it created, from bench/MISTAKES.md.