Research from zero
We gave a swarm of AI agents a field none of them was told anything about: making GPU kernels faster on an NVIDIA H200. No expert, no playbook. The system had to find the physical limits, form its own ideas, test them, and write down what it learned. This page shows what it built from first principles, which research methods worked, and the benchmark we made from it.
The more experiments it ran, the closer it got to the hardware limit: every tenfold increase in experiments closed about of the gap, with r = to on each of the five problems.
What it knew at the start, and what it built
At the start the system had a task description, a reference kernel and a rented H200. Everything else below it wrote itself, in plain files, while it worked. Each folder kept a notebook of what moved the number, what did not, and what it derived from first principles.
The method, one loop at a time
Every idea went through the same loop. Step through it.
Which research methods found wins
Each idea was tagged with where it came from. Here is how often each source passed a full check with a same-box control. Deriving from the physical limit and reading working code beat sweeping knobs, and beat reading papers.
One problem at a time
Pick a problem. Left: every scored case as a share of its physical floor at our best run. Right: the task as the agents saw it, the answer, and the reasoning, built only from measured data.
bench/tools/gap.py.The benchmark: what a model must build
Each task is one scored case of a real serving kernel. The agent gets a naive starter that passes every gate, a reference in plain PyTorch, and the floor of every case. It returns one file. The grader times it on the H200 and reports the speedup and the share of the physical floor.
Which models find kernel wins
Two views. The fleet is 14 days of agents working the real competition tasks, with every lane run, GPU run and dollar logged. Cold start is the controlled protocol: one fresh agent, the naive starter, six graded steps, and a confirmation run in a fresh process with new secret seeds.
Tiers set by real competitors
The old first tier was 1.10x the stock PyTorch library. Stock attention and MoE are so slow that an unchanged starter passed it. The contest ladder measures share of the physical floor against the field at the close: beat SGLang, match our 14-day swarm, match the #1 team, then go past it.
The hard set
The loads where a 14-day, 58,618-run swarm stayed farthest from the floor. Each is a Harbor task.
How grading works
The submission runs in its own process. The reference, the secret seeds and the baseline live in processes that never load it. A result counts only if every gate passes, in this order.
Known cheats, all caught
Each cheat below is a real file in the red-team suite. It must come back INVALID, and the honest starter must pass, before any task ships.
Explore an episode
Every graded step keeps its speedup with a 95% interval and the time of each case against its floor. Pick an episode, then a step. The agent's own code and notes ship only in the full dataset, to the provider whose model wrote them.
The dataset
The trajectories are measured, not synthetic. Each attempt carries its label, its speedup, its per-case ratios, the code hash and the job id of the GPU run that measured it.
Get the sample
The sample runs on any CPU in about two minutes: every gate, the red-team suite, and the scoring code for 6 of the 9 task families. Timing needs an H200. Private references, held-out seeds and agent-written code stay out.
What is inside
Checked before release
For labs
Three ways to use FloorBench. Pricing on request.
Evaluation requests: a contact address goes here before this page is published.
Citation
@misc{floorbench2026,
title = {{FloorBench-H200}: Kernel Tasks Graded Against the Physical Floor},
author = {Lukoszko, Robert},
year = {2026},
note = {Version }
}