Contribute a task
Bring a real GPU kernel: its inputs, a reference in plain PyTorch, and a physical floor you can derive. Accepted tasks join FloorBench with author credit. The submission address goes here before this page is published.
What makes a good task
Paged FP8 attention decode at 80K context
- Starting files
- A fused qkv row, a paged FP8 E4M3 KV cache, a page table and sequence lengths. A naive starter that passes every gate.
- The work
- Write
kernel_fn(data, state)for one decode step and one 8-token verify block, over every key in range. It is replayed as a CUDA graph. - Deliverable
- One file,
/app/submission.py, withprepare(weights)andkernel_fn(data, state). - Grading
- Correct within rel L2 2% of an FP32 reference on fresh inputs, then timed per case. The score is the geomean speedup and the % of the physical floor.
Real shapes
Take the shapes from a production serving path, not a textbook op. One case can hold a whole decoder layer.
A floor you can derive
Give each case its limit: bytes at HBM bandwidth or FLOPs at the power-capped tensor rate, plus launch cost. The floor does not move when a library improves.
Gates a cheat cannot pass
The task must reject output caches, baked pointers and timer tricks. Our red-team suite runs on every task before review.
Task files
Put each task in its own folder. The format is the Harbor task format, so the same folder runs in our harness and in yours.
| task.toml | Name, loads, the number of cases, the floor per case in microseconds, time limits, GPU type, and network_mode = "no-network". |
| instruction.md | The problem the agent sees: the math, the dtypes, the interface table, the correctness bound and the score. |
| environment/ | The Dockerfile, the naive starter.py and the reference in plain PyTorch. The reference is readable, never importable. |
| tests/test.sh | The grading entry point. It runs the gates in a separate process, times the graph, and writes the reward. |
| solution/ | Your best known kernel. Send it to reviewers only; it never ships with the task. |
- Keep secrets out. No keys, no internal hostnames, no client code. Use public model shapes.
- Keep the answer outside the task folder. Reviewers ask for the solution on its own.
Review
- Structure check. The validator reads every file and the floor table.
- Red team. The naive starter must pass. The known cheats (output cache on the data pointer, baked TMA descriptors, timer patching, graph skip, reference import and others) must come back INVALID.
- H200 timing. We measure the A/A noise and check that every tier sits at least 5% or ten times the noise above the starter.
- Decision. We accept the task, ask for changes, or say why it was declined. Push a new commit to resubmit.
Contribution terms
Draft, pending legal review. You confirm that you have the right to share the task and that it holds no confidential client or employer material. Accepted tasks may ship with the benchmark, with their starter files and graders, under the benchmark licence.