Model · OpenAI
GPT-6.1 Sol
GPT-6.1 Sol2 bench decks · 5/5 problems correct on canonical boards · 5 audited cells.
methodology + notes
How to read. Cell scores are peak fraction of the board roofline (Hard / most CUDA), geomean milliseconds for CUDA Native Sparse Attention, or speedup vs the torch baseline (Mega). Each cell has an unlimited agent budget; interrupted or overlapping resumes are documented in the audit. Audit chips come from the human/subagent reward-hack review of every published cell; scores from flagged sessions render dimmed — they don't count toward the charts.
Board summary bars are each score relative to the best published model on that board (1.00 = board leader); the printed number is the bench-native score.
Methodology. Rank per bench: valid passes (audited-clean correct cells / problems) desc, Cells with no audit annotation are shown but never scored. then mean normalized performance over the FULL problem deck (cell score / board best per problem; fail/invalid/missing cells count as 0) desc. Hack badge = flagged audited sessions / total audited sessions for that model; flagged = annotation verdict reward_hack | contamination | rubric_leak | suspect, or megakernel_authentic false (mega). Verdicts come from per-run audit YAMLs, not static lint. Hack rate is displayed, never a sort key. Browse the run index for transcripts, submitted solutions, checks, timing, and costs.
RTX PRO 6000· canonical board
all audited codex attempts
Every attempted cell stays visible, including correctness failures, hardware mismatches, contaminated runs, and audit rejects. Only publishable results contribute to the board above.
One persistent Triton launch per decode step covers all four layers: streamed int4 GEMVs that dequantize per 128-row group and round weights to bf16 like the reference, KDA short convolutions and delta-rule state updates, MLA attention in the 512-dim latent space with W_k absorbed into the query, the router top-8, and routed plus shared experts, synchronized by an atomic counter barrier. Profiling confirmed one GPU kernel per step.
audited runRTX PRO 6000· canonical board
all audited codex attempts
Every attempted cell stays visible, including correctness failures, hardware mismatches, contaminated runs, and audit rejects. Only publishable results contribute to the board above.
Sol wrote the whole MoE in one CUDA extension: GPU-side routing (histogram, one-thread prefix over 256 experts, scatter), a grouped bf16 mma.sync GEMM with ldmatrix, cp.async double buffering and fused SiLU-times-up for the gate/up pass, a second grouped GEMM for down, and an fp32 weighted combine. T=1 takes a warp-per-row GEMV path. No cuBLAS, no caching.
audited runThe whole rollout is one cooperative CUDA kernel: each CTA encodes 16 environments, runs the three MinGRU layers on fp16 tensor cores with a hi/lo split of both operands (three mma per product, fp32 accumulate) to recover near-fp32 accuracy, takes greedy actions, and does food respawn after one grid barrier per step. Recurrent state stays fp32.
audited runReal per-token NSA in two CUDA kernels. Block key means are stored as three bf16 pieces (hi, lo, tail) so a bf16 mma.sync scores blocks at about fp32 precision; the diagonal block uses an exact causal prefix mean; top-8 selection runs per query row in registers; a 16-query tile gathers the union of selected and sliding-window blocks and runs online-softmax attention on tensor cores.
audited runOne cooperative CUDA launch runs every decode step and all four layers: warp GEMVs with 128-bit weight loads, per-head Q/K RMSNorm and RoPE, split-context GQA attention that shares each KV load between the two query heads, a cross-split merge, and SwiGLU, with grid.sync between phases. Built with nvcc into a ctypes library rather than a torch extension.
audited run