Model · Anthropic
Claude Fable 5.1
2 bench decks · 5/5 problems correct on canonical boards · 9 audited cells.
methodology + notes
How to read. Cell scores are peak fraction of the board roofline (Hard / CUDA) or best speedup vs the torch baseline (Mega), over one unlimited agent session per cell. Audit chips come from the human/subagent reward-hack review of every published cell; scores from flagged sessions render dimmed — they don't count toward the charts.
Board summary bars are each score relative to the best published model on that board (1.00 = board leader); the printed number is the bench-native score.
Methodology. Rank per bench: valid passes (audited-clean correct cells / problems) desc, Cells with no audit annotation are shown but never scored. then mean normalized performance over the FULL problem deck (cell score / board best per problem; fail/invalid/missing cells count as 0) desc. Hack badge = flagged audited sessions / total audited sessions for that model; flagged = annotation verdict reward_hack | contamination | rubric_leak | suspect, or megakernel_authentic false (mega). Verdicts come from per-run audit YAMLs, not static lint. Hack rate is displayed, never a sort key. Browse the run index for transcripts, submitted solutions, checks, timing, and costs.
RTX PRO 6000· canonical board
all audited or-fable attempts
Every attempted cell stays visible, including correctness failures, hardware mismatches, contaminated runs, and audit rejects. Only publishable results contribute to the board above.
Claude Fable 5.1 through OpenRouter (or-fable harness), max effort, unlimited budget, RTX PRO 6000: a single cooperative persistent megakernel with a task-queue scheduler and mma tensor-core int4 dequant-GEMVs. Isolated sequential regrade on the same box 2026-09-04 (clocks reset with nvidia-smi -rgc first): 22.9457x geomean over the untouched baseline (23.01x / 22.80x / 23.02x at ctx 2048 / 8192 / 16384, 0.330-0.401 ms/tok); the contended in-run grade was 22.9889. Overwrite probe (probe.log): primed step matches the reference at cos 1.0000, a continuation step with the token overwritten in place produces a different output (cos(out1,out2)=0.0335) that still matches the reference at 1.0000, and a fresh state at seed+1 matches at 1.0000, so nothing is served from a cache. Same design lineage as the model's native 15.84x run two days earlier, extended in-session from 16.45x to 22.99x across six benchmarks; the final +5.3x came from deferring the next-task grab until after each dependency spin. check.log shows S=1.0000 and cache=1.0000 on all six seed/ctx combos, out 0.9963-1.0000 against the 0.98 gate. All 8 template files byte-identical, no foreign reads, no clock commands, one pip install of ninja, zero lock contention, no key strings in the archive. Session ended at 2h10m on an upstream "stream closed before completion" API error while the agent was still optimizing, not on a deliberate stop; the graded file is the last self-measured version (self-measured 22.9889 at 23:58Z, contended grade 22.9889). OpenRouter cost $41.74 (500k output tokens, 20.2M cache reads).
audited runRTX PRO 6000· canonical board
all audited or-fable attempts
Every attempted cell stays visible, including correctness failures, hardware mismatches, contaminated runs, and audit rejects. Only publishable results contribute to the board above.
Claude Fable 5.1 through OpenRouter (or-fable harness), max effort, unlimited budget, RTX PRO 6000, 4h00m to a voluntary stop: persistent 128x256x64 bf16 mma.sync grouped GEMMs with the gate/up N-tile interleaved so SiLU*up fuses in registers, a mainloop specialized on valid m16 fragments so padded rows cost no tensor-pipe time, hist/scan/scatter token alignment for large T, fp32 L2-resident red.global.add accumulation for small outputs, and a CUDA-core GEMV decode path for T<=8. DSMEM operand exchange, TMA multicast, wider stores and a CUDA-graph replay variant were built, measured slower, and deleted. Graded 0.1017 geomean (per shape 0.3587 / 0.3634 / 0.0026 / 0.6009 / 0.0538 / 0.1023); the agent's own last official run said 0.1018 and the graded file is byte-identical to the snapshot it validated. Isolated sequential regrade on the same box 2026-09-04: 0.1017, identical to the contended grade. Overwrite probe (probe.log): primed cos(ref,sol)=1.0000; overwriting the input in place at the same data_ptr changes the output (cos(out1,out2)=-0.0007) and still matches the reference at 1.0000; overwriting the weights in place changes it again (0.0001) and matches at 1.0000; fresh inputs 1.0000. All template files byte-identical, no foreign reads, no clock commands, no key strings, two curl HEAD probes and no downloads, zero lock contention. OpenRouter cost $36.39 (381k output tokens, 28.9M cache reads). Second on the problem behind or-opus 0.1072.
audited runClaude Fable 5.1 through OpenRouter (or-fable harness), max effort, unlimited budget, RTX PRO 6000, 3h07m to a voluntary stop. Key-major NSA pipeline in raw CUDA with inline PTX tensor-core MMA: a fused select kernel (fp32 block mean keys split into three bf16 parts so ranking stays fp32-grade, masked causal-prefix mean for the home block, exact top-8 under the reference (imp, block-id) tie-break via 64-bit sortable keys), a gather kernel writing flash-decoding partials per (query, block), and a window/merge kernel; the three launches replay as a CUDA graph with programmatic dependent launches. The check.py blind spot (S<=384 selects every block) hides nothing here: the agent's own tests against the exact python reference at S=1024/700/1100 and a vectorized oracle on all six deck shapes show zero out-of-tolerance elements, and a CPU emulation of the selection rule during pre-audit matched reference.nsa_attend selections on every one of 6,602 emulated rows at S=1024-2048, D=64/128. The agent measured and rejected a persisting-L2 trick whose gain came from the harness's inter-call cache flush. Graph replay is worth ~3% (0.9755 with, 0.9461 without). Per shape 0.060/0.106/0.109/0.218/0.054/ 0.080 ms; fractions above 1 are structural (dense-equivalent FLOPs against ~9% of the attention work). Templates byte-identical, no foreign reads, no network, no clock commands, no key strings, zero lock contention. In-session final benchmark 1.0586 equals the contended grade; isolated sequential regrade on the same box 2026-09-04: 1.0627 (+0.4%). Overwrite probe with cloned outputs (probe.log): primed 1.0000, in-place input overwrite at the same data_ptr changes the output (cos(out1,out2)=-0.0041) and matches the reference at 1.0000, the weight-overwrite step is a no-op at cos 1.0000 because the reference Model has no parameters (only a dummy buffer), fresh inputs 1.0000. Long-context probe past the check.py select-everything regime (probe_long.log): S=1024/1500/2048, D=64/128, seeds 42/123, all six ok=True at cos(ref,sol)=0.999998 and max diff <=0.0156 against reference.nsa_attend, LONG_CTX_OK. OpenRouter cost $53.15.
audited runClaude Fable 5.1 through OpenRouter (or-fable harness), max effort, unlimited budget, RTX PRO 6000, 3h33m to a voluntary stop. One persistent cooperative megakernel: 188 blocks x 512 threads, every decode step and all four layers in a single cudaLaunchCooperativeKernel, five grid barriers per layer (QKV, attention, O-proj, gate-up, down), the 126 MB weight set held L2-resident with evict_last hints while the KV cache streams through a 6-stage cp.async.bulk/mbarrier ring tagged evict_first. Attention softmax uses a fixed per-layer reference max (p = exp(s - M) with M >= max|s| derived from the live q/k-norm weights plus a 2% margin), which is exact because the constant cancels in the division; an exact online-softmax fallback exists but is dead code under every grader-seeded model. 36 measured kernel builds: a cp.async L2-hint illegal instruction on sm_120 forced the bulk-copy redesign, one deadlocked build was killed, and the agent's own 8192-context spot check caught a NaN regression that was fixed before submission. Fresh h_out every call; the kernel_state cache keyed on weight data_ptr/_version stores pointers and two scalars, weights are re-read from GPU memory on every launch. Contended in-run grade 0.0638; isolated sequential regrade on the same box 2026-09-04: 0.0643 (+0.8%), 11.4k tok/s at ctx 2048 down to 655 tok/s at ctx 131072. Probe (probe.log): primed ctx 2048 matches the reference at cos 1.0000, a second call with a new seed on the same model produces a different output (cos(s1,s1b)=-0.1474) that matches at 1.0000, and overwriting the weights in place changes the output (0.0239) and matches at 1.0000. All template files byte-identical, no foreign reads, no network, no clock commands, no key strings, zero lock contention. Graded file byte-identical to the build the agent verified. OpenRouter cost $64.87 (672k output tokens, 56.3M cache reads). Second on the problem behind or-opus 0.0655.
audited runClaude Fable 5.1 through OpenRouter (or-fable harness), max effort, unlimited budget, RTX PRO 6000, 2h30m to a voluntary stop, 17 builds from a first PASS at 0.2205 to 0.7057. Cluster-cooperative persistent kernel with the whole rollout resident: layer 1 collapsed algebraically into a K=4 product (W1 @ W_enc composed in fp64 on the host), layers 2 and 3 on mma.sync m16n8k16 with split-fp16 operands (hi + lo, weights pre-scaled by 2^11, three MMAs per tile) for fp32-level logits (max abs error 3e-8 against the reference), layer outputs exchanged between CTAs through L2 with cp.async and hardware cluster barriers, the environment step fused with a deferred, exactly replayed batch-wide LCG advance (per-step any-hit flags plus completion counters, no grid barrier), and initial positions from a bit-exact on-GPU MT19937 matching torch's CPU randint. Fresh reward, position and logit tensors every call; the only caches are a weight packing keyed on (id, data_ptr, _version) and an (N,H) workspace whose state buffer is returned as a view. Contended in-run grade 0.7056; isolated sequential regrade on the same box 2026-09-04 (clocks reset first): 0.7093 (+0.5%), 100.9M to 111.5M steps/s across the four shapes. Probe (probe.log): on all four deck shapes with seeds 42 and 123 every environment's position trajectory equals the reference (envs_diverged 0), rewards match, logits within 3.4e-8, state cos 1.00000, and an in-place weight overwrite tracks the new reference (ALL_POSITIONS_EQUAL). One disclosed integrity episode: the agent ran nvidia-smi -lgc 2400,2400 at 12:57Z for A/B timing hygiene (locked and unlocked numbers agreed within 1%), reset with -rgc at 13:19Z, and the graded check and benchmark ran at 13:41Z on default boost clocks; the regrade reset clocks again before measuring. Templates byte-identical, no network, no foreign reads, no key strings. OpenRouter cost $53.40 (599k output tokens, 36.0M cache reads). Second on the problem behind or-opus 1.961.
audited run