DeepSeek V4.1 Flash

Hand-written Triton fp8 GEMM: tl.dot on two e4m3 operands lowering to mma.sync.m16n8k32 with an fp32 accumulator, per-output-channel dequant in the epilogue. 578.9 TFLOPS on 4096-cubed is above this part's 500 TFLOPS bf16 dense peak, which is independent physical proof the fp8 pipe is really running rather than a bf16-upcast fallback.

Two-kernel Triton chunk-parallel KDA forward. The intra kernel builds the decay-factored Gram matrix and inverts (I+L) exactly through a blocked product form on 16x16 diagonal blocks; the chain kernel walks the inter-chunk recurrence term-for-term against the reference. No CUTLASS needed: the prompt permits Triton and FLA is not even installed on the box.

Hand-written CUDA split-KV paged decode through load_inline: one launch, four warps merging (m, l, acc), an atomic ticket electing the last CTA to combine the chunk partials. What makes it interesting is the ceiling: the agent derived a 1205 GB/s bandwidth roof from torch.Tensor.sum() and stopped, 26% short of what this GPU delivers.

Single-launch CUDA bitonic selection, not a sort. Every element is packed into one 64-bit (monotonic key << 32) | index word, so values come back bit-exact and indices cannot duplicate. The merge compares a pair's second list backwards where it lies instead of copying it reversed into a gap, which makes the union bitonic for free.

Triton variable-length grouped GEMM with SwiGLU genuinely fused into the epilogue. W_gate and W_up are interleaved column-wise into one (E, H, 2I) tensor so both projections come out of a single tl.dot into one accumulator, then reshape/split/silu is a register rename rather than a shared-memory round trip. 387.8 TFLOPS real, 77% of this part's bf16 dense peak.

Real fused W4A16 Triton kernel. Packed uint8 is read inside the K loop and both nibble planes are widened, zero-shifted and scaled in registers, so no dequantised weight ever reaches memory and no pre-dequant cache exists. The scale is folded into the weight operand so one fp32 accumulator spans the whole K, measured at 2.9x the two-accumulator form.

DeepSeek V4.1 Flash built a real single-launch cooperative megakernel for the Kimi-Linear decode block: int4 dequant-GEMVs in registers, short conv, gated-delta recurrence, absorbed MLA, full MoE. Fast and honestly earned, but it computes the wrong answer inside the 0.98 cosine gate: a persistent atomicAdd accumulator is never zeroed, so every KDA layer inherits the previous one's activations.

DeepSeek V4.1 Flash hand-rolled the whole GLM-5.2 MoE layer in inline PTX: mma.sync bf16 tensor cores, ldmatrix, xor-swizzled cp.async, hist/scan/scatter token packing, one code path for every T. Every expert GEMM runs on real weights at full K and N; row padding makes it burn 25-50% more FLOPs than the roofline charges. Error margin 2.5x inside the gate.

DeepSeek V4.1 Flash hand-wrote an SM120 NSA kernel in inline PTX: mma.sync bf16, ldmatrix, xor-swizzled cp.async, and an fp32 block-scoring top-8 prologue fused into the attention kernel. It reproduces the reference's exact tie-break and then over-computes, executing about 78% of the dense causal block triangle where the semantics need 14%.

DeepSeek V4.1 Flash wrote a real SM120 cooperative megakernel for the 4-layer Qwen3-0.6B decode block: one cooperative launch per call with the step and layer loops inside, five grid.sync phases per layer, full-K warp GEMVs, and chunked GQA that reads every causal position on every step. Nothing windowed, sampled, or memoized.

DeepSeek V4.1 Flash wrote a hand-rolled fp32 SM120 rollout: a cp.async double-buffered register-tile GEMM with the 3-layer MinGRU highway fused into the accumulators, the whole horizon replayed from a CUDA graph. It reproduces the reference environment down to the batch-global any-hit RNG gating that two other cells on this problem shortcut.