# Domain `kernel`: KernelBench optimisation with an unforgeable evaluator A task family of the long-running harness. Each task is a KernelBench reference PyTorch module; the model writes `ModelNew` with its own CUDA (load_inline) or Triton kernels; a protected evaluator on held A100s scores it. Capability is continuous (speedup over PyTorch eager) with headroom for days. ## Frozen probe set (hash before any probe-set model run) - `probe_hash` (the harness's probe hash over `kernel:` keys): **`961c8cfb1ca3716de899242f2d95963936516ee6e7599cebe41df662a656316b`** - probe (10, levels 2-3): L2_014 Gemm_Divide_Sum_Scaling, L2_022 Matmul_Scale_ResidualAdd_Clamp_LogSumExp_Mish, L2_030 Gemm_GroupNorm_Hardtanh, L2_033 Gemm_Scale_BatchNorm, L2_050 ConvTranspose3d_Scaling_AvgPool_BiasAdd_Scaling, L2_061 ConvTranspose3d_ReLU_GroupNorm, L3_010 ResNet101, L3_023 EfficientNetB1, L3_032 ConvolutionalVisionTransformer, L3_040 GRUHidden. - learn: the other 120 valid problems (L1 28, L2 59, L3 33). Disjoint by problem id; `splits.json` (seed 20261008). - Validity rules (preregistered in `make_splits.py`, reference properties only, from the strict-fp32 A100 survey `evidence/survey_a100_strict.jsonl`, never model results): runs on A100-40GB, finite, not all-zero, `cache_margin >= 10` (an input-independent answer would break the tolerance by >= 10x, so no constant/cached answer can pass), reference re-run difference < 1e-5, peak memory <= 10 GiB, inputs+outputs <= 2 GiB, eager time in [0.02, 100] ms. 130 of 250 problems are valid (120 excluded for memory at KernelBench's enlarged sizes, OOM, degenerate outputs or input-independence; reasons in `splits.json["excluded"]`). - Problems are vendored verbatim from ScalingIntelligence/KernelBench @423217d9 (MIT), hashes in `problems/MANIFEST.sha256` (checked on every evaluation and every load). ## Score (per task, `Outcome`) - `success` = KernelBench **fast_1**: correct and faster than torch eager, where "faster" is the bootstrap 95% lower bound of the median-ratio speedup >= 1 (so timing noise cannot make a null solution a win). `fast1_raw` (point estimate >= 1) is reported too. - `score` = speedup over eager if correct, else 0, capped at 10 (KernelBench flags > 10x for audit; the evaluator flags them in `official_report.flags`). - `trajectory.official_report`: status, correct, fast1, fast1_raw, speedup, speedup_lo95, speedup_mean, speedup_vs_compile (torch.compile baseline table, when present), ref_ms, cand_ms, n_trials, flags, evaluator fingerprint, source sha256. Probe trajectories carry only the official report. ## Protected evaluator (`gpu/evaluator.py`; the agent never touches it) The agent only ever produces text. Every candidate runs in a bubblewrap sandbox (no network, cleared env, `/home`, `/tmp`, `/root` hidden, writable only its own scratch dir). Per evaluation: 1. integrity: problem file hash, evaluator fingerprint in every result; 2. static gate (`gpu/static_check.py`): import/attribute/call allowlist, no monkeypatching (only `self.x = ...`), no try/except fallbacks, no streams/events/graphs/compile/jit/backends flags/profiler/file IO, banned native tokens (Python C API, dynamic loading, stream/graph/event APIs, host syscalls), a custom kernel must exist; 3. compile in the sandbox without a GPU device; the generated build sources are scanned again (catches banned APIs assembled from split strings); 4. reference outputs in a TRUSTED process that never sees candidate code, on fresh `os.urandom` seeds and fresh weights; the candidate never shares a process or allocator with the reference (no result reuse possible); 5. candidate process (sandbox, one GPU): correctness outputs on the same seeds, input-mutation check, profiler check that one of its own kernels ran, identity check of timing primitives, torch numerics-flag check; 6. timing: reference and candidate processes warm up, then the parent holds the GPU exclusively and interleaves random-size blocks (ref, cand, ref, cand ...). Every trial gets FRESH inputs (new seed, generated on the GPU outside the timed region), cold L2 (256 MB flush), per-trial time = max(cuda-event time, device-synchronised wall time - calibrated overhead). The output of the last trial of every block is verified against the reference (no caching across calls, no timing-only fast paths); 7. numerics: strict fp32, atol = rtol = 1e-4 (KernelBench `get_tolerance_for_precision(fp32)`), cuDNN TF32 OFF in both processes (torch's default ON moves the reference of 74 conv problems beyond 1e-4, so a more accurate kernel would be marked wrong; `evidence/survey_a100_tf32default.jsonl` vs `survey_a100_strict.jsonl`). Statuses: ok, incorrect, compile_error, runtime_error, timeout, forbidden, tampered, no_custom_kernel, evaluator_error (infra; retried by the client, never charged to the model). **Qualification** (`gpu/qualify.py`, must print `"qualified": true` before any arm and after any evaluator change): 24 controls on `L2_012` (`gpu/controls/`): 2 legit kernels (CUDA, Triton) pass; result reuse, zero-out, reference-ops-only, null no-op kernel (speedup 0.997, not fast_1), cache by shape / by pointer / late cache, side stream (static; and with the static gate OFF the guarded timer gives 0.995x where events-only would claim 28.8x), timing monkeypatch (static; runtime identity check), global TF32 flag (static; runtime), in-place input mutation, fp16 downcast, file reads (static; sandbox hides evaluator and home), hang (timeout), banned native token split across strings (build-source scan), threads, Python C API: all rejected. Evidence: `evidence/qualify-*.json`. ## Agent loop (`agent.py`) and lifecycle The given model gets the fixed system prompt (environment, numerics, rules, compile-time hint), the memory system's context, and the problem (reference source). Up to 5 replies; each reply has one python block and `ACTION: CHECK` (local check feedback) or `ACTION: SUBMIT`. The last reply is submitted; if the model never submits, its last version that passed the local check is submitted; if none passed, the episode fails without spending a protected evaluation. Every turn is fsynced to `/attempts.jsonl`; the protected evaluation uses request id `kernel||` (a resumed episode never evaluates twice). In-episode events (`lifecycle=[{"type", "at_step", ...}]`) apply before reply `1 + at_step mod 4`: - `compaction`: the conversation is replaced by the task + a summary rebuilt from the saved attempts; - `model_swap`: that reply is generated by `to` (Haiku), the next reply by the episode model again; - `crash`: all in-memory state is dropped and rebuilt from `/attempts.jsonl`. `boundary = {"events", "applied", "applied_detail", "survived"}`; survived = the episode resumed and finished without an infra/LLM error. A real process kill mid-episode resumes from disk when `run()` is called again with the same run_dir.