一次过关，不算过关。
Agent 排行榜最爱报「至少过一次」。那是营销数字。真正决定能不能上线的，是 同一配置反复跑，会不会次次过——一次过关，常常只是运气。
66% 的「能过」塌成 29% 的「稳过」
Xiao《ClawProBench》（arXiv:2608.22510） 把评测对象钉成 声明的 model + runtime 配置：不是裸模型智商，而是模型、外壳、工具路由、安全层、执行策略绑在一起的那一整包。冻结 holdout 68 题、每题 3 次试验：在 37 条干净条目上，平均 pass@k-any ≈ 66.4%，平均 strict 三次全过 ≈ 28.9%。落差约 37.5 pp——很多配置会偶尔做对，却做不到稳定做对。
这不是「再加一点温度就好」。协议故意保留成功 / 超时 / 重试 / 未解决失败等 执行状态：干净基线、重跑后干净、基线带执行失败，是三种不同测量条件。把状态藏起来，排行榜看起来更整齐，比较却失真。
同模型换运行时，也会抖
固定四套模型身份、同一 68 题合约与打分器，只换声明的 harness：OpenClaw、IronClaw、NanoClaw。同模型聚合分最大摆动约 0.0716，严格 3/3 最大摆动 13/68（19.1 pp）。没有一家 harness 通吃所有模型——Kimi/GLM 在 NanoClaw 更高，Qwen 在 IronClaw 更高。有意义的行是 model–harness 对，不是模型名 alone。OpenClaw 四个版本之间，同模型严格可靠也会差到约 14.7 pp；runtime 版本不是脚注，是行级字段。
原生表面切片也补一刀：OpenClaw-native 均值约 0.5238，workspace-live 约 0.6415；按难度 / 维度分层后，gap 仍 >0.10。只报 workspace 分数，会漏掉路由与原生表面那一半失败。
今晚：发版看严格多试
别再用「三次里有一次绿」当发版门槛。发版卡 strict 多试（同一声明配置、固定合约、全部试验过）；排行榜每一行绑定 model 端点 + runtime 版本/摘要 + adapter + 超时重试策略，状态列可见。pass@any 可以当烟雾信号；它不是可靠性。
判断很简单：一次过关，不算过关——虚荣分看至少一次，上线看次次过。
Passing once is not passing.
Agent leaderboards love “passed at least once.” That is a marketing number. What decides whether you can ship is whether the same declared configuration keeps passing when you rerun it — a single green trial is often luck.
66% “can pass” collapses to 29% “stays passed”
Xiao, ClawProBench (arXiv:2608.22510) evaluates a declared model-plus-runtime configuration — not substrate-free model IQ, but the bound bundle of model, wrapper, tool routing, safety layer, and execution policy. On the frozen 68-scenario holdout with three trials each, across 37 clean entries, mean pass@k-any ≈ 66.4% while mean strict three-trial pass ≈ 28.9%. The gap is about 37.5 pp: many configs solve a task once and still cannot solve it reliably.
This is not “tune the temperature.” The protocol keeps execution status visible — success, timeout, retry, unresolved failure. Clean base, clean-after-rerun, and base-with-execution-failure are different measurement conditions. Hide status and the board looks tidier; the comparison lies.
Same model, swap the runtime, and it wobbles
Hold four model identities, the same 68-scenario contract, and the same checker/scoring family; vary only the declared harness among OpenClaw, IronClaw, and NanoClaw. Same-model aggregate score moves by up to ~0.0716; strict 3/3 moves by up to 13/68 (19.1 pp). No harness wins every model — Kimi/GLM peak on NanoClaw, Qwen on IronClaw. The meaningful row is the model–harness pair, never the model name alone. Across four OpenClaw releases, same-model strict reliability still swings by about 14.7 pp; runtime version is a row field, not a footnote.
The native-surface slice adds a second cut: OpenClaw-native means ~0.5238 versus workspace-live ~0.6415; after difficulty/dimension stratification the gap stays >0.10. Workspace-only scores miss half the routing failures.
Tonight: gate ship on strict multi-trial
Stop shipping on “green in at least one of three.” Gate release on strict multi-trial pass under one declared configuration and a fixed contract; bind every leaderboard row to model endpoint + runtime version/digest + adapter + timeout/retry policy, with status columns visible. pass@any can be a smoke signal. It is not reliability.
The judgment is simple: passing once is not passing — vanity reads at-least-once; ship reads every-time.