别再比模型了。同一套权重，换一层外壳，分数可以从 13% 跳到 95%。
GPT-5.6 Sol 在 ARC-AGI-3 Public 上至少有三个数字：官方通用 harness 13.3%；打开 ChatGPT / Codex 实际在用的保留推理和 compaction，38.3%，输出 token 少大约 6 倍；第三方 Schema 报到 约 95%。权重没改。
官方 harness 是故意「写残」的：不给工具，每步丢掉私有推理，历史用滚动截断。它的工作是把模型短板暴露出来，不是把产品打满。OpenAI 自己把这事写进了说明。
所以 95%、99%、100% 不是「模型突然聪明了」。Schema、VISTA、AVO 改的是观察怎么变成世界模型、计划怎么被验证、失败怎么被接着打。这些近满分大多未经 ARC Prize 核验，也不能拿来跟 13.3% 横比。看到「某模型 98%」先问：哪个 harness、哪个数据集、有没有官方核验。不问这三句，你比的是宣传材料。
更麻烦的是：模型自己还写不好这层外壳。ByteDance Seed 的 HarnessDev 不测「这道题答对了没有」，测「你能不能造出可运行的执行系统」。写作和 ML 实验勉强打平人类参考；代码和搜索差一截。同一套权重，Terminal-Bench 2.1 在 Terminus 2 里 35.2%，换到 Codex CLI 是 49.6%。Data 任务里 77.8% 的失败被归到 harness 缺陷。看得见的「自我进化」换个执行模型经常退回去。生成出来的代码 harness，两万多条轨迹里几乎看不到 checkpoint——状态根本没存住。
Harness 是系统软件。不是 while True: llm(); tool() 外面再包一层聊天框。
DeepSeek 把第二半开源了：dsh，一切都是插件。OpenClaw 2.0 把 harness 收成带 hostCapabilities 的运行时合同。Tardigrade 更干脆：harness 是对不可变事件日志的纯函数，崩溃了就重放。这三件事说的是同一句话——工程主战场在外壳，不在下一个模型 ID。
所以真要干活，就三条：
- 冻结 harness 再换模型。 评测写成
model @ harness @ dataset。两样一起换，那个数字不是比较。 - 把 loop / tools / context / verify 写进 git。
AGENTS.md、skills、最小 CI。skills 要版本化，别让模型自己改全局规则。 - 别默认聊天框就是写代码的形态。 外壳该长在编辑器里，而不是再做一个滚动窗口。
下一篇我会把这套收成可执行的最小 harness：选题、核源、成稿、你审、再发。
Stop comparing models. Same weights, different harness, and the score jumps from 13% to 95%.
GPT-5.6 Sol has at least three numbers on ARC-AGI-3 Public: 13.3% on the official generic harness; 38.3% with the retained-reasoning and compaction settings ChatGPT / Codex actually use, and about 6× fewer output tokens; ~95% from third-party Schema. The weights did not change.
The official harness is intentionally thin: no tools, private reasoning dropped after every action, rolling truncation. Its job is to expose model shortcomings, not to max the product. OpenAI wrote this down.
So 95%, 99%, 100% is not "the model got smart overnight." Schema, VISTA, and AVO change how observations become a world model, how plans get tested, how failures get continued. Those near-ceiling scores are mostly unverified by ARC Prize, and they are not a matched comparison against 13.3%. Next time a slide says "model X, 98%," ask: which harness, which dataset, was it officially verified. Skip those and you are comparing press kits.
Worse: models still cannot write that shell. ByteDance Seed's HarnessDev does not score "did the task succeed." It scores "can you build a runnable execution system." Writing and ML experiments can match the chosen human references; code and search still lag. Same weights, Terminal-Bench 2.1: 35.2% inside Terminus 2, 49.6% inside Codex CLI. 77.8% of Data-task failures were harness defects. Visible "self-evolution" often collapses when you swap the executor. Across tens of thousands of generated-harness trajectories, there is almost no checkpoint — state was never saved.
A harness is systems software. It is not while True: llm(); tool() with a chat pane glued on.
DeepSeek open-sourced the second half: dsh, everything a plugin. OpenClaw 2.0 turned the harness into a runtime contract with hostCapabilities. Tardigrade goes further: the harness is pure functions over an immutable event log; a crash replays. All three are the same sentence — the engineering fight is the shell, not the next model ID.
If you actually have to ship:
- Freeze the harness, then swap the model. Write evals as
model @ harness @ dataset. Change both and the number is not a comparison. - Put loop / tools / context / verify in git.
AGENTS.md, skills, a tiny CI. Version the skills so the model cannot silently rewrite the rules. - A scrolling chat pane is not the default shape of coding. The shell belongs in the editor, not in another window.
Next post I'll compress this into a runnable minimum harness: pick a topic, check sources, draft, you review, then ship.