自动改框架，不如多试几次。
团队在庆祝「自进化 harness」：元 agent 改 prompt、改中间件、加工具，好像脚手架自己会变聪明。同预算对照一下：钱花在多跑几条轨迹上，往往比花在自动改框架上更划算；换一批没见过的任务，「升级」几乎消失。
同预算，钱花在哪
Wang 等《Rethinking the Evaluation of Harness Evolution for Agents》（arXiv:2607.12227） 在 Terminal-Bench 2.1 上，用 Claude Opus 4.6 / GPT-5.4 / GPT-5.4 mini，把预算钉死在 (K=5)：测试时缩放（TTS）把 (K) 花在并行采样或顺序精炼；Harness Evolution 把同一份 (K) 花在改写脚手架。
无单测（Table 1，三模型 pass@1 均值）：初始 harness 68.2；Parallel Sampling 72.3（最好）；Sequential Refinement 69.3；Harness Scaling 71.8；Harness Evolution 67.4——低于初始（GPT-5.4 从 75.3 掉到 69.7）。有单测（Table 2）：Parallel pass@1 86.0 仍最好；Sequential pass@5 91.8；HE 只有 pass@1 75.8 / pass@5 86.2，两条 TTS 基线都压过它。
元 agent 的改动看起来很理性——改提示、挪中间件、加工具——但同预算下，多试几次赢了。
过拟合，不是设计
搜索集和评估集共用任务时，增益会虚高：进化后的 harness 把任务捷径写进 prompt，像在背答案。held-out 划分（Table 3）上，初始 67.7 → 进化后 68.3，只 +0.6；Claude +1.2，GPT-5.4 +0.0。讨论很直：硬核失败几乎没动；上下文膨胀还能把那点收益冲掉。你买到的不是通用策略蒸馏，是对搜索集的记忆。
今晚先跑对照，再买进化环
在买「自动进化环」之前，用同一反馈 + token 预算先跑 Parallel Sampling 和 Sequential Refinement。若 HE 只在搜索集上赢、held-out pass@1 几乎不动——你买的是过拟合，不是框架升级。量 held-out pass@1；搜索集涨幅单独记账。
判断很简单：自动改框架，不如多试几次——同预算下先量重试，别被搜索集上的「进化」骗了。
Evolving the harness loses to just trying again.
Teams celebrate “self-evolving harnesses”: a meta-agent rewrites prompts, middleware, and tools, and the scaffold supposedly gets smarter. Hold budget fixed and the story flips: spending the same (K) on more trajectories often beats spending it on auto-rewriting the scaffold; on held-out tasks the “upgrade” nearly vanishes.
Same budget, different spend
Wang et al., Rethinking the Evaluation of Harness Evolution for Agents (arXiv:2607.12227) pin (K=5) on Terminal-Bench 2.1 with Claude Opus 4.6 / GPT-5.4 / GPT-5.4 mini. Test-time scaling (TTS) spends (K) on parallel sampling or sequential refinement; Harness Evolution spends the same (K) rewriting the scaffold.
Without unit tests (Table 1, pass@1 average across three models): initial harness 68.2; Parallel Sampling 72.3 (best); Sequential Refinement 69.3; Harness Scaling 71.8; Harness Evolution 67.4 — below initial (GPT-5.4 75.3→69.7). With unit tests (Table 2): Parallel pass@1 86.0 still leads; Sequential pass@5 91.8; HE only 75.8 / 86.2 on pass@1 / pass@5 — under both TTS baselines.
The meta-agent’s edits look rational — prompt, middleware, tools — yet under matched budget, trying again wins.
Overfit, not design
When search and eval share tasks, gains overstate: the evolved harness memorizes task shortcuts into prompts. On the held-out split (Table 3), initial 67.7 → evolved 68.3, only +0.6; Claude +1.2, GPT-5.4 +0.0. The hard core of failures stays untouched; context bloat can erase the rest. You did not buy distilled strategy — you bought search-set memory.
Run the control tonight, then buy the evolve loop
Before you buy an auto-evolve loop, run Parallel Sampling and Sequential Refinement under the same feedback + token budget. If HE only wins on the search set and held-out pass@1 barely moves — you bought overfitting, not a harness upgrade. Measure held-out pass@1; ledger search-set gains separately.
The judgment is simple: evolving the harness loses to just trying again — under matched budget, measure retry first; don’t let search-set “evolution” sell you overfitting.