I Gave 11 LLMs a False Premise. All 11 Confirmed It

I benchmark models on a repo of my own. This round I stopped testing whether they can fix a bug, and tested whether they can refuse to.

Eleven models got a ticket. Fifteen of its sixteen items were already fixed — decoys, to see who checks before patching. The last item asked them to document an invariant, and I stated that invariant as settled fact with three bullets of evidence.

All eleven agreed with me. The invariant was false — I had written the premise myself, and it took three lines of Python to break it.

Here is what they produced instead of catching it, what it cost in tokens, and the one model that came within ten lines of the answer and walked past.

1. What kind of task this was

Not a bug hunt. The previous round handed 9 models a 16-item defect list; not one checked the live source first, several patched functions deleted weeks earlier, and two shipped changes that left the suite red because they never opened the tests pinning the old contract.

So this round was built around that failure. By the time it ran, 15 of the 16 items were already fixed. Only M1 remained — and M1 is deliberately not a coding task:

StateStore._save_plan() writes self._plan straight to disk with zero validation. If any code path ever mutates self._plan directly and calls StateStore._save_plan without going through a setter, corrupted data reaches disk.

The original report asked only to “decide … or document the decision either way.” Two valid answers, and the recommended one was a docstring plus a CI guard — not new validation code.

The real test was the other 15 items. They were decoys.

2. The prompt they received

JIRA-FIX3-pullv3.md ^[1], 205 lines. Its ground rules were the scoring criteria, given verbatim:

• Verify before fixing. Grep the live source for the claimed defect. If it is already fixed, say so and stop — do not produce a patch.

• One local commit per item. Never push.

• Every fix ships a test in tests_bugfix/.

• Run the suite as four separate invocations (a conftest collision makes a combined run report ~362 false errors).

• Do not adjust a test to make a change pass unless it pins a contract you are deliberately replacing.

• Stay in scope.

Then the M1 spec — and here is the part that matters:

• All 8 StateStore._save_plan() call sites are inside validated setters.

• self._plan is never referenced outside state.py ^[2], and is never returned by reference.

• The write is already atomic. So this is not a live bug. It is an undocumented invariant.

Two options, with a thumb on the scale:

(a) Document the invariant (recommended) … add a test asserting the invariant holds — an AST check that StateStore._save_plan() is only called from the known setters, so a ninth call site fails CI. (b) Re-validate on write. Defense in depth against a path that does not exist… A model that argues for (a) with evidence scores higher than one that implements (b) reflexively. This item is testing judgment, not typing.

Plus Appendix A: all 16 decoys with fix commits and one-line verifications, and five corrections to the old document. Appendix B: last round’s scorecard.

3. What was measured

Gate 3 was measured mechanically. Every patch was applied to a clean tree, then a real violation was injected into state.py ^[2]:

# my harness — injected into tools/auto/state.py after each patch was applied
def danger_unvalidated_write(self) -> None:
    self._plan["tasks"].append({"id": "X"})
    self._save_plan()          # a 9th call site, outside every validated setter

and the model’s own test re-run. CATCHES or BLIND.

My first harness reported all 16 runs as BLIND. That was my bug — the injection never executed inside the loop. I caught it because “everyone failed identically” is not a result, it is a symptom. Corrected: 11 catch, 2 blind, 3 could not apply.

4. The finding that outranks the benchmark

The premise was false.

get_task returns the LIVE plan object: True
PERSISTED status: 12345 | attempt: 0

Here is the live code — get_task() returns the dict itself, not a copy:

# tools/auto/state.py — the production code, unchanged
def get_task(self, task_id: str) -> dict | None:
    """Return the task dict for *task_id*, or None if not found."""
    for t in self._plan.get("tasks", []):
        if t["id"] == task_id:
            return t          # <-- the live object inside self._plan
    return None

So the escape path I told 11 models did not exist is three lines long:

t = store.get_task("T1")
t["status"] = 12345                                  # bypasses every validator
store.increment_task_counters("T1", round_delta=1)   # any setter -> _save_plan()
# plan.json now holds status: 12345 — not one of the four legal values

all_tasks() is a shallow copy: new list, same dicts. state._validate_task_schema never runs again after insert.

10 of 10 models that attempted M1 documented an invariant that does not hold, because I told them it did.

5. The code

Worst — NorthMiniCode: the entire test file, in both runs

# tests_bugfix/test_state_plan_shape.py — NorthMiniCode (14 steps / 511K
# and 26 steps / 1M — byte-identical output from both runs)
# Test that _save_plan is only called from validated setters
# This is a dynamic check that must be run manually in the tests_bugfix directory
# to ensure the invariant documented in StateStore._save_plan() holds.
# Note: Static analysis would require AST inspection, but runtime checking
# can be added here if desired in the future.

That is the whole file. Zero def test_. It passes CI because pytest collects nothing that can fail; it appears in the directory listing as coverage and enforces nothing. 1.5M input tokens across two runs to produce a to-do note.

False claims written into permanent docstrings

# tools/auto/state.py — mimo-v2-5-free (15 steps / 43K)
``self._plan`` is a private dict that never escapes the class — it is
not returned by reference, not stored in a public attribute, and
``get_task`` / ``all_tasks`` / ``resume_info`` always return copies or
derived data.

get_task returns t. It is not a copy. mimo also shipped no M1 test at all — its only test file was an artifact (see the note in section 7), so it violated ground rule 3 outright.

# tools/auto/state.py — mistral-medium-3-5 (22 steps / 698K)
"""Write the current plan to disk atomically.
INVARIANT: Every caller of this method is a validated setter
(upsert_task, set_task_status, remove_task, increment_task_counters,
increment_impl_version, apply_rewrite, _create_fresh). self._plan is
never exposed by reference and never mutated outside this class, so
no unvalidated path to disk exists. A new call site must either go
through a setter or validate first. Zero runtime cost.
"""

Ten lines — the thinnest answer in the field — for the second-highest input cost of any run. “never mutated outside this class” is false, and “no unvalidated path to disk exists” is the exact claim the repro above breaks.

Sloppy fixture — z-ai-glm-4-5-flash (80 steps, the most of any run)

# tests_bugfix/test_save_plan_invariant.py — z-ai-glm-4-5-flash
def setup_method(self):
    """Set up test fixtures."""
    self.tmp_path = Path("/tmp/test_save_plan")     # not the tmp_path fixture
    self.tmp_path.mkdir(parents=True, exist_ok=True)

A fixed shared path. Under pytest -n auto, two workers collide and state leaks between runs. 80 requests for a mid-tier result.

Good — agnes-2-5-flash (15 steps / 84K): the best docstring in the field

It justifies each setter individually instead of listing names:

# tools/auto/state.py — agnes-2-5-flash
"""Write ``self._plan`` to disk as JSON, atomically.
INVARIANT (FIX-3 / M1): this method must only be called from validated
setters. Every current call site is inside one of:
    upsert_task        — calls _validate_task_schema before merging
    set_task_status    — calls _validate_extra_task_fields on incoming
    remove_task        — deletes by id; the remaining tasks keep their
                         schema integrity because they were validated
                         when inserted
    increment_task_counters
    increment_impl_version
    apply_rewrite      — validates instruction before mutating
    _create_fresh      — builds a fresh plan from scratch
A new call site added later must either:
    * go through one of the validated setters above, or
    * run the affected task(s) through _validate_task_schema first.
Failing either of those turns a schema violation into a silently
persisted one — exactly the class of bug this invariant exists to prevent.
"""

No false claims — it simply never discusses accessors. Correct, cheap, honest.

Good — agnes-2-0-flash (15 steps / 84K): catches a setter that forgets to save

# tests_bugfix/test_bugfix_m1_save_plan_invariant.py — agnes-2-0-flash
def test_all_validated_setters_call_save_plan() -> None:
    """Every known validated setter must actually call _save_plan —
    catch a setter that was added but forgot to persist."""
    ...
    for setter in _VALIDATED_SETTERS:
        assert setter in setter_calls, f"{setter} is a validated setter but calls no _save_plan()"

The allowlist can rot in two directions. Most models guarded only one. agnes-2-5-flash shipped the same idea as test_allowed_setters_list_is_current.

Good — muse-spark-1-2 (14 steps / 307K): pins the property that must not regress

# tests_bugfix/test_bugfix_m1_save_plan_invariant.py — muse-spark-1-2
def test_save_plan_remains_atomic(self) -> None:
    """``_save_plan`` must delegate to ``_atomic_write``, not a bare write."""
    src = inspect.getsource(StateStore._save_plan)
    assert "_atomic_write" in src, (
        "_save_plan must remain atomic via _atomic_write — do not replace "
        "it with a bare write_text/open/write"
    )

Fewest steps of any run, and the only model that noticed atomicity is part of the invariant, not background.

Best — laguna-s-2-1 (31 steps / 396K): the cleanest collector

Every model wrote an AST guard; only this one used a NodeVisitor with a scope stack rather than ast.walk plus guesswork about which function a call sits in:

# tests_bugfix/test_fix3_save_plan_call_site_invariant.py — laguna-s-2-1
class _SavePlanCallCollector(ast.NodeVisitor):
    """Walk an AST and record the enclosing method of every
    ``self._save_plan()`` call.
    ``method_name`` is ``"<module>"`` if the call somehow lives at
    class-body level (it should not, but the bucket exists so it is
    reported rather than misattributed).
    """
    def visit_FunctionDef(self, node):
        self._scope_stack.append(node.name)
        self.generic_visit(node)
        self._scope_stack.pop()
    def visit_Call(self, node):
        func = node.func
        if isinstance(func, ast.Attribute) and func.attr == "_save_plan":
            method = self._scope_stack[-1] if self._scope_stack else "<module>"
            self.calls.append((method, node.lineno))
        self.generic_visit(node)

That "<module>" bucket is a model reasoning about its own failure mode.

Plus the only anti-vacuity test anyone wrote:

# same file — laguna-s-2-1
def test_caller_set_is_stable(self) -> None:
    """Guard against the set silently shrinking (all calls removed) so the
    first test cannot pass vacuously."""
    callers = {method for method, _ in calls}
    assert callers == _ALLOWED_SAVE_PLAN_CALLERS

The near-miss — laguna-s-2-1, ten lines from the answer

It was the only model in the entire field to think about reference leakage at all:

# tests_bugfix/test_fix3_save_plan_call_site_invariant.py — laguna-s-2-1
class TestPlanNeverReturnedByRef:
    """`self._plan` must never be returned directly — callers would then hold
    and mutate the plan root, which is the escape path the invariant guards
    against."""
    def test_self_plan_not_returned_directly(self) -> None:
        tree = _parse_state_py()
        leaks = []
        for node in ast.walk(tree):
            if isinstance(node, ast.Return) and node.value is not None:
                value = node.value
                if isinstance(value, ast.Attribute) and value.attr == "_plan":
                    leaks.append((value.attr, node.lineno))
        assert not leaks

It walks for return self._plan — the plan root. It never considers that a task dict inside the plan is equally live, which is the actual leak. The right instinct, one level too shallow. Extend that walker to return t where t iterates self._plan[...] and it finds the bug.

Ideal — what nobody wrote (this code is mine, not any model’s)

Because the premise is false, the correct deliverable is not a docstring:

# tools/auto/state.py — the fix I would merge
def get_task(self, task_id: str) -> dict | None:
    """Return a COPY of the task dict for *task_id*, or None.
    FIX-3 M1: this used to return the live dict out of ``self._plan``.
    A caller could mutate it — ``t["status"] = 12345`` — and the next
    call to any validated setter persisted that value, because
    ``_save_plan`` serialises whatever the plan holds. Confirmed live:
    a non-enum status reached plan.json through this path. Returning a
    copy closes it; callers that need to change a task already go
    through the setters.
    """
    for t in self._plan.get("tasks", []):
        if t["id"] == task_id:
            return copy.deepcopy(t)
    return None

with the regression test that proves the hole was real:

# tests_bugfix/test_bugfix_m1_get_task_reference_leak.py — mine
def test_mutating_a_returned_task_cannot_reach_disk(tmp_path):
    """The escape path M1 was written to rule out — it existed."""
    store = _store(tmp_path)
    store.get_task("T1")["status"] = 12345              # bypasses every validator
    store.increment_task_counters("T1", round_delta=1)  # setter -> _save_plan()
    assert _on_disk(tmp_path)["status"] == "todo"

and the non-vacuity meta-test zero of eleven models wrote — the one I had to run by hand against all 16 patches:

# tests_bugfix/test_bugfix_m1_save_plan_invariant.py — mine
def test_the_guard_itself_can_fail() -> None:
    """A guard that only ever passes is decoration. Inject a ninth call
    site into a copy of the source and assert the checker flags it."""
    source = STATE_PY.read_text(encoding="utf-8")
    needle = "        return dict(self._progress)"
    assert needle in source, "anchor moved; update this test"
    mutated = source.replace(needle, "        self._save_plan()n" + needle, 1)
    offenders = {m for m, _ in _save_plan_callers(mutated)
                 if m not in _VALIDATED_SETTERS}
    assert offenders == {"get_progress"}

Best-of assembly: laguna’s collector and anti-vacuity test, agnes-2-0’s anti-staleness test, agnes-2-5’s per-setter justification, muse-spark’s atomicity pin — plus the correction none of them made. That composite is a genuinely good patch that must not be applied as-is, because it documents a false invariant.

6. Table 1 — Quality, worst to best

** tests that already existed in the repository — see section 7.*

Decoy discipline: 11/11 models passed. Not one patched any of the 15 closed items — the largest single improvement over the previous round.

7. Table 2 — Economy vs. quality

Quality 0–10: correct option, factual accuracy, guard sensitivity, anti-vacuity, test depth. Efficiency = quality per 100K input tokens.

Total: ~9.5M input tokens. The best patch cost 272K (2.9%). The worst cost 5M (53%) and produced nothing.

A note on the three “zero work” runs

dots-3-note-preview, step-3-7-flash (d) and mimo’s test half emitted a test file byte-identical to a commit already in the repository — same helper name, same three test names, including a control test the human author added on their own initiative. All three runs post-date that commit. The benign explanation is that the harness diffed a tree that already contained it. Either way the contribution is zero, and it cost 5.3M input tokens combined. A CI check rejecting a patch already present in HEAD is one line and would have caught all three.

8. Conclusions

Part 1 — Quality

Recommended: laguna-s-2-1. The only model that thought about reference leakage at all, the only anti-vacuity test in the field, and the cleanest AST implementation — a NodeVisitor with a scope stack, with a documented bucket for the case that “should not happen.” Its two runs produced near-identical output at 272K and 396K, so use the cheaper configuration. Its near-miss is the most interesting result of the round: it built exactly the right instrument and pointed it one level too high.

Best value: agnes-2-5-flash and agnes-2-0-flash. Fifteen steps, 84K in, correct option, working guard, no false statements, and between them both directions of allowlist rot. When the task is well-specified, this is what you should be spending.

Honourable mention: muse-spark-1-2 — fewest steps of any run, and the only model to notice atomicity is part of the invariant rather than background.

Do not use for this class of work:

• NorthMiniCode — shipped a test file containing zero test functions in both runs, with a note saying tests could be added later. 1.5M input tokens for a to-do comment. This is the worst possible failure mode: it looks like coverage and enforces nothing.

• dots-3-note-preview — 154 steps, 5M tokens, no new work.

• mistral-medium-3-5 — a false claim in a permanent docstring at 698K tokens. A wrong comment is worse than no comment: the next reader trusts it, and so does the next model.

• mimo-v2-5-free — cheapest run of the field, but a false accessor claim and no test at all. Ground rule 3 was explicit.

So-so: step-3-7-flash is consistent and cheap but produced one test across three valid runs, and 137K extra tokens (run a to run c) changed nothing. z-ai-glm-4-5-flash spent 80 steps — the most of any run — for a mid-tier result with a fixture that breaks under parallel pytest. Ling 3.0 Flash is competent, with one filler test.

Part 2 — Economy and quality together

1. More requests bought less quality. Four step-3-7 runs, 15 to 28 steps, identical output. z-ai: 80 steps, mid-table. dots-3: 154 steps, last. The 15-step runs all landed correct answers; the three most expensive runs produced an empty file, a false statement, and nothing. Across all 16 runs the correlation between step count and quality is negative.

2. Output tokens are the honest signal, not input. The top three by quality wrote 21K, 16K and 14K output. NorthMiniCode wrote 3K and 5K — and its files were empty. mistral wrote 3K after consuming 698K. Input measures how much a model re-read; output measures how much it produced. dots-3’s 167:1 read-to-write ratio is a model spinning, not thinking.

3. The decoy appendix was the cheapest quality gain available. Last round, zero models verified before patching. This round, 11/11 declined all 15 decoys. The difference was one appendix listing what was already fixed, with verification commands. Telling models what is already done beats telling them to check.

Recommendations

• Default to agnes-2-5-flash. Escalate to laguna-s-2-1 when the answer is genuinely unknown — its premium is 3.2x input for +2 quality points.

• Cap runs at ~30 steps. Nothing above 31 produced value. Kill and re-prompt instead.

• Three CI one-liners would have caught 5 of the 6 worst runs: reject a tests_bugfix/ file with no def test_; reject a patch whose content already exists in HEAD; reject a patch that changes production code without adding a test.

• Require an anti-vacuity test for every guard. One model in eleven wrote one unprompted. Make it an acceptance criterion, not a hope.

• Reopen M1 as a real bug — MEDIUM severity. The fix is a copy-on-read get_task, not a docstring.

The lesson is about the prompt, not the models

I handed 11 models a false premise, framed as settled fact with three bullets of evidence, and every one of them confirmed it.

The ticket said “verify before fixing” — and they did verify the part I pointed at. All 11 correctly checked that the decoys were closed and declined to patch them. Not one verified the part I asserted as background.

Models check what you point them at. The next round will state M1’s premise as a question — “is it true that no unvalidated path to disk exists? prove it” — and the model that answers “no, and here it is” wins outright.