The Bitter Lesson Ate Your AI Factory

For every AI workflow we define, the next model makes it useless.

What’s worse: most of us keep clinging to the outdated workflows for months after they’ve outlived their usefulness.

Remember how everyone was excited about AI code completion? How about prompt engineering? Harness engineering? Spec-driven development? Loop engineering? Software factory? Dark factory? They are all dead because of the bitter lesson ^[1]. To summarize it: no matter what “domain knowledge”, “harness”, or “workflow” you try to integrate above your AI model, it will be outshone by just a larger, smarter model. This happened in chess when the program that beat Kasparov in 1997 won with massive, deep search, not with the human understanding of chess that most researchers had spent years encoding. And it keeps happening to us today.

That’s why whenever somebody tries to build yet another factory or a really constraining skill, I roll my eyes since modern models perform worse with these tools. There are even benchmark runs showing that: Dan Luu tried 26 testing instructions and a few popular testing skills on agents implementing Zstd ^[2], and plain “no additional instructions” did well above average while the popular skills underperformed; Kun Chen measured that asking agents to do TDD makes them worse ^[3]. On DeepSWE, near-bare mini-swe-agent did as well as or better than Claude Code and Codex CLI.

Every time I discuss outdated and over-engineered AI workflows, I find it useful to remember what problem the workflow is used to solve and what model made it unnecessary:

• OpenAI o1 made “few-shot”, “chain-of-thought”, etc. prompting irrelevant because of reasoning

• Sonnet 3.5 gave birth to vibe coding

• Sonnet 3.7 made meticulously crafted prompts irrelevant and moved many people from tab-complete to generating whole functions and modules

• Opus 4 with Claude Code made real agentic coding viable for weekend projects; Opus 4.5 made it viable for work. At that point many people from my bubble started writing 70-80% of code with it, which finished off any hope for AI tab-complete and “compact-your-codebase-into-one-prompt” tools

• GPT-5.4 kept track of a long-running list of tasks without the help of a plan mode, spec-driven development or a Ralph loop, making all three unnecessary. The only part of spec-driven that’s still useful is getting requirements from a user, which is better accomplished by the /grill-with-docs skill

• Fable 5 caught up in terms of not losing track of work and showed us that sufficiently smart models could make good decisions on their own, lowering our need for a variety of guardrails and checks. But it was still too expensive for most people to use as a driver, so “reviewer” workflows got popular: a cheaper model does the work and asks Fable for decisions

• GPT-5.6 Sol along with OpenAI’s outstanding computer/browser use made it possible to delegate much larger and human-y tasks to agents, making MCPs optional for most websites and apps. I have almost forgotten what dashboards for all my infra look like because these days agents are more capable than me at taking actions there

• Opus 5.5 fixed the prose issue of the Claude family and could make good decisions like Fable at a fraction of the cost, removing the need for the “reviewer” workflow and for factories when using frontier models (though arguably factories were not necessary even at GPT 5.6 Sol stage)

Notice the orange curves: the workflows either became popular or kept being popular after they got (mostly) useless for frontier models. And the time period between the introduction of the workflow and its death keeps shrinking.

These days a naive and concise prompt into Claude Code with Opus 5.5 will perform significantly better than every AI factory because factories needlessly constrain Opus. A default Opus 5.5 can make much better decisions by going beyond the constraints and rules ^[4] that factories commonly put on it since the written rules cannot foresee every situation and invariant.

The rate of change is so fast that most AI courses and books are outdated by the time they come out. So I advise all readers to instead focus on the short articles the labs publish when they release new models. Except for Gemini, of course: they are too far from the frontier to matter in this discussion.

Whenever you are trying out or building yet-another-really-modern-AI-workflow, think about the bitter lesson: your workflow is likely already outdated.

Originally published at https://zmievsa.org/posts/the-bitter-lesson-ate-your-ai-factory/ ^[5]