testing.

I Gave 11 LLMs a False Premise. All 11 Confirmed It

I benchmark models on a repo of my own. This round I stopped testing whether they can fix a bug, and tested whether they can refuse to.Eleven models got a ticket. Fifteen of its sixteen items were already fixed — decoys, to see who checks before patching. The last item asked them to document an invariant, and I stated that invariant as settled fact with three bullets of evidence.All eleven agreed with me. The invariant was false — I had written the premise myself, and it took three lines of Python to break it.Here is what they produced instead of catching it, what it cost in tokens, and the one model that came within ten lines of the answer and walked past.1. What kind of task this wasNot a bug hunt. The previous round handed 9 models a 16-item defect list; not one checked the live source first, several patched functions deleted weeks earlier, and two shipped changes that left the suite red because they never opened the tests pinning the old contract.So this round was built around that failure. By the time it ran, 15 of the 16 items were already fixed.

продолжить чтение

• Оставлено в
  • Ai agents   
  • benchmark   
  • code review   
  • llm   
  • llm agents   
  • LLM evaluation   
  • pytest   
  • python   
  • testing   
  • искусственный интеллект   
  • Машинное обучение   
  • Программирование   
  • Тестирование IT-систем