llm.

Anatomy of a Broken Benchmark Runner: How Seven AI Models Fixed (or Didn’t Fix) run-code.sh

IntroductionThere is a question quietly buried inside every task handed to a language model, and it rarely gets asked out loud. Do you trust one model? Do you go looking for the best model? Do you run several models side by side and keep whatever each of them happens to catch? Or — one turn further still — do you hand that whole pile of partial answers to yet another model and ask it to combine them into one? Each answer sounds reasonable on its own. Each is also, on its own, incomplete — and the only way to find out which incompleteness actually hurts you is to run the experiment rather than assume the answer.This article runs it, in four steps that build on each other.One model. A single broken bash script, with seven distinct bugs of varying severity, was handed to four models — Sonnet 5, HY3, Qwen3-Max, DeepSeek-V4-Flash — each working alone, each with no knowledge of what the others were doing. That's the first framing: one model, unaided, fixing only what it personally noticed.Best model.

продолжить чтение

• Оставлено в
  • *nix   
  • agentic ai   
  • ai   
  • bash   
  • benchmark   
  • code review   
  • coding   
  • Debugging   
  • linux   
  • llm   
  • openrouter   
  • искусственный интеллект   
  • Настройка Linux