I Gave an AI the Same 20 Coding Tasks With Short and Detailed Prompts — More Context Didn’t Always Produce Better Code

There is a habit I picked up after using AI for coding for a while. Whenever the generated code was not quite right, I added more context. Then a little more. Input formats, edge cases, performance requirements, error handling, preferred architecture, things the function should not do. Eventually a two-line request could turn into a small technical specification.

It feels logical. A developer cannot read your mind, so why should an AI model be able to? More information should remove ambiguity and give better code. But after a few cases where a detailed prompt produced something strangely overengineered, I started wondering whether this assumption was actually true for small programming tasks.

So I made a small experiment. I prepared 20 coding problems and sent every problem to the same model twice. The first version was intentionally short. The second explained almost everything I could reasonably explain without giving away the solution. There were 40 generated solutions in total. No follow-up messages, no asking the model to fix a failing test, no manually repairing imports. The first answer was the answer.

The result was less clean than I expected. Detailed prompts were slightly better overall, but on several tasks the extra information made the generated code worse. And those failures were probably the most interesting part of the experiment.

How I set up the experiment

I wanted this to look more like normal programming than an artificial benchmark. The tasks were small enough for one file, but not just LeetCode-style algorithms. Some required state, some had annoying edge cases, some dealt with text or collections, and a few involved asynchronous code.

Everything was written for Python 3.12. Each generated solution was placed in a clean directory and tested using pytest. The model did not see the tests. For every task I prepared two prompts. The short version was usually between about 20 and 40 words. It described the required behavior and little else. The detailed version was normally several times longer and included input assumptions, edge cases, restrictions, expected complexity, error behavior, and implementation details that would be completely reasonable to include in a real ticket.

The model and generation settings remained the same. Every prompt started in a fresh context. That was important because I did not want one solution influencing the next one. I tested 20 problems: TTL cache, LRU cache, interval merging, log parser, CSV deduplication, recursive dictionary merge, retry helper, bounded async mapper, sliding-window rate limiter, file extension counter, URL normalizer, event deduplicator, batch generator, dependency resolver, tree flattener, simple scheduler, configuration merger, pagination helper, token bucket and JSON diff.

That mix is obviously not enough to say anything universal about AI coding. Twenty tasks are twenty tasks. But it was enough to expose a pattern I had already noticed during normal development. For scoring I cared mainly about whether all tests passed. I also recorded the number of failed tests, source lines, imports and whether the solution introduced components that the task never really needed.

The test runner

I did not want to judge code by looking at it and deciding that one version felt cleaner. That becomes subjective very quickly. A surprisingly ugly function may be correct, while a beautiful abstraction can fail on the third edge case.

So most of the scoring was automated. Each task directory contained the hidden tests plus two generated implementations. The runner copied one implementation into the expected module location, executed pytest in a subprocess, collected the result and then repeated the process for the other version. The runner itself was deliberately boring. Boring test infrastructure is usually a good sign.

from __future__ import annotations

import json
import shutil
import subprocess
import sys
import time
from dataclasses import asdict, dataclass
from pathlib import Path


ROOT = Path(__file__).parent
TASKS_DIR = ROOT / "tasks"
RESULTS_FILE = ROOT / "results.json"


@dataclass
class Result:
    task: str
    variant: str
    passed: bool
    duration_ms: float
    source_lines: int
    return_code: int
    output: str


def count_source_lines(path: Path) -> int:
    count = 0

    for line in path.read_text(encoding="utf-8").splitlines():
        stripped = line.strip()

        if not stripped:
            continue

        if stripped.startswith("#"):
            continue

        count += 1

    return count


def run_solution(task_dir: Path, variant: str) -> Result:
    generated_file = task_dir / f"solution_{variant}.py"
    active_file = task_dir / "solution.py"

    if not generated_file.exists():
        raise FileNotFoundError(generated_file)

    shutil.copyfile(generated_file, active_file)

    started = time.perf_counter()

    process = subprocess.run(
        [
            sys.executable,
            "-m",
            "pytest",
            "test_solution.py",
            "-q",
        ],
        cwd=task_dir,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
        timeout=30,
    )

    duration_ms = (time.perf_counter() - started) * 1000

    result = Result(
        task=task_dir.name,
        variant=variant,
        passed=process.returncode == 0,
        duration_ms=round(duration_ms, 2),
        source_lines=count_source_lines(generated_file),
        return_code=process.returncode,
        output=process.stdout[-4000:],
    )

    active_file.unlink(missing_ok=True)

    return result


def discover_tasks() -> list[Path]:
    return sorted(
        path
        for path in TASKS_DIR.iterdir()
        if path.is_dir()
        and (path / "test_solution.py").exists()
    )


def print_summary(results: list[Result]) -> None:
    for variant in ("short", "detailed"):
        current = [r for r in results if r.variant == variant]

        passed = sum(r.passed for r in current)
        lines = sum(r.source_lines for r in current)

        average_lines = lines / len(current)

        print(
            f"{variant:8} "
            f"passed={passed}/{len(current)} "
            f"avg_lines={average_lines:.1f}"
        )

    print()

    by_task: dict[str, dict[str, Result]] = {}

    for result in results:
        by_task.setdefault(result.task, {})[result.variant] = result

    for task, variants in by_task.items():
        short = variants["short"]
        detailed = variants["detailed"]

        if short.passed != detailed.passed:
            winner = "short" if short.passed else "detailed"
            print(f"{task}: {winner} only")


def main() -> None:
    results: list[Result] = []

    for task_dir in discover_tasks():
        print(f"Running {task_dir.name}")

        for variant in ("short", "detailed"):
            result = run_solution(task_dir, variant)
            results.append(result)

    RESULTS_FILE.write_text(
        json.dumps(
            [asdict(result) for result in results],
            indent=2,
        ),
        encoding="utf-8",
    )

    print()
    print_summary(results)


if __name__ == "__main__":
    main()

There is nothing AI-specific in this runner. That was intentional. Once the code had been generated, I wanted to treat it exactly like code written by anyone else: put it in a repository and see whether it survives the tests.

It also removed one tempting mistake from the experiment. When AI-generated code looks almost correct, it is very easy to fix one line mentally and count the answer as successful. I did not do that. If one character made a hidden test fail, the attempt failed.

The result was much closer than expected

The short prompts passed all tests on 15 of the 20 tasks. The detailed prompts passed 16. At first glance, that seems to confirm the obvious answer. More information helped. But the individual results were more interesting than 15 versus 16. Both versions passed 12 tasks. There were four tasks where the detailed prompt produced a passing solution and the short prompt failed. But there were also three tasks where the opposite happened: the short prompt passed everything while the detailed version introduced a bug.

One task defeated both versions. So after making the prompts several times longer, adding constraints and carefully describing edge cases, the final improvement was one additional successful task out of twenty. That is not evidence that short prompts are better. It is not even a large enough experiment to prove that detailed prompts are slightly better. With only twenty paired tasks, a difference of one success would be silly to treat as some universal rule.

The useful part was figuring out why the outcomes changed. The detailed prompts helped when important behavior was genuinely missing from the short specification. They were especially useful for ambiguous boundary conditions. If the prompt explicitly said what should happen to duplicate timestamps, empty input, malformed data or expired entries, the generated implementation was less likely to guess incorrectly. But extra detail also changed the shape of the generated program. That turned out to matter more than I expected.

More instructions often produced more software than I asked for

The short solutions were usually boring. One function, maybe one helper, a dictionary or deque, return the result. The detailed prompts encouraged the model to construct something closer to a miniature library. There were additional classes, custom exceptions, type aliases, helper methods and defensive checks. Some of them were useful. Others created new places for bugs to hide.

Across this tiny sample, the detailed solutions were noticeably longer. That alone is not bad. A 90-line implementation can be far better than a 40-line implementation if those extra 50 lines are doing necessary work. The problem was that some of the extra code existed only because the prompt mentioned concepts rather than because the solution needed them.

Mention thread safety and a lock appears. Mention extensibility and suddenly there is an abstract base class. Mention production use and validation begins happening at several layers. Mention performance and the model may replace an obvious linear operation with a more complicated data structure before checking whether the data structure changes the semantics.That behavior sounds strangely familiar, because humans do exactly the same thing. A specification can tell a developer what matters, but it can also accidentally suggest what the architecture should look like. The model appeared to react to both.

One TTL cache made the problem obvious

The TTL cache task was one of the clearest examples. The short prompt asked for a small in-memory cache with set and get operations. Every key had a TTL. Reading an expired value had to behave as if the key did not exist. The detailed version additionally mentioned thread safety, bounded capacity, lazy cleanup, monotonic time, O(1) average access, updating existing keys, and deterministic eviction.

None of those requirements are absurd. In fact, if this were going into a real service, several would be useful. The short answer produced a tiny dictionary-backed implementation. It passed the tests used for the basic task. The detailed answer produced something much more ambitious based on OrderedDict and a lock. Most tests passed too, but an interaction between expiration and capacity caused it to evict a live entry while expired entries were still occupying capacity.

The interesting part was not that AI made a mistake. That happens. The interesting part was that the mistake only existed because the richer prompt pushed the implementation toward a more complex design. Here is a reduced version of the task and the kind of implementation I ended up testing around that failure.

from __future__ import annotations

from collections import OrderedDict
from dataclasses import dataclass
from threading import RLock
from time import monotonic
from typing import Generic, TypeVar


K = TypeVar("K")
V = TypeVar("V")


@dataclass
class Entry(Generic[V]):
    value: V
    expires_at: float


class TTLCache(Generic[K, V]):
    def __init__(self, capacity: int) -> None:
        if capacity <= 0:
            raise ValueError("capacity must be positive")

        self._capacity = capacity
        self._items: OrderedDict[K, Entry[V]] = OrderedDict()
        self._lock = RLock()

    def set(self, key: K, value: V, ttl: float) -> None:
        if ttl < 0:
            raise ValueError("ttl cannot be negative")

        now = monotonic()
        expires_at = now + ttl

        with self._lock:
            if key in self._items:
                del self._items[key]

            self._items[key] = Entry(
                value=value,
                expires_at=expires_at,
            )

            self._items.move_to_end(key)

            # This looks reasonable, but it can evict a live entry
            # even when older expired entries still exist elsewhere
            # in the dictionary.
            while len(self._items) > self._capacity:
                self._items.popitem(last=False)

    def get(self, key: K) -> V | None:
        now = monotonic()

        with self._lock:
            entry = self._items.get(key)

            if entry is None:
                return None

            if entry.expires_at <= now:
                del self._items[key]
                return None

            self._items.move_to_end(key)
            return entry.value

    def __len__(self) -> int:
        now = monotonic()

        with self._lock:
            expired = [
                key
                for key, entry in self._items.items()
                if entry.expires_at <= now
            ]

            for key in expired:
                del self._items[key]

            return len(self._items)


def test_expired_entries_should_not_force_live_eviction(monkeypatch):
    import solution

    current_time = 1000.0

    def fake_monotonic():
        return current_time

    monkeypatch.setattr(solution, "monotonic", fake_monotonic)

    cache = solution.TTLCache[str, int](capacity=2)

    cache.set("old", 1, ttl=1)
    cache.set("live", 2, ttl=100)

    current_time = 1002.0

    # old is expired, so logically only one live entry remains.
    # Inserting new should not evict live.
    cache.set("new", 3, ttl=100)

    assert cache.get("old") is None
    assert cache.get("live") == 2
    assert cache.get("new") == 3


def test_get_removes_expired_entry(monkeypatch):
    import solution

    current_time = 2000.0

    def fake_monotonic():
        return current_time

    monkeypatch.setattr(solution, "monotonic", fake_monotonic)

    cache = solution.TTLCache[str, str](capacity=10)

    cache.set("session", "abc", ttl=5)

    assert cache.get("session") == "abc"

    current_time = 2006.0

    assert cache.get("session") is None
    assert len(cache) == 0


def test_update_refreshes_position_and_ttl(monkeypatch):
    import solution

    current_time = 3000.0

    def fake_monotonic():
        return current_time

    monkeypatch.setattr(solution, "monotonic", fake_monotonic)

    cache = solution.TTLCache[str, int](capacity=2)

    cache.set("a", 1, ttl=10)
    cache.set("b", 2, ttl=10)

    current_time = 3005.0

    cache.set("a", 10, ttl=20)
    cache.set("c", 3, ttl=20)

    assert cache.get("a") == 10
    assert cache.get("b") is None
    assert cache.get("c") == 3

The fix is not difficult. Before capacity eviction, expired entries need to be removed, or the data structure has to track expiration separately. What interested me was how we got there. The short problem did not contain capacity at all, so this entire bug class could not exist in the short solution. The detailed specification made the implementation more useful, but it simultaneously increased the state space that needed to be correct. That sounds obvious when written down. It was less obvious when I was adding more and more requirements to prompts because more detail felt automatically safer.

Detail helped most when it removed ambiguity

There were several tasks where the opposite happened. The URL normalizer was one. The short prompt simply asked the model to normalize URLs for deduplication. That leaves a surprising amount of room for interpretation. Should fragments disappear? Should default ports disappear? Is a trailing slash significant? Should query parameters be reordered? Should percent-encoded characters be decoded?

The short solution made reasonable guesses. Some were different from the behavior expected by the tests. The detailed prompt explicitly defined those rules and the model passed. The same thing happened with interval merging. The phrase overlapping intervals sounds simple until two intervals touch at an endpoint. Should [1, 3] and [3, 5] merge? It depends entirely on what the intervals represent. Once the detailed prompt explicitly defined the boundary behavior, the implementation became correct.

That gave me a more useful distinction than short versus long. Information that removes semantic ambiguity is valuable. Information that merely describes how serious, robust or production-ready the code should be is much less predictable.Tell the model exactly what an empty list means and you probably helped it. Tell it to produce a scalable, maintainable, extensible implementation and you may have just purchased 80 extra lines of code.

Some constraints competed with each other

Another pattern appeared when the detailed prompt contained several individually sensible requirements. One task asked for bounded asynchronous mapping. The function had to process work concurrently, preserve input order, never run more than N workers at once, propagate exceptions and avoid creating a task for every input item.

That is a much better specification than simply asking for concurrent mapping. It is also a more difficult programming problem. One generated answer satisfied the concurrency limit and preserved order, but could stall after a worker exception because the queue shutdown path assumed that every scheduled item would eventually call task_done.

The short version used asyncio.gather with a semaphore. It was less memory-efficient because it created all coroutine tasks up front, but it passed the functional tests. Which one is better?

That depends on the actual problem. The detailed implementation was trying to satisfy a requirement that the short solution completely ignored. Calling the short code better would therefore be misleading. At the same time, if the detailed implementation deadlocks under an exception, its architectural ambition does not make it usable.

This became a recurring theme. Detailed prompts did not merely provide additional hints for solving the same problem. Sometimes they quietly transformed a small problem into a harder one. That sounds like a trivial distinction, but it changed how I interpreted the benchmark.

Long prompts also make failures harder to diagnose

There is another practical cost that pass/fail statistics do not show. When a 25-word prompt produces incorrect code, the debugging conversation is straightforward. The requirement is probably missing, misunderstood, or the generated code simply contains a bug. When a 250-word prompt fails, there are many more possible causes.

Was one requirement buried between less important constraints? Did two requirements conflict? Did an example imply behavior that the text contradicted? Did the model over-focus on performance? Did asking for clean architecture encourage unnecessary abstractions? At some point the prompt itself becomes another program. It has inputs, implicit priorities, dependencies and edge cases. Unlike normal code, we do not have a compiler telling us that two sentences disagree with each other.

This is why adding another paragraph after every failed AI response started to feel wrong to me. The prompt became longer, but not necessarily clearer. A shorter rewrite was occasionally much more effective than an additional explanation.

What I changed in my own coding prompts

After this test, I stopped trying to write a complete specification before every AI coding request. For small isolated tasks, the first prompt is now usually compact. I describe the behavior, the important input and output types, and any constraint that would change the solution fundamentally. Then I run the code.

If it fails because the model guessed a missing semantic rule incorrectly, I add that rule. If it fails because the implementation itself is broken, I would rather show the failing test than add another paragraph of prose. Tests turned out to be a much cleaner language for many constraints.

Instead of writing a long explanation about what should happen when a TTL expires exactly at the current timestamp, one assertion can define it precisely. The same applies to duplicate records, empty inputs, ordering, exceptions, Unicode and boundary values. For larger codebase work this changes again. If a model is modifying existing architecture, context about surrounding code can be much more valuable than prose. Showing the interface, tests and neighboring implementation often communicates more than describing all of them. So the lesson I took from the experiment is not to use short prompts. It is to stop measuring prompt quality by prompt length.

More context is useful only when it contains useful information

The most surprising result was probably how unsurprising everything looked after the experiment was finished. Of course a precise rule about duplicate timestamps can improve an implementation. Of course adding five new requirements makes the programming problem harder. Of course more branches and more state create additional opportunities for bugs.

Yet when working with an AI assistant, it is strangely easy to collapse all of those things into one idea: more context should produce better output. My 20 tasks did not support such a simple rule. The detailed prompts won 16 tasks and the short prompts won 15. More importantly, each approach had cases where the other one succeeded. The best detailed prompts were not good because they were long. They were good because they removed ambiguity that mattered to the tests.

The worst detailed prompts were long because I had kept adding desirable properties until a small utility function had become a tiny systems-design exercise. Now, before adding another paragraph to a coding prompt, I ask a simpler question: does this information reduce ambiguity, or does it just make the request sound more complete? Those are not the same thing.

And if the model already has the function signature, neighboring code and a good test suite, another 200 words may be the least useful context I can give it.