Chapter 2: From spec to tests to code

Test-driven development was always a good idea that most teams did not have time for. With an assistant, the economics flip: the tests are cheap to write and they are the only thing standing between you and a confident, plausible, wrong implementation. This chapter is the test-first contract, adapted for a world where the code is generated.

Why the order matters more now

When a human writes code and then tests, the tests are biased by the implementation: you test what you built. When an assistant writes code and then tests, the bias is worse - the model will write tests that pass against the code it just produced, including tests that enshrine its mistakes.

Tests written before the implementation, from the specification, are independent of it. They are the executable form of the spec, and they let you use the strongest prompt available:

Make the failing tests in tests/test_reset.py pass. Do not modify the tests.

That sentence replaces a page of prose, and it is checkable: the tests either pass or they do not, and the test file either changed or it did not.

The contract

Four rules:

1. Tests come from the spec, not from the code. One acceptance criterion, one test. One example, one assertion.
2. Tests must fail first, for the right reason. A test that fails with ImportError because the module does not exist yet is fine. A test that fails because of a typo in the test is not - fix it before generating anything.
3. The implementation step may not touch the tests. Say it in the prompt. Verify it in the diff. If the assistant argues that a test is wrong, it might be right - but the decision is yours, made by editing the spec and then the test, not by letting the implementation step rewrite its own exam.
4. A passing suite is necessary, not sufficient. Review the diff against the spec anyway (Module 1, verification layers 3-6).

Who writes the tests?

The assistant can, and should, write most of them - in a separate step from the implementation.

Step A - tests. Give the assistant the spec and ask for the tests only:

Here is the specification for the password-reset flow. Write pytest tests covering every acceptance criterion and every example. Do not implement anything. Each test name should say which criterion it covers.

Review the tests as carefully as you would review production code. Check that each criterion is covered, that unhappy paths are present, that assertions are specific (assert response.status_code == 404, not assert response.status_code != 200). Run them: they must fail.

Step B - implementation. New conversation, or at least a clear break:

Make the failing tests in tests/auth/test_reset.py pass. Constraints: [paste the constraints block from the spec]. Do not modify the tests.

Separating the steps matters. A model asked to write tests and implementation together will make them agree with each other; a model asked for tests first has only the spec to agree with.

Tests that pull their weight

Not all tests are equal guardrails. In rough order of value against generated code:

Test type	What it catches	Cost
Behavioural / integration - call the endpoint, check the response	Wrong behaviour, missing auth, wrong status codes, fabricated APIs that blow up at runtime	Medium
Example-based unit tests from the spec's examples	Edge cases the model skipped	Low
Property-based tests (Hypothesis, fast-check)	Whole classes of input the examples did not cover	Medium
Characterisation tests of existing behaviour before a refactor	The assistant "improving" something that was load-bearing	Low to write with an assistant, high value
Snapshot tests of large outputs	Unintended changes to rendered output	Low, but noisy
Mock-heavy unit tests that assert on internal calls	Mostly the implementation you just wrote; brittle; low value against generated code	Low cost, low value

A generated test suite will skew towards the last row unless you steer it. Ask for behavioural tests first.

When the spec is wrong

It will be. Half the value of writing tests first is discovering, before any code exists, that two acceptance criteria contradict each other or that an example is impossible. When that happens:

1. Stop generating. Do not let the assistant "resolve" the contradiction by picking one.
2. Fix the spec. Edit the written specification. This is a decision, and decisions are yours.
3. Fix the tests to match. Then re-run: they should fail again, for the right reason.
4. Resume.

The failure mode to avoid is the moving target from Chapter 1: resolving the contradiction in the chat, so that the spec file, the tests and the code all end up slightly different.

Keeping the spec alive

A spec that is thrown away after merge was documentation. A spec that is kept is a contract. Three light-weight habits:

• Spec lives next to the code. docs/specs/password-reset.md in the repository, not in a ticket that will be archived. Reference it from the PR description.
• Tests reference the spec. A test name or docstring that names the criterion (test_unregistered_email_returns_identical_response) makes the link visible.
• Changing behaviour means changing the spec first. Same loop, next iteration. The spec file's history is the history of what the system is supposed to do.

Some teams go further and make specs the primary artefact, with code treated as a derived output that can be regenerated. Whether or not you go that far, the principle holds: the spec is the thing you argue about, the tests are the thing that settles the argument, and the code is the thing that happens to satisfy both.

Worked example, continued

Continuing the password-reset flow from Chapter 1.

Step A. The assistant produced nine tests. Review found: the timing test used a single run (fixed to 100 runs with a tolerance); the "already-used token" test asserted 400 but not the token_invalid code (added); no test for an empty email body (added, 422). Ran: 10 failures, all ImportError - correct.

Step B. New session. Prompt as above. First attempt: 9 passing, 1 failing - the timing test. The assistant had taken the early return on unregistered email. Second prompt: "The timing test fails because the unregistered branch returns early. Both branches must do equivalent work; see the spec." Second attempt: 10 passing, test file unchanged (verified with git diff --stat tests/).

Review. Diff: two routes, one helper, 84 lines. Reuses mailer.send() and auth_tokens - matches constraints. One finding: the token comparison used ==; changed to secrets.compare_digest. That was a spec gap (constant-time comparison was not stated); added it to the constraints block for next time.

Merge. Spec committed at docs/specs/password-reset.md, linked from the PR.

Summary

• Write the tests from the spec, before the code; make sure they fail for the right reason.
• Separate the test-writing step from the implementation step so the model cannot grade itself.
• "Make the failing tests pass; do not modify the tests" is the strongest prompt you have.
• When the spec is wrong, fix the spec, then the tests, then resume - never resolve it in chat.
• Keep the spec in the repository; it is the contract.