Chapter 2: Adoption and measurement
Most teams adopt AI tools by accident: someone installs one, others follow, and six months later the team has eight workflows, no shared rules and a review queue full of large diffs. This chapter is the deliberate version - three phases, each with artefacts - and the measurement that tells you whether it is working.
Three phases
| Phase | Duration | Goal | Artefacts |
|---|---|---|---|
| 1. Explore | 2-4 weeks | Pick one or two tools on real tasks | Evaluation report, tool decision |
| 2. Standardise | 4-8 weeks | One written, enforced workflow | Rules file, review norms, policy, first eval gate, spec template |
| 3. Scale | Ongoing | Survive turnover; measure; improve | Metrics dashboard, quarterly review, onboarding path |
Phase 1: Explore (two to four weeks)
Goal. Find out what the tools do on your code, with your engineers, and pick one or two.
How. The evaluation from Module 3: five real tasks, a shared spec for each, rotated across engineers and tools. Keep a log of every correction made in review - this becomes the first draft of the rules file.
Artefacts. A two-page evaluation report (what worked, what did not, what rules were needed) and a tool decision. Resist the urge to skip this phase; the report is what you will point at when someone asks "why this tool?" a year from now.
Anti-pattern. Exploring forever. Set the end date at the start.
Phase 2: Standardise (four to eight weeks)
Goal. One workflow, written down, that everyone on the team follows.
How.
- Rules file (Module 3), committed to each repository. First draft from the exploration log; iterated weekly.
- Review norms. Diff size limit. Tests reviewed before implementation. Comments reference
lines. "LGTM" without a comment is not a review. Written in
CONTRIBUTING.md. - The one-page policy (Module 5) and the configuration that enforces it.
- First evaluation gate (Module 4) if the team ships any model-backed feature; otherwise, the test-first contract from Module 2 as the gate.
- One shared spec template in the repository.
Artefacts. The five items above, in the repository, reviewed and merged like any other change.
Anti-pattern. Standardising by memo. If it is not in the repository and enforced by configuration, it is not a standard.
Phase 3: Scale (ongoing)
Goal. New team members onboard in a day; the practice survives turnover; measurement drives the next change.
How.
- Onboarding path. Read the rules file together. Run the loop on a real ticket with a pair. Review the first five diffs out loud. Module 6 Chapter 1.
- Quarterly review. Rules file: delete what no longer prevents anything. Policy: same. Metrics: what moved, and why.
- Cross-team sharing. Rules files, spec templates and eval harnesses are copyable. Copy them.
Artefacts. A dashboard (below), a quarterly review document, and an onboarding checklist.
What to measure
The purpose of measurement is to answer one question: is the team delivering more value, safely, than before? Measure delivery, not tool usage.
Start from DORA
The four DORA metrics are the right baseline because they measure outcomes and because many teams already track them:
| Metric | What AI assistance should do to it | What to watch for |
|---|---|---|
| Deployment frequency | Rise | Rising frequency with rising failure rate means you are shipping faster and worse |
| Lead time for changes | Fall | Falling lead time with rising review time means generation got faster and review became the bottleneck |
| Change failure rate | Stay flat or fall | The single most important guard. If it rises, stop and look at review norms |
| Time to restore | Stay flat or fall | Should improve if assistants help with diagnosis; watch for the opposite if nobody understands the code |
Add three AI-specific signals
| Signal | How to get it | Why it matters |
|---|---|---|
| Review findings per 100 lines on AI-assisted PRs vs. others | Label PRs (most tools add a trailer); count review comments | The direct measure of whether generated code is arriving in reviewable shape. Should fall as the rules file matures |
| Spec-first ratio | Fraction of PRs that link a spec file or whose tests were committed before the implementation | Measures whether the workflow is actually followed, not just the tool used |
| Eval gate pass rate and score trend (for model-backed features) | From CI | Whether the feature is getting better or worse |
What not to measure
- Acceptance rate of suggestions. It measures how often people press Tab, not whether what they accepted was good. Vendors love it; it tells you nothing about outcomes.
- Lines of code generated. More lines is not more value, and it rewards exactly the large diffs you are trying to prevent.
- Hours "saved". Self-reported, unverifiable, and it makes engineers feel surveilled. If lead time fell, you saved hours; measure that.
- Individual developer productivity. Measure the team. The moment individuals are ranked on AI metrics, the metrics get gamed and the trust is gone.
Talking about the results
Three audiences, three framings:
- To the team: the review-findings trend and the change failure rate. "We are shipping 40 % more and failure rate is flat. Review findings per 100 lines are down from 6 to 2 since the rules file. Here is what we are changing next quarter."
- To leadership: DORA metrics before and after, and cost. "Lead time from 5 days to 2. No change in failure rate. Tool cost is X per engineer per month; the equivalent lead-time improvement by hiring would cost Y."
- To security and compliance: the policy, the enforcement configuration and the audit trail (Module 5). "Here is what is allowed, here is how it is enforced, here is the evidence."
Three objections and honest answers
"It will make our engineers worse." It can, if they stop reading what they merge. The workflow in this course - spec first, tests first, verification as the job - is designed so that the engineer's judgement is exercised more, not less. Measure review findings; if they climb, the objection is right and the fix is review norms, not removing the tool.
"It will leak our code / our customers' data." It can, if you use consumer tiers with training on inputs and no exclusions. Enterprise tiers with zero retention, excluded secret files, and a data-classification rule address the realistic version of this risk. Module 5 Chapter 1.
"The code will be unmaintainable." It will, if ten engineers generate in ten styles. The rules file exists to prevent exactly this. A team with a good rules file produces more consistent code with assistants than without, because the conventions are applied on every change rather than remembered on some.
None of these objections is wrong. Each describes what happens without the workflow. The answer to each is the workflow.
Summary
- Three phases: explore (pick), standardise (write it down and enforce it), scale (onboard, review, measure).
- Measure delivery outcomes - DORA plus review findings, spec-first ratio and eval scores - not tool usage.
- Never measure individuals on AI metrics; never report acceptance rate as a success.
- The three standard objections are correct descriptions of adoption without a workflow. The workflow is the answer.