The Core Idea: Agents Need an Environment, Not Just a Prompt
An agent is only as reliable as the loop around it. A useful loop has five parts:
- A specification precise enough to check.
- Context about the codebase, conventions and commands.
- Fast feedback: tests, type checks and linters the agent can run itself.
- Bounded permissions: a sandbox where mistakes are cheap.
- Human review before anything reaches production.
Skip any one and quality drops predictably: vague specs give plausible-looking wrong code; no tests mean nobody knows it is wrong; broad permissions turn mistakes into incidents; no review lets defects accumulate.
Step 1: Size Tasks So They Can Be Verified
Good agent tasks are small, self-contained and checkable. Compare:
| Poor task | Better task |
|---|---|
| Improve the search feature | Add a sortBy query parameter to GET /products accepting price and rating; add tests for both and for an invalid value (returns 400) |
| Fix the bug in checkout | Orders with a 100% discount code fail with a null pointer in PriceCalculator.total(); write a failing test first, then fix |
| Migrate to the new library | Replace OldClient with NewClient in the billing package only; keep behaviour identical; existing tests must pass |
A useful template for a task: goal, scope (files or modules that may change), acceptance criteria (tests that must pass or be added), and constraints (do not change public APIs, do not add dependencies). If you cannot state acceptance criteria, the task is not ready for an agent, and probably not for a human either.
Step 2: Write the Context File
Most coding agents read a repository-level instruction file (commonly AGENTS.md, or a tool-specific name such as CLAUDE.md). Put there what a new engineer would need on day one, concisely. Long files get ignored or contradict themselves; keep it under a couple of pages.
# AGENTS.md
## Project
Order service (Java 21, Spring Boot). Modules: api, domain, persistence.
## Commands
- Build and test: `./mvnw -q verify`
- Run a single test: `./mvnw -q -Dtest=PriceCalculatorTest test`
- Format: `./mvnw spotless:apply`
## Conventions
- Domain logic lives in `domain`; controllers stay thin.
- Use constructor injection; no field injection.
- Every public method has a unit test; bug fixes start with a failing test.
- Errors: return problem+json via `GlobalExceptionHandler`; never expose stack traces.
## Boundaries
- Do not modify `db/migration/*` files that are already merged; add a new migration.
- Do not add dependencies without a comment in the PR explaining why.
- Never read or write `.env` or anything under `secrets/`.
## Definition of done
- `./mvnw -q verify` passes with no new warnings.
- New behaviour has tests; the PR description lists what changed and why.Keep this file in version control and review changes to it like code. When an agent repeatedly makes the same mistake, add one line to the file rather than repeating the correction in every prompt.
Step 3: Make Tests the Guardrail
- Test first for bugs. Ask the agent to write a failing test that reproduces the bug, confirm it fails for the right reason, then fix it.
- Review the tests, not just the code. Agents sometimes make tests pass by weakening assertions or special-casing inputs. Treat deleted or loosened assertions as a red flag.
- Add property, contract and integration tests around boundaries where unit tests miss behaviour.
- Keep the suite fast. Agents iterate many times; a ten-minute suite makes the loop useless.
- Mutation testing on critical modules reveals tests that pass without checking anything.
Step 4: Sandbox and Limit Permissions
- Run agents in a container or ephemeral workspace with only the repository checked out.
- Give read-write access to the working branch, read-only elsewhere, and no production credentials.
- Restrict network access to package registries and approved hosts. Prompt injection through web pages, issues or dependencies is a real risk: an agent that reads untrusted text and can run commands or push code can be manipulated.
- Require approval for destructive actions (deleting branches, force pushes, running migrations, deploying).
- Log every command and file change for audit.
Step 5: Review Like It Is Someone Else\'s Code, Because It Is
Agent-produced diffs are often larger and more confident than their quality warrants. Keep review effective:
- Keep changes small. Ask for one logical change per pull request; reject sprawling diffs.
- Read the description first, then the tests, then the implementation.
- Check for the classic agent failures: invented APIs or methods that do not exist, silently swallowed errors, duplicated logic instead of reuse, outdated library usage, security shortcuts (disabled certificate checks, string-built SQL), and unrelated "cleanup" edits.
- Run it. Check out the branch or use a preview environment (preview environments per pull request) and exercise the feature.
- The human who merges owns the change. "The agent wrote it" is never an acceptable explanation in an incident review.
Step 6: Add CI Gates That Do Not Trust the Author
Whether a human or an agent wrote the code, the pipeline should enforce the same bar:
# .github/workflows/ci.yml
name: ci
on:
pull_request:
permissions:
contents: read
pull-requests: read
jobs:
verify:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: "21"
cache: maven
- name: Build, test, and check formatting
run: ./mvnw -q verify spotless:check
- name: Fail on new dependencies without review label
run: |
git fetch origin "${{ github.base_ref }}" --depth=1
if git diff --name-only "origin/${{ github.base_ref }}" | grep -E '(^|/)pom.xml$' >/dev/null; then
echo "pom.xml changed - requires the 'deps-reviewed' label"
gh pr view "${{ github.event.number }}" --json labels --jq '.labels[].name' | grep -qx deps-reviewed
fi
env:
GH_TOKEN: ${{ github.token }}Add secret scanning, dependency vulnerability checks, static analysis and licence checks. Protect the main branch so that the checks and at least one human approval are required.
Step 7: Measure What Changes
| Metric | Why it matters with agents |
|---|---|
| Lead time for changes | The headline benefit; confirm it improves end to end, not just coding time |
| Change failure rate | Early warning that speed is costing quality |
| Review time per PR and PR size | Growing diffs overload reviewers |
| Rework rate (changes reverted or fixed within a week) | Measures hidden defects |
| Cost per merged change (tokens, compute) | Agents can burn budget on loops; set per-task limits |
| Test coverage and mutation score trend | Guardrail health |
Compare the same team before and after, and prefer your own data to vendor claims. Self-reported productivity gains are often larger than measured ones, so measure delivery outcomes, not impressions.
Where Agents Fit Well, and Where They Do Not
- Good fits: bug reproduction and fixes with clear tests, boilerplate and scaffolding, refactors with strong test coverage, test generation, documentation, dependency upgrades, code explanation and exploration.
- Weaker fits: ambiguous product decisions, novel architecture, performance-critical or security-critical code without expert review, and areas with poor tests.
A Rollout Plan
- Week 1: pick one repository with decent tests; write the context file; run agents on five small, well-specified tasks and record results.
- Week 2 to 3: add the CI gates and PR template; agree review rules; add sandboxing.
- Week 4: review the metrics and failures; refine the context file; expand to a second repository.
Frequently Asked Questions
What makes a good task for a coding agent?
A good task is small, self-contained and checkable: it states the goal, the files that may change, acceptance criteria such as tests that must pass, and constraints like no new dependencies. If you cannot state how to verify it, it is not ready.
What goes in an AGENTS.md or CLAUDE.md file?
Project overview, build and test commands, coding conventions, boundaries on what must not be changed, and a definition of done. Keep it short and concrete, review changes to it like code, and add a line whenever an agent repeats a mistake.
How do I keep agent-written code from lowering quality?
Use tests as guardrails, review tests as carefully as code, keep pull requests small, run the same CI gates for agents as for humans, and have a human who owns each merge. Measure change failure rate and rework.
What are the security risks of coding agents?
Prompt injection through untrusted text such as issues, web pages or dependencies can manipulate an agent that can run commands. Run agents in a sandbox with no production credentials, limit network access, and require approval for destructive actions.
Do coding agents replace code review?
No. Agents often produce larger, more confident diffs, which makes review more important. The human who approves a merge is accountable for it.