This is not a theoretical guide. It is the path we actually walk when a client asks for an agent, ordered by the steps we take — and the mistakes we paid for in the first agent we put into production in 2024.
Step 0: define the task in one sentence
If you cannot write what the agent does in one sentence with one verb and one output, you are not ready to build. «Reads an invoice and proposes a balanced entry for approval» is a valid sentence. «Helps the accounting department» is not.
Step 1: tools before the model
An agent without tools is a chatbot. Tools are what let it do something — and they are also its permission boundary. Our rule: every tool does one thing, with the narrowest permission that suffices, and returns a clear error when it fails.
// one tool, one permission, one output
const proposeEntry = tool({
name: 'propose_journal_entry',
description: 'Proposes a balanced journal entry from extracted invoice fields. Does not post.',
input: z.object({
vendor: z.string(),
total: z.number().positive(),
vat: z.number().min(0),
date: z.string().date(),
}),
// the entry stays a draft until a human approves it
execute: async (inv) => ledger.draft(balanceEntry(inv)),
});Note that the tool does not post. Posting is a decision with a financial effect, and every decision with a financial effect passes through a human. That is not hesitation about trusting the model; it is responsibility by design.
Step 2: choose the model by the task, not the fashion
There is no «best model». There is a best model for this task, at this cost, in this time. Extracting fields from an invoice does not need the largest model on the market; judging a legal contract might. Design the system so the model can be swapped without a rebuild — one interface, and a provider changed from a config file.
Step 3: memory is not free
Everything you put in the context costs money and time and distracts the model. Good memory is selective: what the agent needs for this step, not everything that ever happened. In the invoice agent, for instance, it needs the chart of accounts and the last five entries for that vendor — not the company's entire history.
Step 4: evaluate before release — always
This is where a product separates from an experiment. Before any user sees the agent, we build an evaluation set from real cases — a hundred actual invoices with their correct entries — and we measure: how many entries did the agent propose correctly? How many errors would have slipped through without a human review?
- The evaluation set comes from real data, not generated data
- A numeric success criterion written before the run (say: 95% correct entries, 0 unbalanced entries)
- The evaluation repeats on every change to the model or the instructions — like a regression test
Step 5: what breaks in production
Three things broke for us and did not show up in the evaluation:
- Ugly inputs: an invoice photographed at an angle, in handwriting, in two languages. The evaluation set was cleaner than reality. The fix: add the ugly cases to the set the moment they appear.
- Over-trust: the user approves the proposed entry without reading it because «the agent is usually right». The fix: highlight what deviates from the norm, and make approval require a look rather than a click.
- Cost: retry calls doubled the bill in the first month. The fix: a call budget per task, and an explicit stop instead of a retry loop.
In short
A production agent is not a clever model; it is a system: narrow-permission tools, selective memory, measured evaluation, a log of every decision, and a human at every financial effect. Whoever starts from the model ends with an experiment. Whoever starts from the task ends with a product.
