Free · 4 checks · ~10 minutes
The AI Automation Readiness Checklist
These are the questions I ask before quoting any automation work. Most projects that fail did so because nobody asked them — not because the model was wrong.
A demo proves a model can do a task once, with clean input, while someone watches. Production asks whether it does that task a thousand times, on data nobody cleaned, while nobody is watching, without doing something expensive when it is confused. These four checks separate the two.
Score the process before you automate it
Not every manual process deserves automation, and the expensive mistake is automating the wrong one first. Score each candidate 1–5 on four axes:
| Axis | Score 5 when… | Score 1 when… |
|---|---|---|
| Frequency | Runs daily or more | Runs a few times a year |
| Rule clarity | You can write the rules down | “It depends, you just know” |
| Cost of error | A mistake is cheap and visible | A mistake is silent and regulatory |
| Input stability | Same shape every time | Every source formats differently |
The counterintuitive one: low cost of error should score high, not low. The best first automation is one where being wrong is obvious and cheap — that is what lets you ship it before you fully trust it. Save the high-stakes process for automation number three.
Total ≥16 → strong candidate · 11–15 → viable with guardrails · ≤10 → fix the process first; automating it just makes the mess faster.
Test the data before you trust the model
Most “the AI is hallucinating” reports are really data problems wearing a costume. Before any build, pull fifty real records — not curated examples — and check:
- Completeness. What percentage have every field the task needs? Under 80% and the agent will improvise for the rest.
- Consistency. Is the same fact written the same way twice? (“USA”, “U.S.”, “United States”.)
- Recency. How stale is the oldest record you would act on? An agent cannot tell that a 2019 price is not today's price.
- Ground truth. For fifty inputs, can you state the correct output? If not, you cannot evaluate the system, only feel good about it.
The test that predicts everything: take ten records where the right answer is genuinely ambiguous. If your team disagrees about those, the model will too — and you have found the part that needs a human in the loop, before you have paid to discover it.
Define the failure path before the happy path
The demo is the happy path. Production is the other four. Decide these in advance, in writing, because deciding them mid-incident goes badly:
- Unsure. When the model has low confidence, does it stop, ask, or guess? “Guess” is a decision — make it deliberately.
- Wrong. When it acts incorrectly, how do you find out? If the answer is “a customer tells us”, add the check now.
- Down. When the provider is unavailable, does work queue or vanish? Queued work is recoverable; dropped work is not.
- Expensive. What is the per-run and per-day cost ceiling, and what happens at the ceiling? An agent in a retry loop can spend a month's budget overnight.
- Compromised. If someone feeds it hostile input, what is the worst action it can take? Whatever that is, it should not be able to take it unreviewed.
The rule I hold to: anything with an irreversible side effect — money moving, an email sending, a record deleting — passes a validation gate before it executes. In regulated work that gate is often a human. Being slower than a demo is the point.
Set the kill criteria before you start
An automation with no stop condition absorbs budget indefinitely, because every month there is a plausible reason to keep tuning. Write these down on day one, when you are still objective:
- Accuracy floor. Below what measured accuracy is it not worth running? Measured on the ground truth from check 2, not on impressions.
- Cost ceiling. Above what cost per completed task does the manual process win?
- Time box. By what date must it beat the manual baseline? “Still improving” at month six usually means “not working”.
- Escape hatch. Can you switch back to manual in a day? If not, that is a dependency you took on without pricing it.
If you only do one thing
Do check 2. Pull fifty real records and try to write the correct answer for each. Teams that can do that ship working automations; teams that cannot are usually about to spend six months learning that their data, not their model, was the project.
What good looks like
A production agent I would sign off on has: a plan it can show you, a budget and timeout per run, tools with least-privilege scope, a validation gate before side effects, a persisted trace for every decision, an evaluation suite that runs on changes, and a documented answer for each of the five failure paths above. If any of those is missing, it is still a demo — a very good one, but a demo.
Want a read on your specific case?
Send me the process you are considering and roughly what your data looks like. I will tell you which of these four it fails, and whether it is worth building — including when the answer is that it isn't.
Muhammad Waqas — AI Agent Engineer & Technical Lead · waqas-2hs.pages.dev