Short answer
Pick one repetitive task with a clear finished state, write down how it runs today and what it costs, then spend four weeks moving the agent from shadow to supervised to limited live, and decide. Measure against the baseline you wrote down, keep a person approving anything that leaves the building, and end the month with yes, no, or a fix list.
Key takeaways
- Choose the pilot task by volume and clarity. How impressive it would look is irrelevant.
- Write the baseline before day one: time per task, errors, and cost. Without it, day 30 is an opinion.
- Week one shadow, week two supervised, week three limited live, week four measure and decide.
- Anthropic’s own guidance is to find the simplest solution possible and test agents extensively in sandboxed environments with guardrails, because errors compound (vendor guidance, 2024).
Most agent projects I see fail in the first week, before the agent has done anything. Someone picks a task because it would make a good demo, nobody writes down how the work runs today, and a month later the only evidence is how people feel about it. A pilot is a small, boring experiment with a written starting point. Treat it that way and 30 days is enough to know.
Which task should an AI agent pilot start with?
A task that happens often, follows steps someone could write down, has a clear finished state, and is cheap to get wrong while you learn. Intake forms, follow-up drafts, appointment reminders and data entry between two systems all fit. Anything that moves money or makes a promise to a customer should wait for the second pilot.
| Test | Good first pilot | Bad first pilot |
|---|---|---|
| Volume | Happens daily or weekly | Happens a few times a quarter |
| Steps | Someone can write them down in a page | Depends on one person’s judgment |
| Finished state | Obvious: the form is filed, the reply is drafted | Vague: “the client feels looked after” |
| Cost of a mistake | A person catches it before it matters | Money moves or a customer is misled |
| Data | Already in systems the agent can read | Scattered across inboxes and memory |
If you have more than one candidate, there is a fuller method for choosing which work to delegate first. Pick one. Two pilots at once means neither gets watched properly.
What do you need in place before day one?
Four things, written down: an owner who reads the agent’s output every day, the steps of the task as a short SOP, a baseline of how long the work takes now and how often it goes wrong, and a stop rule that says what makes you pause the pilot. Without the baseline, you cannot tell whether it worked.
The baseline does not need to be precise. Time ten instances of the task this week, count the errors you already catch, and note what the person doing it costs per hour. Writing the SOP is usually where people discover the task had steps nobody knew about, which is worth finding out before the agent does. There is a separate guide to writing one an agent can follow.
- An ownerReads the output every dayWithout one, problems surface in week four
- A short SOPThe steps of the taskWriting it shows steps nobody knew about
- A baselineHow long the work takes nowAnd how often it goes wrong. Without it, you cannot tell if it worked
- A stop ruleWhat makes you pauseWritten down before the pilot starts
What happens in each week of a 30-day agent pilot?
Week one the agent shadows: it does the task alongside the person, and nothing it produces is used. Week two it is supervised: its output is used after a person approves each item. Week three it runs live on a limited slice with spot checks. Week four you measure against the baseline and decide.
| Week | The agent | The person | Move on when |
|---|---|---|---|
| 1. Shadow | Does every instance in parallel; output is not used | Does the task as usual, compares results daily | Most outputs match what the person did |
| 2. Supervised | Does the task; output goes out only after approval | Approves, edits or rejects each item, and logs why | Edits are small and getting rarer |
| 3. Limited live | Runs on a defined slice without approval | Spot-checks a sample daily; handles the rest | No serious errors in the slice |
| 4. Decide | Keeps running under week-three rules | Compares time, errors and cost to the baseline | You have written down yes, no or fix |
The log from week two is the most valuable thing the pilot produces. Every edit and rejection, with a one-line reason, tells you where the SOP was unclear and where the agent needs a rule. Most of those get fixed by changing the instructions rather than the tool. For what a person should approve after the pilot ends, see where approval belongs.
How do you know if the pilot worked?
Compare week four to the baseline on three numbers: time the person spends on the task, errors that got past review, and total cost including the agent and the review time. If time dropped, errors held steady or fell, and the cost is lower, say yes. If one failed, you have a fix list.
- TimeTime the person spendsShould have dropped
- ErrorsErrors that got past reviewShould have held steady or fallen
- CostTotal costIncludes the agent and the review time. Should be lower
Count the review time too. An agent that saves two hours of work and adds ninety minutes of checking has saved thirty minutes. That can still be worth it, but only if you know it. The measures to keep watching after day 30 are in the guide to telling whether an agent is working.
Is there a standard framework for managing AI risk in a pilot?
The closest thing to a standard is NIST’s AI Risk Management Framework, released January 26, 2023. It is voluntary and organized around four functions: Govern, Map, Measure and Manage. For a small pilot that means an owner and rules, a list of what could go wrong, a baseline, and a decision.
NIST added a profile for generative AI on July 26, 2024. You do not need to read either document to run a 30-day pilot. It helps to know the four-part shape exists, because it is the same shape a larger company’s compliance team will ask about if you ever sell to one. On the build side, Anthropic’s engineering guidance for agents makes the same point from the other direction: start with the simplest thing that works, because autonomy brings higher costs and compounding errors, and test in a sandbox with guardrails.
What makes a 30-day pilot fail?
- No baseline, so the result is an opinion.
- No owner reading the output daily, so problems surface in week four.
- A task picked for the demo instead of the volume.
- Skipping the supervised week because the shadow week looked good.
- Changing the task halfway through, which resets the comparison.
If nobody in the business can give the pilot thirty minutes a day for a month, wait until someone can. The agent will run either way. What makes it a pilot is a person watching it.
Questions people ask
Is 30 days long enough to pilot an AI agent?
For one repetitive task, yes. Four weeks gives you a shadow week, a supervised week, a limited live week and a week to measure. Tasks that happen only monthly will not produce enough instances, so pick something that runs daily or weekly for a first pilot.
What should I measure in an agent pilot?
Three numbers against a baseline taken before day one: time spent on the task, errors that got past review, and total cost including the agent and the review time. Keep a log of every edit or rejection in the supervised week, with a reason.
How many tasks should a first pilot cover?
One. Two pilots at once split the owner’s attention and blur the comparison. Once the first one has a yes and has run cleanly for a while, start the next.
When should the agent run without approval?
Only after a supervised stretch where edits became small and rare, and then only on a limited slice with spot checks. Anything that moves money or makes a promise to a customer should keep a person approving it after the pilot too.
What if the pilot fails?
Look at the week-two log first. Most failures trace to unclear instructions or missing rules, which you can fix and rerun. If the problem is the task itself, like judgment nobody can write down, pick a different task instead of a different tool.
Sources
- National Institute of Standards and Technology, AI Risk Management Framework (AI RMF 1.0, January 26, 2023; generative AI profile July 26, 2024): Release dates, voluntary status, and the four functions Govern, Map, Measure and Manage.
- Anthropic, “Building effective agents” (December 19, 2024, vendor guidance): Start with the simplest solution and add complexity only when needed; agents bring higher costs and compounding errors; test extensively in sandboxed environments with guardrails.
Published . Last reviewed .
Want this in your business?
Book a call and we will scope exactly which part of your operation an agent should take first.
Book a call with the team