“Let's add AI to the product” is a choice of tool, not a goal. Before picking the tool you need to define the job. We ask these five questions at the start of every project. None of the answers needs technical knowledge, only a close knowledge of the job.
1. Who does this task today, and how long does it take?
You can't prove you improved something you never measured. If a support agent takes four minutes to answer a question, those four minutes are the target.
Measuring needs no special tool. Ask the people doing the task to keep a simple tally for a week: how many requests came in, how many minutes each took, which were repeats. The tally shows two things: the time you could win back, and how much of the work really follows a rule. If only a small share of requests repeat, AI may be the wrong tool; a better form or an FAQ page might do.
2. What happens when the model is wrong?
A bad product recommendation is annoying; a wrong invoice amount costs money. The higher the cost of an error, the more a human-approval step becomes mandatory.
- Low-cost errors: an irrelevant product suggestion, a slightly odd summary, a mislabelled support ticket. Someone notices, fixes it and work carries on.
- High-cost errors: a wrong invoice amount, a promised refund the customer isn't entitled to, a date misread from a contract, a confident answer on a health or legal matter.
In the first group the model can work alone. In the second it drafts and a person has the final say. Cases in between need a rule too: when unsure, the model should hand the conversation to a person instead of answering.
3. Which data holds the right answer?
The model doesn't know your return policy. Where are the documents an answer should rest on, how current are they, and who updates them? On most projects the real work is putting that data in order. The information usually sits in four places:
- Website pages and PDF files: public, but often out of date.
- Files in shared folders: several versions of the same document sit side by side.
- Old support conversations: real questions, but also wrong or outdated answers.
- People's heads: the most accurate information, written down nowhere.
Where does the data go?
Decide in writing, at the start of the project, which data goes to which system. Sensitive fields such as ID numbers, card details or health records should be masked before they reach the model.
4. How will you measure success?
Before launch, assemble a test set of a hundred real examples. Run the same set after every change. “It feels better” is not a measurement. A good test set has four traits:
- The questions are real, not invented: taken from support records or customer emails.
- Each question comes with the expected answer, written by someone who knows the job.
- Hard cases sit alongside easy ones: questions with missing details, double meanings or typos.
- Questions the model shouldn't answer belong in the set too; the right behaviour there is to say “I don't know” or hand off.
Boil the results down to one tally: of a hundred examples, how many were right, wrong or handed off. That tally stays useful after launch, when you sample real conversations at regular intervals and judge them the same way.
5. What will it cost per month?
Model costs grow with usage. Work out cost per message and expected volume up front, and route simple questions to a smaller model if needed. Four things drive the cost:
- Volume: how many questions or documents will be processed each month?
- Text length: the longer the text sent to and returned by the model, the higher the fee. A ten-page contract costs far more than a one-line question.
- Model size: more capable models cost more, and not every task needs one.
- Number of steps: if one answer takes several calls to the model, the cost multiplies.
Set these four against the tally from the first question. If the monthly cost is below the value of the time saved, the project makes sense; if not, narrow the scope.
First steps once the answers are clear
- 1
Start with one narrow task: not “all of support” but, say, only shipping and returns questions.
- 2
For the first weeks, have a person approve every answer. The model writes the draft; the agent sends it.
- 3
Log every answer: what was asked, what the model said, what the person changed.
- 4
After a few weeks, review the logs and the test set together. If errors are few, relax the approval step or widen the scope; if many, fix the data first.
This order may look slow, but it builds trust: the team sees for itself where the model is strong and where it is weak.
When all five answers are clear, picking the model is usually an afternoon's work.
If one of the answers is missing, complete it first. AI speeds up a well-defined task; with a vague one it only produces mistakes faster.
