Before adding AI, give the pilot a useful boundary

Choose one decision or task

‘Add AI’ describes a technology preference, not an outcome. A stronger starting point is one recurring task: find the right internal document, draft a response for a person to review, or organize a set of incoming requests. Explain how the task is handled today and where it becomes slow, unclear or unreliable.

Also ask whether simpler rules, search or a better interface could solve the problem. An AI pilot should earn its place by improving an agreed task, not by increasing the number of model calls in the product.

Make a small set of examples you can actually judge

Before tuning prompts, collect representative examples with permission to use them. Include ordinary inputs, incomplete information and cases that should be refused or sent to a person. Write what a useful answer must include, what it must not invent and when it should say it does not know.

Reserve examples that are not used during iteration. Track the model and prompt version with each evaluation. A demo working on its rehearsed inputs is encouraging, but it is not evidence that the workflow is ready for everyone.

• A task owner who can judge usefulness
• Representative examples and explicit failure cases
• A baseline from the existing process
• A versioned evaluation rubric and an agreed review set

Set data and action boundaries early

Decide which data may leave your infrastructure and which provider terms need review. Use synthetic or redacted material while these decisions are unresolved. Do not treat a model’s output as authority to expose another customer’s records or execute an irreversible action.

For an assistant that suggests changes, keep preview, approval and execution separate. Add the smallest set of permissions needed for the pilot. Document a manual path when the provider is unavailable or an answer fails validation.

Choose the stop conditions as carefully as the success conditions

Observe usefulness alongside cost per completed task, response time and the effort required for human correction. Define an acceptable budget and a clear way to disable the feature. Do not substitute a good-looking average for checking the failures that could harm your users.

The outcome may be a wider rollout, a narrower workflow or a decision not to use AI. All three are useful if they are supported by evidence. Pilot results cannot promise future accuracy, savings or revenue without a repeatable measurement context.

• Quality and safety checks with a named reviewer
• Cost and latency budgets for the bounded workflow
• Human fallback and a tested off switch
• A documented go, revise or stop decision