Insights · AI implementation

Our AI pilot works. What does it take to run it reliably?

By Michael Joseph · 10 September 2026

A successful pilot shows that an AI system can perform a task under the conditions tested. Everyday operation requires evidence that it handles the work it will actually receive, acts within its authority, and has a clear response when something goes wrong. It also needs people responsible for operating and improving it.

Define the job before judging the system

“It answers correctly” is too vague for a system that updates records, schedules work, or recommends a consequential action. Define the permitted task, the information it can use, and which actions require approval. Decide what successful completion means to the business.

For a customer-service workflow, that might mean resolving a defined class of inquiries with accurate source information and a usable handoff for exceptions. For an internal agent, it could mean preparing a complete transaction for approval without creating duplicate records.

Test the workflow, including the uncomfortable cases

Use representative tasks alongside missing information, contradictory instructions, unavailable tools, and requests outside the intended scope. Assess whether the whole task completed correctly. A plausible response can accompany an incorrect database update.

Agree acceptance criteria before reviewing results. Include the frequency and consequences of errors, the human review burden, response time, and operating cost. Keep test inputs and results so that a later change can be compared with the same evidence.

Keep authority proportionate to the task

An agent that drafts a recommendation does not need the same access as an agent that executes it. Scope access to the work, identify approval points, and provide a way to stop actions when evidence or permissions are insufficient. Test those boundaries rather than assuming the model will respect them.

The NIST AI Risk Management Framework provides a voluntary structure for governing, mapping, measuring, and managing AI risks. It can inform this work; adopting a framework does not certify an individual system as safe or compliant.

Monitor outcomes after launch

A service can remain online while its usefulness declines. Monitor task outcomes and exceptions as well as availability. Examine whether performance differs across the cases that matter to your organization. When confirmed outcomes arrive later, retain the connection between the original output and what subsequently happened.

My implemented healthcare AI governance work includes prediction-to-outcome reconciliation, performance metrics, alerts, and scheduled monitoring. Those are concrete operating capabilities. They are not, by themselves, evidence of clinical benefit, regulatory approval, or a particular level of accuracy.

Google’s production monitoring guidance explains how data differences and pipeline problems can affect model performance. For an agentic workflow, I would also examine tool failures, incomplete actions, escalation quality, and recovery.

Assign ownership before expanding use

Name who reviews incidents, approves changes, maintains integrations, and decides whether operation should continue. Document the fallback process and test that people can use it. Any support agreement should make responsibilities, coverage, and escalation arrangements explicit.

A staged release lets you evaluate those arrangements on a bounded workload before expanding access or authority. The evidence should determine the next step, including when to revise the workflow or retain human control.

I connect AI development and implementation with evaluation and governance for businesses, governments, and institutions across the US and EMEA. The aim is a system whose behavior, limits, and ownership are clear enough to operate.

The linked references support the general practices discussed. They do not endorse Epirroi or validate the implementation examples.