AI Proof of Concept: Prove the Decision

An AI proof of concept must answer one funding decision

Start with a visible workflow and a decision that has an owner. A support team may want to know whether AI can draft accurate replies from approved documentation without exposing private account data. The decision is not “Can a model write support text?” It is “Should we fund a limited drafting tool for these ticket types, with this review process?”

Write the decision before choosing a model or interface. Name the user, input, desired outcome, unacceptable outcome, current process, review point, and date when the evidence will be judged. If the problem is still disputed, run a focused product discovery workshop first. A POC can test a hypothesis, but it cannot rescue a missing product question by adding a chat box.

Keep the boundary narrow enough to learn. “Summarize renewal risks in ten account notes for a customer success manager” is testable. “Transform customer success with AI” is a strategy slogan wearing a lab coat.

Define success before the first impressive output

Measure the complete task, not model eloquence. For the renewal example, success might require identifying the relevant risk, citing the source note, avoiding unsupported claims, finishing inside the manager's working time, and making review faster than reading the notes manually. A fluent paragraph that misses the cancellation request is a failure.

Establish a baseline from the current process. Record completion, correction effort, time, escalation, and cost on the same representative cases the POC will face. OpenAI's evaluation guidance recommends defining the task, running it on test inputs, and analyzing results before iterating. The exact tooling may change, but the discipline is stable: define expected behavior before optimizing prompts.

Use representative cases, including the ones that ruin the demo

Build a small evaluation set from the work the product will actually receive. Include common inputs, rare but costly cases, ambiguous requests, missing context, conflicting sources, long documents, prompt injection attempts, and cases that should be refused or escalated. Separate this test set from the examples used to tune the workflow.

OpenAI's agent evaluation guidance distinguishes traces used to debug a workflow from repeatable datasets used to compare changes. That distinction is useful even with another provider. Inspect why one run selected the wrong tool, then preserve the case so the next prompt, model, or retrieval change cannot quietly reintroduce it.

Consider a hypothetical invoice extraction POC. Ten clean PDFs may all pass. The useful test also includes a photographed receipt, two currencies, a handwritten correction, a duplicated page, and an invoice number hidden in a footer. The edge cases reveal whether the product needs confidence thresholds, human verification, or a narrower promise.

Prototype the whole workflow, not only the model call

The user experiences upload, waiting, evidence, correction, approval, and recovery. Prototype those states. Show where source material came from, what the system is uncertain about, how a person edits the result, and what happens when the provider is unavailable. An accurate response inside an unusable review flow is not a successful product.

Google's People + AI Guidebook organizes AI design around human needs, onboarding, explanation, feedback, and data practices. Apply that thinking to the specific job. In the invoice example, highlight extracted fields beside the source region and make uncertainty actionable. Do not make the reviewer hunt through a confidence dashboard that looks like an aircraft cockpit designed by accountants.

Test with the people who will do the work and the people who handle failures. A frontline user can expose correction friction. An operations owner can explain what must be logged. A security or legal specialist should review the relevant risks when data or consequences justify it.

Treat risk and operating cost as product requirements

Map what the system can read, infer, store, send, and change. Add least-privilege access, data retention, provenance, review, deletion, and incident handling to the POC where they affect viability. NIST describes AI risk management across governance, context, measurement, and response. A tiny pilot does not need a theatrical governance program, but it does need named risks and owners.

Measure cost per accepted task, not cost per model request. Include retrieval, tool calls, retries, human review, failed runs, storage, observability, and support. A cheaper model that doubles correction time may be the expensive route. A slower model may still win if it removes a costly review step without increasing risk.

For a hypothetical legal document triage tool, a safe POC might only classify a bounded document set and show supporting passages. It should not send advice or update a case record. Limiting the action surface can prove useful retrieval and categorization without pretending the team has validated autonomous legal work.

Do not smuggle prototype code into production

Decide which artifacts are evidence and which are candidates for reuse. The task definition, evaluation set, risk register, interface findings, and architecture decisions should survive. A quick integration, hard-coded account, permissive data store, or manually repaired output may not.

GOV.UK's prototyping guidance makes the boundary explicit: prototypes help teams test ideas quickly, but their code does not need production security, performance, or scale. Before reuse, review authentication, authorization, privacy, availability, accessibility, observability, vendor limits, and failure recovery as production requirements.

The serious objection is that this preparation slows learning. It does if every production control is built on day one. It also slows learning when the team discovers after a successful demo that the required data cannot leave a region, reviewers cannot inspect sources, or the cost only works without retries. Test the assumptions that could kill the product, and postpone the ones that cannot.

End the POC with scale, change, or stop

Book the decision review before the build starts. Present the baseline, evaluation results, observed failures, user feedback, risk controls, operating cost, technical constraints, and unanswered questions. Keep facts separate from recommendations. Then choose one outcome: fund a production slice, run one narrower experiment, or stop.

A production slice needs an owner, budget, acceptance criteria, monitoring, staged rollout, rollback, and a date for re-evaluation. A narrower experiment needs a new question, not another round of prompt polishing. A stop should preserve what was learned so the same idea does not return six months later wearing a different vendor logo.

This decision package also makes an AI product development partner easier to judge. Ask whether the partner can name the baseline, failure set, review process, cost model, and stop condition before proposing an architecture.

AI proof of concept buying checklist

• Write one workflow, user, decision owner, decision date, and explicit exclusions.
• Capture the current baseline on representative normal, difficult, and unsafe cases.
• Define success for user value, task quality, risk, operations, latency, and full cost.
• Prototype evidence, correction, approval, failure, and recovery states around the model.
• Record data rights, access, retention, provenance, providers, and human responsibilities.
• Agree which artifacts may be reused and which prototype shortcuts must be replaced.
• Schedule the scale, change, or stop review before work begins.

Prepare a one-page brief with the workflow, current process, available cases, sensitive data, unacceptable failures, constraints, and decision date. Send it to Seuil Digital to scope the smallest AI proof of concept that can change a real investment decision.