What a Real AI Pilot Report Looks Like

The fastest way to waste an AI budget is scoping 'an AI assistant for the company'. Company-wide means every data source, every permission model, every edge case — a surface so large that nothing gets measured properly and everything gets blamed vaguely. Scope one task: find passages in the policy library with citations, or sort the support inbox into quote, complaint and spam with draft replies.

One dataset means a bounded, representative sample: a few hundred real documents or a few thousand real messages, warts included. Scanned pages, typos, duplicate threads — the sample must embarrass you slightly, or it is not representative. Clean the access model first (who may see what), because a search pilot that leaks restricted documents fails on day one regardless of accuracy.

Write the scope as one page: the task in one sentence, the dataset described, the users who will judge it, the two numbers that decide (accuracy target, cost ceiling), the 2–4 week window, and the decision that follows. Both sides sign. Anything outside that page is a different pilot, priced separately.

Pick the task with the clearest pain and the cleanest measurement. Document search wins when staff lose hours hunting files; triage wins when inboxes drown. If neither pain is sharp, say so — a pilot without a sharp pain measures nothing worth paying for.

Vendor accuracy claims are measured on benchmarks, not on your scanned PDFs. Your pilot needs its own scoreboard: a fixed set of 50–100 real queries with known-good answers, agreed before the pilot starts. Every system answer gets graded against this set — right with citation, right without citation, wrong, or refused.

Citations are the load-bearing column. An answer with a passage link a human can verify in ten seconds is a working feature; an answer without one is a rumour with good grammar. Track citation rate separately from raw accuracy — a system at 80% accuracy with citations beats 90% without, because the 20% wrong is catchable and the 10% wrong is invisible.

Log weekly, not just at the end. Accuracy moves as prompts, chunking and retrieval settings change — the log shows which change helped and which regressed. Aim the target honestly: 85%+ cited accuracy on a bounded task is a green light; below 70% after tuning means the task, the data, or the model choice is wrong, and the report should say which.

Keep a human in the grading loop throughout. Automated grading of AI output by another AI model is convenient and circular — spot-check at least a fifth of graded answers by hand, especially the ones the system got confidently wrong.

AI pricing hides in three places: per-token API charges that scale with document size, per-query retrieval costs, and the human minutes spent checking, correcting and tending the system. Log all three from the first day, or the pilot's economics are fiction.

Per-use cost is simple arithmetic once logging exists: total API spend for the week divided by resolved queries. Compare it against the current cost of the task — fully loaded staff minutes per search or per triaged message. A pilot that resolves queries at one-tenth the staff cost with 85% cited accuracy is an obvious yes; one at half the cost with 60% accuracy is an obvious no. Most pilots land between, which is exactly why the numbers must be real.

Watch the failure economics hardest. Wrong answers cost double: the API spend plus the human cleanup. A system that is cheap per query but wrong often is more expensive than doing nothing — the report must show corrected cost, not sticker cost.

Project the pilot numbers to production volume before deciding. Token costs, retrieval latencies and tending minutes rarely scale linearly; a pilot handling hundreds of queries a week behaves differently at tens of thousands. State the scaling assumption in writing so the go decision carries its conditions with it.

Document search and inbox triage are the two highest-payback pilot shapes for service businesses, and they fail in different ways — pick the one whose failure mode you can tolerate. Search fails by missing: the answer exists but retrieval doesn't surface it, and staff lose trust after three misses. Triage fails by misrouting: a complaint labelled as spam is a customer lost, while a search miss is merely a wasted minute.

Choose search when the pain is time: staff hunting through drives, shared folders and PDFs for answers that provably exist. It needs a bounded library (hundreds of documents, not millions), a cleaned access model, and queries with verifiable answers. Success is easy to grade because the right passage either appears with a citation or it doesn't.

Choose triage when the pain is volume: an inbox where quotes, complaints and spam land unsorted and the urgent hides among the routine. It needs labelled history (a few thousand past messages is plenty), clear routing rules to learn from, and a human reviewer for the first weeks. Success is measured in correctly routed share plus drafts accepted with light edits.

Run one, not both. The discipline of a single task, single dataset and single scoreboard is what makes the report trustworthy — two pilots at once split attention, double the tending time, and produce two inconclusive reports instead of one decision. The winner earns the second pilot; the method, not the code, is what transfers.

Agree in writing what failure looks like before the pilot begins: cited accuracy below an agreed floor after two tuning rounds, corrected cost above staff cost, or tending time that never falls. Kill criteria written upfront turn an emotional argument at week four into a calm comparison against numbers everyone already accepted.

Watch for the two early warning signs in week one. First, the sample proves unusable — access rules too tangled to clean, or documents so inconsistent that no retrieval strategy finds them. Fix the data before continuing, or stop honestly. Second, the team stops checking outputs — trust without verification is how confident errors reach customers.

The report has six fields and fits on three pages: scope (task, dataset, window), method (model, retrieval setup, grading set size), accuracy table (right-cited, right-uncited, wrong, refused — with the human spot-check rate), cost table (API spend, queries resolved, cost per resolved query, tending minutes), recommendation (scale, adjust, or stop), and conditions (what would change the answer).

Scale means: accuracy at or above target, corrected cost comfortably below staff cost, and a named owner for tending. Adjust means: one specific change with a re-measurement window — different chunking, tighter scope, better source docs. Stop means: the numbers say no, with the reasoning preserved so nobody re-runs the same pilot in a year.

Attach the raw logs as an appendix, not decoration: the graded query set, weekly accuracy and cost tables, and the list of confidently-wrong answers with what caused each. Future work — yours or anyone's — starts from evidence instead of vibes.

Circulate the report to everyone who will live with the decision, especially the sceptics. A pilot report that only the enthusiasts read is marketing; one the sceptics accept is an asset. Their sign-off is the real deliverable.