Your AI answers customers. How do you know it answers well?

The first draft of the change made the assistant refuse more often: refusals rose from 28 to 37 on the same questions, and two more small edits that night brought them back down. That is the reason the rows stay as a test. A fix for one complaint can quietly cause another, and only a rerun on the same questions shows it.

Zero is what the runs showed; it is not a promise it will never happen again. This is one site and one business, not a benchmark. What matters is the chain it proves: a reason in plain words, a rule anyone can read, a change, and a number that moved.

Ask a business owner to describe a good answer and you get adjectives: accurate, helpful, on brand. Show them real answers to a real customer question and they judge them in seconds. Ask "why?" and you get something useful: "it sounds like us", "it never gives a link to book a call", "it answers a question the customer did not ask". That sentence is a requirement nobody could have written up front.

Barry Boehm popularised the phrase "I'll know it when I see it" (IKIWISI) in 2000, for requirements that only show up once people use a system, and the forward-deployed engineering guide builds on the same idea.

AI answers make it sharper. In a 2024 study, Shreya Shankar and colleagues found that people need criteria to grade outputs, but grading outputs is how they find their criteria; they called it criteria drift. The study was small (nine practitioners), but it matches the case above: you still write a definition of good, after the reactions, from the reasons people gave.

That one sentence per answer is what this guide calls the why column, and an engineer or a vendor can act on it the next morning. Hamel Husain calls it a critique, Shankar's work talks about notes, and some tools call it an explanation; the idea belongs to all of them.

Keep the score simple: pass or fail for one answer; for two or three, rank them 1, 2, 3, ties allowed, a blank for a fail. Eugene Yan, who writes on product evaluation, advises: "Focus on binary pass/fail or win/lose labels." Avoid a 1-to-5 scale, where nobody agrees on the difference between a 3 and a 4.

Not every row will get a reason. On the real-estate site, 49 of the 62 rated rows carried a written why; the rest still counted in the score, but only the rows with a reason said what to change.

Start smaller than you think. Anthropic's engineers call 20 to 50 simple tasks drawn from real failures "a great start", and Husain and Shankar suggest reviewing that many outputs after every significant change. The real-estate site's set is 51 questions (50 of them were in the rated sitting above; one was added the next day), which is the start line, not the finish.

Start from questions real visitors ask (follow-ups your search platform suggests are a quick source), then add questions the assistant should not answer well, and sort everything into the seven categories below. They have served two unrelated assistants, the real-estate search and a brand-guidelines chatbot built on public brand pages. The examples in the table come from the second one, with the brand's details removed.

On the should-decline rows, a clear decline scores as a pass and an answer is the fail, just as on a price or legal question. The one exception is the rude row: decline the tone, answer the valid request, and score whether the valid part was answered.

Count two things, like a smoke detector, which should go off for every real fire and stay quiet when you make toast. Of the questions it should have declined, how many did it decline? Each miss is an answer your pages do not back. Of everything it declined, how many should it have answered? Each one is a false alarm, and enough of them make the assistant useless. Borderline rows and follow-ups stay out of both counts: correct is not settled for the first, and the second are judged turn by turn.

Start with 3 to 5 questions per category, and more for Covered, so a safer assistant does not become a useless one: 21 to 35 rows in all, a follow-up sequence counting as one. The real-estate site's 51 map onto the same table: 10 Covered, 15 Not in your pages, 10 Borderline, 10 Off-topic and 6 Rude, which also held the attempts to make it misbehave. A set that small shows where the failures are, not how often, so do not quote a percentage per category.

Before you trust an AI checker on its own, you need more: Husain and Shankar plan 100 to 200 labelled examples for each kind of failure, and Eugene Yan aims for 50 to 100 failures, because "a dataset with hundreds of labels but only five failures isn't useful." Even then a set cannot rule out rare failures: Yan shows that a 3 percent defect rate on 200 samples leaves a margin of about 2.4 points either way, too wide to confirm a target under 5 percent.

Rerun the set whenever anything changes, including what you did not change yourself. Vendors update and retire models on their own calendar, so an answer that passed last month can fail after an update you never saw. Pin the model version where you can, rerun on a schedule as well as after your own edits, and keep the rows somewhere you control. When a row goes from pass to fail, the why column tells you which rule broke.

One person owns what good looks like. Husain and Shankar call this person the "benevolent dictator": "one domain expert who understands your users". Committees average out the reasons until nothing is left to act on.

Scoring can be shared, but only on the business side: people who know the customers. Write the guidelines down, and have the owner review the disagreements and any new kind of failure.

An AI checker can pre-sort answers at volume once a person's labels exist to check it against; Shankar's study found AI evaluators "simply inherit all the problems of the LLMs they evaluate." Ask the checker for its own reason too, so you can read why it disagrees with your reviewer.

"This is just standard error analysis." Yes, and that counts in its favour: Hamel Husain and Shreya Shankar teach it to engineers under that name, and it works. What changes here is the reader: the person who owns the answers and will never read an engineering blog.

"Our vendor's dashboard already does this." A dashboard counts what someone told it to count. The useful questions are who told it, in whose words, and whether those words came from looking at your real answers.

"Experts disagree with each other." They do. Anthropic's test for a good question is one where "two domain experts would independently reach the same pass/fail verdict." Where yours do not agree, that is a finding: the rule is not written down yet.

On the real-estate site, the early prompts were tuned by hand: read the rankings, change the wording, rerun. That works, and it is slow. Once the ranked sheet existed, an AI coding agent got it (the answers, the ranking and the why) with one goal: match it. It changed the prompt, ran the whole set, scored the answers against the sheet, and went again. It took a couple of hours, and its prompt went live.

The pattern comes from Andrej Karpathy's autoresearch, where an AI agent edits, runs, measures and keeps or discards a change with no person in the loop. His version measures a model against a benchmark; this one measured a prompt against the business's own rankings and reasons.

Two things make a loop like this safe to leave running: people do their part once, up front (the ranked rows and the why, which the AI cannot invent), and the loop gets a cap, a maximum number of rounds and a spending limit, so a run that stops improving also stops costing. Karpathy's loop runs until someone stops it. A loop that calls a paid model on a live business should not.

The sources below are cited for the specific points used here and nothing beyond them.

Hamel Husain and Shreya Shankar, "Evals FAQ" (living page, retrieved 2026-10-15): error analysis, 20 to 50 outputs per significant change, the benevolent dictator, and 100 to 200 labels per failure mode.

Hamel Husain, "Using LLM-as-a-Judge For Evaluation" (2024-10-29): critiques; "A Field Guide to Rapidly Improving AI Products" (2025-03-24): the tools trap.

Shreya Shankar, J.D. Zamfirescu-Pereira, Björn Hartmann, Aditya Parameswaran and Ian Arawjo, "Who Validates the Validators?" (UIST 2024): criteria drift, and AI evaluators needing human validation. A qualitative study with nine practitioners.

Andrej Karpathy, "autoresearch" (GitHub repository): the pattern behind the automatic loop, an AI agent that edits, runs, measures and keeps or discards a change on its own.

Anthropic Engineering, "Demystifying evals for AI agents" (2026-01-09): 20 to 50 tasks to start, two experts reaching the same verdict, and testing both sides of a behaviour.

Eugene Yan, "Product Evals in Three Simple Steps" (2025-11-23): binary or win/lose labels, failure counts for checking a checker, and the sample-size arithmetic.

Barry Boehm, "Requirements that handle IKIWISI, COTS, and rapid change" (IEEE Computer, July 2000): requirements that emerge with use rather than being specified up front.

Nelson F. Liu, Tianyi Zhang and Percy Liang, "Evaluating Verifiability in Generative Search Engines" (2023): citation support rates and the inverse link between perceived usefulness and citation precision.

OpenAI, "Evaluation best practices": the vibe-based evals anti-pattern.

Moffatt v. Air Canada, 2024 BCCRT 149 (Civil Resolution Tribunal of British Columbia, 2024-02-14): an airline held responsible for what its website chatbot told a customer, and ordered to pay $812.02 in damages, interest and fees.

The worked example comes from one small real-estate site's August 2026 ranked sheet and the change log that followed. You can try its AI search and test the result yourself.