How do you know an AI agent is ready for the public?
An AI agent is ready when the professionals who will answer for it say so, judging its answers blind, against criteria they wrote themselves. Not when the team that built it reports a high accuracy score. That distinction is the difference between a number and evidence, and an audit of New York City’s official chatbot, published this year, shows what happens when the two get confused.
What the New York City audit found
The MyCity chatbot launched in September 2023 and was extended to 311 content in March 2025. The agency running it reported accuracy of 95 to 99 percent.
The Comptroller’s audit looked at how that rate was computed. The denominator counted the prompts the system had recorded, not the number of questions people actually asked. Recomputed on the right denominator, August 2025 comes out between 84.8 and 92.7 percent. Same system, same month, a swing of ten points depending on who divides.
The user feedback from the same period points somewhere else entirely. Of the 70 people who left feedback in July and August 2025, 50 were negative, and 19 said the answer had not helped them.
On evaluation before launch, the agency could produce a one-page summary of a 2023 test, too thin for the auditors to tell whether inconsistent answers had been tested at all. So the auditors ran their own questions, and found the chatbot “could not consistently provide information when asked the same question” (Office of the New York City Comptroller, audit report on the MyCity system).
Nobody lied here. A number is only as independent as whoever counted it.
Why a self-reported accuracy rate is not evidence
Three things go wrong when the team that builds an agent also grades it.
The builder picks the denominator. Every accuracy rate is a fraction, and the fraction is a design decision. Counting logged prompts instead of questions asked is not fraud, it is a reasonable-looking choice made by people with an interest in the result.
The builder picks the questions. An agent is tested on the cases its builders imagined. The cases that matter are the ones a caseworker meets on a Tuesday afternoon, phrased the way the person in front of them phrased it.
A single rate hides where the agent breaks. 92 percent accurate tells you nothing about which 8 percent, and the failures are never spread evenly. They cluster on the questions nobody wrote a document about, which are often the questions that matter most.
This is why a high reported rate and a wall of negative feedback can both be true at once. One measures what the system logged. The other measures whether someone got the answer they came for.
What a review campaign is
On the Bayes Platform, an agent does not reach a user until a review campaign has come back good enough. The mechanism has four parts.
The service writes the criteria, in its own words. Not a generic quality rubric. The questions reviewers answer are written by the institution that will be accountable for the service, in the vocabulary of its own profession.
Testers query the agent on real cases. The people who do the work put the agent through the questions they actually get, not a curated test set.
Reviewers rate the answers blind. They do not know which version, model, or configuration produced the answer in front of them. That removes the quiet bias toward the newest configuration and makes two versions genuinely comparable.
The verdict gates the launch. If the reviews do not come back good enough, the agent does not go live.
A campaign typically mixes three kinds of question, because quality is not one thing:
- Retrieval quality, as a single choice: was the right document returned? An agent grounded in an organization’s documents can retrieve the wrong source and still write a fluent answer, and that is the most dangerous failure mode in a public service.
- Response relevance, on a 1 to 5 scale: did the answer address the question? Correct and useless are compatible.
- Suggested improvements, as free text. This is where the reviewers tell you what they know and the system does not.
What a bad score is actually telling you
The instinct, when a review comes back weak, is to change the model. That is almost never the fix.
An agent grounded in an institution’s validated documents fails where the documents are missing or contradictory. Every weak answer points at something nobody has written down yet: a procedure that lives in one person’s head, two circulars that disagree, an eligibility rule updated everywhere except the handbook. Read that way, a review campaign is not a grade. It is a work list for the service, and the knowledge it surfaces stays useful long after the agent ships.
Self-reported rate against review campaign
| A self-reported accuracy rate | A review campaign | |
|---|---|---|
| Who defines the measure | The team that built the system | The service accountable for the answers |
| What gets counted | What the system logged | Answers to real questions from the work |
| Who judges | The builder | The professionals of the service, blind |
| What a failure tells you | That the number went down | Which document is missing |
| When it happens | After launch, as reporting | Before launch, as a gate |
| What it produces | A percentage | A verdict and a work list |
After launch, it keeps going
A campaign before launch is a gate, not a certificate. The Bayes Platform handles the full lifecycle of a public-service agent, from rapid prototyping to real experimentation, then production and monitoring over time, and the same review mechanism runs again on a new version, a new document base, or a new set of cases. An agent that was validated in March on one set of criteria is not validated in October on a different service.
Frequently asked questions
Can an automated benchmark replace human review? Automated evaluation is useful for catching regressions between two versions, and we use it that way. It cannot tell you whether an answer serves the person who asked, because that judgment is domain expertise: whether a care protocol was read correctly, whether a benefit rule applies to this situation. That call belongs to the professionals.
How many reviewers does a campaign need? Enough to cover the real range of cases rather than a sample of easy ones, with each answer seen by more than one reviewer. The number that matters is the coverage of situations, not the count of ratings.
What happens if the reviews come back bad? The agent does not go live. The free-text answers and the weak retrieval scores point at what is missing, the service fills those gaps, and the campaign runs again on the same criteria. That loop is the product working, not the product failing.
Who owns the review results? The institution. The Bayes Platform is open-source and deployable on the institution’s own infrastructure, so the campaigns, the answers and the ratings stay in-house like the rest of the data.
The verdict is not ours to give
We build the platform, and we are confident in what it does. Deciding that an agent is good enough to answer a citizen belongs to the people who will answer for it.
You can see the review step running, along with the rest of the lifecycle, in the platform tour, and the fuller picture of how these agents are built and governed is here: AI agents for public services.