Picture a meeting room at city hall. The vendor plugs in a laptop and asks its AI assistant five questions. All five go well. It answers in seconds, writes clean prose and summarizes a forty-page regulation in three paragraphs. Someone at the table says what everyone is thinking: "This already works."
Six months later, the system has logged thousands of interactions. The treasurer asks how much time it saved. The comptroller asks how many answers were wrong. The opposition council member asks why it is worth what it costs. Nobody has the figure, because nobody measured how the work was done before and nobody defined what it meant for the system to work.
A demo is not an evaluation. A demo shows what a system can do with hand-picked questions, in one afternoon, in front of people who want it to go well. An evaluation answers something else: what the system does with the institution's real cases, against what point of comparison, with what errors, and who finds out when it starts to fail.
In June 2026, the OECD published its Digital Government Outlook 2026, with data from its 36 member countries on policies in place between January 2023 and December 2024. The contrast between adoption and control is hard to ignore:
- Use: 35 of 36 countries already use AI in at least one area of government.
- Risk assessment before deployment: only 14 of 36 require it (39%).
- Internal review committees: 12 of 36 (33%).
- Post-deployment audits: 11 of 36 (31%).
- Impact measurement: only 10 of 36 (28%) report any financial or non-financial study of an AI use case.
The report also points to a contradiction: 50% of countries say they base AI adoption decisions on evidence of potential savings or efficiency, even though only 28% have measured impact in any case. The OECD asks what evidence those decisions rest on and whether adoption is driven more by expectations and pressure than by demonstrated value. Without measurement, it warns, governments cannot tell whether a pilot succeeded or justify scaling it, and the result is a proliferation of pilots with little potential to grow.
Mexico appears in the annexes of the same report. For the period analyzed, it reported an internal review committee, but no auditing processes or pre-deployment risk assessment tools, and no measurement of the impact of AI in government. The figure has a date: it closes in 2024, before the current federal plan.
That plan points in the right direction. The National Artificial Intelligence Plan, dated April 2026 by Mexico's Agency for Digital Transformation and Telecommunications, proposes defining guidelines for responsible AI use in the public sector and creating impact evaluation mechanisms. It also sets the starting questions for each use: what public problem it seeks to solve, what data it operates on, who supervises its results, what risks it may create and what mechanisms exist to correct errors.
A municipality does not need to wait for those guidelines to ask the same questions before signing.
The UK government published the results of two experiments that show why enthusiasm is no substitute for measurement.
The first was GOV.UK Chat, a generative AI assistant built on the official portal and tested with 1,000 invited users. Nearly 70% found the answers useful and just under 65% were satisfied. A demo would have stopped there. On reviewing the answers, the team concluded that accuracy did not reach the level required by a site where factual precision is crucial. There were a few cases of hallucination, and some users played down the risk of error, partly because of the credibility of the GOV.UK brand. A wrong answer with an official seal gets believed more, not less.
The second was the Microsoft 365 Copilot trial at the Department for Business and Trade, with 1,000 licenses between October and December 2024. Among respondents, 72% said they were satisfied or very satisfied. Writing tasks saved some time, but others, such as scheduling meetings or generating images, took longer than without the tool. Some 22% reported spotting hallucinations. And the report found no evidence that time saved translated into higher productivity, although it notes that this was not the study's central aim.
The lesson is not that those systems are useless. It is that satisfaction and measured usefulness are two different figures, and only an evaluation separates them.
A useful evaluation does not require a lab. It requires four things defined before launch.
A baseline. How long first response takes today, how many cases get reopened, how many entries need correcting. Without that number, any later improvement is an opinion. Memorandum M-25-21 from the US Office of Management and Budget, from April 2025, asks agencies to document the expected benefit of high-impact AI against their existing processes. The comparison is the heart of the exercise.
A set of real cases. Not the vendor's five questions, but one or two hundred from the institution itself: the frequent ones, the badly written ones, the ambiguous ones and the ones with no answer in the sources. Those last ones are the most revealing, because they show whether the system recognizes its limit or improvises. It is the same test we described in an answer is not a source.
A threshold agreed in writing. What error rate is tolerated, and in what kind of case. Misclassifying a streetlight report does not weigh the same as misquoting a legal deadline or writing up the facts in a detention file. The same system can pass one task and fail another.
An accountable person who signs. The US memorandum requires an identified person to accept the risk with a signature, and whoever reviews the assessment must not have taken part in development. It does not apply in Mexico, but the principle travels well: someone with a name and a title decides the system is ready, and someone else reviewed that decision.
The bar should be proportionate. The OECD recommends tailoring controls to the risk of each use case, to avoid both misuse and inaction. A meeting-minutes summary does not need the same as a tool that classifies offenses.
A system approved in March is not the same system in September. The vendor's model changes, regulations change, a new administration arrives, citizens start asking other things. The OECD calls these changes in data, behavior or performance over time drift, and holds that post-deployment audits and ongoing monitoring are essential to detect them. Checking only before deployment, it says, can be insufficient.
It also warns against badly designed audits that create false confidence. An annual report nobody reads watches nothing.
In practice, monitoring means three plain routines:
- Periodic sampling. Each month, a person checks a random sample of answers or suggestions against the source and logs errors by type.
- Re-evaluation after each change. If the model, the regulation or the catalog changes, the set of real cases is run again. M-25-21 asks for scheduled periodic reassessments and reassessment after significant modifications.
- A planned exit. If errors exceed the threshold, it is clear who turns the function off and how operations continue without it. The memorandum is direct: an agency that does not meet the minimum practices must safely discontinue use of the function.
The opinion of the people using the service counts as evidence too. According to the OECD, only 8 of 36 countries have citizen feedback or complaint mechanisms on AI in government.
Many municipalities will not have access to the code or the model, and that does not leave them without options. M-25-21 anticipates this: without access, the agency must use alternative methods, such as querying the service and observing its outputs, or giving evaluation data to the vendor and obtaining the results.
That translates into concrete clauses. The OECD notes that only 21 of 36 countries provide central support for buying AI, so much of this work falls on each institution. The contract should cover:
- The right to test before and during operation with the institution's own cases, not only the vendor's.
- Advance notice of model changes, so re-evaluation happens before they reach the public.
- Access to the records of inputs, outputs and human decisions, in a format the institution can read without the vendor.
- Agreed outcome metrics, compared against the baseline and not against the sales pitch.
Without records no evaluation is possible, as we explained in a suggestion is not a decision: what was not written down cannot be reviewed.
In Agora, Intello's citizen services platform, implementation includes loading information on procedures, requirements, offices and the legal framework, and testing the assistant with real conversations before turning it on to check that it does not make things up. Then comes a pilot in which timelines and concepts are adjusted before expanding to the rest of the municipality.
In operation, a judge reviews each answer before it is sent, the assistant only cites what is in its sources, and the conversation moves to the staff inbox if the citizen asks or the case gets complicated.
Agora's analytics provide material for comparing periods: resolution times, first response, reopenings, backlog and percentage handled on time, filtered by date and department. Per-ticket surveys add the resident's opinion, and the log keeps who changed what, when, before and after. The baseline and the threshold are still for the institution to define.
In Tribuna, the public safety and civic justice platform, INTELLO AI helps draft the facts, suggests the classification of offenses and checks the consistency of the file, but the decision belongs to the person. Implementation is validated with real users in a pilot and rolled out in phases. The log records reads, writes and risk events, and the timeline for each file makes it possible to review what was done and who did it.
The OECD sums up the problem in a line that holds at any level of government: the 28% of countries that assess the impact of AI today should be seen as a floor, not a ceiling. Evaluation should be the normal expectation.
The question for leadership is not whether the demo went well. It is which number is going to change, how it is measured today, who signs off that the system is ready and who finds out when it stops being ready. If those four answers do not exist before contracting, six months later the treasurer's answer will not exist either.
If your institution wants to adopt AI with prior testing, a pilot and a record of every action, see how Agora and Tribuna operate, or request a demo. A demo is a good start; the evaluation is what comes next.
- OECD, Digital Government Outlook 2026: From Foundations to Transformational Impact, June 2026
- Agencia de Transformación Digital y Telecomunicaciones (Mexico), Plan Nacional de Inteligencia Artificial, April 2026
- Government Digital Service (UK), The findings of our first generative AI experiment: GOV.UK Chat, January 18, 2024
- Department for Business and Trade (UK), Evaluation of the M365 Copilot pilot, August 2025
- Office of Management and Budget (US), Memorandum M-25-21, April 3, 2025