How Government Agencies Can Actually Evaluate AI Tools

Turn administrative records into benchmarks and contracts into real oversight.

By Carrie HirschReviewed by PAP Editoral TeamUpdated November 16, 202616 min read

What you’ll learn in this article…

  • Federal agencies reported 3,611 AI use cases in 2025, a 69% jump.
  • Convert existing administrative records into hidden benchmarks for testing.
  • Write third-party evaluation and randomized A/B tests into procurement contracts.

AI procurement in government has outrun the capacity to test what gets bought. In 2025, 56 federal agencies reported 3,611 AI use cases in operations, a 69% increase over the prior year. California's Executive Order N-9-26, which accelerates an independent verification framework for AI systems, shows the same pressure at state level.

For public managers the work is concrete: pick a test set, write evaluation rights into the contract, run a live field test, and staff human review. Stanford HAI's November 11, 2026 issue brief on evaluating AI in public administration gives that work a method, built largely from records agencies already keep.

Why AI Evaluation Is Now a Core Administrative Competency

Evaluation capacity, not access to models, is now the binding constraint on public-sector AI.

The inventory is the oversight problem

In 2025, 56 federal agencies reported 3,611 individual AI use cases inside government operations, a 69% increase over the previous year. Treat that as an arithmetic problem for oversight rather than a novelty statistic: every use case is a live deployment that some office must be able to test, document, monitor, and switch off.

California puts dates on the framework

Executive Order N-9-26, signed by Gov. Gavin Newsom and effective September 18, 2026, directs the Government Operations Agency to build an independent verification organization framework. GovOps owes safety and security recommendations by November 16, 2026; application criteria for verification bodies by May 1, 2027; and completion of the Gov. Code 11549.82 work, with implementation underway, by December 1, 2027. The order leaves the mechanics open: no verification organization is approved, and final criteria remain outstanding. Federal practice is already more prescriptive. GSA's 2026 directive requires an AI Impact Statement, an independent evaluation plan, and real-world test results before a high-impact use case deploys, plus an Authorization to Operate, annual re-registration, and public listing in an agency inventory.

Agentic risk stops being hypothetical

On June 18, 2026, an autonomous agent researching public medicines spending reached Services Australia's Medicare statistics portal. Officials said no personal records were accessed and that the portal held aggregate statistics. Even so, it is the scenario public managers now have to price into procurement.

The gap the brief names

Stanford HAI's November 11, 2026 brief by Danielle Sutton, Vera Lummis, and Daniel E. Ho states the complaint directly: agencies already operate under legal mandates to evaluate AI but lack methods that work in practice. Read the blueprint at https://hai.stanford.edu/policy/a-blueprint-for-evaluating-ai-in-public-administration.

What the Stanford HAI Blueprint Actually Proposes

Buying the model that tops a public leaderboard and buying the model that survives your own casework are two different procurement strategies. The Stanford HAI issue brief "A Blueprint for Evaluating AI in Public Administration," by Danielle Sutton, Vera Lummis, and Daniel E. Ho (November 11, 2026), argues that most evaluation practice in use today is built for the first strategy and does not serve the second. The document is short and worth reading in full at https://hai.stanford.edu/policy/a-blueprint-for-evaluating-ai-in-public-administration.

Two Risks, Translated for Agency Use

  • Safety risk: a system manipulated or attacked, or an agentic tool taking unanticipated actions with access to government data.
  • Performance risk: the tool fails to perform the intended function, or fails to deliver the benefit the business case promised, after deployment.

Why Outside Benchmarks Do Not Transfer

Government work is legally constrained, procedurally documented, and consequential even at low volume. A model that scores well on a general leaderboard has never been tested against your statute, your notice requirements, or the case file your staff actually touch. Generic methods were not designed for that setting, and the brief says plainly that they fall short of what agencies are already legally obligated to do.

Four Levers

The recommendations come down to four operational moves, taken in order:

  • Mining administrative records into hidden benchmarks
  • Requiring third-party evaluation as a contract condition
  • Running randomized A/B tests during rollout
  • Rebuilding quality assurance around human-AI collaboration

Hidden Benchmarks: Turning Administrative Records Into Test Data

A hidden benchmark is a test set built from your own agency's paperwork: the intake forms, case notes, and final decisions already sitting in your systems of record. Stanford HAI's November 2026 issue brief, "A Blueprint for Evaluating AI in Public Administration," argues that these administrative records are the evaluation data vendors cannot manufacture, because only the agency holds the ground-level history of how its work actually gets done.

A four-step method

  • Pick the task: choose high volume plus a recorded outcome trail, such as eligibility redetermination, FOIA triage, or permit screening.
  • Extract and de-identify: pull the inputs a reviewer saw and the outcome recorded, then strip identifiers before anything leaves the building.
  • Hold out a sample: reserve cases the model never trains on, so the score reflects unfamiliar work.
  • Define "correct" first: write the standard with program staff before any model is scored, not after the results arrive.

Where benchmarks go wrong

Administrative records show what the agency decided, not what was right. Flag appealed and overturned cases, discretionary exceptions, and known error classes, then exclude or reweight them before scoring.

Document missing fields and skewed caseloads as well. A benchmark drawn from one office's records tests that office's practice, not the agency's.

The brief adds a second payoff: releasing benchmark sets would both measure and drive AI progress across the field.

Keep one limit in view. A benchmark of past decisions teaches a model to reproduce past discretion, so use it to test accuracy and never to set policy.

Writing Third-Party Evaluation Into Procurement

Writing third-party evaluation into procurement means adding contract language that gives an outside party, an accredited lab, a university partner, or the state auditor, the right and the access to test a system before go-live, and again whenever the model changes materially.

What the paperwork already demands

Federal guidance has moved part of the way. OMB M-26-04 requires vendors in large language model solicitations after December 2025 to submit model, system, and data cards, bias-evaluation methods and results, benchmark scores, and red-team findings.1 M-25-21 steers agencies toward contract terms covering continuous monitoring and evaluation, with post-award monitoring where appropriate.2 GAO-25-107933 asks agencies to test proposed solutions during proposal review, to the greatest extent practicable, using agency test environments. GSA's 2026 high-impact requirements push hardest, requiring an independent evaluation plan and real-world test results before deployment, reviewed by the Chief AI Officer or a designee.

None of that is yet a standard clause conditioning award on an outside evaluator, so the public manager has to write it.1

Clauses to insist on

  • Access: a right to independent evaluation, plus model documentation and prior evaluation results.
  • Change notice: written notification when the model or its training data shifts materially.
  • Audit rights: cooperation with post-deployment testing and access to logs.
  • Indemnification: remedies tied to failures the evaluation should have caught.

Scoring the evidence

Run a pass/fail gate before price is scored: no disaggregated accuracy, no adversarial test report, no bid. Then score evaluation-evidence quality at roughly 20 to 30 percent of the technical total.

The condition in practice

Evaluation completed by an accredited lab, university partner, or state auditor before go-live, repeated after any material model change, with renewal contingent on it.

A Worked Weighted Scorecard: Accuracy, Fairness, Privacy, Safety, Cost, Usability, Public Value

The OECD's review of generative AI experimentation in government sorts evaluation into five areas: model performance and input/output quality, projected impact and value, cost and institutional integration, usability and acceptability, and risk and compliance. It sets no weights and no numeric scale, and neither does the NIST AI Risk Management Framework, which is voluntary and defines no pass/fail threshold. The scorecard below is therefore an agency's own instrument, anchored to those published rubrics rather than derived from them. Weights and anchors are illustrative and should be ratified by your legal, equity, and program offices before any procurement uses them.

DimensionRecommended WeightWhat to MeasureMinimum Acceptable Score (0–3)
Accuracy and input/output quality20Spread in accuracy between the best and worst demographic subgroup on a held-out test set; for generative tools, groundedness of citations and task completion rate2: no subgroup falls more than 5 points below the overall figure, and outputs trace to a retrievable source
Fairness and contestability15Appeal-overturn rate on decisions the tool touched, compared with the pre-deployment baseline2: overturn rate stays within the baseline range and appeals are logged with reasons
Privacy and data quality10Representativeness of the records used to test the model; count of data elements collected beyond the stated purpose2: test data documented and representative, and no field collected without a stated purpose
Safety and security20Number of unanticipated agentic actions in a supervised sandbox trial; results of adversarial testing on manipulation attempts2: every privileged action requires a human authorization step, and no unmitigated high-severity finding remains
Cost and institutional integration10Fully loaded cost per transaction, including staff review time, licensing, and integration work2: cost per transaction at or below the manual baseline, or a documented public-value case above it
Usability and acceptability10Review minutes per case for staff using the tool versus staff working without it; front-line and affected-community feedback2: no net increase in review time, and feedback channels are open and answered
Public value and projected impact15Share of cases reaching a correct outcome faster, plus documented benefit to the statutory purpose of the program2: benefit is measured against a stated program goal, not a vendor claim

From Score to Decision: Deploy, Human-Review, Suspend, or Retire

A composite score only matters if it maps to a decision someone is authorized to make. The bands below pair each score range with a named decision authority, the human oversight required, and a documented trigger of the kind that has forced agencies to act. Small municipalities can run a shorter dimension set with external peer review, state agencies should file results against executive-order requirements, and federal programs should report on the OMB-aligned schedule; in every case a 0 or 1 on safety, fairness, or privacy vetoes deployment regardless of the composite.

Composite Score BandRequired ActionHuman Oversight LevelTypical Trigger
2.5 or higher, with no dimension below 2Deploy with monitoring. Authority rests with the agency head or CIO, and the use case is logged in the agency inventory before launch.Sampled human review of a fixed share of outputs, with an escalation path to the program manager.Scorecard cleared, no veto gate triggered, and the use case recorded in the inventory.
2.0 to 2.49, or any single dimension below 2Human-review mode. Authority rests with the program manager and QA lead, and scope is limited to a controlled pilot.Case-level human decision required before any adverse action reaches a resident.A weak dimension on the scorecard, or a pilot test showing error rates above the program baseline.
0 or 1 on safety, fairness, or privacySuspend. Authority rests with the agency head and legal counsel, with notice to the oversight body inside a set window.Full manual processing until the defect is remediated and the tool is re-scored.Court finding that automated terminations violated due process (Tennessee Medicaid eligibility system, 2024); public and political backlash one day after launch (America.gov chatbot, 2026).
Below 2.0 after remediation, or a veto gate that cannot be fixedRetire. Authority rests jointly with the agency head and CIO; contract termination runs through procurement.Decommission with records preserved for audit, appeals, and claimant remediation.Software error producing incorrect fraud findings, followed by litigation and a $20 million settlement covering 3,000 claimants (Michigan MiDAS, 2024); presidential directive requiring agencies to stop using a vendor's technology and its removal from a government schedule (GSA, 2026).

Randomized A/B Testing During Rollout, Even Without a Research Office

Can an agency test an AI tool on live cases without a research office on staff?

Yes, if the design is simple enough to survive a busy intake queue.

The Design That Holds Up

Randomize eligible cases at intake, not by worker preference: half move through the AI-assisted workflow, half through the standard one. Before the first case is assigned, write down what you will measure and how: decision accuracy against a reviewed sample, processing time, appeal and reconsideration rates, and outcomes broken out by subgroup so equity gaps surface early rather than in litigation. Pre-registering those outcomes is what separates an evaluation from a retrospective narrative. One analyst with a spreadsheet can run this; the discipline matters more than the tooling.

When Randomization Is Not Available

Small agencies can stagger rollout by office or by week, then read the results as an interrupted time series. A methods course or public policy program at a nearby university can often supply design and analysis help at little cost, and the Stanford HAI brief recommends exactly this kind of partnership.

Stopping Rules, Named in Advance

Decide before launch what result pauses the rollout: an error rate above the manual baseline, a widening gap between subgroups, or a spike in appeals. Name the official empowered to call it in the deployment plan, not after a bad month.

The Failure to Avoid

Most agencies measure speed and user satisfaction and never measure decision quality or error rates. Instrument the workflow so accuracy can be audited on a sample of closed cases. Speed is the easiest number to collect and the least informative one.

The binding constraint on responsible AI in government is not access to models but the capacity to evaluate them, and the will to put existing administrative records behind that evaluation.

Redesigning QA Around Human-AI Oversight

With 56 federal agencies reporting 3,611 AI use cases in 2025, a 69% jump in a single year, oversight volume now grows faster than headcount. That arithmetic belongs in the QA plan, not in a footnote.

Do the workload math first

Multiply review minutes per case by daily case volume to get FTE hours. An office processing 800 determinations a day at a 20% sampling rate flags 160 cases; at three minutes each, that is eight hours a day, a full position's worth of review before anyone is hired. Run that number before the pilot, not after. Treat "human in the loop" as a staffing line item, not a procurement checkbox.

Protect discretion, not just presence

Require reviewers to record their reasoning, then monitor override rates over time. A rate that drifts toward zero suggests automation bias, where staff rubber-stamp the model's output and the human layer becomes decorative. Audit overturns the way you audit original decisions.

Build redress into the workflow

Residents should learn that AI was used, receive a plain-language explanation of the decision, and have a right to human re-review with a published appeals path. Publish overturn rates and average time to a re-review decision; together they are the honest test of whether oversight works.

Measure the frontline

Track time per case, override and escalation rates, staff confidence, and shifts in which case types staff actually handle. A rising escalation rate is not automatically failure. The Stanford HAI blueprint's central recommendation is to rebuild quality assurance around human-AI collaboration rather than bolting a model onto legacy sample audits.

How MPA and MPP Programs Are Teaching AI Governance

What the curriculum actually looks like

At Harvard Kennedy School, AI governance is best documented on the edges of the residential degree rather than at its center. "The Foundations of AI Governance: From Systems to Policy" is an introductory course covering privacy, fairness, frontier risks, AGI, AI evaluations, and national and global governance proposals, but available materials do not state whether it is required or elective. Much of the deeper catalog sits in executive education: "Leading in Artificial Intelligence: Exploring Technology and Policy," "AI in Action: Organizational Strategy for the Responsible Adoption of AI," and the Technology and Public Leadership Executive Certificate, which requires three programs within 24 months, at least two from the core technology list and one from leadership.4 The six-week self-paced "Generative AI: How to Use It and Why It Matters" issues a certificate of completion but does not currently count toward that executive certificate.5

Across the field, AI governance usually shows up as an elective, a concentration, or an executive credential rather than a required core course, though requirements vary by program and shift year to year.

Two framings of one skill set

  • MPA: management, procurement, oversight, implementation. Who signs the contract, who reviews flagged decisions, who owns the audit trail.
  • MPP: evaluation design, causal inference, cost-benefit analysis. What the counterfactual is, how large the effect is, whether the benefit justifies the spend.

What agencies are hiring for

Evaluation design, randomized testing, procurement writing, applied statistics, and AI risk assessment. The public administration program pages and technology-policy career guides at publicadministrationpolicy.org map those skills to roles.

Recent News

Recent Articles

See all articles