Skip to content
Founderz: Responsible Use of AI home

in collaboration with Microsoft

All case studies

Case study · Government evaluation

UK government: human review of AI-sorted consultation responses

An AI tool sorted public consultation responses into themes. People checked every one of its decisions, and the results were published.

Principles involved

When and where

2025

United Kingdom

What happened

When the UK Department for Work and Pensions consulted the public on its Pathways to Work proposals, it received tens of thousands of responses. The government’s Incubator for AI used a tool called Consult to suggest themes and sort each response into them.

People stayed in charge at every step. Policy teams edited the AI’s list of themes and signed it off. Then 125 reviewers checked and corrected how the tool had sorted 87,738 responses. For 17,000 of them, two people reviewed the same response, so the evaluators could compare the AI with human judgment, and people with each other.

Reviewers left 73% of the tool’s sortings unchanged. On the evaluation’s agreement measure, the tool agreed with a reviewer (0.816) more closely than two reviewers agreed with each other (0.710). The median check took 19 seconds per response.

What the sources establish

The evaluation, published by the government in October 2025, establishes these figures for one consultation. It timed the checking, not the whole analysis. Reviewers saw the tool’s themes already selected. In our reading, that may have made them more likely to agree, a form of anchoring the report doesn’t test.

Implications

Here AI did the first pass at a scale people couldn’t match, people checked every sorting and signed off the themes, and the government published the measurements. The case also raises a harder question: when people review AI’s suggestions, how do you know they’re judging rather than agreeing?

Recommended practice

  1. 01Design the check before you run the AI: who reviews what, and how you’ll know the review is real.
  2. 02Measure agreement between people as well as between people and AI. It tells you what good looks like.
  3. 03Keep people on the decisions that matter, and record how you checked.

Sources

What each source establishes, and its limits. The practices and recommendations on this page are ours, and the facts come from the sources. See every source we use.

  1. Consult Evaluation: DWP’s Pathways to Work consultation UK Incubator for AI (i.AI) · Dated August 27, 2025; published October 30, 2025 · Government evaluation Policy teams edited and signed off the AI’s themes; 125 reviewers checked the tool’s sorting of 87,738 responses and left 73% unchanged; the tool agreed with reviewers (F1 0.816) more than reviewers agreed with each other (0.710); the median check took 19 seconds. Limits: One consultation. It timed the checking, not the whole analysis.

Cite this page

Founderz (2026). UK government: human review of AI-sorted consultation responses. The RUAI Standard, 2026 edition. Developed by Founderz in collaboration with Microsoft. https://responsibleai.founderz.com/toolkit/case-studies/uk-consult-human-review

Licensed under CC BY 4.0: share and adapt with attribution.