Skip to content
Did They Answer?

Methodology

What the scores measure, how they are produced, and how far to trust them.

What is assessed

The government responds in writing to every petition on the UK Parliament petitions site that reaches 10,000 signatures. This site covers those responses from the 2010–15 parliament to today. It assesses the written response only: not the petition, not any later debate, and not whether the government's position is right.

The scores describe how a response engages with a petition. They say nothing about the intent of the people who wrote it.

The rubric

Each response is scored from 0 to 2 on five criteria. The five scores are added together, with equal weight, for a total out of 10.

  1. Addresses the ask: Does the government response engage with the specific request the petition makes?

    • 0: Ignores or sidesteps the request
    • 1: Touches on the request indirectly
    • 2: Directly addresses the specific request

  2. Evidence: Does the government response support its position with data or evidence?

    • 0: No data or evidence
    • 1: Vague appeals to evidence, research or expert advice
    • 2: Specific figures, studies, reviews or sources

  3. Policy reasoning: Does the government response explain the policy reasoning behind its position?

    • 0: No rationale given
    • 1: Restates existing policy without explaining it
    • 2: Explains the rationale and the trade-offs involved

  4. Outcome: Does the government response commit to action, or give a reasoned refusal? Describing existing policy or saying it will keep the matter under review is not a commitment, and a refusal only counts as reasoned if it explains why the request will not be met.

    • 0: Neither commits to anything nor explains a refusal
    • 1: Vague commitment, or a refusal with little reason
    • 2: Concrete action, or a refusal with clear reasons

  5. Specificity: Is the government response written for this petition rather than boilerplate?

    • 0: Generic boilerplate that could answer many petitions
    • 1: Partly tailored to this petition
    • 2: Clearly written for this petition

The total maps to one of three labels:

  • Low engagement: below 4
  • Some engagement: 4 to under 7
  • Thorough engagement: 7 or more

Each response is also given a stance:

  • Accepts: The government agrees to do what the petition asks
  • Partly accepts: The government agrees to part of what the petition asks
  • Declines: The government says no to what the petition asks
  • Deflects: The response neither agrees nor says no, e.g. it only describes existing policy

How the scores are made

The scoring is automated. Each response goes to typesafe/jev-1.14, a decision model that answers fixed questions by choosing between the options it is given. It writes no text of its own, so it cannot invent reasons or quotes. The highlighted evidence on each petition page is a sentence from the response itself, picked by the model.

The model reports its confidence in each answer. An assessment is marked low confidence when the model is less than 60% sure of the stance, or of three or more of the five criteria. Uncertainty on one or two criteria is common and is not flagged. Low-confidence assessments are still shown, because hiding them selectively would be worse than showing the uncertainty.

This is prompt version 2. When the questions change, every response is assessed again and the old results are kept for comparison.

Checking the model against independent labels

200 responses were labelled with the same rubric, then scored by the model. The labels were not written by people. Two Claude AI agents labelled every response independently, without seeing the model's scores or each other's labels, and a third agent settled any disagreements. This checks the model against a different AI. It is not a human review. Agreement with the labels:

Agreement between the model and the check-set labels
MeasureAgreement
Addresses the ask88%
Evidence78%
Policy reasoning83%
Outcome74%
Specificity71%
Stance84%
Overall label85%

Run 2026-10-17 with typesafe/jev-1.14-20261014. A criterion agrees when the model's score, rounded, equals the label.

Where they disagreed

Disagree with an assessment?

Every petition page has a Dispute this assessment link. It opens a public issue on GitHub, where the code, rubric and check-set labels also live.

Data

Petition and response text comes from the UK Parliament petitions site and is updated daily. Contains Parliamentary information licensed under the Open Parliament Licence v3.0.