Evaluations decide what a good AI answer looks like. Models are trained to hill-climb them, so they shape how AI behaves for millions of people. The public almost never gets a say in writing them.
We tried something different. In a study conducted in collaboration with Anthropic, about 2,200 people in the United States, the United Kingdom and India built a free expression evaluation from the ground up. They proposed the principles, chose the topics where they think AI restricts expression and wrote the criteria for judging answers. Each stage built on the work of the one before.
We set a high bar for agreement. A principle only counted if it reached bridging consensus: more than 85% agreement in every country, cultural group, gender and age group at once. No single group could carry a principle on its own.
Explore the interactive report: free-expression.weval.org
Read the full report (PDF): free-expression.weval.org/free-expression-evaluation.pdf
Here are the key takeaways from the project. For the full results, visit the interactive website or download the full report.
1. People want accurate, sourced answers. Of 100 principles participants proposed, 44 cleared the bar. The strongest were about information quality. Focusing on evidence-based information drew 98.4% agreement, and stating uncertainty drew 97.4% agreement. People also wanted AI to cite its sources and say what it left out.
2. Public input can test a model’s constitution. Claude is trained against a Constitution that Anthropic writes. We put 30 of its principles on free expression to the same test. People backed its stance on openness, such as presenting the strongest version of each view and keeping political opinions out. All seven principles that weigh safety against openness fell short. When a situation is uncertain, the public leans more cautious than Anthropic’s existing Constitution.
3. Deliberation turns input into a finished evaluation. Each stage narrowed what came before it. Participants proposed 1,438 topics and cast 34,814 votes, and we tested the 60 with the broadest agreement. They proposed 100 principles, and 44 held up across every group. Then 1,092 people read real model answers and wrote 7,347 criteria for what a good response should do, which we synthesized into 3,791. Every criterion applies a principle the public voted on. And 430 of them flagged concerns those principles hadn’t anticipated, the kind of knowledge a rulebook written in advance would miss.
4. Claude hedges more than its peers. We tested seven frontier models on prompts drawn from the topics participants chose. Claude Sonnet 4.6 added the most caveats and safety referrals, while participants asked for warnings matched to what the user needs. Anthropic has said it is using these findings to change when and how Claude adds them.
What’s next
The knowledge needed to evaluate AI well already exists in communities and among domain experts. The work is building the process to allow them to set the criteria and rubrics for the evaluations themselves.
We are building the technology to make that possible with Weval Studio. Now in pilot, it lets communities and domain experts build evaluations like this one through deliberation, without a research team. If you want to build one with us, write to hi@cip.org.
Explore the interactive report at free-expression.weval.org, or read the full report as a PDF