Summary
A new arXiv paper examines user complaints that AI assistants such as ChatGPT are getting worse. The authors argue that human evaluation has not tested a second possibility: that the people judging the answers may have changed rather than the models. They study 1,260,967 blind votes from Chatbot Arena, a public platform where users compare answers from two anonymous large language models and pick the better one or say both are bad, with votes cast from May 2023 onward. The source snippet is cut off before the full findings are given.
AI-assisted summary based on the listed source.
What happened
Users often complain that AI assistants such as ChatGPT are getting worse. Some versions do change, but human evaluation leaves a second possibility untested: the people judging may change. We separate the two on Chatbot Arena, a public platform where people compare answers from two anonymous large language models...
Why it matters
Builders and businesses that track chatbot quality through user ratings may be measuring evaluator standards as well as model performance. If the paper's approach holds, teams could read complaints about declining quality with more care before assuming a model regressed. Readers should check the full paper for its results and how it separates model changes from judge changes.
Signal Intelligence
Signal Strength 95%
Technical label SOURCE-BACKED
Public Interest 35
Category RESEARCH
Reader Depth TECHNICAL
Event context 1 source
Signal Strength reflects source quality, relevance, freshness and evidence. Public Interest helps organize discovery; it is not proof of truth.
Public Interest components
Recognizable Entity Score 51
Practical Impact Score 0
Novelty Interest Score 70
Consequence Score 18
Curiosity Score 0
Shareability Score 52
