Crowdsourced AI benchmarks have serious flaws, some experts say

Quick Summary:

AI labs are increasingly relying on crowdsourced benchmarking platforms such as Chatbot Arena to probe the strengths and weaknesses of their latest models.

But some experts say that there are serious problems with this approach from an ethical and academic perspective.

Hadgu and Kristine Gloria, who formerly led the Aspen Institute’s Emergent and Intelligent Technologies Initiative, also made the case that model evaluators should be compensated for their work.

---

More Pictures

---

Read Quick SummarySave Time

or

Article Details

Author / Journalist: Kyle Wiggers

Category: Technology

Markets: United States

Topics: AI Benchmarking

Source Website Secure: Yes (HTTPS)

News Sentiment: Neutral

Fact Checked: Legitimate

Article Type: News Report

Published On: 2025-04-22 @ 12:30:00 (11 hours ago)

News Timezone: GMT -5:00

News Source URL: techcrunch.com

Language: English

Article Length: 798 words

Reading Time: 5 minutes read

Sentences: 19 lines

Sentence Length: 42 words per sentence (average)

Platforms: Desktop Web, Mobile Web, iOS App, Android App

Copyright Owner: © TechCrunch

News ID: 28148539

View Article Analysis

About TechCrunch

Main Topics: Technology

Official Website: techcrunch.com

Update Frequency: 7 posts per day

Year Established: 2005

Headquarters: United States

News Last Updated: 8 hours ago

Coverage Areas: United States

Ownership: Independent Company

Publication Timezone: GMT -5:00

Content Availability: Worldwide

News Language: English

RSS Feed: Available (XML)

API Access: Available (JSON, REST)

Website Security: Secure (HTTPS)

Publisher ID: #20

Publisher Details

Frequently Asked Questions