PhD Students Become AI Industry Judges: LMSYS Org's Chatbot Arena Revolutionizes Model Evaluation
PhD students Lianmin Zheng and his LMSYS Org team have transformed from academic researchers into the de facto judges of the AI industry.
Featured on a recent TechCrunch podcast, they recount their improbable rise through innovative benchmarks that now dictate AI model rankings.
From Berkeley Labs to Global Influence
The journey began at UC Berkeley's Sky Computing Lab, where the students sought better ways to evaluate large language models.
In March 2023, they unveiled Vicuna, an open-source chatbot achieving over 90% of ChatGPT's quality using just 7 billion parameters.
This success led to Chatbot Arena, a crowdsourced platform pitting anonymous AI models in blind pairwise battles voted on by millions of users.
Impact on AI Development
Chatbot Arena quickly became the gold standard, with top labs like OpenAI and Anthropic tuning models to climb its Elo-based leaderboard.
Its real-world preference data offers a more reliable gauge than traditional benchmarks, influencing billions in AI investments.
However, controversies arose, including accusations that some labs gamed the system through targeted optimizations.
Funding and Future Horizons
In May 2025, LM Arena—the rebranded LMSYS project—secured $100 million in seed funding to scale operations.
Looking ahead, the team plans to expand into multimodal evaluations and harder benchmarks like Hard Prompts.
Their story underscores how grassroots innovation by PhD students can reshape an industry dominated by tech giants.
With LMSYS.org at the forefront, the future of AI judging promises greater transparency and competition.
