Princeton Center for Information Technology Policy

princetoncitp.bsky.social

09 May 2025 02:53

>> Read the paper in full: arxiv.org/pdf/2504.20879 @sarahooker.bsky.social @cohereforai.bsky.social @mziizm.bsky.social @sayash.bsky.social @shaynelongpre.bsky.social @beyzaermis.bsky.social @sanmikoyejo.bsky.social @mit.edu

Princeton Center for Information Technology Policy

princetoncitp.bsky.social

06 May 2025 01:10

⚠️ Leaderboard Illusion: "We find that undisclosed private testing practices benefit a handful of providers who are able to test multiple variants before public release & retract scores if desired..the ability of these providers to choose the best score leads to biased Arena scores" Paper out now!🔻

Jan van Gemert

jvgemert.bsky.social

04 May 2025 01:54

Amazing work on evaluating LLM evaluation.

Raphaël Merx

rapha.dev

04 May 2025 01:53

Incredible paper, finding that large companies can game the LMArena through statistical noise (via many model submissions), over-sampling of their models, and overfitting to Arena-style prompts (without real gains on model reasoning) The experiments they run to show this are pretty cool too!

Kapalı içerik