Hanna Wallach
hannawallach.bsky.social
VP and Distinguished Scientist at Microsoft Research NYC. AI evaluation and measurement, responsible AI, computational social science, machine learning. She/her. One photo a day since January 2018: https://www.instagram.com/logisticaggression/
👤 Joined on 31 July 2023
2146
Followers
247
Following
73
Total Posts
📜 Kullanıcı Geçmişi
- 31 Tem 2023, 15:45
🎉 Bluesky'e katıldı - İlk kullanıcı adı:
@hannawallach.bsky.social - 10 Kas 2023, 00:05
🔄 Profil bilgileri güncellendi
🚫 Blokladığı Kullanıcılar
❤️ Beğendiği Gönderiler
Come find us at COLM, I’ll be presenting this paper next Thursday afternoon (10/8, oral session 6 and poster session 6). Full paper here: arxiv.org/abs/2609.08812
Show PostI learned so much working on this with @angelinawang.bsky.social , @hannawallach.bsky.social , Sang Truong, Alex Chouldechova, @afedercooper.bsky.social, Jean Garcia-Gathright, Daniel Ho, @azjacobs.bsky.social, @sanmikoyejo.bsky.social, and Nick Pangakis!
Show PostOverall, we show that convergent and discriminant validity are valuable lenses for systematically assessing benchmark validity. Our approach can be useful for benchmark developers in assessing individual benchmarks, as well as researchers looking to assess groups of benchmarks.
Show PostWe also use our approach to interrogate a few individual benchmarks, and find evidence that BBQ, a bias benchmark, may be inadvertently measuring reasoning rather than bias. This is especially important because BBQ is often the only bias benchmark used in commercial model releases.
Show PostIn some cases, benchmarks correlate more strongly by shared design elements than the concepts they claim to measure. For example, bias benchmarks that claim to measure gender and race bias cluster by task rather than these demographic targets.
Show Post