albertge.bsky.social

08 May 2025 21:14

Online data mixing reduces training costs for foundation models, but faces challenges: ⚠️ Human-defined domains miss semantic nuances ⚠️ Limited eval accessibility ⚠️ Poor scalability Introducing 🎵R&B: first regroup data, then dynamically reweight domains during training!

changho.bsky.social

09 February 2025 22:27

4/ 🔑 Key insights: - Weak models can make accurate pseudolabels based on easy patterns - Strong models leverage these labels to generalize on hard patterns. - More overlap → better generalization

changho.bsky.social

09 February 2025 22:27

2/ The intuition is simple: generalization tracks the data points containing both “easy” patterns (learnable by a weak model) and “challenging” patterns (only learnable by a stronger model), as with such points, weak predictions create signal to learn challenging patterns with stronger models.

changho.bsky.social

09 February 2025 22:27

What enables a strong model to surpass its weaker teacher? 🚀 Excited to share our ICLR 2025 paper: "Weak-to-Strong Generalization Through the Data-Centric Lens"! 🧵

Fred Sala

fredsala.bsky.social

11 December 2024 23:16

First up at #NeurIPS2024 from our group, our work on labeling via programmatic distillation (a spotlight!). Label your data orders of magnitude faster and cheaper — come join us today at Poster Session 2 East for a demo!